Hey everyone! I'm the founder of Stakpak, I just published this comparing 5 LLMs on a specific Terraform code generation tasks (and of course included ourselves at the end 😅) We're trying to figure how to make infra configurations easier without abstractions, and code generation alone is not enough. I really appreciate your feedback, or if you'd like to share more edge-cases where existing LLMs screwed up I'd love to hear your story

Good share. It's not fair to compare llms this way in my opinion. They can never (for now) match a developers expectation of how the code should be (organized, modules, variables, global, child etc). Will check stackpal on what you guys too to see if you also can execute (terraform apply) post code generation or expect the code to be used outside. Disclaimer. Our #qmcloud platform does generate any code format you want, generates visualization and then does execution too. It's quite difficult to get the code in all languages the way devs think they need.

That's what we're working on 😄 infra tools powered by LLMs that do match devs' expectations. Specifically for configuration management. We're very early, but I think there's a lot of room for improvement compared to using vanilla LLMs for infra tasks. Even if Terraform plan succeeds apply could fail for a multitude of reasons. It's a complex problem that requires much more than accurate code generation. Stakpak doesn't execute the code, we focus on what's earlier in the lifecycle. There is a great variety of TACOs we can integrate with for this bit https://itnext.io/infrastructure-as-code-landscape-overview-2024-a066124e5989