This message was deleted.

I, typically, use cloud-init and when I need to change anything I destroy the server and rebuild it. I just make sure that everything is resilient enough to stand that kind of abuse.

Thanks @Troy Knapp, appreciate your feedback 👍 Actually, I have a GitLab instance in my instances pool so not very safe/resilient to fully reprovision it frequently 😕

That sounds scary... you've got a single point of failure that you can't practice losing and recovering. I'd be thinking about what you can do to make that a non-issue... like putting it in a scale set or something.

I remember solution while ago where we used terraform with user-data in instances where simple bash would install ansible and then pull and execute playbook. This allowed us to not have to change script all the time and since playbook was always in same location we could apply standard release processes to ansible. Now you have decoupled nicely Infra provisioning with config management

You typically want to aim for immutable infrastructure. It's best practice to recreate the instance if your cloud init script changes.

You can definitely provision Ansible with a cloud-init... and Ansible can be hard to wrap your head around sometimes, but its fine for *nix instances. It's a bit more of a pain for windows boxes.

I would prefer SaltStack than chef 😉 but in principle idea is same. Provision infra in terraform and leave config management to tool which is designed for it 😉

Provision with Terraform then configure with Ansible. I’ve used the TF aws_instance user-data to pass in and run initialization scripts (update os, rename hostname, create user) so that it’s ready to be managed by Ansible going forward. Both sides are all in code but the config files, os admin tasks, reboots, etc makes more sense to automate in Ansible

I'd say it depends how complex your instances get, because tools have their sweet spots. In my simple view, TF is mainly for managing cloud resources (as black boxes) but can do a bit more; while other tools are mainly for managing the "contents" of those resources, once created. For machines that's a clear difference, for other types of resources it's more of a grey area. I've used a lot of puppet over the years (after the instances got created with terraform, with a bit of cloud-init to install puppet and trigger a first run). For "complex" machines (lots of packages / config file changes) it's nice to have a tool that provides a bit of abstraction and re-use. Puppet, Ansible, Chef could all do this and is a matter of taste, and what best suits the (experience of the) team maintaining it. For machines that only have trivial configuration one could stick with terraform's remote-exec provisioner (fine for idempotent actions) and use the null resource to make it remember in its state file that it already did a particular action once (for non-idempotent actions). And when possible, rebuilding machines over maintaining them for a long time also saves on patching effort. We've always ensured that provisioning machines (both the machine creation and their config mgmt) was fully automatic and unattended, but sometimes that process is too slow to do regularly (i.e. if it takes puppet an > 30 mins to download and install all the relevant things because we only need a tiny instance type, then it may not be convenient to replace machines e.g. weekly). We started to use Kubernetes more in the last years; and also there we found the practical limitation of Terraform. We do use the helm provider a little bit to bootstrap new clusters, but to manage all app deployments after that we didn't think it was convenient enough, so we use more fit-for-purpose tools once a cluster is running.

Cloud init is wonderful but "less is more". It can do a lot of things but you get yourself into trouble when you try to do too much. I spent way too much time with cloud init to ultimately use it to copy shell scripts and kick them off. Terraform is nice bc you can use regular files or TF templates in your code and use TF to compress and base64 encode them for the cloud init payload

Also making a Golden Image Pipeline is definitely a thing. A cloud-init should be super simple and just have a custom tweaks at most for a particular server. IMO, you should do all the package management in a golden pipeline, not a cloud-init.