Platform engineering is the discipline of designing and building toolchains and workflows that enable self-service capabilities for software engineering organizations in the cloud-native era.

Platform Engineering

This is very triggering :joy: lots of flashbacks.

We once deleted an entire production cluster via Argo in a similar way. Ours was an accidental rename of one argo application to the same name as another existing argo application. Took out an entire region for our customers (this was at <http://influxdata.com|influxdata.com>). We got it all back though (lots of backups).

My colleagues did a talk about it at Kubecon <https://www.youtube.com/watch?v=xDGjmav8UBg>

I understand why this is triggering then! Let me check the talk, seems super interesting (thanks for sharing). I can't imagine the feeling of discovering that even the backup were not good...

That must’ve been a nightmare for all involved :100:

Since im just getting into Argo, is there a way to avoid this type of thing or is it a you got to be really careful of what you do lol?

People often forget that powerful tools make powerful footguns.

<@U05MEALKAGP> That was an interesting talk, thanks.  I’m sure that was a No Good Very Bad Day, but props to influxdata for being able to fully recover.  Those are some nice, simple actions in the post-mortem as well that may save other teams worlds of pain.  That’s exactly the kind of talk I love to see at *con events!

Pleasure! And I’ll be sure to pass that on to Rick and Wojciech when I speak to them next :smile: I agree, these are the kinds of talks I like too. The battle scars :joy: Definitely where the most learning happens.

We suffered the loss of entre envs sometimes due to terraform code being out of wack with pt branches vs mainbranches being merged without updates causing non-backward compatible change to be overqritten and AKS getting nuked irreversibly ;å

Namespace deletion is a dangerous thing. In our ArgoCD we make sure ArgoCD wont delete a namespace even if someone tries to.

<https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#no-prune-resources|https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#no-prune-resources>

You can even go as far as giving ArgoCD all RBAC except DELETE on Namespaces