Hey, After an on call fix from a dev, we just real...
# general
r
Hey, After an on call fix from a dev, we just realised that he has modified an IAM policy to fix the issue but that 5 days later a bunch of database backups were not dumped and we lost 1 week of data... So now just realised that our IAM management is just a mess. Curious to hear if you have similar stories
k
Definitely. It's why we have alerts if our backups don't run
But the sad truth is you kind of have to have the scenario sometimes before you think to detect it. Make sure you conduct a blameless retro so you can do better next time
a
Definitely come across similar problems, but 1 thing it does usually highlight is IAM Roles/Service Accounts being used for more than 1 purpose (usually too many!) so that changes for purpose A accidentally affect purpose B