Hello fellow Platformers. Hope everyone is having ...
# general
j
Hello fellow Platformers. Hope everyone is having a lovely friday 🙂 I have a question: Are you using the same branching strategy for IaC repo's, scripts repos and configuration repos as you do for you application development repos. I know of course it depends if you have IaC and application split into different repos; Let's assume you have. As an example, would you use the same branching strategy for your DevOps/Platform repos as you do for your Application development repos ?
a
I lean towards very short lived (a day or two max) branches for all, so yes? 😅
Curious why you're asking? My experience is where this differed it based on the tools we used. E.g. using Atlantis for terraform applies may affect decisions
j
Just to make sure that we understand one another. You think that infrastructure code repo's should use the same branching strategy as an application development repo, is that what you are saying ? And the reason I am asking is because infrastructure provisioning/development code does not really have the same phases as an application development, at least not from my perspective, but if you can please change my mind :) I am the main responsible for making IaC where I work, sometimes it is transforming existing Cloud Infrastructure to IaC and sometimes it is designing new cloud solutions and the code for it. And as I see, it is difficult to compare the way you change the functionality of a button (application side) to creating an IaC solution (infrastructure side). I guess it might be a maturity thing since we are only 3 DevOps/SRE guys and the Cloud for our company is still fairly new. So I asked to what other people in here have experienced and what have worked well for them 🙂
a
Yup! My experience is that infra code needs as much testing / roll out as any software code. While it may not have as mature tooling in place, it has been possible in my experience. I would suggest maybe looking at things Kief Morris has written in this space?
And just to be real, I have absolutely worked at places that SSH’d to the servers or tested by just slapping something together then merging that code to prod rather than progressing it through envs. I just don’t prefer it 😅
j
I’ve done a fair amount of IaC and I personally don’t think you should follow the same strategy as your application code. What I did in the past was having different folders for different environments.
And we put all IaC code into a separate repository.
and also different folders for different application…
j
alright that makes sense. As of right now, what I have done to structure the IaC I have create a folder structure like so: • IaC-repo ◦ Bicep ▪︎ infra-1 ▪︎ infra-2 ▪︎ infra-3 ◦ Terraform ▪︎ infra-1 ▪︎ infra-2 ▪︎ infra-3 And as for branching I am primarily using main branch, but that is mostly because it does not make sense right now to do PR's because of missing hands. But I do believe that when creating new or changing existing IaC there should be some way to track the changes/additions. Like: 1. Work item created that requests new infrastructure 2. Create branch based on that work item to track work 3. When work is done create PR 4. PR approved and branch merged to main And testing of course happens through the whole process both before, in between and after the PR.
j
That’s right…
Another approach is to define your variables in separate files or json…all the main code remains the same but for each environment, you pass the variable file
j
IaC or config management? two different things although they seem to get conflated… i think the code for infra should be reusable and should follow the same branching strategy as code. The problem (i think) comes in when the “infra” or platform team doesn’t have a dev and staging environment… the dev and staging environments are more like production for them then for anyone else… if i break dev i stop work my customers are impacted… so i need an account to validate and unit test my code before pushing to prod which is all of the environments including prod that the business relies. if the same IaC isn’t used to make resources in dev as in staging and as in prod then the environments will be different and then you have snowflakes and have a horrible work / life balance … so same code to deploy everything in all environments with as much similarity to code deployments as possible… which i think includes avoiding a monolith of resources with entire accounts resources defined in one repo and instead have micro services with resources clustered into specific repos… etc so feature/chore/bug branch deploys to an account that is not “production” and then merge to main where that same code is deployed and configurations are different based on environment vars (or how ever config is stored json or param store or dynamobd etc)
j
you’re right! the least error prone is to have the monolith approach, but the best approach is to extract the variables, but doesn’t make sense to have a dev.json file in your main branch
pros and cos as always
j
if you don’t have dev.json in the main branch then you have more than one repo not a branching strategy….
a feature branch /dev/ should be different from main only as long as it takes to prove that the new code doesn’t break anything and then merged in the smallest amount of change possible - building to the final product
eventually main.py or main.tf will also become different between branches if dev.json isn’t in main
at which point there will be fear making a change to anything - as there should be also at which point there will be lack of clarity and who ever last worked on the repo(s)/branches will own that solution and it won’t be available to anyone to make changes - thus the work life balance i now am on call but cannot make a change to code with assurance it will work so i am now just part of a calling tree to reach the last person to merge to that branch/repo
j
You don’t need a branching strategy at all…branching strategy implies whatever is in that branch is specific for that environment
j
correct - we don’t need to do what is right - but i don’t want to work at the auto parts company before phoenix project i want to work there three years later
i think branching strategy implies a way to get any branch into main so that main can be deployed to any environment and work… so that new work can be started from main not the last known branch that worked in an environment…
j
It’s whatever makes sense for you…you could take another approach like this:
2 repos
1 is the actual code with no variables/configs
that follows branching strategy
another repo where you define your configs
and that you have it be a more monolith single branch
j
i do agree it is what ever makes sense to the development teams and those doing the work - but i don’t want to see them called at 2 am to fix something because a change to main in a feature branch cannot be merged back to main and proven to work and be pushed where it is needed without some siloed knowledge… i want team A to be able to go work on something else and team B to pick up the work so while it is what ever makes sense to “you” it is also what makes sense to the entire engineering org - scale breaks at some point -
i could also store my code in an NTF share and just ask people not to change it unless they talk to me first - i did 12 years of perl that way and spent 5 years trying to undo that mess
but it made sense to me and managed the config of 20,000 windows computers for a company making 420,000,000 a year… but just because it worked for me it didn’t allow me to do something else and for someone else to pick up where i left off… it was just me sad panda
😄
j
Got you! Makes sense! And often times the whole org needs to be on the same page…
j
“relative reality” we are both likely right 🙂 culture and context are likely more important than any of my musings on this in this thread 🙂
s
IAC is a code repo, so just treat everything as code. It is a principal and should apply to requirement, dev, testing, rollout, and everything related.
i
Since in different companies the strategies differ for the branching scheme based on their specific SDLC, I believe that it is safe to say that: you should choose the strategy that better makes sense to your current workflow. Are there best practices? Surely, but remember that best practices are guidance, not orders.