https://platformengineering.org logo
#platform-toolbox
Title
# platform-toolbox
l

Leonardo Romanato

02/10/2023, 2:28 PM
Hello! Anyone of you dealt with a complex Jenkins environment (more than 600 devs across multiple Jenkins masters) to rationalize and standardize pipeline code? Any tool / technique to suggest? We have an historical Jenkins environment where we gave a lot of freedom to devs and now it is hard to manage it, especially when it comes to confidence in upgrading without breaking things and when there is the need to manage massive pipelines update (eg: introduce mandatory stage, refactor specific commands part of shared library, …)
a

Arie Heinrich

02/10/2023, 3:09 PM
I dont have that huge of a scale but i can tell you what i did. 1. Shared Libraries 2. Controllers in containers Infront of loadbalacers so we can replace and upgrade. 3. agents in containers in on prem VM or in k8s. 4. I had only one controller that was busy 24/7 by 5 different teams. So each team got a new controller. 5. All agents were prepared before hands, with docker images and all the tools. 6. security reports, on who has permission where. moving to groups instead of users, base it on you AD if needed. 7. Training and patience. Its not a sprint, its a marathon.
8. Use Jcacs plugin (https://plugins.jenkins.io/configuration-as-code/) to create a configuration that you then reuse instead of manually applying changes 9. go over ALL the plugins and remove what is not needed. Dont assume its needed. Use the plugin usage plugin to give you some insight (not 100% accurate) 10. Incorporate the plugins into the process of creating your controller image
11. Try to see if JTE - Jenkins Template engine gives you more control over the pipeline itself 12. Move to a new system, that requires 0 intervention from the developers. Make sure you have top management back for this as this could lead to "war"
l

Leonardo Romanato

02/10/2023, 3:18 PM
A lot of good suggestions @Arie Heinrich! Thanks so much!
Some of those are already applied to our Jenkins environment
However it was more on how to approach a massive refactoring rather which I think is something in parallel with your points (apart from point 12, which is were I would go, but at the moment don’t have the authority / capacity to do it)
a

Arie Heinrich

02/10/2023, 3:23 PM
When we worked on our shared library, instead on building it using fully groovy code, i actually created it mostly with python with just a small groovy wrapper to pass parameters and secrets. Since my devs "hated" groovy but loved python it made it easy for me to onboard them in an "Inner Source" mentality where they were creating tools and opening pull requests to the main shared library repo after we gave them a short training on how to create them and added some basic testing so to have a certain level of quality to it. Then its the hard work of replacing the pipelines to use the new shared libraries, that is where JTE can help as its adds another layer that is missing so much in Jenkins and that's governance. You could potentially swap from the regular jenkins "distro" to the cloudbees one, that has its cost but might be better suited to force control and governance.
l

Leonardo Romanato

02/10/2023, 3:25 PM
Thanks Arie!! I will have a look! Also the suggestion for the “shared ownership” on the “shared library” it is great
a

Arie Heinrich

02/10/2023, 3:25 PM
its a way to create engagement so its not something done "behind thier backs" or without any input from them.
r

Romaric Philogène

02/11/2023, 9:49 PM
cc @Solomon Hykes
s

Solomon Hykes

02/11/2023, 10:38 PM
👋 we built dagger.io around exactly the pattern you describe @Arie Heinrich 😁 You write pipeline logic in Python, Go, JS… Then you write the minimal jenkins/gitlabCI/github Actions /circleci configuration to execute that logic with the correct inputs. Bonus it works on the dev machine too. Definitely the way to go if you want to keep complexity in check
a

Arie Heinrich

02/12/2023, 1:07 PM
Yes i have read about dagger and was in middle of testing what i can do with it. Im not sure introducing another "abstraction layer" / another DSL is necessarily the right way. I mean why would i need any of the orchestration engines if i can do all execution logic within Dagger. Yes I need to worry about passing variables / secrets between the engine and the command line but im still no sure doing parallel steps inside Dagger is "better" than using the native parallel / matrix abilities.
25 Views