Hello! Anyone of you dealt with a complex Jenkins environment (more than 600 devs across multiple Jenkins masters) to rationalize and standardize pipeline code? Any tool / technique to suggest? We have an historical Jenkins environment where we gave a lot of freedom to devs and now it is hard to manage it, especially when it comes to confidence in upgrading without breaking things and when there is the need to manage massive pipelines update (eg: introduce mandatory stage, refactor specific commands part of shared library, …)

I dont have that huge of a scale but i can tell you what i did. 1. Shared Libraries 2. Controllers in containers Infront of loadbalacers so we can replace and upgrade. 3. agents in containers in on prem VM or in k8s. 4. I had only one controller that was busy 24/7 by 5 different teams. So each team got a new controller. 5. All agents were prepared before hands, with docker images and all the tools. 6. security reports, on who has permission where. moving to groups instead of users, base it on you AD if needed. 7. Training and patience. Its not a sprint, its a marathon.

A lot of good suggestions @Arie Heinrich! Thanks so much!

When we worked on our shared library, instead on building it using fully groovy code, i actually created it mostly with python with just a small groovy wrapper to pass parameters and secrets. Since my devs "hated" groovy but loved python it made it easy for me to onboard them in an "Inner Source" mentality where they were creating tools and opening pull requests to the main shared library repo after we gave them a short training on how to create them and added some basic testing so to have a certain level of quality to it. Then its the hard work of replacing the pipelines to use the new shared libraries, that is where JTE can help as its adds another layer that is missing so much in Jenkins and that's governance. You could potentially swap from the regular jenkins "distro" to the cloudbees one, that has its cost but might be better suited to force control and governance.

Thanks Arie!! I will have a look! Also the suggestion for the “shared ownership” on the “shared library” it is great

its a way to create engagement so its not something done "behind thier backs" or without any input from them.

cc @Solomon Hykes

👋 we built dagger.io around exactly the pattern you describe @Arie Heinrich 😁 You write pipeline logic in Python, Go, JS… Then you write the minimal jenkins/gitlabCI/github Actions /circleci configuration to execute that logic with the correct inputs. Bonus it works on the dev machine too. Definitely the way to go if you want to keep complexity in check

Yes i have read about dagger and was in middle of testing what i can do with it. Im not sure introducing another "abstraction layer" / another DSL is necessarily the right way. I mean why would i need any of the orchestration engines if i can do all execution logic within Dagger. Yes I need to worry about passing variables / secrets between the engine and the command line but im still no sure doing parallel steps inside Dagger is "better" than using the native parallel / matrix abilities.