Hey platformers! How do you onboard teams into your clusters? Currently using artisanal free-range bash scripts but I am thinking of diving into Helm. The scenario is basically, you have to onboard a dozen teams who each need their own namespaces/service accounts/limits/etc on shared dev, stage, and prod clusters.

We make Helm charts to provision various things like that, and use Argo CD ApplicationSets to configure them per-cluster/per-team as we need with different values files. So each new use-case requires an ApplicationSet tweak and a new values file. Not as self-serve as an IDP would be but reduces toil enough for now

Thanks for the reply! Yeah I feel like we are too small for full on service catalog and IDP so looking for the "good enough" copy-paste type solutions people have made

Hmmm… that’s a tough one but I am inclined to say that C&P is not “good enough”. It’s really not that hard to move your scripted onboarding into a CI based form, e.g. Github Actions and then call a scaffolder, or if smaller is desired, a templating engine, from there. This will not make your life much more complicated when creating the thing but will allow you to scale it much more easily in the future. It’s also your first step into turning this into a self-service by making this a manually run pipeline with inputs. If you choose to upgrade to a portal in the future, you can let the portal take over the GUI and call the action - done. It’s also easy to inject an orchestrator into this later because you will naturally separate out the run-once activities into the manual pipeline. At the same time, all other tasks will go into auto-triggered pipelines. Making the orchestrator take over parts of your auto-triggered pipelines to make infra creation and management that much easier is more refactoring than re-design.

We have crossplane do this. we have a "management cluster" with crossplane configured against all clusters and have a custom XRD called tenant where we define all resources that need to be created per tenant and that is run against all kubernetes providerconfigs in that cluster. it gives an easy single point of management and control, and we manage that cluster via flux

Hey @Scott Rosenberg! Out of curiosity - how far does that pattern scale in your case and what were the design decisions to go beyond pure Flux for cluster management and let Crossplane take over? Is it OSS Crossplane or Upbound?

We have it currently with hundreds of tenants and it works great with oss corssplane.

Cool! I had “change of interface” and “separation of concerns” at the back of my mind. “Preventing abstraction leaking impl details” was not on my list but makes total sense! Thanks for sharing!

I just checked out crossplane and I like it! Too much for my needs right now but I see how I might evolve to using it

Crossplane/Kratix is our next step of abstraction and am excited to try it out and simplify the interaction between our feature teams and some of the more complex charts/TF we have

Following up Helm is the way to go. Found a pattern to use both terragrunt and helm with the same inputs. Would like to move off terraform eventually though

If you're going deep in helm, try helmfile, really nice convenience functions

@SeanOG looking at helmfile, why did no one tell me sooner??? thank you

Its the best hidden secret