Currently determining some solutions to something I’ve been wanting to conquer within our environment for sometime. Wanted to see if anyone here has implemented anything similar or have other ideas: We run on AWS EKS using Helm for all of our K8s installs. The Helm Charts are specific under every service’s repository so that the developers themselves have access to modify all of the aspects of those Helm Charts. We shifted away from a “the container is the service” to a “the Helm Chart is the service” model. The only issue with that though is if we want to make changes to something that only we (platform) really would touch within the pod spec, such as Topology Spread*, we have to touch that within every repo or have the owning teams make those changes (which takes them a while due to priority work). I’m wanting to shift away from some of those aspects being controlled via the services’ Helm Chart though and would like to inject configuration into the pod spec after the Helm Chart has been installed. Currently I’m going down the path of Mutating Webhooks but not sure if there’s an easier way, like PodPreset(🪦) would have given us. Ideally all we would need to do is change one template and push out whatever Platform changes we wanted to all services. * Topology Spread is my example here because this is specifically the issue I’m trying to solve. Given we are on EKS, we have no ability to customize the scheduler which would be my quick path forward although this new goal helps accomplish much more and would make our life much easier

I'd possibly suggest looking into Kyverno to handle this, allows you to write mutating policies https://kyverno.io/docs/writing-policies/mutate/

Creating an org chart(s) that your services consume is a good option to help unify things a bit.

@Andrew Kirkpatrick That looks extremely promising. Thanks!

There are loads of examples https://kyverno.io/policies/?policytypes=mutate including something like https://kyverno.io/policies/other/add-node-labels-pod/add-node-labels-pod/

@Troy Knapp Can you expand a bit more on what you mean? You talking about creating a single parent chart for all of the services?

Yes. For example, we have a single chart for all our dotnet microservices that I maintain. When a change needs to be done, they make a ticket and I update the chart with whatever needs to be done. All the values files live with the service, but the helm chart is mine

That’s would be nice to have that type of control. Not sure if it’d fly here though since we are pushing the developers to own the chart (even though it’s like pulling teeth). Thanks for the suggestion!

We're probably looking to move to a hybrid approach between those 2 ideas; having some centralised charts that services can use if they like, but let teams roll their own if they need something more complex/want more control

Mutating rules sound like a very efficient way of creating footguns (so I've never done it). I'm not sure how I'd manage that in such a way as to make it apparent to a dev that the config they are expecting in the chart won't be the config they get on the cluster.

My thought is that the mutated part would only be things the Platform team cares about. Basically remove all cluster specific elements out like HPA, PDB, TopologySpread, NodeAffinities, etc. and let the Helm chart the devs have be just enough to run on minikube if they so desired.

So I save all the artifacts in an artifacts repo. During CI, I update the new image in a values file, and copy over the relevant dev-values.yaml, stg-values.yaml, prd-values.yaml file into a repo that looks like this:

─── charts
    ├── root chart
    |   └──<lots of chart template stuff>
─── artifacts <everything in here is auto generated>
    ├── dev
    |   ├──chart1
    │   |   └── values.yaml
    |   └──chart2
    |       └── values.yaml
    ├── stg
    |   ├──chart1
    │   |   └── values.yaml
    |   └──chart2
    |       └── values.yaml
    └── stg
        ├──chart1
        |   └── values.yaml
        └──chart2
            └── values.yaml

During the release process I update the image tag in stg and prd when integrations tests have passed and they've been deployed to lower levels. You could also copy the version of the chart that you used as well if you wanted to.

Thanks for the explanation! Sadly we don’t use Argo but that’s another pain I’ll hopefully get rectified at some point. We’ve already been bitten so many times by Helm but it does what it needs to with our current setup. I’ll have to POC it out a bit for others but I have a few new (less painful) paths to go down for my initial goal. Thanks for the help!

If you're using Helm charts, helmfile and GitHub codeowners could provide a way to manage control between developers and platform operators. https://helmfile.readthedocs.io We deploy microservices, with each service having its own team of developers. The developers handle basic adjustments to Kubernetes resources. We use a helmfile to control which charts are used and what values are injected, which are common across microservices. Control over values related to Kubernetes resources is delegated to the developers.

i would use helm for this have all the configurations that the platform team needs to define as variables with iterations and conditionals where needed, you will need to commit that change on the team repo but then you will manage everything on the values of each environment that i would assume you are managing

I had similar issue and we have ended up with single chart which platform managed and directory structure similar to above. We had separate chart for say infra related stuff like AWS orchestration and then we used Helmsman to wrap around both and give extra functionality which helm lacks. If you go with Kyverno devs will say I have no idea what is going on and all that magic hidden in cluster will bite you in long run. Eventually you will have multiple policies and your lifecycle managed by multiple processes. When things go wrong you will pull your hair out trying to figure out what is wrong and where.

Totally agree with what Troy is saying above. We have a platform that abstracts and automates sync waves and Argo CD into a single solution.

One of the things that I like about the approach that I laid out is that once you have all your artifacts figured out its still really useful for Helm, but it flows beautifully into Argo... so you don't have a lot of sunk cost refactoring your approach and you don't have to boil the ocean right away, either.

late to this conversation.. like Troy, we have a concept of base charts that we (plateng) maintain and serve to our devs. for the most part they only interact by way of a values file, if they can't do something they want to, it becomes a kind of feature request for the helm chart. we have different bases for the different types of patterns we need to support (springboot app, python app, etc) - devs are able to own their own services, configure to their liking, but have some infra guardrails in place that are abstracted away from them. they're helm-educated and they're using helm to put together/deploy their services but we are the owners of the charts (although we barely have to touch them). its a sort of self-service