For those who write your own Kubernetes controller...
# kubernetes
j
For those who write your own Kubernetes controllers/operators, what platform/language do you prefer to write them in? My team has needed a handful of custom controllers over the years and we’ve used a lot of tools to make them (bash, operator-sdk/go, operator-sdk/helm, operator-sdk/ansible, crossplane). None have had the right balance of ease-of-use, performance, and flexibility. Everything except bash and operator-sdk/go have limitations in the use cases they can be used for, and the others have proven difficult to build and maintain. I recently tried out Kopf and it seems to be the most promising option so far! Python is easier for us than Golang, and the function decorators are quite nice for writing reconciliation logic and letting the platform deal with all of the Kubernetes watches and caches. Maybe there are other tools I should know about and try out?
t
it sounds great... but its not passing tests... 😢
m
Looks cool. As for tests, the latest release tag did pass. Seems something in main/dev is not passing. https://github.com/nolar/kopf/actions/runs/5698265627
Or.. a flaky time-based test.
t
yeah, its just the unit test for python 3.7 that's failing
j
Yeah I don’t think that current status is indicative of a general quality issue. My bigger concern is that the project seems to be primarily built by a single developer which generally has sustainability issues.
a
Thanks for sharing Jake! I am actually super interested by this as I am currently writing a controller based on Kubebuilder. Did you ever use that to scaffold golang? Also curious what you mean by using Crossplane to build your controllers/operators 🤔 do you mean creating your own provider using their framework?
j
Regarding Crossplane, I just mean using their “composition” mechanism, which I see as a controller framework. You could get effective use out of Crossplane without using compositions and creating your own CRDs/controllers which manage resources installed by the Crossplane providers.
The only kubebuilder stuff I have done is via the operator-sdk. I realize kubebuilder works for a lot of people. For me personally, I am not a huge fan. I’ve struggled to get proper tool chain installed to make it all work, and I find code generation to be really hard to maintain. I find the approach of using decorators in Python or macros in Rust to be much more appealing, because it is all handled from within the mechanisms of the language.
But it seems like the right approach for Golang! And I’m glad the Golang users have a fairly standard approach to rally around. I’m just not a Golang guy, except for when I have to be.
a
Yeah I get you both on Golang and kubebuilder. I think those are very fair concerns. It is interesting how you described using compositions as a stand in for a more complex controller/operator option. I thought about that a bit more and I think I get what you mean if it's around using a composition to drive a pre-packaged provider in an allowable way. But my (limited) experience with compositions is that they are yaml wranglers, and often what I'm doing with my controllers and operators is more business logic and workflows. Is that what you meant?
j
Yeah, the capabilities of compositions do come down to yaml wrangling, but a lot of our controllers do as well. We often just want to capture an abstraction of a commonly used pattern or apply some subtle adjustments to existing use cases. So much more of a “controller” than an “operator”, if we want to make the distinction.
a
Definitely my experience as well. Often controllers / operators are pretty minimal logic but take a lot of upkeep.
Thank you so much for sharing your journey. It really helped me realise that the path I am on makes a lot of sense for an even more diverse (and often much simpler) use case. The way Kratix uses “Workflows” to run an ordered set of containers is really powerful as a low cost controller/operatator as well. I look forward to continuing to explore examples!
j
A coworker told me about shell-operator, which allows you to write a bunch of hooks in whatever language you like, have them subscribe to combinations of events (which can get pretty specific thanks to jq filters), and perform kubernetes changes on your behalf based configuration your hooks write to temporary files. I haven’t used it much yet, but it actually looks like the most promising option for my team so far.