Hey Platform Engineers, how you deal with monitor...
# gitops
Hey Platform Engineers, how you deal with monitoring, e.g. Grafana. Is there a good GitOps way to deploy Grafana Dashboards? Ideally I want to deploy a new Microservice that would bring Grafana Dashboards, Prometheus setup etc. as CRDs or similar?
I can only give an anti-recommendation. I have experience with https://github.com/grafana/grafonnet-lib. It sucks because it quickly becomes unmanageable-jsonnet-hell. Probably, (for Grafana at least) UI way is still the best option, unfortunately. And some recurring job dumping dashboards in JSON format could help you to backup dashboards and rollback if needed If someone here had a better experience, would be happy to learn 🙂
I was hoping for https://github.com/grafana-operator/grafana-operator
• Efficient dashboard management through jsonnet, plugins and folder assignment, which can all be done through
but seems like also jsonnet ?
however, we made one minor change, we use a custom grafana image that includes all our dashboards as part of the image
basically we “develop” dashboards in the UI, then when happy with them export the JSON and commit them to our repo
and it will get picked up and deployed by ArgoCD
we use the prom-operator and we have an helm chart with all the dashboards. It's horrible as people won't really care at making new dashboards/tweaking what is there because it is so annoying to commit and have it deployed. But if you don't do so your dashboard is gone at the next restart of grafana...
Suppose it also depends on the size of your org and how often dashboards change. In our case it can be months before we'd need a change, so the process isn't onerous enough to go with something more complex
Thanks for the feedback, that sounds like a valid approach to develop them in UI and put them in the Image or Deployment 🙂
It can also be done with a Terraform provider pushing dashboards on a git repo.
+1 terraform grafana provider