Hey Platform Engineers, how you deal with monitoring, e.g. Grafana. Is there a good GitOps way to deploy Grafana Dashboards? Ideally I want to deploy a new Microservice that would bring Grafana Dashboards, Prometheus setup etc. as CRDs or similar?

I can only give an anti-recommendation. I have experience with https://github.com/grafana/grafonnet-lib. It sucks because it quickly becomes unmanageable-jsonnet-hell. Probably, (for Grafana at least) UI way is still the best option, unfortunately. And some recurring job dumping dashboards in JSON format could help you to backup dashboards and rollback if needed If someone here had a better experience, would be happy to learn 🙂

I was hoping for https://github.com/grafana-operator/grafana-operator

• Efficient dashboard management through jsonnet, plugins and folder assignment, which can all be done through

.yamls

!

but seems like also jsonnet ?

we use Kube-prometheus-stack: https://github.com/prometheus-operator/kube-prometheus

we use the prom-operator and we have an helm chart with all the dashboards. It's horrible as people won't really care at making new dashboards/tweaking what is there because it is so annoying to commit and have it deployed. But if you don't do so your dashboard is gone at the next restart of grafana...

Suppose it also depends on the size of your org and how often dashboards change. In our case it can be months before we'd need a change, so the process isn't onerous enough to go with something more complex

Thanks for the feedback, that sounds like a valid approach to develop them in UI and put them in the Image or Deployment 🙂

It can also be done with a Terraform provider pushing dashboards on a git repo.

+1 terraform grafana provider