Hello to all, any experience on Observability Architecture using Kafka, OpenTelemetry, Thanos, Cassandra, Loki? I need to implement this architecture and need some experience to prepare the trip to on board of those technologies. Thanks so much. Please if you experience is throw hate, left bad words, pathetic response and try to show off your knowledge in a boastful and bullish way, pass by, this is not for you.

@Carlos Romero I am trying to set up a similar architecture but with slightly different components • Redpanda instead Kafka. It’s much easier to operate and it’s fully compatible • Quickwit instead of Loki. Idem it’s much simpler than Loki to operate and has a better performance for the search • FluentBit instead of Otel collector (for logs collection). It has a better performance and less resource usage Our volume: 800 millions of logs/day for 1 server.

Hi 👋 @Hung B great my friend!!! Maybe could share experience 😉

TBH not so much things to say 😂 It works well. Very few incidents. Mostly network and storage issue

We migrated from ElasticSearch (Graylog) to Loki a while back and honestly, it was a mistake. Issues with Loki: • Bad performance on larger queries. • Parts of Loki just OOMs/crashes instead of failing gracefully if a query is to large/complicated. • Hard to tune to mitigate the above issues and performance is still light years away from ElasticSearch. • The teams that are happy with Loki are the teams that only output structured Logs, but if you have third party systems that aren't too amenable to that - you'll have issues. • We've seen instances of missing logs (we have two instances and they would be missing different logs) and there are (were last time I looked) open issues since a year plus back on Loki regarding this, but no answer from the developers.