Hey All, What is the best practices nowadays for c...
# general
s
Hey All, What is the best practices nowadays for collecting logs? In our organization, there are hundreds engineering teams has their own kubernetes cluster. As a platform team, we would like to standardize the logs via centralized way. Filtering logs, prevent unstructured logs and etc. How can we achieve this via otel? Any suggestions will be valuable.
d
I can’t speak for all orgs but at Wayfair they had a centralized observability team within platform engineering. Since 2018 to now this has mainly been orchestrated by https://www.tremor.rs ( disclaimer - i’m a maintainer of the project ). In 2018 influx was used for metrics with elastic for logs - mostly on premise. In 2023 Wayfair are mostly cloud native. With otel - you could start with the otel-collector and map your other log formats to this ( using a tool like tremor, or vector ) and then forward it to a cloud provider that supports otel like Honeycomb ( or Axiom.co where i work ) or to your internal systems of record if they support otel natively.
I know @Endre Karlson has built an otel based system for this at his last organisation - he was using
tremor
and i believe victoria metrics and opensearch in that environment. I believe otel collectors were used in some of the source systems
It is getting easier every day to use otel for this purpose. Would love to hear about your successes and any gotchas if you try this route!
e
OTELCOL id think
d
Yes. Best place to start if there’s no remapping or transformations to do and very quick to test
s
@Endre Karlson @Darach Ennis thanks a lot for valuable information
So can we use otel-collector with tremor or vector to filter the logs?
d
You can with tremor as it has otel sinks and sources. Vector IIRC doesn’t have otel sinks
s
I think, otel-collector does not have any mechanism to filter the logs, am I right?
And one more question, what is your experience, does it make sense to deploy otel-collector of each kubernetes cluster( +500 clusters) and collecting logs from each of them to centralized one?
e
No it does have Processors for this purpose
to Filter & Transform using what they call Opentelemetry Transformation Language -> https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md
s
So no need to use different vendor, am I correct?
e
I think it totally depends on what you want to do and if OTC supports what you want to do 😉
d
This. If OTC gives you what you need then its the simplest workable solution
s
tbh, we are trying to find the best practices. In our organization. as I mentioned before we have ~400 engineering teams using their own k8s cluster
Main goal is centralized the logs in single place
With log management policies
d
If you have diversity of sources and no consistency around normalization - than having the ability to normalize the various feeds is useful. This is where something like tremor becomes useful
e
IHMO, try OTC and see how far you get
d
For example - if you need to classify logs and do some kind of traffic shaping etc..
Starting with OTC is still a great start
e
You can do alot of remaps / tranform just by using OTL instead of Tremor, and introducing Tremor is then a diff piece the puzzle to learn, maintain, deploy etc... and if you need something more advanced for the task Tremor is great I think