This message was deleted.

I can’t speak for all orgs but at Wayfair they had a centralized observability team within platform engineering. Since 2018 to now this has mainly been orchestrated by https://www.tremor.rs ( disclaimer - i’m a maintainer of the project ). In 2018 influx was used for metrics with elastic for logs - mostly on premise. In 2023 Wayfair are mostly cloud native. With otel - you could start with the otel-collector and map your other log formats to this ( using a tool like tremor, or vector ) and then forward it to a cloud provider that supports otel like Honeycomb ( or Axiom.co where i work ) or to your internal systems of record if they support otel natively.

OTELCOL id think

Yes. Best place to start if there’s no remapping or transformations to do and very quick to test

@Endre Karlson @Darach Ennis thanks a lot for valuable information

You can with tremor as it has otel sinks and sources. Vector IIRC doesn’t have otel sinks

I think, otel-collector does not have any mechanism to filter the logs, am I right?

No it does have Processors for this purpose

So no need to use different vendor, am I correct?

I think it totally depends on what you want to do and if OTC supports what you want to do 😉

This. If OTC gives you what you need then its the simplest workable solution

tbh, we are trying to find the best practices. In our organization. as I mentioned before we have ~400 engineering teams using their own k8s cluster

If you have diversity of sources and no consistency around normalization - than having the ability to normalize the various feeds is useful. This is where something like tremor becomes useful

IHMO, try OTC and see how far you get

For example - if you need to classify logs and do some kind of traffic shaping etc..

You can do alot of remaps / tranform just by using OTL instead of Tremor, and introducing Tremor is then a diff piece the puzzle to learn, maintain, deploy etc... and if you need something more advanced for the task Tremor is great I think