Log aggregation and analysis We are looking for a cost effec

Question

Log aggregation and analysis We are looking for a cost effective scalable and reliable log analysis solution We currently use a mix of cloud watch data dog and sumo Is there anything you can share about how you resolved a similar issue What was your architecture Any open Source tooling What paid tools did you use Thanks

Nathan Hruby · Answer

This is guided by log volume and rate use cases for queries end users security devs ops etc structure of events retention time

Nathan Hruby · Answer

probably you have some mix of folks needing fast reliable eventing that create metrics and metadata on the stream volume highly detailed and relatable events liek traces and bulk queries to answer security and product type questions which probably means you may need multiple tools datastores

Swarna Mani · Answer

What tools would you recommend

Swarna Mani · Answer

Thanks Nathan

Endre Karlson · Answer

if you need something that seems promising in as a newer alternative to opensearch checkout quickwit

Nathan Hruby · Answer

Grafana Loki and Graylog are two tools not in your list that I ve used before and work pretty well

Nathan Hruby · Answer

oh hahah and and of course slightly smiling face

Nathan Hruby · Answer

and if you wanted to jsut do it yourself in aws <https www aha io engineering articles log management using aws athena|this> is a good high level overview

Swarna Mani · Answer

Thanks < Nathan Hruby>

Ric McLaughlin · Answer

Seen a growing use of Athena as described in that Aha post for the log analysis use case at a cost way way lower than CW Logs

Ric McLaughlin · Answer

For some element of real time analysis I d be doing something like kinesis gt stream filters to trigger a lambda gt firehose gt OpenSearch collection and then OpenSearch serverless for analysis

Nathan Hruby · Answer

opensearch just gets expensive at massive data quantity this is where a firehose that splits the data into s3 for longer duration months years etc and opensearch for short term days weeks gets very cool

Daniel Serodio · Answer

Coming late to this thread since I just found it via the newsletter but <https chaossearch io |ChaosSearch> SaaS seems really interesting for this I ve watched a demo but haven t implemented it yet

Endre Karlson · Answer

< Daniel Serodio> Isnt that like Quickwit does then they to do scale out on s3 elastic ish

Daniel Serodio · Answer

Never heard of Quickwit before but looking at its homepage is looks like ChaosSearch is 100 SaaS Serverless no need to worry about servers while Quickwit is self hosted self managed

Endre Karlson · Answer

depends on qhat you need want

Ashish Hanwadikar · Answer

We <http kloudfuse com|kloudfuse com> have a unified observability platform that addresses logging metrics traces kubernetes events etc all in a single database that is custom designed from grounds up for storing timeseries and event records We have a free to download version that you can use forever I will be happy to demo the product you can also a live demo from our website Because of the storage efficiency and ability to run in your VPC the solution is extremely cost effective providing 70 90 cost savings compared to SaaS vendors Hope you will try it out