Anyone have suggestions on how to help your users see that their logging is out of control?
01/19/2022, 6:39 PM
are you able to share the associated costs of said logging? eg resource/Datadog billing with them?
01/19/2022, 6:42 PM
That can be shared for sure. Just trying to figure out if i'm missing something or if someone's found a better way that what we're doing haha
01/19/2022, 6:56 PM
We did something similar once. In order for Grafana to be able to manage all the logs and return query results, we needed a lot of compute for it, and we showed them the AWS costs
”So every time you wanna run those debug queries we spend $50 for each query” kind of way
01/19/2022, 6:57 PM
01/20/2022, 11:25 AM
I've also used costs in the past. We ran a monthly Splunk report to get the amount of data per index. Those indexes were mapped to the teams that owned them, and based on the Splunk costs for that month we were therefore able to calculate the costs per team.
01/21/2022, 11:14 PM
Show them CW logs costs as an example
We noticed one day that one app was logging the fact that it logs which logged the fact that it logs 😅
01/22/2022, 1:36 AM
Limit retention duration to something you can tolerate.