Anyone have suggestions on how to help your users see that their logging is out of control?
l
Leena
01/19/2022, 6:39 PM
are you able to share the associated costs of said logging? eg resource/Datadog billing with them?
👍 2
a
Alan Barr
01/19/2022, 6:42 PM
That can be shared for sure. Just trying to figure out if i'm missing something or if someone's found a better way that what we're doing haha
a
anttip
01/19/2022, 6:56 PM
We did something similar once. In order for Grafana to be able to manage all the logs and return query results, we needed a lot of compute for it, and we showed them the AWS costs
”So every time you wanna run those debug queries we spend $50 for each query” kind of way
a
Alan Barr
01/19/2022, 6:57 PM
whoa gnarly
s
Sander Knape
01/20/2022, 11:25 AM
I've also used costs in the past. We ran a monthly Splunk report to get the amount of data per index. Those indexes were mapped to the teams that owned them, and based on the Splunk costs for that month we were therefore able to calculate the costs per team.
👍 1
j
Joseph Khafaji
01/21/2022, 11:14 PM
Show them CW logs costs as an example
We noticed one day that one app was logging the fact that it logs which logged the fact that it logs 😅
h
Hal Lesesne
01/22/2022, 1:36 AM
Limit retention duration to something you can tolerate.