I’m curious to hear some examples of how observabi...
# observability
I’m curious to hear some examples of how observability teams guard log access to log data that contains PII. Do you allow all devs to query any log or do you have rules/config to lock down access so that they can only access log data for services they own?
It depends on your requirements. Sometimes PII is scrubbed during export e.g. with fluentbit or some custom filtering. This is easiest and I think the most common approach. Depending on your volume, you can also do fancy filtering to replace sensitive userids with unique ids so you can still correlate log entries to a user but decouple that from the actual user. You can also send raw logs to one destination and filtered logs to another. And then limit access to the raw logs. This is more or less my approach as PII is not usually needed by anyone except in special circumstances (e.g. lawyers). If you need the PII in the logs, then you can restrict access via access controls in viz tools like grafana, tableau, sentry, whatever. You can also create queries that filter out PII for dashboards and other panels (but only in combination with access controls). Also not all tools have built in query audit logs so if you do keep PII accessible to devs in your logs, you may need to add an audit log mechanism depending on your rqmts.
And dont forget to make sure your log storage is compliant with your goals including any backup or versioning mechanisms.