Hey everyone 👋🏼 A question for the community members here. What tool do you use to access production database for debugging purposes? Eg let’s say a bug is reported that only appears in prod environment and you need to look up the exact object that is causing the issue, what do you use to go digging? Also, do engineers in your company have static database access keys or is the access permissions somehow granted on-demand? I am trying to find the best solution for our team since our current process/tool is not ideal.

My current workplace is a wip, but at the last place I worked we used a tool called db labs to provide access to clones. https://postgres.ai

Thanks for your input @Taylor! I did not look into data fuzzing tools. I will do more digging. Any other alternatives?

Tools like Dbeaver, TablePlus, etc… DBs are backed by VPN and AAD access control (with AD groups) and we are looking into using Azure PIM for privilege elevation.

@Heiki Riesenkampf we support companies with database simulation testing /debugging… is a Production environment a hard requirement to debug?

@Craig Alfieri what do you use for database simulation?

At a previous company we had a self-service tool that would allow engineers to get dump of a database, with all the sensitive and personal data redacted. Developers would send a request, it would get reviewed and approved, then a mysqldump was executed and the developer was able to download it from S3

@Taras Mankovski company just came out of beta, name is Antithesis ; former team from FoundationDB started it. I’ll DM you some GitHub details

@Heiki Riesenkampf - we at groundcover support your use case of identifying the exact object causing problems (regardless to DB or any other object) thanks to our eBPF based solution (monitors everything on kernel mode, no integration is needed). Feel free to check our free tier license or alternatively check our blogs where we share use cases. Regarding your question of “static” keys to prod - as a former cyber security consultant I can suggest looking at PAM solutions, providing on demand access, recorded, password change, approval processes etc…

Hi @Heiki Riesenkampf we have implemented Boundary + Vault with fine grained access control to allow connecting prod db. Lets say someone needs to debug into prod, db they can be allowed access temporarily by attaching specific role to their user, which would authorize them again corresponding boundry and vault acl, and let them generate temporary credentials and tcp connection to db