https://platformengineering.org logo
Title
h

Heiki Riesenkampf

08/26/2022, 1:03 PM
Hey everyone 👋🏼 A question for the community members here. What tool do you use to access production database for debugging purposes? Eg let’s say a bug is reported that only appears in prod environment and you need to look up the exact object that is causing the issue, what do you use to go digging? Also, do engineers in your company have static database access keys or is the access permissions somehow granted on-demand? I am trying to find the best solution for our team since our current process/tool is not ideal.
t

Taylor

08/26/2022, 2:59 PM
My current workplace is a wip, but at the last place I worked we used a tool called db labs to provide access to clones. https://postgres.ai
That way devs did not need to access the live prod database for things. We typically required whatever fix to be written as a migration, except for some specific circumstances where it didnt make sense to do that
h

Heiki Riesenkampf

08/26/2022, 3:28 PM
Thanks for your input @Taylor! I did not look into data fuzzing tools. I will do more digging. Any other alternatives?
a

Aleksandar Borkovac

08/26/2022, 4:11 PM
Tools like Dbeaver, TablePlus, etc… DBs are backed by VPN and AAD access control (with AD groups) and we are looking into using Azure PIM for privilege elevation.
I also used Self Service Group ID before - replacement for Azure PIM
c

Craig Alfieri

08/26/2022, 5:35 PM
@Heiki Riesenkampf we support companies with database simulation testing /debugging… is a Production environment a hard requirement to debug?
t

Taras Mankovski

08/27/2022, 6:09 AM
@Craig Alfieri what do you use for database simulation?
d

Daniel Solsona

08/27/2022, 10:12 AM
At a previous company we had a self-service tool that would allow engineers to get dump of a database, with all the sensitive and personal data redacted. Developers would send a request, it would get reviewed and approved, then a mysqldump was executed and the developer was able to download it from S3
c

Craig Alfieri

08/27/2022, 11:35 AM
@Taras Mankovski company just came out of beta, name is Antithesis ; former team from FoundationDB started it. I’ll DM you some GitHub details
o

Oshri Paryanti

08/28/2022, 9:07 AM
@Heiki Riesenkampf - we at groundcover support your use case of identifying the exact object causing problems (regardless to DB or any other object) thanks to our eBPF based solution (monitors everything on kernel mode, no integration is needed). Feel free to check our free tier license or alternatively check our blogs where we share use cases. Regarding your question of “static” keys to prod - as a former cyber security consultant I can suggest looking at PAM solutions, providing on demand access, recorded, password change, approval processes etc…
d

Dishant Pandya

09/02/2022, 9:15 AM
Hi @Heiki Riesenkampf we have implemented Boundary + Vault with fine grained access control to allow connecting prod db. Lets say someone needs to debug into prod, db they can be allowed access temporarily by attaching specific role to their user, which would authorize them again corresponding boundry and vault acl, and let them generate temporary credentials and tcp connection to db