Hey everyone, Can any one help me understand how should i be thinking about what metrics to look for when it comes to test automation for improving developer experience? Context: I am trying to create a mental model on how developer experience teams should pick initiatives to improve testing experience for backend developers. I believe beyond NPS as a primary indicator there could be other metrics like Test coverage, build quality, CI failures etc.. that could help create a more rationalise approach to decide on investments and measuring success of investment.

Please check

From a platform perspective, a big metric for me is: What percentage of tests fail for reasons that aren't "developer's fault"? I'd also be interested in test execution time, especially how early in the pipeline we can detect failures. (e.g. why couldn't this have been found before it got to CI?). There are a ton more metrics I'd be interested in as a development team who owns the test (coverage, pass rate, etc), but that seems outside of the scope of what I'd expect platform to monitor.

An interesting metric to track is the stability of a test suite. Flaky test suites encourage people to ignore failures and just “rerun until green”. Overall suite time is also important, as if the test runtimes start getting too long people start taking shortcuts. Test coverage is critical, but it’s a double edged sword as it can be a vanity metric. It can encourage people to write huge amounts of low value tests just to increase that coverage metric. To @Justin Abrahms’s point above, having independent failure metrics per stage are useful rather than just tracking the overall failure rate. You’d expect and probably accept a much higher failure rate earlier in the pipe, so you want to know how early you are catching things

Thank you Justin, Kyle and Jay, I took a shot at it to define a mental model here. Sharing it here to hear what you think. Goal of test automation platforms is to remove toil on developers to create, maintain, execute and learn from tests in a semaless way so that they can move with faster velocity and have high confidence on outcomes. Therefore, for a developer any initiative needs to measured on how it is improving velocity or building confidence in developer on what they are building. And guiding metrics here could be: 1. Improved Velocity a. Test environment setup time b. Test execution times c. % of tests failures (not because of developer's fault) 2. Improved confidence a. Test Adoption - Shift Left b. Quality and coverage of tests at each stage c. Stability and scalability of features d. Outages What do you all think?

Is your intent to measure the efficacy of your platform or your developers? Test execution time is the latter as is your whole second section.

Hey Nikhil, I would check out some of the work that Abi Noda does at DX - he produces some very detailed articles on Dev Ex - https://www.linkedin.com/in/abinoda/