@Alison Rosewarne You mentioned that one of the PMs in your org was measuring an “hours saved” metric for your OKRs … can you give any detail about how you approached measuring this?

We didn't use it as OKR but did track an approximate metric for this in some tools. One example: we built some automation to auto-merge PRs when approved, send reminders to reviewers, or send PRs with dependency updates (this was some time before GH implemented many of these). We just took conservative estimates of how much each of those actions took on average (e.g. "sending a PR with a lib update takes 10 min") and multiply by executions of the tool. 10 mins may look small, but if you're automating this across the org the numbers add up to quite a lot already. And, being conservative gives you the luxury of not having to defend the number (in fact, you can easily argue that it actually saves a lot more than 10 mins). We didn't use this so much for OKRs, but it did appear in a few presentations to execs in this form: "Look, it took N engineer-days to build this tool, and now it's saving 10*N engineer-days per month." It helps a lot to justify getting headcount.

It's an evolving practice but involved: 1. Comparison of before platform and after platform efforts for the work no longer required, e.g. if you use our web micro frontend approach you save 90 days up front and then ongoing maintenance costs. 2. Multiplying effort to build components as savings everytime you use them. e.g. a button in our design system took so many hours to build so everytime you use a button you've saved those hours. 3. Reduction of toil time savings - some automation saves ongoing manual effort (especially per AWS account which really adds up).

To what extent did you find yourself using potential time saving to prioritize vs estimated savings to demonstrate benefit after the fact?

This type of very objective metric was quite useful for us to focus the team and related to the autonomy topic we discussed a few days ago. A typical problem we has was "engineer has an idea about $tool, wants time to build it, will get annoyed if we say no". This type of conversation can lead to engineers feeling frustrated because we impose features etc. So we tried to say "ok, bring us the data: show how much time your tool is expected to save per execution, how is it going to get adopted, etc.". If you bring a business case and it works out, let's do it.

Yeh that makes sense. Used carefully it sounds like a useful practice - thanks for the info!

Orgs also need to decide how they want to balance throughput vs latency. There’s a lot of tools that reduce engineers cycle time locally but have no actual impact on the throughput of the organization. Using time saved metrics works well at macro blocks but breaks down at the micro bc that time isn’t recapitalized into something else.

To what extent did you find yourself using potential time saving to prioritize vs estimated savings to demonstrate benefit after the fact?

Historically I'd say we were definitely focused on the latter - demonstrating benefit after the fact. Moving forward we want to do both (prioritise according to hours saved as well as report actuals).

In our team called Ethos at Adobe, we measure zero to hello world and zero to production readiness (cycle time) as key metrics which we report on a quarterly basis. We have a semi-automated way of capturing these for now so we ask developers who go through the onboarding documentation to track these times in a spreadsheet (not perfect) which we use as a baseline and calculate the mean time over the year to see performance.

We did this too ^ I'd note something I mentioned in my talk though: I found it dangerous to frame the platform's benefits as cost reduction. So when we used that type of metric we gave it a spin and presented those numbers from a different angle. We did not save 10% engineer hours. We added 10% engineering capacity.

@Galo Navarro Very interesting. Were you able to quantify it in $ amount? Wondering what adds the most impact for leadership

You can do hours saved * avg salary

Got it, you mentioned you did a talk. Did you share some of these best practices there? Would love to watch it!

Here is the link

Thank you!