https://platformengineering.org logo
#general
Title
# general
s

Simone Casciaroli

11/12/2022, 10:22 PM
Hi I wrote a piece on what metrics should drive Platform Engineering, it sounds related to what @Arun Gantagoru was asking and given that is quite an opinionated piece I would love some feedback.  https://medium.com/@simonecasciaroli/platform-as-a-product-how-to-set-your-objectives-ee798e65a7f0
a

Arun Gantagoru

11/13/2022, 3:19 AM
@Simone Casciaroli, Thanks for sharing your article. I like your KPI's for a platform team. trying to summarize: Happiness: Increase customer satisfaction 15% on our monthly survey Efficiency: Decrease time your team spent onboarding a customer Adopters: Task-Success: Decrease 30% how long it takes to create a new service using the platform My feedback & takeaways: Happiness can be subjective, there has to be a clear constant feedback loop - a process to measure Happiness? I assume you are proposing a survey like structure to measure that emotion? The questionnaire for survey must be defined - Just for sake of terminology may be I would state this as satisfaction Efficiency : Absolutely like the efficiency metric. This is an important factor and I assume opening tickets and teams engagement on those tickets would be able to measure efficiency. Task Success: This KPI can again be subjective, based on what new technology, capability you are trying to develop and build for organization.
s

Simone Casciaroli

11/13/2022, 9:13 AM
Thanks for the feedback, yes the summary of the metrics (that are an adaptation from the Google HEART metrics) are: Happiness: how satisfied is your customer (equal to internal teams from now on) with your product? [This is certainly a qualitative metric, so I normally have a monthly survey that includes it] Efficiency: how long does it take for your team to onboard, support and enable the service you provide? [This is purely quantitative and is based on support tickets or how much you are involved to onboard new customers or to enable their features] Adoption: how many customers are using your platform or committed to use it? (a very key metric) [again, a quantitative metric based on usage or commitment] Task Success: this is the flip side of efficiency. How long it takes for a new customer to learn your platform? how long it takes for them to be able to solve their problems with it? how often does it use capabilities of your platform to solve his needs? [this can be quantitative or qualitative based on what you want to measure and what you provide, I had some questions on this in our survey]
As mentioned because it's an adaptation of an existing framework (HEART) that is used in the product team I kept Happiness although I have the same "feeling" about the word and prefer satisfaction.
a

Arun Gantagoru

11/13/2022, 11:44 AM
Thanks @Simone Casciaroli, makes sense.
t

Thomas Grünewald

11/14/2022, 1:51 PM
Hi @Simone Casciaroli, thanks for sharing. From my perspective all mentioned indicators are useful, however, I think it is important to emphasize one thing: if you do not ask for customer satisfaction regularly (e.g. combined surveys with quantitative/qualitative output like you mentioned) all your other KPIs go done the drain. In other words, if you have a process in place that manages and reacts upon this single metric, 90% of the job are done. However, then comes the hardest bit: meeting the challenge of asking for feedback and accepting it as your guiding indicator.
r

Russ Nealis

11/16/2022, 1:56 PM
I think DORA metrics are the most critical ones to watch. They are the ultimate “outcome” metrics and have the most research behind them (e.g. statistically predictive of positive business outcomes). Beyond that I use CSAT across a number of platform capabilities (e.g. building, operating, and delivering software agnostic to specific technologies). Remember that if you’re running an internal platform your customers are not the other engineers at the company (a common framing mistake IMHO), your customers are the same as everyone else at the company: the customers of the company. You need to do what’s best for them (e.g. maybe putting more focus reliability at the expense of velocity, which may upset internal developers but be what’s best for your paying customers). If you lose sight of this things can fall apart. Check out this old gem https://www.svpg.com/platform-product-management/
s

Simone Casciaroli

11/16/2022, 5:03 PM
This is super interesting and the opposite of what I think. Thanks for sharing and let me mule it though a bit before responding.
Thanks @Russ Nealis again for the great read. It helps me understand where I'm at and my biases in terms of Internal Platforms' metrics. The first thing I do agree with is the article. I used to run a SaaS company and in B2B is extremely common to consider both the customer and the user needs separately. It also makes sense that for an external platform (what the article refers to), there is a third set of needs to consider. Where my thought starts to potentially clash with what you wrote is that internal platforms are very different from external ones. And my advice is only battled tested for Internal platforms. For example, across all the companies I worked with, the voice of the developers was hardly heard, while instead, mostly what was driving platform development was the senior leadership's perceived need applied in a controlling way: • we need more resilience; we will enforce it at the platform level • we need more security, ... • we need to improve some of the DORA metrics; the platform should be responsible While I believe that part of a platform's job is ALSO to create an environment where the right behaviours are incentivised and the wrong behaviours are avoided, my drive to push for Product/Developer focused metrics stems from those three considerations: • The focus on DX is strongly connected with the lack of it when building internal tools. I would love to hear if your experience differs, but this is what I observed in places that are not BigTech. • Ultimately all the non-functional requirements above need product teams and platform team focus. If I have them as Product teams metrics, they will trickle down to the platform (again, this is not strictly true for every product the platform maintains, so I certainly have non-functional metrics for platform teams when needed) • And ancillary to the above most of the behavioural changes described above are non-functional requirements (security, reliability, DORA) that can't be plugged in and require co-ownership. An excellent way to make it happen is to turn it into a need for the customer (internal teams) Your message helped me realise that my bias is that I can operate and think at the system level because I'm responsible for the broader engineering function, so I can optimise the whole system. Product/Customer metrics are valid (IMHO) even if your other teams don't operate with a co-ownership mindset like described above. However, I would love other people's points of view on this because it's an important aspect and is quite foundational to my "thesis". What do you all think?
r

Russ Nealis

11/19/2022, 1:20 PM
(My background in case there’s confusion - I’m completely focused on Product for Internal Developer Platforms, not external) I’m not saying developer happiness is unimportant to keep tabs on, I pay very careful attention to this too. For example, I consistently run quarterly internal developer NPS/CSAT surveys and collect feedback on individual products + capabilities. However, I consider a great internal developer platform experience as a means to an end with respect to the ultimate goal of an internal platform: finding the optimal balance of velocity + reliability + cost so developers can optimally ship value to our customers. The DORA metrics better capture that ultimate outcome on the velocity and reliability front. As an aside, if you look through the DORA research it does talk about how optimizing for those metrics maps to all sorts of benefits (e.g. less burnout) that are related to developer happiness. I think the relatively new SPACE framework is more deliberately capturing this as “satisfaction.”
Satisfaction is how fulfilled developers feel with their work, team, tools, or culture; well-being is how healthy and happy they are, and how their work impacts it. Measuring satisfaction and well-being can be beneficial for understanding productivity20 and perhaps even for predicting it. For example, productivity and satisfaction are correlated, and it is possible that satisfaction could serve as a leading indicator for productivity; a decline in satisfaction and engagement could signal upcoming burnout and reduced productivity.
https://queue.acm.org/detail.cfm?id=3454124 In the end this boils down to inputs -> outputs -> outcomes. inputs - the things you work on in your developer platform outputs - your developer platform, inclusive of happiness attained for developers outcomes - developers are optimally leveraging / your platform to deliver value for the business Outcomes are the most important.
98 Views