Measuring Platform Team success through Developer Happiness

Our platform team provides services underpinning the work of something in the order of 1,000 people spread across different roles - developers, architects, support engineers and so on. Each month, there are hundreds of thousands of interactions with our tools - almost all of which are automated through CI. As the Product Lead, I am often asked how i know what to prioritise next in delivery and the answer is simple - we measure Developer Happiness. now don’t get me wrong, i want our developers to be happy, but it’s not that i specifically care about developer happiness in itself. happy developers are great, but in this case what i’m looking for is things that are annoying them - i’m using Developer Happiness as a proxy for all the things which we’re not measuring (or simply can’t measure) but which annoy the teams on the ground as they try to get their job done.

Measuring Platform Team success

Most traditional measures of the “success” of a platform team look at metrics like adoption rate, service uptime, mean time to onboard, or recommend frameworks like SPACE or DORA, and we run a bunch of these as standard Operational KPIs - are our platforms up and running, performing as expected, are tickets getting a response within SLA and so on. While these are useful (SPACE in particular includes “developer satisfaction”) i think that measuring developer productivity is not all that helpful when it comes to informing Platform direction. While productivity measures might be able to tell us that something has changed, they cannot tell us what or why, especially in the age of AI coding agents, and although we know that uptime and ease of use are important, just measuring these doesn’t help us identify new, valuable services or improvements which would push our Platform Team forward.

Developer Happiness is a proxy for Developer Experience

In 2016, Oliver Hart shared the Nobel Prize in Economics with Bengt Holmström for his contributions to Contract Theory which the Nobel committee summarised as:

Because it is impossible for a contract to specify every eventuality, this branch of the theory spells out optimal allocations of control rights

I find this a useful analogy for the limits of measuring developer experience i.e. we cannot anticipate everything that will change. New technologies, ways of working, preferences and people all introduce unforeseeable obstacles to developer productivity. Because we can’t foresee what might be blocking developers, we can’t measure it up front. And in any case, not all obstacles are directly measurable - cognitive load, workplace interruptions, squeaky chairs, cold coffee are all things which contribute to a poor developer experience, and in any case, Goodhart’s Law tells us that when we try and measure these things, there’s a risk that they become a target and stop being a good measure, and we’ve seen this ourselves. When we started measuring “ticket resolution time”, developers complained that tickets were closed on the first reply from an engineer, and they had to open a new ticket to continue the conversation even if the problem wasn’t resolved.

Measuring Developer Happiness

I’m ok with “happier developers” becoming a target - it’s better than “lines of code per day” as a goal, so “happy developers is our goal” is ok by me. But how do we know how happy our developers are - or more importantly, how do we know what’s stopping them being as happy as they could be? Sending out quarterly emails is one way, but our experience is that response rates are very low (our response rate was around 8%, well below industry average), you tend to get the same people responding over and over, and the quality of the feedback is poor - it requires too much cognitive effort, so we opted for short, 15 minute peer-to-peer chats instead.

As the internal organisation is “closed”, we know which teams are using our tools (around 100), and which teams aren’t (around 50). Each month, we randomly¹ sample around 10 teams², differentiating those who’ve interacted with our platform and those who haven’t, pick one or two people in each and set up a quick, 15 minute call with them - around 10 calls a month. Each of our platform engineers takes it in turn to run the call, and they open with one simple question: how happy are you?

We dont limit their responses to the Platform Team offerings - we sometimes hear feedback i need to bring to the attention of other leaders in the organisation - but more often than not they’re things which are in our control. They’ve told us about API error responses which are confusing or unhelpful. We’ve discovered design gaps in our APIs which caused manual steps in their workflow. And often we hear of new services or technologies they’d like us to consider. Their feedback, along with their own product goals and KPIs, and general industry, market and technology trends, directly inform how our Product Owners think about evolving their products, or developing new offerings - think about it this way: we’re not actually measuring Developer Happiness, we’re searching for things which make developers unhappy.

Developing plans

Overall, we break work down in to 4 categories, and fill the team’s capacity in this order:

1. Tickets and support - people contacting us for help or about things which arent working as they should.
2. Routine maintenance - patching, minor database optimisations, reviewing access control exception alerts etc.
3. Planned maintenance - major upgrades (e.g. a new Kubernetes or database version), major platform changes (such as swapping out ingress controller etc.)
4. All other work

Obviously we need to ensure that we have sufficient capacity to work on bucket four, but after that’s done, what do we deliver? Well, we run a Feature Prioritisation Board - this advisory body contains representatives from our four delivery branches: Engineering, Architecture, Support and Security. They meet monthly, and express priority preferences over a range of items, some of which they suggest, but many of which come from the people on the ground.

We take the roadmaps and recommendations to the Prioritisation Board who either endorse the roadmaps or recommend changes. As i noted above, the feedback is advisory (the Product Owners have the final say over their roadmaps), but the Prioritisation Board has skin in the game - as we progress the roadmap, they are required to identify one or more teams to pilot new or changed services with, and are jointly responsible for our Platform Uptake KPI.

Ultimately, i dont really expect to discover everything that’s making developers unhappy, and we certainly can’t solve everything that is, but I feel that our approach has a number of benefits over widely distributed, untargeted emails.

• We reach quieter teams who would otherwise simply slog on or even just abandon our tools.
• Because we’re not just looking at our standard operational KPIs, we constantly discover things we would otherwise never have known to look for, and we do this without having to know what questions to ask up front.
• Our Platform Engineers are constantly exposed to the people using their services - meaning they understand their day to day experience much more closely, letting them develop better, more helpful services.

Footnotes