DevOps metrics is a trending topic among software engineering managers. These metrics became very popular after the publication of Accelerate — The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations and the yearly The State of DevOps reports.
If you’re not familiar, here’s are the 4 DevOps Metrics proposed by the work:
- Delivery Lead Time, which is also known by Lead Time for Change or Cycle Time
- Deployment Frequency, also known as Delivery Throughput
- Time to Restore Service
- Change Fail Rate
If you search for “DevOps metrics,” you’ll quickly find many articles listing much more than four of them. Although I wouldn’t say they’re wrong, that’s something we need to pay attention to. Overseeing too many metrics — or DevOps KPIs, to cite a common term — at once can bring more problems than solutions.
For this article, I prepared a brief list with three benefits of using those metrics so that you can accelerate engineering deliveries at your place.
Benefit #1 — They consider only the engineering scope.
In the article A primer on engineering delivery metrics — Scaling software & systems, Buritica states that the product conception and design phase varies too much and then defends measuring what the engineering team is responsible for: from commit to deploy.
It makes sense. The product design phase defines what should be built, and considering it as engineering work distorts its efficiency. On the other hand, focusing on the product delivery phase allows managers to understand precisely how shipping code to production works.
In short, DevOps metrics show the big picture of your whole engineering process.
Benefit #2 — It promotes collaboration and alignment.
I don’t believe in developer or engineer productivity. Metrics such as the number of Lines of Code added, commits, and user story points delivered usually bring anxiety, depression, and burnout — to cite a few of them mentioned in Isaac Lyman’s article Can developer productivity be measured?
Modern Software Development requires collaboration. More than that, it relies on teamwork. It’s like a complex system that needs a hint of chaos to work appropriately. Because of that, it’s useless to measure individual productivity. If you measure each individual by inputs or outputs, the system will adapt to it, and then you’ll see a local improvement.
However, what about the other vital aspects of engineering success, such as quality, stability, and maintainability? The team probably won’t take them into account during the daily activities because what matters at the end of the day is to reach a certain number of inputs or outputs done.
In my opinion, this is a management anti-pattern. If we want engineering teams to collaborate, then we need to see how they work together. Every institutionalized practice, process, or automation changes how the system behaves altogether.
That’s why you need metrics that fully represent the big picture to understand if they have improved or worsened the team’s dynamic. As a result, teams work with high autonomy guided by the alignment provided by metrics that tell what’s expected from them.
Benefit #3 — Improving the system is more effective.
If you compare two individuals, given they’re humans, it very likely they won’t perform the same. Depending on the type of demand — whether it’s a bug, a new feature, or PoC -, their motivation, how clear the task, the related experiences they had, and many other aspects can cause divergences in the numbers.
In a Leaderboard, there will always have someone at the bottom. Even if you start a mentorship program to leverage individual productivity, it doesn’t mean all engineers will produce the same.
However, you can find whether a mentorship program is working by looking at the system. If the system is delivering more, with quality and reliability, it’s working. It doesn’t matter who is at the bottom or the top of the Leaderboard.
Daniel Markovitz wrote in an article for the Harvard Business Review called Productivity Is About Your Systems, Not Your People:
As legendary statistician and management consultant W. Edwards Deming argued in his book Out of the Crisis, 94% of most problems and possibilities for improvement belong to the system, not the individual. I would argue that most productivity improvements belong there as well. Personal solutions can be useful, but the most effective antidote to low productivity and inefficiency must be implemented at the system level, not the individual level.
Understanding the impact of engineering initiatives, projects, and daily activities on the system outcomes makes it much more likely to managers drive their teams to high-performance.
You’ll probably need more metrics later.
DevOps metrics are enough for having the big picture. But sometimes, we need to zoom in and see how the system behaves in a specific part of the process. In this case, you’ll need more metrics.
You may have a better idea of what I’m talking about reading Smruti Patel’s article Debugging engineering velocity and leading high-performing teams — Productivity & Eng velocity. Breaking the process into smaller steps is fundamental to understand where bottlenecks and issues are. Then, managers can complement their view by instrumenting those steps.
These complementary metrics give managers visibility and allow the teams to detach from inputs and outputs. So, they can elaborate on sophisticated solutions that impact the whole system.
However, the team should align every action with those 4 DevOps Metrics.
In short
The 4 DevOps Metrics are excellent to give managers a big view. However, to debug the process, they need to instrument the system using other metrics. It’s like Observability, but for the process.
For more ideas of how and what to measure, I strongly recommend our latest e-book, The Engineering Manager’s Play for debuggable and observable processes. It has quality content with examples of what to measure in each delivery phase.
See Engineering Metrics Live: Click here to get a full demo of SourceLevel Engineering Metrics and how they can help your team.
Originally published at https://sourcelevel.io on January 14, 2021.
