I’ve been using Engineering Metrics for a while, and I am amazed by the significant impacts on engineering teams. Metrics is a topic I genuinely care about. During my days at Plataformatec, we used to discuss a lot about them. At the time, we used to focus on an Agile perspective.
Our team was not satisfied in measuring Velocity in Story Points, nor we thought that estimating was helpful. So, we dug into Kanban philosophy and learned new statics techniques.
We found out that measuring the cards’ flow through the board was more accurate and useful and that forecasting using Monte Carlo could give us more realistic estimations.
Despite having many customers, we discovered that most of them had similar problems: lack of visibility and alignment. Once we introduced our metrics, the product teams had a north star.
Besides, we also noticed that many CTOs had no metric to show to the board. The CMO could prove they were bringing qualified leads, the CFO had data to show their team reduced costs, but CTOs usually had no concrete metrics.
Well, in summary, that’s the story behind SourceLevel. Over time, we received lots of valuable feedback, and we’re confident our product is in the right direction. We are helping companies to accelerate their deliveries sustainably and effectively.
For this article, I separated three signs that you need Engineering Metrics in your teams. Check if you identify with any.
#1 — You don’t know in which step of the product delivery bottleneck is.
For a commit to reach production, the code goes through a lot of steps. Many of them rely on human activities, like coding and peer-reviewing. The nature of this kind of work varies in quantity, time spent, and many other intrinsic variables.
When you analyze the big picture, engineers in all levels of experience collaborate to solve a diverse type of demand, from bugs to refactorings, from typos to new releases, and so on.
Systems are more complex than the sum of the parts. That’s why measuring individuals make no sense. The outputs produced vary too much in sizes, shapes, and purposes.
Besides, overseeing everything is very difficult. It increases significantly when the team scales up. So, you need to instrument each step by gathering data from commits, pull requests, and deploys — as a start -, so that you can inspect them to find improvement opportunities.
If you think there’s no bottleneck in your process or don’t know where it is, I strongly recommend instrumenting your system with Engineering Metrics.
#2 — You don’t have any metric to report upwards.
Every area has its standard metrics. You work for many companies, but the CMO or CFO, for instance, will always report pretty much the same metrics. However, the engineering area still doesn’t have a consolidated metric to report upwards.
I’ve seen many CTO fail because they couldn’t prove the team was making progress or that demand was over the capacity. Many executives still rely on gut feelings and outdated estimating methods.
By measuring the whole process and then being able to drill down on specific parts of the system, CTOs and managers can understand their initiatives’ impact and create better reports.
#3 — You don’t know whether the team is making progress towards desired outcomes.
When you focus on outputs, the results usually don’t go along with the outcomes. If the team’s productivity is measured by how many Pull Requests get merged, smart engineers will close as many Pull Requests as possible. It doesn’t matter whether the team reviewed it properly.
The organizational culture is abstract and changes frequently. Metrics are a crucial factor. Measuring the outputs, as the number of Merged Pull Requests, isn’t enough. Leaders need to measure the outcomes.
The engineering outcome is not only “code in production.” As a delivery system, we want engineering to deliver clean, well-tested, and reliable code to production faster in a sustainable — and predictable — way.
When the leaders spread this message across all teams and give the team autonomy to do whatever change they need, we have the benefits of measuring outcomes: teams are not attached to outputs to experiment and innovate. The chosen set of metrics provides the alignment.
For example, a way to deliver faster would be moving from a Pull Request approach to a trunk-based one. It can accelerate deliveries, but on the other hand, would the code still be well-tested and reliable? If the team is not mature enough, there are great chances it would not.
That said, after defining the outcomes for your engineering success, I strongly recommend looking for a set of metrics that reflects the results.
Do you want to know more about Engineering Metrics?
If you want to read more about Engineering Metrics, I’ve written a lot about them recently. You can find them under this category.
Feel free to schedule a 15-min chat with me to talk about the challenges at your company. Maybe, I can handle more resources for you.
Originally published at https://sourcelevel.io on January 28, 2021.
