Many Engineering Managers promote Pull Requests as part of the development workflow. It’s a consolidated practice that brings lots of benefits. It consists of comparing the changes of a branch with the repository’s base branch (conventionally called master).
Pull Requests provide useful and actionable metrics. However, following the wrong metrics cause distortions and bring more disadvantages than benefits. In this blog post, I list some metrics I consider useful.
Measuring process, not individuals
Before getting into the metrics, I want to make a disclaimer: don’t use these numbers to compare individuals.
Sometimes a hard-to-find bug requires a single line of code to be fixed, and that single line took a week of work. I’ve witnessed it many times in my career.
I’ve also witnessed Engineering Managers encouraging developers to open pull requests with too many changes that it was unpractical to review. They usually reinforce that by telling everyone these developers are productive, that they are doing the hard job when others are taking the easiest ones.
Measuring individuals through pull requests may even be unfair. A developer dedicated to maintaining a legacy codebase tends to be slower than another one, which works on a greenfield project.
That’s why measuring Pull Requests is tricky. Engineering Managers can’t use pull request data to asses individuals. If you do pull requests, then you want your team to collaborate. In this practice, collaboration is the core value. Developers’ effort can’t be measured just by how many pull requests are open or merged. Even worst, effort does not represent its size.
However, managers can use pull requests metrics to understand the team dynamics and act appropriately to correct behaviors before things get out of track. Here are 5 suggestions of what to measure:
- Time to Merge
- Pull Request Lead Time
- Pull Request Size
- Pull Request Flow Ratio
- Pull Requests Discussions
Pull Request Lead Time
The lead-time metric gives you an idea of how many times (usually in days) pull requests take to be merged or closed. You can calculate lead time by repository or considering all repositories of the organization. The latest option gives you a clearer idea of your team dynamics.
To find this number, you need to track every pull request. Save the date and time for each pull request when opened, and then, when it’s merged, save it too. The formula is easy. Just perform a simple average for the difference of dates.
This metric is especially useful for raising questions and start investigations before it’s too late. A good practice is to measure this number over time so that you can spot trends and behaviors more pragmatically. SourceLevel automatically does that. We calculate lead time by repository and organization daily so that you can compare periods.
Time to Merge
In general, pull requests are open with some work in progress, which means that measuring pull request lead time doesn’t tell you the whole story.
Time to Merge is how much time it takes for the first commit of a branch to reach master. In practice, the math is simple. It’s the timestamp of the oldest commit of a branch minus the timestamp of the merge commit.
Comparing these two metrics (pull requests lead time and time to merge) can be surprising. Let’s see an example.Pull Request Lead Time = 3 days
Time To Merge = 15 days
In the above scenario, a pull request is taking an average time of 3 days to be merged. It is pretty good. But the Time to Merge is 15 days. It means that developers work an average of 12 days (15–3) before opening a pull request. It means a developer opens a pull request once in two weeks. I definitely would investigate why.
Let’s see another example.Pull Request Lead Time = 8 days
Time To Merge = 13 days
In this case, developers open one pull request by week (13–8 = 5 days), much better than the previous example. However, pull requests are taking 8 days to be merged or closed. There are tons of reasons why the lead time is that high. The number is not telling you the reason, but it indicates where the problem may be.
Pull Request Size
Time to Merge can vary, and the causes are many. The average pull request size is one of them. Depending on how many lines of code changes, it requires more or less effort to review.
Ironically, developers tend to merge long pull requests faster. People get kind of lazy to perform thorough reviews when there are too many things going on. So, they immediately approve changes.
There are two problems here: your time to merge goes up, and your quality goes down.
Short pull requests are usually particular. They add a single feature, fix a single bug, or improve a little part of the codebase. They are very straightforward. It makes developers to review with attention and guided by a goal, lessening pull request lead time, and time to merge. It also mitigates the risks of bugs or security breaches.
The Pull Request Size is the average of total lines of code added plus the total lines of code removed. The higher the number, the bigger the pull request is.
Pull Request Flow Ratio
The Pull Request Flow Ratio is the sum of the opened pull request in a day divided by the sum of closed pull requests in that same day. This metric shows whether your team works in a healthy proportion.
Merging pull requests and deploying to production is a good thing. It adds value to the final user. However, when your team closes more pull requests than opens, soon pull request queue starves. It means there may be a hiatus in the delivery.
On the other hand, the more pull requests are open, the higher the probability of increasing its lead time. Creating a task force to review and merge them in batches is the usual practice I have seen in the market. Sooner or later, a hiatus in delivery emerges.
You need to make sure your team merges pull requests in a ratio as close as they open, the closer to 1:1, the better. That’s why to keep an eye on the ratio is essential. A healthy pace makes your flow to be smoother and more predictable. The challenge here is to deliver value to the final user consistently.
Pull Requests Discussions
When I say discussion, I mean the number of comments and reactions per pull request. On the contrary to social media posts, too much engagement in pull requests leads to inefficiency, and I show you why.
Measuring the number of comments and reactions for each pull request gives you an idea of how your team collaborates. Collaboration is great, and as leaders, we want to endorse it. However, after a certain level, discussions slow down development.
Some pull requests require more attention than others, and the team must contribute to the threads. But if it happens too often, there may be something wrong. Maybe your team is not aligned, or maybe the software requirements are not precise enough. Misalignment discussions are not collaboration. It is time waste.
In the opposite scenario, having almost zero engagement means code review is not part of the team’s habits. It’s your responsibility to foster this culture.
In summary, you need to find an ideal number based on your team size and distribution. It can’t be too much, and can’t be too little. More importantly, you need to act on pull requests with useless discussions to keep your team productive.
Conclusion
Engineering Managers can extract lots of useful metrics from pull requests. In this blog post, I shared 5 metrics extracted from Pull Requests and how they help engineering managers to understand how their team works. Although they are pretty simple, they give you visibility of your team’s dynamics. That’s crucial for acting before problems arise.
SourceLevel currently provides some of these metrics. Others are coming soon. If you got interested in these metrics, contact us by email or chat. You may get earlier access to them!
You can also subscribe to updates using the form below. We send product updates, news, and featured content of our blog regularly.
Originally published at https://sourcelevel.io on November 14, 2019.
