50 shades of Lead Time. Measuring each part of the development process

January 29, 2020 in Code Review, Engineering Metrics

I have read multiple definitions for Lead Time and Cycle Time over the last years. Recently, I shared why I prefer using the first over the later.

That said, when I refer to lead time, I mean the time spent from any given point of the process to another further point of the process. As simple as Lead Time = end_date - start_date.

In summary, I want to handle in this blog post is: how can I use the lead time to understand the working system, and which are the most useful lead time metrics?

Measuring the development process

I consider the development process what happens from the moment developers start writing code to the moment that code reaches production.

Of course, this definition does not consider all the tasks required in a software life-cycle, for instance, having a User Story ready for development and prioritized in the backlog. It requires much work that is hard to measure automatically and rarely is at the technology team’s hand. They are the product team’s responsibility.

After deploy, there is lots of work too. Operations usually take over the risks. In many companies, sysadmins or DevOps perform rollback in production. Usually, it is not at developers’ hands too. The Product Team also measures the impact of the deployed changes to feed the backlog cyclically.

That’s why I like to visualize this part of the process separately. It gives more visibility to the development team and isolates the development from the full system.

The many lead times of the development process

Of course, there are not 50 lead times. But you can calculate a few of them, and I want to explain those briefly. I am mentioning those I think are the most relevant ones.

Simplified development flow and possible lead time measurements

Commit to Deploy Time: it measures how much time a commit takes to reach production. Every time you run a deploy, you can list all the shipped commits since the last deploy. Then, calculate the lead time for each of them. You can analyze the average, median, or a given percentile.
Commit to QA or Stage Time: Same as Commit to Deploy Time, but applied for other environments.
First Commit to Deploy Time: The idea here is similar to Commit to Deploy, but it measures how much time a branch takes to reach production. It requires a little more git abilities to filter the correct commits. Lead time is calculated based on the timestamp of the first commit against the timestamp of the deploy. You can analyze the average, median, or a given percentile.
First Commit to Pull Request Time: This metric is especially useful if your culture favors opening pull requests as soon as possible for early feedback. It calculates the time spent from the first commit to pull request’s creation date.
Time to Review, Time to Merge, or Pull Request Lead Time: This lead time measures how many times developers need to review a pull request. To find the number, just do closed_or_merged_at — opened_at. It considers all the steps of code review in it, which gives you a specific cut in the whole process.
Time to First Review: After a developer opens a pull request, how much time the team takes to publish the first review? It’s an excellent metric to understand whether your team adopted code review as part of their daily activities.
Last Review to Merge Time: For each closed or merged pull request, calculate how many time elapsed since the last review. It gives you a sense of “how much time a done job got stuck.” If the number is considerably high for your context, then you should ask yourself, why are people afraid or avoiding to merge?

How to know which to measure?

You should always measure what makes sense to the business. You can start by measuring where you feel the problem is. If you think your team is taking too much time reviewing pull requests, measure it! If you think your team is taking too long to open a pull request, measure it!

As time goes, your understanding of how the system behaviors increases. It unlocks new metrics and new possibilities of measurements. Add new metrics to your dashboard to fulfill missing spots and take actions based on real and reliable data.

Conclusion

The agile community refers to Lead Time and Cycle Time confusingly. I explained what is Lead Time for me, and presented some different calculations you can do to understand how your team is performing.

All presented lead times reveal a different facet of the development flow, which in this case, I consider from the first commit to its deploy in production.

Currently, SourceLevel supports measuring Time to Review (Pull Request Lead Time), throughput (merged pull requests by week), and a histogram showing lead time distribution over the last months.