productivity

Is Lines of Code a good way to measure programmer output?

Background

First, some background: several studies (Sackman, Erikson, and Grant – 1968; Curtis – 1981) have shown that there are large variations in productivity levels among the best and worst programmers. While the numbers from the studies are controversial, I tend to agree with the basic premise that a super programmer can significantly outperform the average programmer. In my real-world projects, I estimate that variations have ranged up to 5/1.

As a manager or technical lead of a project, it’s important to have a good idea of how productive your programmers are. With a good idea of productivity levels, you can make better estimates for time and resources, and you can manage the individual developers better. Knowing that Programmer A has relatively lower productivity than his teammates, you can assign him smaller features and save the more complex ones for more productive/better programmers. Or, in the case of the negative-productivity programmer, you can identify him quickly and react appropriately instead of letting him continue to negatively impact your project.

So, is Lines of Code (LOC) per Day by itself a good way to measure productivity? I think the answer is a resounding no for many reasons:

• A good programmer is able to implement the same feature with much less code than the average programmer.
• Code quality is not taken into account. If you can write a thousand lines of code more than the average programmer, but your code is twice as buggy, that’s not really desirable.
• Deleting and changing code, activities that are associated with important tasks such as re-factoring and bug-fixing, are not counted, or even counted negatively.

A New Method to Measure LOC

If LOC is not a good way to measure productivity, why am I writing about it? Because it’s still a good metric to have at your disposal, if you use it correctly, carefully, and in conjunction with other data. I also propose a revised method to calculate LOC that can better correlate with productivity. This “new-and-improved” LOC, in conjunction with other data (such as a Tech Lead’s intimate knowledge of his programmers’ style, efficiency, and skill level), may allow us to gain a better picture of programmer productivity.

The traditional way of calculating LOC has always been to count the lines of source code added. There are variations, such as not counting comments or counting statements instead of lines, but the general concept is the same: only lines or statements of code that are added are counted. The problems with the old method are:

• Buggy code counts as much as correct code.
• Deleting or changing code is not counted. Deleting/changing code is often done when you are re-factoring, or fixing bugs.
• Optimizing a 20,000-line module to make it 10,000 lines actually impacts the LOC negatively.

At a conceptual level, my new method to calculate LOC (let’s call it “Lines of Correct Code” or LOCC) only counts correct lines of code, and code that is deleted or changed. Short of reviewing each line of code manually, how does a program know if a line of code is correct? My answer: if it remains in the code base at the end of the produce cycle, then for our purpose, it is “correct” code.

Algorithm for Counting Lines of Correct Code

Below is the proposed algorithm for calculating the LOCC. It should be possible to automate every of the steps described here using a modern source control system.

• Analyze the source code at the end of the product cycle and keep a picture of the code that exists at the end. This is our base-line “correct” code.
• Go back to the beginning of the project cycle and examine each check-in. For each check-in, count the lines of code that is added or changed and remains until the end. Lines of code that are deleted are also counted.
• Auto-generated code is not counted or is weighted appropriately (after all, some work is involved).
• Duplicate files are only counted once. In many applications, some files are mirrored (shared in SourceSafe-speak) in multiple locations. It’s only fair to count these files only once.

Ways to Use Lines of Correct Code

Here are a few ways I am planning to use LOCC in my projects:

• Look at the LOCC per day (week/month) of the same developer over time.
• Compare the LOCC per day between different programmers of equal efficiency and skill level.
• Compare the total LOCC between different projects to get an idea of their relative size.
• Correlate the LOCC of a programmer against his/her bug rate.
• If a programmer writes code that is often deleted or changed later on, try to find out why.

Tell me what you think. Is this LOCC metric something that you would consider using in your project? I am writing a utility to calculate LOCC automatically from SourceSafe and if there’s sufficient interest, I will consider making it available.