Goodhart’s Law and software development

October 3, 2011 at 10:23 am | Posted in Blogroll, Software Engineering | Leave a comment

Charles Goodhart, a former advisor to the Bank of England and emeritus professor at the London School of Economics had proposed a law in 1975 that “once a social or economic indicator is made a target for the purpose of guiding policy, then it will lose the information content that originally made it useful”. It was later simplified and made more generic by Professor Marilyn Strathern as “When a measure becomes a target, it ceases to be a good measure”.

Most of the examples published about its applicability are from communist countries. I was looking for something closer to our domain. Last week I saw a good article by Lee Copeland in the latest issue of “Better software magazine” which I am sure may interest you. He puts it very nicely as “Connecting rewards and punishments to specific goals can negatively impact a metric’s usefulness”. Here are some of the relevant conclusions and examples from this article.

Management chooses a metric in an attempt to understand the behavior of a system or process. Later, this same metric becomes a goal (“Attain this value or else”). When this occurs, the “or else” part motivates people to change their behavior to achieve the goal. Some examples,

When LOC/day becomes a goal, developers may be enticed to write more lines of less efficient code.

One of the indicators of the quality of system design is the subclass: superclass ratio—sometimes called the specialization ratio. At one company, not only was the specialization ratio measured, but a goal was set—the specialization ratio had to be 3 or above. Designs not meeting that goal were sent back for rework. Developers, not liking rework, simply added cleverly disguised empty classes until the ratio was met.

In other cases, rewarding testers for the number of test cases resulted in many poorly written test cases; rewarding testers for the number of bugs they found resulted in a high number of unimportant or duplicate bugs reported; and penalizing testers for bugs rejected by the development staff resulted in important bugs going unreported.

And he concludes, “Goodhart’s Law reminds us that connecting rewards and punishments to the achievement of specific goals can create unintended consequences. Some will strive to reach those numbers without concern for anything else. If the person being measured is affected by the outcome, she is likely either to lie, thus subverting the usefulness of the measurement, or to focus on what is being measured without regard for the consequences”.

What is the solution? Some of the options found in other more generic articles are,
Specify the information required, but don’t disclose how it will be interpreted (To me this seems to be fraught with trust issues)
Depend more on human discretion (This gives the flexibility but may not scale)
Balanced scorecards (Useful if the interpretation considers the balancing effect, but would have same problems if each measure is taken by itself and judged)
Theory of constraints – which is about focusing on bottlenecks (This has the benefit that once a bottleneck is taken care, it exposes other bottlenecks and the measures could now be moved to the new bottlenecks)

Your views and inputs are valuable; please share.

 

Advertisements

Leave a Comment »

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.
Entries and comments feeds.

%d bloggers like this: