librelist archives

Metrics (was: What I Hope to Learn)

From:

Pat Maddox

Date:

2009-12-06 @ 21:25

On Dec 5, 2009, at 4:22 PM, Seth Ladd wrote:

> Metrics might include: number of bugs (as defined as functionality
> that does not match expected, often predefined, behavior), number of
> people involved in a feature implementation, Test First or Test
> Sometime, Customer Happiness (??).

Here are some metrics that I find useful from an agile planning perspective:

* Conception to completion total time

A story gets entered into the backlog.  Eventually it's accepted and 
released into production.  How long did that take?  The average of all 
stories completed to date is the lead time.  This metric does two things 
for me.  First, it allows me to give an estimate of when a new feature 
will be implemented.  "Well, if we put it at the end of the backlog 
without changing any prioritization, it will be out in about 6 weeks."  
Second it gives you a sense of your stories' staleness.  The less change 
you anticipate the longer the lead time can be.  For example if you're 
working on a rewrite, or most of the project's functionality is known 
early on, you can have a long backlog.  There's little risk that when you 
finally get to a story it will be obsolete.  In a startup scenario you 
want to keep that backlog smaller if you plan 6 months out, and the 
product changes entirely, then much of your planning becomes waste (as was
the case on one project I was on that changed its central focus three 
times in a year)

* Story start to release total time

Another one to look at is how long it takes from when a story is planned 
for the current iteration and when it's actually released.  Same as the 
prior metric, just without the time in the backlog.  There's a lot of 
stuff I look for in here, depending on what's going on with the team.  
Sometimes a story will get worked on in one iteration but not get 
completed.  It should probably have been split into smaller stories, or 
you need to look for some other inefficiency that caused it.  Also you can
start to ask questions about frequency of releases.  Let's say the 
developers are rolling along, they're doing good planning and executing so
well that nothing slips into the next iteration.  Yet the average time 
between queuing a story in the current iteration and releasing it to 
production is 3 months.  What's preventing you from getting it out the 
door sooner?

Those to me are frankly the most interesting and useful and what I would 
want to track on every project.  Doing so requires that you have a simple 
tool story-tracking tool (Pivotal Tracker is my fav in this area) and are 
100% dedicated to making sure that all work goes through it.  Too often it
becomes easy to just do small things here and there and fit them in.  Not 
only do the metrics become BS, I think it's completely unfair to both the 
developers and the customer.  Being lax with story-tracking hides the 
actual cost of development.  Any time you do this you're setting yourself 
up for some conflict down the road.

There are plenty of other metrics I've tracked.  Mostly they are 
situational.  If we start to get a lot of bugs, I track # of bug.  If 
quality seems to be slipping, I track # of story rejections in an 
iteration (representing work that the developer said was done but the 
customer didn't accept for whatever reason).  If I notice that the current
iteration isn't very stable, with new stories being added to and removed 
from it, I track that.  Too often a story that gets added in the middle of
an iteration will miss out on proper planning & collaboration, and you end
up with lower quality than usual.

So adapt the metrics to your team, as well as the current state of 
affairs.  You've heard the phrase "what gets measured gets managed."  A 
better approach for me is "what gets minded gets mended."  Simply by being
aware of things will allow you to correct their course.  In the non-tech 
world, once I started tracking every penny I spent and earned, I found 
myself with a lot more money at the end of each month than I had before.  
I never made a conscious decision to spend less & save more.  I just made 
it visible and intuitively altered my behavior to better reflect my 
values.  Same thing happens with weight.  Stepping on a scale each week 
doesn't mean you'll necessarily lose weight.  If you step on a scale each 
week and write it down somewhere, and review the weekly data side by side,
well you tend to magically lose weight.

The most important thing in all of this is identifying values.  Why do you
want to look at metrics?  Well because it gives you insight into a part of
the process that value.  Maybe it's time to market, maybe it's product 
quality, maybe it's communication.  So start there.  Identify the team's 
values, write them down and review them periodically.  When you find 
yourself getting off course you can think up a metric that applies to a 
particular value.  At that point you will either intuitively correct 
yourself, or set the stage for planning ways to tackle the problems.

I'm definitely interested to know what metrics other people track, how, and why.

Pat

Aloha Pat,

> * Conception to completion total time
> * Story start to release total time

While those are interesting, I'm not sure they are the right knobs to
turn.  Ask any business owner and they'll say, "Well, of course I want
it done faster if I don't have to sacrifice quality."  So, it's
interesting to see your velocity (from the metrics above) I don't know
what you'd be able to tweak from tracking those.

I would consider the above metrics useful for determining if you are
improving, because we as software developers need to continue to: Add
Business Value (did the software actually help the business save
money/time or add to profits?) at Ever Increasing Velocities.

I'm most interested in metrics about The Process.  Such as:

- How long did I communicate with a client?
- How many tests did I write?
- Did I write the test first?
- Did I write a Cucumber test?
- Did I have a CI server installed?
- Did my team have a dedicated QA person?
- Was a customer involved in beta testing?

I want to see which knobs we can turn to to deliver Faster Business Value.

>
> The most important thing in all of this is identifying values.  Why do 
you want to look at metrics?  Well because it gives you insight into a 
part of the process that value.  Maybe it's time to market, maybe it's 
product quality, maybe it's communication.

Bingo, I think tracking metrics is important so we can learn how to
deliver Faster Business Value.  That phrase sounds lame, but that's
what we get paid to do!  And I want us, as a community, to be able to
speak in concrete terms about What We Do and Why We Should Do It.  I
don't want to hear that TDD is better, I want to be shown the
empirical evidence from controlled scientific studies demonstrating
that TDD results in Faster Business Value.

Now of course the question is, how do you measure Faster Business
Value?  The Faster part is a bit easier, and this is where your above
metrics come into play... measurements like Story Start to Finish
time.  We'll need to somehow quantify how difficult the Story is,
across projects, to get any useful data.  And I have no idea how to do
that.  Business Value is pretty tough to measure, but I would start by
tracking number of user generated Stories.  The assumption is that a
Story delivers Business Value, which I hope is a fair assessment.

But in narrowing my focus, I still maintain that the community needs
to be able to speak more concretely about Process such that we can
prove why certain things work and when they work.  Where are all the
scientific studies?  Where is the Agile Measurement and Improvement
movement?

Seth