Story points vs numbers of stories. What's the best way to predict your project completion date? Or are estimates worth it?

Do story points actually help in estimation and the prediction of project milestones and end dates? Given the inherent inaccuracy of points can we save time by skipping the estimation process?
We estimate our stories in story points (2, 4 or 8 points representing small medium or large). We try and do this when the stories get added to the back log so that we can better plan the upcoming work.

But personally I don't really think the effort required to estimate the stories is worth it, maybe we should plan just with the numbers of stories instead?

Id like to highlight my thoughts with actual data rather than hearsay and conjecture. Opinion pieces are all very well but sometimes you just need to 'show me the data'.

Comparison of the two different metrics

We have been going with our current project for 16 iterations/weeks now, we had one major release at iteration 6 and we are now approaching the next major release for this project. I wanted to take this opportunity to reflect on the no-estimates debate, is there any point in estimates, I will let the numbers do the speaking for me.

Here is our velocity first in terms of story points then in terms of stories completed:

Velocity in story points

Velocity in numbers of stories

The following 2 graphs tracking the amount of work done vs remaining. The first is an agile burn up chart, the second a lean cumulative flow chart

Burn up (story points)

Cumulative flow (numbers of stories)

And here are the statistics around the numbers of stories that we have:
13 stories/spikes with no points. 39 with 2 points. 29 with 4 points. 5 with 8 points

Conclusions

Apart from the fact that there has been a lot of scope creep and our velocity has fluctuated wildly, what do the graphs tell us?

Well at the beginning of the project, like almost every project I've ever worked on we thought we had most (but not all) of the requirements captured, and like most of the projects I've ever worked on we were wrong, very wrong. The scope doubled from 120 points (40 stories) to 240 points (80 stories) from the start of the second phase up to now. So using points or numbers of stories gave a very misleading picture of the scope and hence estimated completion date, No difference in metrics here then.

Our velocity trend is either 15 points or 6 stories, whichever measurement of velocity you use the estimated end date (as of this writing) is the 20th July. Both the burn up and the cumulative flow diagrams show how scope has been added with each iteration as we discover new requirements. As with everything we do we have to ask is that thing that we do adding value?

So I think we could ditch estimates. But this is the thing, we MUST sill analyse stories properly, must break them down into small independent stories (always remember to INVEST in your stories). And we as a team must still discuss the stories before we pick them up to work on.

Appendix

You might ask what happened in early may when the graphs flat line (iteration 20)? Well half the team was needed for an urgent fix/deploy to another system at the same time as holidays from 2 other team members, then we had to ramp up the team again (with different team members).
And why the fluctuations in velocity? well the team changed quite regularly, the time when we were the most productive was (unsurprisingly) the time when the team was the most stable and we weren't getting distracted with other projects and maintenance work.

All graphs are produced curtsey of the Mingle agile project management software from ThoughtWorks, its a great tool for managing your agile projects. We moved from Trello to Mingle around about the new year but that is a different story for a different blog post.

Indecently I'm not even going to mention burn down charts, I've long since abandoned them, they are so limited for trying to visualise the actual picture of what is going on in a given release. If you want to see a good explanation of burn ups vs burn downs just Google it, or look at this example here: http://brodzinski.com/2012/10/burn-up-better-burn-down.html
I realise not everyone sees it my way, and I guess if you have stable/full requirements and want to track work in detail in a given iteration then burn down might work for you but in our experiences it just causes confusion to management and stakeholders. They may end up asking the question 'why is your burn down going up?' or 'why is it flat?'. They don't know if you did nothing or were adding features at the same rate as knocking them off.
Ok, ok, sorry I did mention burn down charts, sorry....

Kanban inspired card wall. Our example

Overview
Some call them 'card walls', some 'story walls', others just 'the wall'. There are many names given to the wall but its purpose is the same, to convey project information to the team and other interested parties / stakeholders. I work on a software development team but Kanban boards can be applied to any process, from manufacturing to household to-do lists. Honestly the uses are as wide ranging as your imagination. But this article concentrates on software development.

Some teams practice 'scrum' and put tasks on the wall, some XP and run on weekly/2 weekly/4 weekly iterations, but we are running in a kanban mode at the moment, applying Work In Progress (WIP) limits as best we can.

I've worked on many different teams, with many different types of wall/board, from scrum to xp, from big walls to tiny boards. I think the Kanban inspired wall we are currently running with is the best I've have the good fortune to use. So I thought I'd share - I'll try to run you through it as best as I can.

The wall

We are fortunate enough to have a very big space for our card wall (3-4 metres wide). I have been on teams that have had to manage with a small white board. Having said that, we have still filled every available space, and it would be nice if we had even more.

The size allows us to show lots of project relevant information and also include plenty of room for up stream activities like information architecture (IA), user experience (UX) and analysis. So we have lots of visibility as to the state of all aspects of the project.

The Columns
The columns are detailed below in their own sections, but to summarise they are:

• Design
• In visual design
• In customer testing
• In front end dev
• Analysis
• In analysis
• Selected for dev
• Development
• In dev
• Dev complete
• QA
• In QA
• QA complete
• Done
• Technical tasks
• Risks

The WIP Limits
Work In Progress limits help us to focus on the tasks currently in play. Too many tasks on the board at any one time can mean the team is spread too thinly. Traffic jams can occur and important bug fixes won't get through quickly.

Example without WIP limits:
Two big stories have just been finished and so QA now gets to work on them, whilst the developers start on new stories. At this point all four dev pairs are busy, as are the two QAs. A day later the testing is still not complete and another pair finishes their story, this story gets queued up in waiting for QA. If there were no WIP limits the pair would start a new story and get on with that work. But what happens if there is then a bug in one of the stories, a big bug? Who should pick it up? A dev pair need to stop what they are doing mid flow, shelve all their current checked out work, context switch to the bug, investigate and fix it. I guess you know where this is going, with effectively three stories in QA and four stories in dev everyone is going to be flip-flopping between stories and bug fixes. Context switching costs time and the whole process reduces our ability to respond to change. The main consequence being that throughput and velocity suffer, not to mention the headaches of leaving half finished code laying about whilst you fix something else.

By limiting the number of stories in each column this does not happen (well, much much less so). Developers are always available for fixes and everyone experiences less context switching and more uninterrupted flow.

The Story Cards
We have four different coloured cards on the wall: yellow, blue, white and pink. The different colours mean different things depending on where they are on the wall, but its quite straight forward.

• Yellow: Design tasks, our UX guys work on the yellow cards in the first 3 columns, either working on the IA, wire frames, or styling of up coming stories.
• Yellow: Development tasks, tech tasks and tech debt. All things that are not stories but need playing, e.g.: Set up performance test environment, or refactor X to remove duplication, or even spike out Y to prove integration points.
• Blue: Features under analysis. These cards only appear in the analysis and done columns and represent whole features, big things. I think there are around 15 in release one.
• Pink: Bugs, simples [sic].
• White: Most of the cards flowing across the wall are white. These denote stories that require dev effort. We try to ensure that all the white cards can be finished in 1-4 days

White Story Card

• Story number for reference in Mingle or TFS.
• Story title, brief, must fit on one card in thick writing.
• Days in play (the dots get added every morning before our daily stand up meeting) This allows us to see how long cards take to flow through the wall.
• Story point estimate (size/complexity)
• Relevant notes can be captured on the reverse if needed.

Yellow Tech Card

• Usually these do not have reference numbers.
• Usually no estimate either.
• Just a brief description of the work required.
• Again relevant notes can be captured on the reverse if needed.

Flow across the wall

Features start their life in the 'In analysis' column. All the features that are designated for release 1 are in this column in priority order. Whilst the BAs are working on splitting stories out, the UX and IA guys are also looking into how best to fulfil the users requirements from a design perspective, and they put their own cards up on the left of the board.
Once a story is ready to be developed, a white card is placed in the 'Selected for dev' column and waits here until a dev pair is ready to pick them up. At which point they are moved in to 'In dev' until the pair has showcased the story to the QA and BA's satisfaction at which point it moves into dev complete (awaiting QA).
The QAs pick up cards and work on them in the 'In QA' column. Cards NEVER move backwards, if there are bugs to fix a dev pair will move to fix it in the QA column, whatever they were doing will remain in the 'In dev' column. Once the QA is happy with the card it will move into 'QA complete' where the BAs now showcase the story to the product owner before moving it to 'Done'.

The Columns In Detail

Design

Walls I've often worked with in the past have not had a design area on them. This project has a heavy design and UX aspect and so it was important to have this area on the wall so we can all see what features are at which point in the process.

The columns we have here are: 'In visual design' for development of the wireframes and IA; 'In customer testing'; and 'In front end dev' for development of the UX including styling and image creating.

Analysis (WIP 6)

The Analysis columns consist of 'In analysis' which list all the features of the current release (release one is about nine months) we keep stickies on the features (blue cards) to show how complete features are. The 'Selected for dev' column is used for the next stories to be picked up by the dev team (white cards), these stories should have been through the analysis process and be ready for devs to pick up, this column also serves as a heads up for the QAs so they can start getting ready for stories, working out acceptance criteria and so forth with the BAs and devs.

Dev (WIP 4)

We currently have four dev pairs and so have set our WIP limit to four for the combination of the two dev columns. At the beginning of the project we set the WIP limit to three as we always had a pair on tech tasks, environment set up and the like, but now we are moving into the second half of the first release most of that type of task has been finished. If there are four white cards in the two dev columns the next pair to finish should help out with the testing effort or work on some tech tasks. This helps keep the actual work in progress on the board to a manageable amount.

QA (WIP 2)

The 'In QA' column has a WIP limit of two as we have two testers. If a story gets blocked due to a bug then devs need to come and help clear the blockage. The card can NEVER move back into dev and the QA can't pick up a new card from dev complete until the issues are resolved. This focuses attention and ensures that stories progress along the wall in a timly manner, always the empasis on getting completed work 'done done' so that we can claim the points, and move on.

Done (Done Done)

This column contains all the stories finished this week. Done Done. These stories are ready to be showcased to the product owner and will be demoed to the entire client user team (all stakeholders) on the Friday of each week. Every Monday all the cards are removed from this column so we can see what has been finished this week and so that the product owner and other stackholders can see the progress since last Friday's showcase.

Metrics

This area contains all the different metrics, the different graphs and guides.

• Cumulative Flow
• Weekly Velocity 
• Points Burn Up
• Risk Count
• Story points Guide
• Done Done Guide

Risks/Issues/Tech Tasks/Tech Debt Wall
There are four columns to this section of the wall. The left most three are for 'tech debt', and tech tasks. The right most is for capturing risks and making them more visible.

Column 1 (Things we must do to make the project successful).
Things like integrate with the corporate authentication system, spike asset management or setup performance testing environment.

Column 2 (Things that would enable us to deliver faster)
Includes things like refactor acceptance tests to reduce build time, and optimise windows configuration on all dev machines.

Column 3 (Things that are not essential to the success of the project but will give us a better solution)
Mainly includes lists of refactorings and areas we want to achieve better test coverage.

Column 4 (Risk wall)
Any risk to the project, from integration points, new technology, people, to computers and hardware. Listed in highest risk at the top. The arrow represents whether the risk is increasing or decreasing as time passes.

Summary
So that is our wall. It saves us time, keeps us focussed on what's important, and neatly tracks our progress.

How does your team work? I would be interested to know what other types of walls are out there. I've seen a few in my time but there is always another way of doing it - what does your wall do for you?


Glossary 
BA - Business Analyst
IA - Information Architecture
Kanban - http://en.wikipedia.org/wiki/Kanban
Mingle - http://www.thoughtworks-studios.com/mingle-agile-project-management
QA - Quality Analyst
Scrum - http://en.wikipedia.org/wiki/Scrum_(development)
Tech Debt - http://martinfowler.com/bliki/TechnicalDebt.html
TFS - Team Foundation Server
UX - User eXperiance
WIP - Work In Progress
XP - eXtreme Programming - http://en.wikipedia.org/wiki/Extreme_Programming