Story points vs numbers of stories. What's the best way to predict your project completion date? Or are estimates worth it?

Do story points actually help in estimation and the prediction of project milestones and end dates? Given the inherent inaccuracy of points can we save time by skipping the estimation process?
We estimate our stories in story points (2, 4 or 8 points representing small medium or large). We try and do this when the stories get added to the back log so that we can better plan the upcoming work.

But personally I don't really think the effort required to estimate the stories is worth it, maybe we should plan just with the numbers of stories instead?

Id like to highlight my thoughts with actual data rather than hearsay and conjecture. Opinion pieces are all very well but sometimes you just need to 'show me the data'.

Comparison of the two different metrics

We have been going with our current project for 16 iterations/weeks now, we had one major release at iteration 6 and we are now approaching the next major release for this project. I wanted to take this opportunity to reflect on the no-estimates debate, is there any point in estimates, I will let the numbers do the speaking for me.

Here is our velocity first in terms of story points then in terms of stories completed:

Velocity in story points

Velocity in numbers of stories

The following 2 graphs tracking the amount of work done vs remaining. The first is an agile burn up chart, the second a lean cumulative flow chart

Burn up (story points)

Cumulative flow (numbers of stories)

And here are the statistics around the numbers of stories that we have:
13 stories/spikes with no points. 39 with 2 points. 29 with 4 points. 5 with 8 points

Conclusions

Apart from the fact that there has been a lot of scope creep and our velocity has fluctuated wildly, what do the graphs tell us?

Well at the beginning of the project, like almost every project I've ever worked on we thought we had most (but not all) of the requirements captured, and like most of the projects I've ever worked on we were wrong, very wrong. The scope doubled from 120 points (40 stories) to 240 points (80 stories) from the start of the second phase up to now. So using points or numbers of stories gave a very misleading picture of the scope and hence estimated completion date, No difference in metrics here then.

Our velocity trend is either 15 points or 6 stories, whichever measurement of velocity you use the estimated end date (as of this writing) is the 20th July. Both the burn up and the cumulative flow diagrams show how scope has been added with each iteration as we discover new requirements. As with everything we do we have to ask is that thing that we do adding value?

So I think we could ditch estimates. But this is the thing, we MUST sill analyse stories properly, must break them down into small independent stories (always remember to INVEST in your stories). And we as a team must still discuss the stories before we pick them up to work on.

Appendix

You might ask what happened in early may when the graphs flat line (iteration 20)? Well half the team was needed for an urgent fix/deploy to another system at the same time as holidays from 2 other team members, then we had to ramp up the team again (with different team members).
And why the fluctuations in velocity? well the team changed quite regularly, the time when we were the most productive was (unsurprisingly) the time when the team was the most stable and we weren't getting distracted with other projects and maintenance work.

All graphs are produced curtsey of the Mingle agile project management software from ThoughtWorks, its a great tool for managing your agile projects. We moved from Trello to Mingle around about the new year but that is a different story for a different blog post.

Indecently I'm not even going to mention burn down charts, I've long since abandoned them, they are so limited for trying to visualise the actual picture of what is going on in a given release. If you want to see a good explanation of burn ups vs burn downs just Google it, or look at this example here: http://brodzinski.com/2012/10/burn-up-better-burn-down.html
I realise not everyone sees it my way, and I guess if you have stable/full requirements and want to track work in detail in a given iteration then burn down might work for you but in our experiences it just causes confusion to management and stakeholders. They may end up asking the question 'why is your burn down going up?' or 'why is it flat?'. They don't know if you did nothing or were adding features at the same rate as knocking them off.
Ok, ok, sorry I did mention burn down charts, sorry....