Wednesday 31 August 2011

How much does your slow machine cost your company?


The problem

I'm currently working at your standard large company. The computers are managed centrally and are replaced every 3 or 4 years or so. The machine I'm working on is a Dell running Windows 7 32bit with 3.21 GB of ram, an Intel core 2 duo CPU at 3GHz and on-board graphics running 2 monitors. Not a terrible machine you may say, but far far from good enough as a developer machine.

We are developing an MVC 3 enterprise web app running on SQL Server 2008, LINQ to SQL, VS2010, Resharper 6, TFS integration. Javascript tests, unit tests, integration tests, feature tests and acceptance tests make up a sizeable testing suite. As you can imagine, this set-up can put quite a load on the machines especially as our app has been growing fast. Our team's velocity is still good, so the code base is growing quickly.

We are finding that the machines hang periodically. All the dev machines are quite slow and all exactly the same spec but for some reason, a few are terribly slow, and we avoid pairing on those machines. Then there's the 30-60 second waits for visual studio builds, the 5 minutes for the check-in builds, the time to compile and run tests as you are doing TDD, the time our tests run for, the 10 seconds+ for visual studio and resharper to refactor things, find things. It all adds up, but how much does it cost? Not only does it cost in terms of wasted time but also in context switching. For example, you're doing TDD, you write a test, do some coding and hit run test but have to wait 30 seconds+ for it to run. This takes long enough to break your flow, you have a quick think about something else and then you realise the test has run and you need to switch you attention back. You might have a quick chat about something else with your pair.

We know it's hurting our velocity but without numbers it's difficult to convince management of the true costs.

So what did we do?

We took a stop watch, kept it with us all day and recorded all the time that where we were waiting for the computer to do something - from opening apps, running builds and tests, searches and refactorings in visual studio - any time at all where the developer had to wait for the machine to work, be it 5 seconds or 5 minutes the stop watch was running. It took quite a lot of discipline. The results were startling.

Results

I did this for a week, every day, and so did a colleague. Our results were very similar in that on average we were sitting unproductive for a collective time of 30 to 60 minutes a day with a couple of days at 15 minutes (mainly due to days with meetings) and a couple of days at a whopping 75 minutes (these were days when perversely we were going quite quick and getting things done, but running builds/tests/check-ins all day takes time).

So lets say 40 minutes wasted per pair per day. That sounds like a lot, but it's how long the stopwatch said. You try it for a day. What are your numbers?

The machine

We are running Windows 7 on a machine that originally was running server 2003, so the requirements on the machine are different. We have turned off all the Aero UI elements from the machines (we have on board graphics). We've turned off the virus checking (controversially, as this is a corporate environment), and followed numerous tutorials on the internet about improving the speed of windows, but all to no avail.

If we had a faster, better, newer machine, then how much would we gain? Well that's subjective, but for arguments sake, say we had a great machine and could cut all the waiting/processing time in half  (*1)

The Costs

What is the cost to the business of one dev per day? It depends on your company, your devs, their seniority, your support staff, building costs, electricity etc. There are many factors, but let's say £200 (this is a conservative estimate *2)

Our team consists of 10 devs, 3 UI/UX/designers, 2 QAs/testers, 2 BAs (analysts), 1 PM (manager) = 18 people. If we assume they each cost a nominal £200 per day, that is £3600 a day for the team as a whole.

Devs are the constraint on the throughput of the system (the bottleneck). That's 10 people losing 40 mins a day, so 400 minutes a day of lost development effort. We said a fast machine might cut this in half, so we have 200 mins of needlessly lost development effort per day. How much does that cost?

We work an 8 hour day, 10 devs x 8 hours = 4,800 minutes of dev time available per day.

So the cost per minute for the throughput is £3,600 / 4,800 = £0.75 per minute (Wait you say, you took the cost of the team, divided by the time of the developers? I shall explain *3)

Finally £0.75 x 200 mins = £150 lost by the business every day

Conclusion

To sum up, it is costing the business £150 per day to have 10 developers using slow machines. Given we are paring on all dev work we only need 5 new fast machines. I know you can get a great machine for £1,000 (*4), which will be future proofed for the next 2 years or so. That's £5,000 in total.

£5,000 / £150 = 33 days. That's the number of days it takes to pay off the new machines by not losing the developers time waiting every day.

So is that £1,000 for a new dev machine worth it? I conclude it is, if the project is longer than 33 days. Is yours?



Appendix
  • Theory_of_Constraints on Wikipedia.
  • The_Goal_(novel) is an excellent book that describes the theory of constraints at a manufacturing plant in a story format.
  • 1 - We will only know how much better a great machine would be when we get one and use it on our project, but I suspect given the state of our current machines it will be half the time.
  • 2 - This is a low estimate, especially as I'm a consultant and get charged out at a higher rate than £200/day, but this is likely to be a conservative estimate for any company employing developers.
  • 3 - The theory of constraints says that the cost of the bottleneck is equal to the cost of the system as a whole. The goal of the system is to produce finished production ready functionality. Which part of the system is the bottleneck, stopping you from delivering more functionality? On our team it's us, the developers. We only release what is coded up every week. Nothing the testers or analysts can do can make us deliver more. So the output of the system is governed by the velocity of the devs. The cost of the team (in our case £3600) is the cost of the 20 story points we deliver that week, and it's the devs who control that velocity. That's not to say the other roles are not important, they are, critically important, but they are not the bottleneck.
  • 4 - Yes you should get a good machine for that. You could actually get a great computer for £500 - £600. We have the monitors and peripherals, and given we are a big company we would qualify for Dell's corporate rates.