Advanced filtering and navigation on Thoughtworks Go Cd with tamper monkey

Our problem

We use Go from Thoughtworks to manage the build and deployment of all our software. Attached below is a screen shot of the current pipelines.

As you can see we have quite a lot going on. In fact we have 390 pipelines (55 services and api builds, 40 nuget package build/publish, and the rest are deployments to our 5 environments from test through preprod and live). There are 730 stages in total (Build, test, deploy, ect). And we have 20 go agents running on different servers.

So you can imagine finding the pipeline you are looking for is tough. We have adopted naming conventions which help but its really quite difficult to find what you are after with the supplied search capabilities.

Also its very tricky to see if anything is broken (red), there is just no way you will notice it in all the pipelines.

As an aside we are using cradiator to give us an information radiator of all our build and deployment pipelines (I blogged about this a year ago) but with over 700 stages its really in need of an overhaul but that's for another blog post.

The solution

We have created a tamper monkey script (found here) that enhances the functionality of the Go pipelines view, allowing you to filter the visible pipelines by status or by keyword.

 

There is also the issue of navigating between the settings and the pipeline history, and visa versa. Often you are investigating a failure, you find the problem in the history and then want to change the settings. Well there is no link, so we created one for you to easily navigate between the 2 parts of the UI easily.

Notice the "Settings" link above and the "History" link below both in the header

The script can be found here: https://gist.github.com/DamianStanger/d1db873e9297b34160ce#file-go-cd-tampermoneyscript-js

Install

Firstly you need to get tamper monkey installed in your browser, you can get it from the chrome web store or http://tampermonkey.net/

Then from the tamper monkey dashboard add the script and tell it what urls to attach itself to. The update url needs to be your instance of Go (our update URL is https://goserver:8154/go/* ). On the settings tab I also add in a couple of user includes to https://goserver:8154/go/home and https://goserver:8154/go/pipelines.

That's it, instantly enjoy better productivity. I imagine this is a good enhancement even if you only have a tenth of the pipelines we have.

Story points vs numbers of stories. What's the best way to predict your project completion date? Or are estimates worth it?

Do story points actually help in estimation and the prediction of project milestones and end dates? Given the inherent inaccuracy of points can we save time by skipping the estimation process?
We estimate our stories in story points (2, 4 or 8 points representing small medium or large). We try and do this when the stories get added to the back log so that we can better plan the upcoming work.

But personally I don't really think the effort required to estimate the stories is worth it, maybe we should plan just with the numbers of stories instead?

Id like to highlight my thoughts with actual data rather than hearsay and conjecture. Opinion pieces are all very well but sometimes you just need to 'show me the data'.

Comparison of the two different metrics

We have been going with our current project for 16 iterations/weeks now, we had one major release at iteration 6 and we are now approaching the next major release for this project. I wanted to take this opportunity to reflect on the no-estimates debate, is there any point in estimates, I will let the numbers do the speaking for me.

Here is our velocity first in terms of story points then in terms of stories completed:

Velocity in story points

Velocity in numbers of stories

The following 2 graphs tracking the amount of work done vs remaining. The first is an agile burn up chart, the second a lean cumulative flow chart

Burn up (story points)

Cumulative flow (numbers of stories)

And here are the statistics around the numbers of stories that we have:
13 stories/spikes with no points. 39 with 2 points. 29 with 4 points. 5 with 8 points

Conclusions

Apart from the fact that there has been a lot of scope creep and our velocity has fluctuated wildly, what do the graphs tell us?

Well at the beginning of the project, like almost every project I've ever worked on we thought we had most (but not all) of the requirements captured, and like most of the projects I've ever worked on we were wrong, very wrong. The scope doubled from 120 points (40 stories) to 240 points (80 stories) from the start of the second phase up to now. So using points or numbers of stories gave a very misleading picture of the scope and hence estimated completion date, No difference in metrics here then.

Our velocity trend is either 15 points or 6 stories, whichever measurement of velocity you use the estimated end date (as of this writing) is the 20th July. Both the burn up and the cumulative flow diagrams show how scope has been added with each iteration as we discover new requirements. As with everything we do we have to ask is that thing that we do adding value?

So I think we could ditch estimates. But this is the thing, we MUST sill analyse stories properly, must break them down into small independent stories (always remember to INVEST in your stories). And we as a team must still discuss the stories before we pick them up to work on.

Appendix

You might ask what happened in early may when the graphs flat line (iteration 20)? Well half the team was needed for an urgent fix/deploy to another system at the same time as holidays from 2 other team members, then we had to ramp up the team again (with different team members).
And why the fluctuations in velocity? well the team changed quite regularly, the time when we were the most productive was (unsurprisingly) the time when the team was the most stable and we weren't getting distracted with other projects and maintenance work.

All graphs are produced curtsey of the Mingle agile project management software from ThoughtWorks, its a great tool for managing your agile projects. We moved from Trello to Mingle around about the new year but that is a different story for a different blog post.

Indecently I'm not even going to mention burn down charts, I've long since abandoned them, they are so limited for trying to visualise the actual picture of what is going on in a given release. If you want to see a good explanation of burn ups vs burn downs just Google it, or look at this example here: http://brodzinski.com/2012/10/burn-up-better-burn-down.html
I realise not everyone sees it my way, and I guess if you have stable/full requirements and want to track work in detail in a given iteration then burn down might work for you but in our experiences it just causes confusion to management and stakeholders. They may end up asking the question 'why is your burn down going up?' or 'why is it flat?'. They don't know if you did nothing or were adding features at the same rate as knocking them off.
Ok, ok, sorry I did mention burn down charts, sorry....

Thoughtworks Go, adding a version text file as a build artifact

Firstly let me be clear that we use Go from Thoughtworks but I'm sure you can use the same technique outlined below for other CI systems such as Teamcity or TFS.

When we deploy our built code to the live servers its good to be able to see what the version of the dlls/exes files is. To do this we put a file in the same directory as the built files called version.txt which contains the details of the build that has been deployed, the build number, the revision of SVN that formed the source for the build.

If you look in the console tab of the build job that you have set up you will see something similar to the following:
[go] setting environment variable 'GO_ENVIRONMENT_NAME' to value 'CI'
[go] setting environment variable 'GO_SERVER_URL' to value 'https://buildAgent01:8154/go/'
[go] setting environment variable 'GO_TRIGGER_USER' to value 'changes'
[go] setting environment variable 'GO_PIPELINE_NAME' to value 'Scoring'
[go] setting environment variable 'GO_PIPELINE_COUNTER' to value '81'
[go] setting environment variable 'GO_PIPELINE_LABEL' to value '81'
[go] setting environment variable 'GO_STAGE_NAME' to value 'Build'
[go] setting environment variable 'GO_STAGE_COUNTER' to value '1'
[go] setting environment variable 'GO_JOB_NAME' to value 'BuildSolution'
[go] setting environment variable 'GO_REVISION' to value '6343'
[go] setting environment variable 'GO_TO_REVISION' to value '6343'
[go] setting environment variable 'GO_FROM_REVISION' to value '6343'
With this data you can create a task which uses powershell to create the file
Command: powershell
Arguments: sc .\version.txt "GO_ENVIRONMENT_NAME:%GO_ENVIRONMENT_NAME%, GO_SERVER_URL:%GO_SERVER_URL%, GO_TRIGGER_USER:%GO_TRIGGER_USER%, GO_PIPELINE_NAME:%GO_PIPELINE_NAME%, GO_PIPELINE_COUNTER:%GO_PIPELINE_COUNTER%, GO_PIPELINE_LABEL:%GO_PIPELINE_LABEL%, GO_STAGE_NAME:%GO_STAGE_NAME%, GO_STAGE_COUNTER:%GO_STAGE_COUNTER%, GO_JOB_NAME:%GO_JOB_NAME%, GO_REVISION:%GO_REVISION%"
This produces a text file like this:

version.txt

GO_ENVIRONMENT_NAME:CI
GO_SERVER_URL:https://buildAgent01:8154/go/
GO_TRIGGER_USER:changes
GO_PIPELINE_NAME:Scoring
GO_PIPELINE_COUNTER:81
GO_PIPELINE_LABEL:81
GO_STAGE_NAME:Build
GO_STAGE_COUNTER:1
GO_JOB_NAME:BuildSolution
GO_REVISION:6343
make sure this file is included in the build output folder along with the build artifacts, good times.

Visualising the Thoughtworks Go pipline using Cradiator, a build information radiator/monitor

You know the old adage, 'out of sight, out of mind'? Like it or not, sometimes the state of the build on CI is forgotten about, and if you can't see the current state without going looking for it, it can stay red for a few days before it's noticed by someone.

I've always been a big fan of information radiators, build monitors, graphs and stats that are in peoples faces, and I just wanted to share our current solution to the whole 'who cares if the CI server is not Green' problem.

We use Go (the CI server from Thoughtworks) for our build and deployment pipeline, which is great. Although it doesn't ship with a build monitor that can be installed on a machine to show the state of the build, Thoughtworks do expose an API that allows you to build your own, for example on http://Server:Port/go/cctray.xml

We found an old-ish project called Cradiator that works with cruse control (remember that old build server? it turns out that Go has the same API, all be it on a slightly different URL). The problem was that we have secured our instance of Go so that you need to be logged in to access it. This caused problems with Cradiator and so we forked it and added the ability to set your own credentials in the config. The fork can be found here.

Below are a couple of screen shots showing the Go build server and also the corresponding Cradiator screen.

Our Go pipeline for this small part of the overall system

The Cradiator build monitor screen

As you can see every stage within a Go pipeline has a corresponding line in Cradiator. To achieve this, we use the following filter in the Cradiator config file: project-regex="^.*::.*::.*"

There is a robotic voice that announces who broke what, with some good catastrophic sound effects to accompany it. It's doing a great job of focusing people on fixing things if/when they go red.

Ever since we have started using this, the average time of a red build to a check in fixing it is less than an hour. Visibility for the win.

References:

https://github.com/DamianStanger/Cradiator

https://github.com/PandaWood/Cradiator

https://cradiator.codeplex.com/

Thoughtworks Go, asynchronously trigger a manual stage from a long running test

We are using Go from Thoughtworks Studios to manage our build pipeline, we have the builds generating artifacts that are then deployed and installed on to UAT servers, automation rocks. We then we have a long running test that runs out of process using lots of NServiceBus queues.

We have a stage that starts a process manager that fires fake messages in to the start of the system, then 3 different services pass messages along doing various things, all connected together via message buses. 

So at the start of the UAT test we kick off some powershell passing in the current pipeline counter (this is a vital detail, it tells the future API call which pipeline to kick)

. .\start_fullpipelinetest.ps1 %GO_PIPELINE_COUNTER%;

in this example the %GO_PIPELINE_COUNTER% variable is 11

Given the nature of the system once the message is sent to kick off the process the Go pipeline goes green and the next stage goes into an awaiting approval manual stage.

The last thing the test do is to send a message out to inform other downstream systems that things are ready to go, we hook into this and fire the following POST into to Go to run the next stage. If you were to do this manually you click the icon circled above, to do it programatically you send the following POST command:

curl --data "" http://user:password@server:8153/go/run/uat_start_FullPipelineTest/11/TestingComplete

Which kicks off the final stage of the pipeline. For us this does some verification as to the expected state of the system and passes or fails accordingly.

I really like it, we get an asynchronous test that does not hog the go agent resources and will instantly tell you about failures once a test run has finished.

Go is very flexible and the API lets you do all sorts of cool things, like uploading artefacts and triggering pipelines.
curl -u user:password -F file=@abc.txt http://goserver.com:8153/go/files/foo/1243/UATest/1/UAT/def.txt
curl -u user:password -d "" http://goserver.com:8153/go/api/pipelines/foo/schedule