Visualising the Thoughtworks Go pipline using Cradiator, a build information radiator/monitor

You know the old adage, 'out of sight, out of mind'? Like it or not, sometimes the state of the build on CI is forgotten about, and if you can't see the current state without going looking for it, it can stay red for a few days before it's noticed by someone.

I've always been a big fan of information radiators, build monitors, graphs and stats that are in peoples faces, and I just wanted to share our current solution to the whole 'who cares if the CI server is not Green' problem.

We use Go (the CI server from Thoughtworks) for our build and deployment pipeline, which is great. Although it doesn't ship with a build monitor that can be installed on a machine to show the state of the build, Thoughtworks do expose an API that allows you to build your own, for example on http://Server:Port/go/cctray.xml

We found an old-ish project called Cradiator that works with cruse control (remember that old build server? it turns out that Go has the same API, all be it on a slightly different URL). The problem was that we have secured our instance of Go so that you need to be logged in to access it. This caused problems with Cradiator and so we forked it and added the ability to set your own credentials in the config. The fork can be found here.

Below are a couple of screen shots showing the Go build server and also the corresponding Cradiator screen.

Our Go pipeline for this small part of the overall system

The Cradiator build monitor screen

As you can see every stage within a Go pipeline has a corresponding line in Cradiator. To achieve this, we use the following filter in the Cradiator config file: project-regex="^.*::.*::.*"

There is a robotic voice that announces who broke what, with some good catastrophic sound effects to accompany it. It's doing a great job of focusing people on fixing things if/when they go red.

Ever since we have started using this, the average time of a red build to a check in fixing it is less than an hour. Visibility for the win.

References:

https://github.com/DamianStanger/Cradiator

https://github.com/PandaWood/Cradiator

https://cradiator.codeplex.com/

Thoughtworks Go, asynchronously trigger a manual stage from a long running test

We are using Go from Thoughtworks Studios to manage our build pipeline, we have the builds generating artifacts that are then deployed and installed on to UAT servers, automation rocks. We then we have a long running test that runs out of process using lots of NServiceBus queues.

We have a stage that starts a process manager that fires fake messages in to the start of the system, then 3 different services pass messages along doing various things, all connected together via message buses. 

So at the start of the UAT test we kick off some powershell passing in the current pipeline counter (this is a vital detail, it tells the future API call which pipeline to kick)

. .\start_fullpipelinetest.ps1 %GO_PIPELINE_COUNTER%;

in this example the %GO_PIPELINE_COUNTER% variable is 11

Given the nature of the system once the message is sent to kick off the process the Go pipeline goes green and the next stage goes into an awaiting approval manual stage.

The last thing the test do is to send a message out to inform other downstream systems that things are ready to go, we hook into this and fire the following POST into to Go to run the next stage. If you were to do this manually you click the icon circled above, to do it programatically you send the following POST command:

curl --data "" http://user:password@server:8153/go/run/uat_start_FullPipelineTest/11/TestingComplete

Which kicks off the final stage of the pipeline. For us this does some verification as to the expected state of the system and passes or fails accordingly.

I really like it, we get an asynchronous test that does not hog the go agent resources and will instantly tell you about failures once a test run has finished.

Go is very flexible and the API lets you do all sorts of cool things, like uploading artefacts and triggering pipelines.
curl -u user:password -F file=@abc.txt http://goserver.com:8153/go/files/foo/1243/UATest/1/UAT/def.txt
curl -u user:password -d "" http://goserver.com:8153/go/api/pipelines/foo/schedule