The fallout of going the microservice route. Build and deploy headaches

Build/Deploy issues with GoCd

As you may have seen in a previous blog post we have a lot of pipelines, over 500 now and its set to grow by another 200 in the next month as we bring another 3 environments up. 700 pipelines is too much for a vanilla go instance to handle, the server can sometimes wait 3-4 minutes before detecting a check-in or to respond to a manual pipeline click.
I'm not having a go at go (no pun intended) i love it i think its very good at its job, but when you get to this scale apparently the embedded H2 DB starts to be a bottleneck and so you need to upgrade to the postgreSQL plugin which adds more power on the back end which would solve some issues I've seen. But there is also an issue with the main pipelines UI which can take 10+ seconds to render with 500 pipelines on the page at once, I put this down to browser performance as you can see the call to the server coming back quite quick with the markup, its just a very big page (15000+ divs, 1600+ forms, 4300+ input fields all with associated styling and javascript). So with all this in mind we decided to split up our go server into six smaller instances.

Go server/agent setup

We have over 70 micro-services mainly written in .net but with a scattering of node and spa apps (detailed here), each is independently deployable and so we have decided to split them based on service boundary which we have roughly 6. So we have decided to create 6 go servers and split the workload across all six so that each server will only be handling 1-2 hundred pipelines each.

There are three consequences to splitting up the servers like this: One is that any templates are going to get duplicated, and you will have to keep track of failing builds on six servers instead of just one which means you will need to log into six servers and remember which services are on which go server instance. But by splitting the work by service boundary we will keep the value stream map intact where by we would loose it if we had split the servers by deployment region of which we now have three (each with a test, preprod and prod environment).

Installing multiple go agents on one VM, pointing at multiple different go servers

So we have six VMs each with a go server installed, and now by the use of powershell six agents on each VM. Each VM has an agent for each of the servers so that the workload can be spread evenly. The problem with this is that whilst a given VM can host several agents (read this) all the agents will point back to the same go server because the install uses an environment variable. The solution to this is to hack the config\wrapper-agent.conf file, change all the instances of '%GO_SERVER%' replacing it with 'go-serviceboundary1.mydomain.com' which is the DNS entry for the go server you are working for.

Given i was creating six agents on six machines i didn't want to do this by hand so i created a script to do it for me (found here) the more interesting bits are summarised below:

Download the latest go agent:
$goSetupExe = "go-agent-16.2.1-3027-setup.exe"
$client = new-object System.Net.WebClient
$client.DownloadFile("https://download.go.cd/binaries/16.2.1-3027/win/$goSetupExe", "C:\go\$goSetupExe")
Command line install the first agent
.\go-agent-setup.exe /S /SERVERIP=go-serviceboundary1.mydomain.com /D=C:\go\agent1
Copy the install to a new instance
new-item "C:\go\agent$agentNumber" -ItemType Directory
Copy-Item "C:\go\agent1\*" -Destination "C:\go\agent$agentNumber" -Recurse
Remove-Item "C:\go\agent$agentNumber\config\guid.txt"
Remove-Item "C:\go\agent$agentNumber\*.log"
Rewrite the config
(Get-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf").replace('go-serviceboundary1.mydomain.com', "go-$boundedContext.mydomain.com") | Set-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf"

(Get-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf").replace('c:\go\agent1', "c:\go\agent$agentNumber") | Set-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf"
Create and start the second service instance
New-Service -Name "Go Agent$agentNumber" -DisplayName "Go Agent$agentNumber (go-$boundedContext.mydomain.com)" -Description "Go Agent$agentNumber (go-$boundedContext.mydomain.com)" -StartupType Automatic -BinaryPathName "`"C:\go\agent$agentNumber\cruisewrapper.exe`" -s `"c:\go\agent$agentNumber\config\wrapper-agent.conf`""

Start-Service "Go Agent2"

It works really well for getting a new setup running quickly, and if i ever need to add a new go server and new agents this will be easy to extend to quickly get up and running.

Feedback

It would be great if i could get some feedback as to whether this is a good idea or not, or else how best to split things up, but i feel this is a nice pragmatic solution with minimal downsides.