The fallout of going the microservice route. Build and deploy headaches

Build/Deploy issues with GoCd

As you may have seen in a previous blog post we have a lot of pipelines, over 500 now and its set to grow by another 200 in the next month as we bring another 3 environments up. 700 pipelines is too much for a vanilla go instance to handle, the server can sometimes wait 3-4 minutes before detecting a check-in or to respond to a manual pipeline click.
I'm not having a go at go (no pun intended) i love it i think its very good at its job, but when you get to this scale apparently the embedded H2 DB starts to be a bottleneck and so you need to upgrade to the postgreSQL plugin which adds more power on the back end which would solve some issues I've seen. But there is also an issue with the main pipelines UI which can take 10+ seconds to render with 500 pipelines on the page at once, I put this down to browser performance as you can see the call to the server coming back quite quick with the markup, its just a very big page (15000+ divs, 1600+ forms, 4300+ input fields all with associated styling and javascript). So with all this in mind we decided to split up our go server into six smaller instances.

Go server/agent setup

We have over 70 micro-services mainly written in .net but with a scattering of node and spa apps (detailed here), each is independently deployable and so we have decided to split them based on service boundary which we have roughly 6. So we have decided to create 6 go servers and split the workload across all six so that each server will only be handling 1-2 hundred pipelines each.

There are three consequences to splitting up the servers like this: One is that any templates are going to get duplicated, and you will have to keep track of failing builds on six servers instead of just one which means you will need to log into six servers and remember which services are on which go server instance. But by splitting the work by service boundary we will keep the value stream map intact where by we would loose it if we had split the servers by deployment region of which we now have three (each with a test, preprod and prod environment).

Installing multiple go agents on one VM, pointing at multiple different go servers

So we have six VMs each with a go server installed, and now by the use of powershell six agents on each VM. Each VM has an agent for each of the servers so that the workload can be spread evenly. The problem with this is that whilst a given VM can host several agents (read this) all the agents will point back to the same go server because the install uses an environment variable. The solution to this is to hack the config\wrapper-agent.conf file, change all the instances of '%GO_SERVER%' replacing it with 'go-serviceboundary1.mydomain.com' which is the DNS entry for the go server you are working for.

Given i was creating six agents on six machines i didn't want to do this by hand so i created a script to do it for me (found here) the more interesting bits are summarised below:

Download the latest go agent:
$goSetupExe = "go-agent-16.2.1-3027-setup.exe"
$client = new-object System.Net.WebClient
$client.DownloadFile("https://download.go.cd/binaries/16.2.1-3027/win/$goSetupExe", "C:\go\$goSetupExe")
Command line install the first agent
.\go-agent-setup.exe /S /SERVERIP=go-serviceboundary1.mydomain.com /D=C:\go\agent1
Copy the install to a new instance
new-item "C:\go\agent$agentNumber" -ItemType Directory
Copy-Item "C:\go\agent1\*" -Destination "C:\go\agent$agentNumber" -Recurse
Remove-Item "C:\go\agent$agentNumber\config\guid.txt"
Remove-Item "C:\go\agent$agentNumber\*.log"
Rewrite the config
(Get-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf").replace('go-serviceboundary1.mydomain.com', "go-$boundedContext.mydomain.com") | Set-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf"

(Get-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf").replace('c:\go\agent1', "c:\go\agent$agentNumber") | Set-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf"
Create and start the second service instance
New-Service -Name "Go Agent$agentNumber" -DisplayName "Go Agent$agentNumber (go-$boundedContext.mydomain.com)" -Description "Go Agent$agentNumber (go-$boundedContext.mydomain.com)" -StartupType Automatic -BinaryPathName "`"C:\go\agent$agentNumber\cruisewrapper.exe`" -s `"c:\go\agent$agentNumber\config\wrapper-agent.conf`""

Start-Service "Go Agent2"

It works really well for getting a new setup running quickly, and if i ever need to add a new go server and new agents this will be easy to extend to quickly get up and running.

Feedback

It would be great if i could get some feedback as to whether this is a good idea or not, or else how best to split things up, but i feel this is a nice pragmatic solution with minimal downsides.

Pattern matching in sublime text with regex

An exercise in Yak shaving

Recently I had a very large xml file where i needed to do some string manipulation and replacements, sublime text is always my go to editor for this type of thing.
Here is an example snippet.
<data name="NumberToWord_1" xml:space="preserve">
  <value>first</value>
</data>
<data name="NumberToWord_2" xml:space="preserve">
  <value>second-value</value>
</data>
<data name="NumberToWord_3" xml:space="preserve">
  <value>3rd text value</value>
</data>

I needed to select the names and then paste them into the values.

Firstly we need to select the name with one of the following regex:
(?<=<data name=").+?(?=")
(?<=<data name=")[^"]+
<data name="\K[^"]+
^.*?"\K\w+
The first uses look behind, look around and a non greedy selector, maybe not the easiest to understand. I evolved this into the second regex by doing away with the lazy selector and the look around at the end. the third is basically the second rewritten with the meta-character \K to reset the start point of the regex (keep). Finally i trimmed away the data by looking for the first double quote on the line, resetting the keep and after that only selecting word characters.

once i had all the name strings highlighted i can use sublime texts multiple copy feature to put the 100+ words into the clipboard.

Next to paste the values back. So we need to select the value
(?<=<value>).+(?=<value>)
(?<=<value>)[^<]+
<value>\K[^<]+
^.+?e>\K[^<]+
The first line again uses a look behind and a look around with a lazy select all. The second example replaces the look around with a more defined character selector of negative <. The third replaces the look behind with a Keep reset character. finally just for fun i selected the first e> in the text with a lazy selector.

So I've now got all the values selected, just paste the current multi clipboard over the top and we have done.

Alternatives

Now there are many ways to skin a cat, i really wanted to play with regex today and so this was a nice exercise to practice with more advanced regex. But i know not everyone likes or gets regex, another approach would be to use the cursor. First select all lines with a data element (multi-select), go home and ctrl+right click till you have the cursor at just after the first quote. Then ctrl+shift+right to select the word, then copy them all. Press the down arrow to get focus to the value element, end and ctrl+left to get the cursor to just inside the end of the element value, ctrl+k, ctrl+space to set a mark, home and ctrl+left to get the cursor just inside the start of the element value, ctrl+k, ctrl+a to select to the mark and then paste. Or for this last step use the ctrl+shirt+a to use sublimes widen selection which works in xml documents.
There are so many variations on this cursor based approach that i cant list them all here, suffice to say sublime text is very powerful at text manipulation, learn your tooling people, and enjoy.

Want to see this in action watch this video:

Recorded on KRUT (an open source screen recorder) with keyjedi (to show the keyboard shortcuts).

If you think there is a simpler (or more clever :-) approach, please leave a comment and share. How bald is your Yak now?