Damo's Podcast Highlights 2016 #21

I subscribe to many podcasts, you can see the list as it was in 2015 here: Developer podcasts v2 but I thought I would start to keep a weekly log of the episodes that I found interesting or useful in some way.

[This Agile Life 109] Fallacies of Software Developers - http://www.thisagilelife.com/109/
[On Books] Episode 15 - How to Learn Anything Fast - http://castig.org/the-first-20-hours-by-josh-kaufman/
[On Books] Episode 18 - Growth Mindset vs fixed mindset - http://castig.org/growth-mindset-vs-fixed-mindset/
[RunAs Radio 473] Hacking a Country with Troy Hunt - http://runasradio.com/Shows/Show/473
[Scrum Master Toolbox Podcast] The role of retrospectives on the path to success - http://www.scrum-master-toolbox.com/2016/05/podcast/allison-pollard-on-the-role-of-retrospectives-to-get-us-on-the-path-to-success/
[Continuous Discussions 41] Creating an Internal Dev/Test Cloud - https://electric-cloud.com/blog/2016/05/key-takeaways-continuous-discussions-c9d9-episode-41-creating-internal-devtest-cloud/
[Freakonomics Radio] How to Be Tim Ferriss - http://freakonomics.com/podcast/tim-ferriss/
[Software Engineering Radio 23] The role of software architecture Pt1. - http://www.se-radio.net/2006/07/episode-23-architecture-pt-1/

Damo's Podcast Highlights 2016 #20

I subscribe to many podcasts, you can see the list as it was in 2015 here: Developer podcasts v2 but I thought I would start to keep a weekly log of the episodes that I found interesting or useful in some way.

[Agile Weekly Crew] Episode #57 – Performance Reviews and one on one feedback
http://integrumtech.com/2012/04/episode-57-performance-reviews-with-kane-mar/

[On books] Thinking fast and slow book review https://www.acast.com/onbooks/episode-11-thinking-fast-and-slow-by-daniel-kahneman-chat-with-alex-miles-younger- and https://www.acast.com/onbooks/episode-12-thinking-fast-and-slow-part-2-on-happiness

[Enterprise devops initiatives] The Importance of Systems Thinking for Cloud Migration (focus more on AWS but still good) http://mikekavis.sys-con.com/node/3386867

[Enterprise devops initiatives] How to Succeed with DevOps? Make Room for Experimentation!
http://www.cloudtp.com/2015/08/27/how-to-succeed-with-devops-make-room-for-experimentation/

[The azure podcast] Azure functions http://azpodcast.azurewebsites.net/post/Episode-128-Azure-Functions

Damo's Podcast Highlights 2016 #19

I subscribe to many podcasts, you can see the list as it was in 2015 here: Developer podcasts v2 but I thought I would start to keep a weekly log of the episodes that I found interesting or useful in some way.

Lets see if i can keep going every week shall we :-)


[Hanselminutes] An overview of containers http://www.hanselminutes.com/527/practical-containers-for-developers-with-aja-hammerly
[devchat tv] Motivation, Impact, professionalism https://devchat.tv/ruby-rogues/244-rr-program-like-you-give-a-damn-with-ara-t-howard-at-rails-remote-conf-2015
[London school of economics] Brexit, should we stay or should we go http://www.lse.ac.uk/publicEvents/events/2016/05/20160510t1830vOT.aspx
[Hello tech pros] Productivity, Motivation, Habits http://hellotechpros.com/joe-zack-productivity/

Damo's Podcast Highlights 2016 #18

I subscribe to many podcasts, you can see the list as it was in 2015 here: Developer podcasts v2 but I thought I would start to keep a weekly log of the episodes that I found interesting or useful in some way.

Lets see if i can keep going every week shall we :-)

[Software engineering radio] Unit Tests  http://www.se-radio.net/2016/05/se-radio-episode-256-jay-fields-on-working-effectively-with-unit-tests/
[Hanselminutes] Management and motivation http://www.hanselminutes.com/526/punishment-driven-development-with-louise-elliott
[Scrum master toolbox] Purpose and management 3.0 http://podplayer.net/#/?id=15224714 from http://www.scrum-master-toolbox.com/
[Devnology] Lean software development http://devnology.nl/podcasts/devnology-podcast-012-mary-and-tom-poppendieck-part-1
[Full stack radio] Git source control tips http://www.fullstackradio.com/41

P.s If you are a little confused about the number in the title its based on the week number. Anyway it looks like I've been doing this longer than i really have ;-)

Deploying Azure websites and webjobs using your CI system with msbuild and azure powershell cmdlets

There are lots of tutorials detailing how to send web apps and web jobs up to azure as part of your CI pipeline but I've found them all lacking, mainly in that they require you to upload to Azure in the same command as performing your build.
Often this command looks like the following:
msbuild.exe foobar.csproj /p:DeployOnBuild=true /p:PublishProfile="azurePubProfile" /p:Configuration=Release
This simultaneously builds, configures and publishes the project to azure.
 
This is unacceptable to me as I want to promote the same built artifact through many environments ci-test-preprod-prod where the only thing to change is the config getting deployed to each. I don't want to keep building the code over and over when I deploy which has many issues ranging from changes in the code between deployments to changes in the environment of the build machine. I want to know that what is getting deployed to prod is exactly the same binaries as was deployed and tested in all previous environments. To achieve this we need to separate the building of the artifact from the deployment.

What follows is the method we are now using to build and deploy our services to Azure. It breaks down into 4 simple steps

1. Build your solution, firstly restoring any nuget packages if needed
2. Configure your service for the target environment and zip up the artifact
3. Configure the powershell session for azure
4. Publish the zipped artifact to azure

App service (websites)

Build

Firstly build your solution, if using any nuget packages restore these first
nuget restore
Perform the build on the solution
msbuild foobar.sln /t:build /p:configuration=release /p:OutDir=BUILD_OUTPUT_PATH
Grab the published website from ./foobar/BUILD_OUTPUT_PATH/_PublishedWebsites/foobar and safe it to your artifact store of choice.

Deploy

Firstly apply the correct config for your target environment for this deployment to the artifact saved from the build. there are many ways to do this so i wont go in to it here.

Next zip up the deployable artifact
Add-Type -Assembly System.IO.Compression.FileSystem
$compressionLevel = [System.IO.Compression.CompressionLevel]::Optimal
[System.IO.Compression.ZipFile]::CreateFromDirectory($sourcedir, $zipfilename, $compressionLevel, $false)
Prepare the powershell session for the upload to azure
Load in your azure cert
Import-AzurePublishSettingsFile azure-credentials.publishsettings
Select the desired subscription
Select-AzureSubscription 00000000-0000-0000-0000-000000000001
Publish the website using the azure cmdlet
Publish-AzureWebsiteProject -Package buildoutput.zip -Name foobar -Slot 'staging'

Webjobs

As above except the location of the built artifact foobarjob/BUILD_OUTPUT_PATH
And the cmdlet for creating the webjob
New-AzureWebsiteJob -Name foobar -JobName foobarjob -JobType Continuous -JobFile .\buildoutput.zip

Conclusion

It turns out its actually really easy to achieve the result we desired, mainly the repeatable deployment of a known, tested build artifact to any environment we like.

We use GoCd from thoughtworks to orchestrate the above process through the pipelines feature. When all this is combined with staging slots, blue green deployments and the like you get a really robust process, excellent auditability, a solid rollback strategy and not to mention peace of mind when pushing to prod.

Id love to hear how you achieve the same results, whether its through version control strategies, azure deployment slot strategies or something else.
Let me know.

Azure app service error 'An attempt was made to load a program with an incorrect format'

I was deploying a new website today and came across an issue running the site on azure. The site ran fine on my dev machine both running under vs2015 and when setup as a website under my local IIS.

But when running on a freshly created Azure app service i got the error:

Could not load file or assembly 'System.Web' or one of its dependencies. An attempt was made to load a program with an incorrect format.

Exception Details: System.BadImageFormatException: Could not load file or assembly 'System.Web' or one of its dependencies. An attempt was made to load a program with an incorrect format.

[BadImageFormatException: Could not load file or assembly 'System.Web' or one of its dependencies. An attempt was made to load a program with an incorrect format.] System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly ....


After digging around on the interwebs for a while it became apparent it was an issue with either the target architecture or the targetting of the dlls. My solution was built to target 'any cpu' but somehow the version of System.Web that was brought down by nuget was the 64bit version.

So i figured that changing the target architecture would have the desired effect and changed the azure platform from 32 bit (the default for new app services) to 64 bit in the application settings of the azure portal (portal.azure.com).


Problem solved. I hope this will point someone else to a quick resolution in the future.

Output nunit2 compatable xml from the nunit3 console runner in your CI pipeline.

Upgrades

We have recently upgraded many of our .net projects to use C#6 and .net 4.6.1. At the same time we upgraded nunit to nunit3 on all our build servers. Problem was that GoCd (our CI server) doesn't recognize the new nunit3 result format, only the older more ubiquitous junit/nunit2 xml format. whilst failing tests would cause the build to break you had to dig into the console output to see the failures rather than looking in the tests tab.

Investigation

The solution to the problem was to get the nunit3 console runner to output results xml in the old nunit2 style, easier said than done. The documentation for the runner is actually quite bad in this regard. All it says is: "The --result option may use any of the following formats. nunit2" but doesn't actually show how. I ended up in the source code for the nunit3-console.tests and saw the answer burred within.

Solution

The final command i ended up with is as follows:
nunit3-console --result:"Result.xml;format=nunit2" --where "cat!=Integration" .\tests.dll
This output was then able to be imported into Go and we again get visibility of the numbers of tests run, passing and failing on the go server.

Thanks to all who responded on the google group discussion pointing me in the right direction.

Applying Azure resource locks to all the databases and storage accounts in a given resource group with powershell

If you have followed any of my previous blogs you will know we have tens of microservices (over 50) in our current architecture. With these microservices goes data (lots of data, valuable data). Each service has storage accounts and/or databases (which we dont really want to loose). We have been going through the process of automating the creation of these resources and in the process need to ensure they are not accidentally deleted (as we have tear down scripts, dangerous in the wrong hands).

Powershell

What follows are some powershell commands that can add resource locks to all your databases and storage accounts, they took a while to build, but are very effective, enjoy.
Write-Host -ForegroundColor Cyan "Adding a CanNotDelete lock to all databases"
Get-AzureRmResource `
 | Where-Object {$_.ResourceGroupName -eq myresourcegroupname -and `
                 $_.ResourceType -eq "Microsoft.Sql/servers/databases"} `
 | Select-Object `
     ResourceName,ResourceType, `
     @{name="name"; `
       Expression={$_.name.replace("myazuresqlservername/","")}}, `
     @{name="lockname"; `
       Expression={"lock-databases-"+$_.name.replace("myazuresqlservername/","")}} `
 | %{New-AzureRmResourceLock -ResourceGroupName myresourcegroupname`
                             -LockLevel CanNotDelete `
                             -LockNotes "Prevent accidental deletion" `
                             -LockName $_.lockname `
                             -ResourceName $_.ResourceName `
                             -ResourceType $_.ResourceType `
                             -Verbose -Force -ErrorAction Stop}
Write-Host -ForegroundColor Cyan "Adding a CanNotDelete lock to all storage accounts"
Get-AzureRmResource `
 | Where-Object {$_.ResourceGroupName -eq myresourcegroupname -and `
                 $_.ResourceType -eq "Microsoft.Storage/storageAccounts"} `
 | Select-Object ResourceName,ResourceType,Name, `
                 @{name="lockname"; `
                   Expression={"lock-storageAccounts-"+$_.name}} `
 | %{New-AzureRmResourceLock -ResourceGroupName myresourcegroupname`
                             -LockLevel CanNotDelete `
                             -LockNotes "Prevent accidental deletion" `
                             -LockName $_.lockname `
                             -ResourceName $_.ResourceName `
                             -ResourceType $_.ResourceType `
                             -Verbose -Force -ErrorAction Stop}
You can customise a bit further and replace the strings "myazuresqlservername" and "myresourcegroupname" with powershell variables and stick this straight in a powershell console or in a script.

Lock removal

As an aside, if you do subsequently want to delete the DB or storage account you first need to remove the lock like this:
Remove-AzureRmResourceLock -ResourceId /subscriptions/00000000-1111-2222-3333-444444444444/resourceGroups/myresourcegroupname/providers/Microsoft.Sql/servers/myazuresqlservername/databases/mydatabasename -LockName lock-databases-mydatabasename

Feedback

Please if you found this useful or you know a better way let me know in the comments below. cheers.

The fallout of going the microservice route. Build and deploy headaches

Build/Deploy issues with GoCd

As you may have seen in a previous blog post we have a lot of pipelines, over 500 now and its set to grow by another 200 in the next month as we bring another 3 environments up. 700 pipelines is too much for a vanilla go instance to handle, the server can sometimes wait 3-4 minutes before detecting a check-in or to respond to a manual pipeline click.
I'm not having a go at go (no pun intended) i love it i think its very good at its job, but when you get to this scale apparently the embedded H2 DB starts to be a bottleneck and so you need to upgrade to the postgreSQL plugin which adds more power on the back end which would solve some issues I've seen. But there is also an issue with the main pipelines UI which can take 10+ seconds to render with 500 pipelines on the page at once, I put this down to browser performance as you can see the call to the server coming back quite quick with the markup, its just a very big page (15000+ divs, 1600+ forms, 4300+ input fields all with associated styling and javascript). So with all this in mind we decided to split up our go server into six smaller instances.

Go server/agent setup

We have over 70 micro-services mainly written in .net but with a scattering of node and spa apps (detailed here), each is independently deployable and so we have decided to split them based on service boundary which we have roughly 6. So we have decided to create 6 go servers and split the workload across all six so that each server will only be handling 1-2 hundred pipelines each.

There are three consequences to splitting up the servers like this: One is that any templates are going to get duplicated, and you will have to keep track of failing builds on six servers instead of just one which means you will need to log into six servers and remember which services are on which go server instance. But by splitting the work by service boundary we will keep the value stream map intact where by we would loose it if we had split the servers by deployment region of which we now have three (each with a test, preprod and prod environment).

Installing multiple go agents on one VM, pointing at multiple different go servers

So we have six VMs each with a go server installed, and now by the use of powershell six agents on each VM. Each VM has an agent for each of the servers so that the workload can be spread evenly. The problem with this is that whilst a given VM can host several agents (read this) all the agents will point back to the same go server because the install uses an environment variable. The solution to this is to hack the config\wrapper-agent.conf file, change all the instances of '%GO_SERVER%' replacing it with 'go-serviceboundary1.mydomain.com' which is the DNS entry for the go server you are working for.

Given i was creating six agents on six machines i didn't want to do this by hand so i created a script to do it for me (found here) the more interesting bits are summarised below:

Download the latest go agent:
$goSetupExe = "go-agent-16.2.1-3027-setup.exe"
$client = new-object System.Net.WebClient
$client.DownloadFile("https://download.go.cd/binaries/16.2.1-3027/win/$goSetupExe", "C:\go\$goSetupExe")
Command line install the first agent
.\go-agent-setup.exe /S /SERVERIP=go-serviceboundary1.mydomain.com /D=C:\go\agent1
Copy the install to a new instance
new-item "C:\go\agent$agentNumber" -ItemType Directory
Copy-Item "C:\go\agent1\*" -Destination "C:\go\agent$agentNumber" -Recurse
Remove-Item "C:\go\agent$agentNumber\config\guid.txt"
Remove-Item "C:\go\agent$agentNumber\*.log"
Rewrite the config
(Get-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf").replace('go-serviceboundary1.mydomain.com', "go-$boundedContext.mydomain.com") | Set-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf"

(Get-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf").replace('c:\go\agent1', "c:\go\agent$agentNumber") | Set-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf"
Create and start the second service instance
New-Service -Name "Go Agent$agentNumber" -DisplayName "Go Agent$agentNumber (go-$boundedContext.mydomain.com)" -Description "Go Agent$agentNumber (go-$boundedContext.mydomain.com)" -StartupType Automatic -BinaryPathName "`"C:\go\agent$agentNumber\cruisewrapper.exe`" -s `"c:\go\agent$agentNumber\config\wrapper-agent.conf`""

Start-Service "Go Agent2"

It works really well for getting a new setup running quickly, and if i ever need to add a new go server and new agents this will be easy to extend to quickly get up and running.

Feedback

It would be great if i could get some feedback as to whether this is a good idea or not, or else how best to split things up, but i feel this is a nice pragmatic solution with minimal downsides.

Pattern matching in sublime text with regex

An exercise in Yak shaving

Recently I had a very large xml file where i needed to do some string manipulation and replacements, sublime text is always my go to editor for this type of thing.
Here is an example snippet.
<data name="NumberToWord_1" xml:space="preserve">
  <value>first</value>
</data>
<data name="NumberToWord_2" xml:space="preserve">
  <value>second-value</value>
</data>
<data name="NumberToWord_3" xml:space="preserve">
  <value>3rd text value</value>
</data>

I needed to select the names and then paste them into the values.

Firstly we need to select the name with one of the following regex:
(?<=<data name=").+?(?=")
(?<=<data name=")[^"]+
<data name="\K[^"]+
^.*?"\K\w+
The first uses look behind, look around and a non greedy selector, maybe not the easiest to understand. I evolved this into the second regex by doing away with the lazy selector and the look around at the end. the third is basically the second rewritten with the meta-character \K to reset the start point of the regex (keep). Finally i trimmed away the data by looking for the first double quote on the line, resetting the keep and after that only selecting word characters.

once i had all the name strings highlighted i can use sublime texts multiple copy feature to put the 100+ words into the clipboard.

Next to paste the values back. So we need to select the value
(?<=<value>).+(?=<value>)
(?<=<value>)[^<]+
<value>\K[^<]+
^.+?e>\K[^<]+
The first line again uses a look behind and a look around with a lazy select all. The second example replaces the look around with a more defined character selector of negative <. The third replaces the look behind with a Keep reset character. finally just for fun i selected the first e> in the text with a lazy selector.

So I've now got all the values selected, just paste the current multi clipboard over the top and we have done.

Alternatives

Now there are many ways to skin a cat, i really wanted to play with regex today and so this was a nice exercise to practice with more advanced regex. But i know not everyone likes or gets regex, another approach would be to use the cursor. First select all lines with a data element (multi-select), go home and ctrl+right click till you have the cursor at just after the first quote. Then ctrl+shift+right to select the word, then copy them all. Press the down arrow to get focus to the value element, end and ctrl+left to get the cursor to just inside the end of the element value, ctrl+k, ctrl+space to set a mark, home and ctrl+left to get the cursor just inside the start of the element value, ctrl+k, ctrl+a to select to the mark and then paste. Or for this last step use the ctrl+shirt+a to use sublimes widen selection which works in xml documents.
There are so many variations on this cursor based approach that i cant list them all here, suffice to say sublime text is very powerful at text manipulation, learn your tooling people, and enjoy.

Want to see this in action watch this video:

Recorded on KRUT (an open source screen recorder) with keyjedi (to show the keyboard shortcuts).

If you think there is a simpler (or more clever :-) approach, please leave a comment and share. How bald is your Yak now?