Monday, 30 November 2015

Do we need to deploy clean up work if there is no additional functionality? i.e no (perceived) business benifit.


We recently did some work to a number of services where all we were doing was removing some old functionality that is no longer used (obsolete code). Removing messages, handlers, classes, tests. I like deleting code, it makes things simpler, less logic to break, less places for bugs to hide. Once we finished, we wanted to get all of this released to prod ASAP. We tested all of the areas affected in a large system level test in the preprod environment, all of the services (and others that had not changed) working together.

But we had push back. 

Push back

The question was: Why do you need to release this clean-up in advance of any further work?
By doing so you are making this an active rather than passive deployment with associated extra risk and double the cost.
If you are removing unused code you can just deploy with the next addition/change to code because by testing that you are implicitly testing that absence of code. Even if the new code isn’t affected, then our deployment checklists cover that situation too – we have already double checked this removal/clean-up of code won’t have an impact on production


I broke this down into a number of sections. A number of questions that I thought were being asked in the statement.

Question: Why do many releases instead of one, isn’t that more risky?
This is a question from the old skool of thought, releases are big bad things we need to do as few as possible.
Answer: I would say many small releases are inherently less risky.
If (very unlikely but) if something goes wrong it will be less clear what caused the issue. Was it new functionality, or the clean-up work that is the culprit?
If we release now, we know what to monitor over the coming days, and have to monitor less
If we don’t deploy all 8 things, someone else will (at some point and in some cases many months in the future). This poor soul will need to decide if the changes we did need testing, what the consequences of the changes are, and worry about if there are other dependent services to deploy.
Each service is push button, so deploy time is small.
Rollback is not hard if we need it with no business consequences at the moment. In future if we release with other functionality and the changes we have made break something we need to rollback new functionality too.

Question: If we are removing code why the need to release anything? There is no new functionality to release.
I guess this is a question about business value, no new business value no need to release.
Answer: That is true, its mostly removing old code and cleaning up. But its just as critical, almost more so to get this out in a small release sooner, as we may (again unlikely but possible, we are human) have removed something we should not have.

Question: Won't it be more to test doing it twice?
This assumes that we manually test everything on every release. where actually we only test what has changed manually in conjunction with automated testing for the rest.
Answer: We have already done good full end to end testing last week of the 8 things affected. If we wait until next week or the week after we will have to do the tests again as the versions of things to be released will all be different by then, so will need to test the 8 deployable things again full stack = extra 1 day

Other reasons for deployment.
The changes, what we did and why we did it, are still fresh in our minds. The longer we leave it the less sure we are that we will be doing the right things.
Its not critical, but I'd prefer not to do a partial deployment (service-X is going to get released soon) I'd like the rest of the clean-up to be deployed too.
Ideally prod and preprod are as same as possible (for environmental consistency and testing reasons). Any differences between the test system of preprod and prod invalidates other testing efforts. Because in prod services will be integrating with a different version of other services than that are in preprod making like-for-like testing impossible.


I maintain there is business benefit in doing the deployment now, and deploying all 8 services at that. To be fair businesses, managers, stakeholders, even developers (especially senior ones) have all seen their fair share of long deployments, failure and difficult rollbacks. Leading, ultimately to a fear of deployment. So it's natural to want to avoid the perceived risks. But perversely by restricting the number of deployments you are actually increasing the likelihood of future fail.
A core philosophy of the devops culture is to release early and often (continuous delivery). By doing the things you find painful more often you master them and make them trivial, there by improving your mean time to recovery.

The business benefit is ultimately one of developer productivity, testability and system up-time

Tuesday, 17 November 2015

Splunk alerts to slack using powershell on windows

We use Splunk to aggregate all the logs across all our services and APIs on many different machines. It gives us an invaluable way to report on the interactions of our customers through new business creation on the back-end servers running on NServiceBus, to the day to day client interactions on the websites and mobile apps.

We have been investing in more monitoring recently as the number of services (I hesitate to use the buzzword micro, but yes they are small) is increasing. At present pace I'd say there is a new service or API created almost each week. Keeping on top of all these services and ensuring smooth running is turning into a challenge, which splunk is helping us to meet. When you add service control, pulse and insight from particular (makers of NServiceBus) we have all bases covered.

We have recently added alerts to splunk to give us notifications in slack when we get errors.

The Setup

We are sending alerts from splunk to slack using batch scripts and powershell.

Splunk Alerts

First set up an alert in splunk, This splunk video tells you how to create an alert from a search results. We are using a custom script which uses arguments as documented here. Our script consists of 2 steps a bat file and a powershell file. The batch file calls the powershell passing on the arguments.

SplunkSlackAlert.bat script in C:\Program Files\Splunk\bin\scripts
@echo off
powershell "C:\Program` Files\Splunk\bin\scripts\SplunkSlackAlert.ps1 -ScriptName '%SPLUNK_ARG_0%' -NEvents '%SPLUNK_ARG_1%' -TriggerReason '%SPLUNK_ARG_5%' -BrowserUrl '%SPLUNK_ARG_6%' -ReportName '%SPLUNK_ARG_4%'"

SplunkSlackAlert.ps1 lives alongside
param (
   [string]$ScriptName = "No script specified",
   [string]$NEvents = 0,
   [string]$TriggerReason = "No reason specified",
   [string]$BrowserUrl = "https://localhost:8000/",
   [string]$ReportName = "No name of report specified"

$body = @{
   text = "Test for a parameterized script `"$ScriptName`" `r`n This script retuned $NEvents and was triggered because $TriggerReason `r`n The Url to Splunk is $BrowserUrl `r`n The Report Name is $ReportName"

#Invoke-RestMethod -Uri -Method Post -Body (ConvertTo-Json $body)

Slack Integration

You can see the call to the slack API in the invoke-restmethod, the slack documentation for using the incoming web hook is here. there is quite a rich amount of customization that can be performed in the json payload, have a play.

Before you can actually use this you must first setup slack integration as documented here which requires you to have a slack account.

The fruits of our labor:

All the script code is given in my gist here.


Thanks to my pair Ruben for helping on this, good work.

Tuesday, 3 November 2015

Developer podcasts v2

A couple of years ago i wrote a blog post about podcasts for developers, this is a follow up as I've now got substantially more. That and a couple of my colleges have asked for my list recently.


.NET Rocks! : Feed Url
Adventures in Angular : Feed Url
All Chariot Podcasts : Feed Url
All Things Pivotal : Feed Url
Azure Friday - Channel 9 : Feed Url
CodeChat (Audio) - Channel 9 : Feed Url
Coding Blocks | Software and Web Programming / Security / Best Practices / Microsoft .NET : Feed Url
Debug : Feed Url
Devnology Podcast : Feed Url
DevRadio - Channel 9 : Feed Url
Full Stack Radio : Feed Url
Functional Geekery : Feed Url
Hack && Heckle : Feed Url
Hanselminutes : Feed Url
Herding Code : Feed Url
Javascript Jabber : Feed Url
Jesse Liberty - Silverlight Geek : Feed Url
MS Dev Show : Feed Url
NodeUp : Feed Url
PowerScripting Podcast : Feed Url
Radio TFS : Feed Url
Ruby Rogues : Feed Url
RunAs Radio : Feed Url
Simple Programmer Podcast : Feed Url
Software Engineering Radio - the podcast for professional software developers : Feed Url
STLTechTalk Podcast : Feed Url
The Azure Podcast : Feed Url
The Cognicast - Cognitect Blog : Feed Url
The Java Posse : Feed Url
The Static Void Podcast : Feed Url
This Week On Channel 9 (MP4) - Channel 9 : Feed Url
ThoughtWorks : Feed Url
WebDevRadio : Feed Url
Windows Weekly (MP3) : Feed Url
YAPP: Yet Another Programming Podcast : Feed Url


Arrested DevOps : Feed Url
DevOps Cafe Podcast : Feed Url
Devops Mastery : Feed Url : Feed Url
Ops All The Things! : Feed Url
The Food Fight Show : Feed Url
The Ship Show : Feed Url

Developer related

Developer On Fire : Feed Url
Get up and CODE! : Feed Url
Mastering Business Analysis : Feed Url
Programmer Vs World : Feed Url
Startups For the Rest of Us » Episodes : Feed Url
The Security Influencer's Channel : Feed Url


Agile Instructor - Coaching for Agile Methodologies such as Scrum and Kanban : Feed Url
Agile NYC : Feed Url
Agile Weekly Podcast : Feed Url
The Agile Coffee Podcast : Feed Url
This Agile Life : Feed Url

Non tech

60-Second Mind : Feed Url
99% Invisible : Feed Url
All items | LSE Public lectures and events | Audio : Feed Url
Freakonomics Radio : Feed Url
Friday Night Comedy from BBC Radio 4 : Feed Url
Haute Couture Podcast - Claudia Cazacu : Feed Url
Monstercat Podcast : Feed Url
NPR: Invisibilia Podcast : Feed Url
NPR: TED Radio Hour Podcast : Feed Url
Planet Money : NPR : Feed Url
Radiolab from WNYC : Feed Url
RI Blog : Feed Url
TEDTalks (audio) : Feed Url
TEDTalks (video) : Feed Url


Bear in mind that some of these podcasts are no longer active. I've kept them in my list because i find past episodes very relevant to the here and now. you can search back to find relevent episodes on anything you care to think of, super useful.

Incidentally my current podcast player of choice is podcastaddict, variable speed playback and great control of how to download files, brilliant search functionality for local podcasts and its very easy to search for and add new podcasts. use it :-)

My opml file extracted from podcast addict is located here: