Advanced filtering and navigation on Thoughtworks Go Cd with tamper monkey

Our problem

We use Go from Thoughtworks to manage the build and deployment of all our software. Attached below is a screen shot of the current pipelines.

As you can see we have quite a lot going on. In fact we have 390 pipelines (55 services and api builds, 40 nuget package build/publish, and the rest are deployments to our 5 environments from test through preprod and live). There are 730 stages in total (Build, test, deploy, ect). And we have 20 go agents running on different servers.

So you can imagine finding the pipeline you are looking for is tough. We have adopted naming conventions which help but its really quite difficult to find what you are after with the supplied search capabilities.

Also its very tricky to see if anything is broken (red), there is just no way you will notice it in all the pipelines.

As an aside we are using cradiator to give us an information radiator of all our build and deployment pipelines (I blogged about this a year ago) but with over 700 stages its really in need of an overhaul but that's for another blog post.

The solution

We have created a tamper monkey script (found here) that enhances the functionality of the Go pipelines view, allowing you to filter the visible pipelines by status or by keyword.

 

There is also the issue of navigating between the settings and the pipeline history, and visa versa. Often you are investigating a failure, you find the problem in the history and then want to change the settings. Well there is no link, so we created one for you to easily navigate between the 2 parts of the UI easily.

Notice the "Settings" link above and the "History" link below both in the header

The script can be found here: https://gist.github.com/DamianStanger/d1db873e9297b34160ce#file-go-cd-tampermoneyscript-js

Install

Firstly you need to get tamper monkey installed in your browser, you can get it from the chrome web store or http://tampermonkey.net/

Then from the tamper monkey dashboard add the script and tell it what urls to attach itself to. The update url needs to be your instance of Go (our update URL is https://goserver:8154/go/* ). On the settings tab I also add in a couple of user includes to https://goserver:8154/go/home and https://goserver:8154/go/pipelines.

That's it, instantly enjoy better productivity. I imagine this is a good enhancement even if you only have a tenth of the pipelines we have.

Script your build and deployment of android cordova apps with powershell

We are developing a new version of our customer facing solution, across web, ios and android, using cordova(phonegap).

Im a big proponent of build automation, and the classical (recommended?) way of using eclipse to build and manage the code base was getting me down, so i decided to write some scripts to build and deploy the app to either a device, an emulator or prepare for release. I also wanted any developer to be able to checkout the code and run the scripts to build the app.

I wrote the scripts in powershell (sorry!) with some batch files to make the various functions easy to run (im developing on windows 8 by the way).

You can find the scripts here: https://github.com/DamianStanger/AndroidBuildScripts

So how does it all work?

firstly it goes with out saying that you need your dev env set up for android development on the command line with cordova http://cordova.apache.org/docs/en/edge/guide_cli_index.md.html#The%20Command-Line%20Interface

As you will know (if you do cordova development) when you use the command line tools for creating cordova apps the folder created is where all your source code is placed and inside there is the www folder that is where you keep your .js and .html files. The problem is that you are keeping your source code along side the automatically built cordova files, not ideal. so I've created my own source folder that is where all the code you edit is kept. We then use powershell to copy these files to the correct places.

The development process

In a powershell (or dos cmd if you prefer)

build.bat
emulate.bat or install.bat

Line 01. run either or, depending on if you are using a real device or not

That's it. Now you might notice that build can take a wile to run because its setting up everything from scratch so i created a shortcut that will only copy your changes across.

quickCopy.bat
emulate.bat or install.bat

This is all good for general day to day dev but eventually you will want to test a production build on a real device for this use the following commands

release.bat
installRelease.bat

To make this work you must only have either a device plugged into usb or an emulator turned on (please use genymotion its so much faster than a standard emulator)
The release process signs and aligns your apk for you :-) so when you are ready you just send the apk you have tested to the play store.

The scripts

Here I'm going to show select lines of code from build.ps1

For building the app in debug and getting that on to your emulator or phone

function create()

cordova create app-cordova-android com.myapp.app myapp
...
cordova platform add android
...
cordova plugin add org.apache.cordova.device

function build()

cordova build android

function emulate()

cordova emulate android -d

function installDebug()

cordova run android -d

To build the releaseable apk and to get that onto your phone use the following:

function release()

cordova build android --release

function sign()

jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore ..\appstore\android-keystore\myapp -keypass myappKeyPassword -storepass myappStorePassword -signedjar .\platforms\android\bin\myapp-release-signed.apk .\platforms\android\bin\myapp-release-unsigned.apk myapp
zipalign -f -v 4 .\platforms\android\bin\myapp-release-signed.apk .\platforms\android\bin\myapp-release-signed-aligned.apk

function installRelease()

adb uninstall com.myapp.app
adb install .\appstore\APKs\myapp-release-signed-aligned.apk

Release versioning

When doing a release to the play store you need to make sure the version numbers are incremented each time, for this i added a helper which will update all the relevent places in the source for you.

just run:

setVersion 102 1.0.2

This will change all the files that need changing in order to properly put a new version of the app onto the play store.

Upload to play store error

Upload failed
You uploaded a debuggable APK. For security reasons you need to disable debugging before it can be published in Google Play

Make sure your manifest is set thus:

<application android:debuggable="false" android:hardwareAccelerated="true" android:icon="@drawable/icon" android:label="@string/app_name">

Resources

• https://github.com/DamianStanger/AndroidBuildScripts
• https://cordova.apache.org/docs/en/3.2.0/guide_cli_index.md.html#The%20Command-line%20Interface
• https://developer.android.com/tools/publishing/app-signing.html
• http://docs.oracle.com/javase/6/docs/technotes/tools/windows/jarsigner.html
• https://developer.android.com/tools/help/zipalign.html
• http://www.genymotion.com/

jquery promises wrapped in javascript closures, oh my..

I recently had a question from one of my fellow devs regarding a problem where the values in a loop were not what they expected. This was down to the deferred execution of the success function after a promise had been resolved.

The following code has been simplified for this explanation:

function (userArray) {
  var i;

  for (i = 0; i < userArray.length; i++) {
    var userDto = userArray[i];
    var user = new UserModel();
    user.displayName = userDto.displayName;
    var promise = service.getJSON(userDto.policies);

    promise.then(function(policyDtos){
      system.log("user.displayname : " + user.displayname);
      convertAndStore(user, policyDtos);
    });
  }
}
Input:
userArray = ['bob', 'sue', 'jon']; //* see footer
Output:
user.displayname : jon
user.displayname : jon
user.displayname : jon

service.getJSON (line 07) returns a jquery promise, the function actually makes a service call to an external API and so can take some time to resolve. Notice how the var userDto and user are declared within the loop, this is not best practice in javascript as the variables are actually hoisted up to the containing function (next to var i). To a none javascript expert it looks like the variables will be created anew inside the loop as they are in c#. In fact there is only one copy of i, user and userDto so obviously the values are overwritten within every loop iteration.

This is the fixed function using closures.

function (userArray) {
  var i,userDto, promise;

  for (i = 0; i < userArray.length; i++) {
    userDto = userArray[i];
    promise = service.getJSON(userDto.policies);
 
    (function(capturedDisplayName) {
      var user = new UserModel();
      user.displayName = capturedDisplayName;

      promise.then(function(policyDtos){
        system.log("user.displayname : " + user.displayname);
        convertAndStore(user, policyDtos);
      });
    }) (userDto.displayName);
 
  }
}
Input:
userArray = ['bob', 'sue', 'jon']; //** see footer
Output:
user.displayname : bob
user.displayname : sue
user.displayname : jon

The introduced function on line 07 creates a capture around the variable userDto.displayName, the variable is a parameter called capturedDisplayName. Now there is a copy of this variable for every iteration of the loop allowing you to use it after the promise has resolved.
You may wonder why the variable promise works as intended given the problems with user and userDto? Well that is because the object referenced within the loop iteration has the .then attached to it before it is overwritten by the next loop, remember the object itself is not overwritten or changed on line 05 only the reference to the object.


//* In reality are more complex objects than simple strings, I'm trying to keep this simple for readability.
//** The names have been changed to protect the innocent.

Using jasmine to test JQuery

Ive been doing lots of javascript work of late (mainly in a durandal SPA app) and wanted to test the following function that uses jquery.

function update () {
    errorVisibleFlag(false);
    $("input.delete:checkbox").each(function() {
        if ($(this).is(":checked")) {
            var idToDelete = $(this).attr("id"),
                status = $(this).attr("status");
            dataService.delete(idToDelete)
                .fail(function(){errorVisibleFlag(true)});
        }
    });
    get();
}
This function is part of a knockout view model within a durandal app.

I came up with the following jasmine test that uses spys to verify that the calls were working and also some DOM manipulation to allow jquery to bind the function to something. Ive found it works really well.

it('When update is called, delete is called with the correct parameters', function(){
  var inputCheckbox = '<input type="checkbox" class="delete" id="bc87d270-95bc-49bc-9cac-3e903d09590b" status="Pending">',
      inputCheckboxId = "#bc87d270-95bc-49bc-9cac-3e903d09590b";
  $("body").append(inputCheckbox);

  var spyDelete = spyOn(vm.dataService, "delete").andCallFake(getData);
  var spyGet = spyOn(vm.dataService, "get").andCallFake(getData);

  vm.update();

  expect(spyDelete).toHaveBeenCalledWith("bc87d270-95bc-49bc-9cac-3e903d09590b");
  expect(spyGet).toHaveBeenCalled();

  $(inputCheckboxId).remove();
});
Note: The viewmodel vm is injected into the test using require.js and the function dataService.delete returns a jquery promise

My Collection Of Podcasts For Developers

[update "2015-11-03"]I've just posted a new version of this list to developer-podcasts-v2[update]

I'm always on the lookout for more development focused podcasts and thought id better share my current crop as i know others are also looking for more podcasts to accompany them on their daily commute.

My interests

Some background on me: Primarily a .net developer with an interest in agile, javascript, ruby, design and devops. So this list derives from these interests.

.Net

http://www.hanselminutes.com/
http://www.dotnetrocks.com/
http://deepfriedbytes.com/
http://channel9.msdn.com/

General Development

http://www.se-radio.net/
http://herdingcode.com/
http://techcast.chariotsolutions.com/
http://devnology.nl/en/podcast
http://thisdeveloperslife.com/

Javascript

http://nodeup.com/

Ruby

http://ruby5.envylabs.com/

DevOps and IT

http://www.runasradio.com/
http://foodfight.libsyn.com/
http://devopscafe.org/

Business

http://www.startupsfortherestofus.com/

Design and Development

I found this resource last month that contains lots of other great podcasts on design and development http://www.smashingmagazine.com/2013/04/19/podcasts-for-designers-developers/?utm_source=feedly

A SPA seed - Javascript stack with node and angular.

Single Page Application built on node.js and angularJS

I've been looking at creating a SPA with a full javascript stack so decided to pull together a seed based on node and angular with jshint to test all the .js files, mocha to run the node tests, karma to run the browser based angular tests and cucumber for BDD (full stack testing/acceptance tests).

I did this because i could not find any examples of how to pull together angular and node in the same project along with testing of everything. This is a good start but until i use it in anger i wont really know if ive got it right, so when i do i will try and update it.

https://github.com/DamianStanger/NodejsAngularSPASeed

Details

Node

node.js, npm, angular, karma, Mocha, phantom.js, jshint, jshintRunner

Ruby (1.9.2)

Ruby is for cucumber that is used for the full stack acceptance testing
ruby 1.9.2, devKit, bundler, cucumber, capybara

Next

The readme.md file gives details on how to get it all running and get the tests working. Then just clone this repo and use it as a starting point for your next node SPA app.

Developing node modules, npm and git. how to publish npm packages from a windows machine whilst preserving the unix style line endings

I recently had a problem whilst publishing a new node.js module id written that is designed to run jshint recursively against a number of directories and or files.

I develop on windows and so the line endings are dos based (crlf) but the file to run your app as stored in the bin folder needs to have unix line endings (lf) for it to run on a mac or nix systems.

i save code in git and github with unix line endings turned on in the repository but on my working directory the file system is dos line endings. So when i publish using 'npm publish' the file in my bin is published with dos line endings.

this means that when you do an npm install -g on a mac or linux you get the error
env: node\r: No such file or directory

To fix this i developed a little batch script that i use to do the publishing of new versions its really simple and will change the line endings just before publishing.

dos2unix --d2u bin\jshintRunner
npm publish

it uses the built in dos2unix (im running windows 8 ultimate) to change the line endings. I hope this helps someone else out with a similar issue.

Link to the project on github : https://github.com/DamianStanger/jshintRunner and on npm https://npmjs.org/package/jshintrunner

A node application to count lines within a files

Node with Mocha, Should and Sinon to count file lines

I've recently had the requirement to count lines of source code in 3 or 4 different code bases, including a couple of single page web apps written in javascript, angularjs and karma, a couple of java server side services and an acceptance test suite again written in java and selenium.

I wanted to compare the code bases and to look into the ratios of test code to production code so we as a team could get some collective feel for the entire code base, which to me was a very valuable exercise.
I decided to write a command line app in node and javascript, mainly because at the moment I’m trying to boost my javascript knowledge and I’m really interested in node. This command line app would count the lines of code in a code base. There is nothing better than a real requirement to spur you into action.
You can find the source code here: https://github.com/DamianStanger/lineCounter
I’m quite pleased how it has turned out but as is the way with every piece of software I’ve run out of budget (free time) before completion. But I would have liked to enhance it further if I could find the time.

Enhancements:

• Return a json string that has file and line counts for every directory in the codebase. This output could then be pushed into a d3 app to visualise the source code and the relative sizes, that would be cool.
• Ability to customise the ignored files and directories.
• Ability to hook into team city, this would need some new output reporter creating so we could track the lines of code over time.

Learnings

• I started off using Karma and jasmine for running the tests but found that they were difficult to get to play well with the node modules I created so I switched to Mocha (http://visionmedia.github.io/mocha/) glad I did, because I love it. I especially like the BDD style tests I can write with many nested describes to get the test context. I’m not sure how I’m going to cope going back to the flat structure of nUnit.
• I started to use Should (https://npmjs.org/package/should) as the preferred mechanism for asserting. The fluent interface is really appealing, it’s very similar to one I’ve been using in .net for a while now.
• I’ve needed to do a bit of mocking in this project and for this I found Sinon (http://sinonjs.org/ ). Very powerful and flexible, its been capable of meeting all my stubbing and mocking needs up to now. Bit of a learning curve but its all good.

node.js an introduction and tutorial to javascript on the server

Presentation

I gave a presentation on node.js on my first day back from the Christmas holidays. It was fun, plenty of people in the room, all eager to learn the basics of node.

Firstly an overview of what node is, what its good for and an introduction to the event driven architecture behind node. I did the classic fast food example and then a coding session.

The live coding session consisted of an introduction to the node REPL, some basic examples of javascript on the server, followed by a simple web server and some performance testing with apache bench (https://httpd.apache.org/docs/2.2/programs/ab.html).

To finish up I demoed a reverse proxy written in one line of node. It's amazing how much power this has, and even more amazing that you can write something like that in such a concise manner but which is still understandable (read maintainable).

I thought it went really well. I don't give many presentations but when I do I like them to be good, relevant, interesting and entertaining (as far as a technical subject can be). Of course I was nervous, especially because i was videoing it. I wanted to actually see what i was like. It's the best way to improve, fast feedback reflection and improvement.

Live coding is always dangerous but it all went remarkably well. Although I did have a small hiccup in that i could not connect to the wireless network, but that only impacted one of my examples, I weathered that storm.

Video

So yes I videoed it and have uploaded to youtube http://youtu.be/vGBk8EB-Yz0 Check it out I'd be really interested to hear your feedback on my presentation style and content.



Attached below are the code examples from the talk so you can test them out if you like.

Enjoy.

Code demo

REPL

1+2;
var add = function(a,b){return a+b};
add(3,4);

process
process.pid
process.env.Path

fs.readFile('foo.txt', 'utf8', function (err,data) {
console.log(data);
});

foo bar code

setTimeout(function(){
console.log("foo");
}, 2000);
console.log("bar");

setInterval(function(){
console.log("bang!");
}, 1000);

hello world web server

var http = require('http');
var server = http.createServer(function (req, res) {
res.end('Hello World\n');
});
server.listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.0.1:1337/');

curl http://localhost:1337/
curl -i http://localhost:1337/

res.write('hello');
setTimeout(function() {
res.end('World\n');
}, 6000);
The following needs apache bench installed and on your path
ab -n 10 -c 10 http://127.0.0.1:1337/

new file express.js

var express = require('express');
var app = express();
app.listen(8080);
app.get('/', function(req, res){
console.log("get");
res.send("Welcome to node!");
});

node package manager

npm install express

app.get('/foo/', function(req, res){
console.log("getfoo");
res.send("Welcome to foo!");
});

a one line reverse proxy...

using request, a module to simplify the making of web requests
var http=require("http"),
request=require("request");
http.createServer(function (req, res) {
console.log(req.url);
req.pipe(request("http://www.xperthr.com/" + req.url)).pipe(res);
}).listen(1337);

How to update/install Node on Ubuntu

I needed to upgrade node.js on my ubuntu dev machine but could not find any good instructions on the internet, i tried several suggestions and finally got it working using an amalgamation of a few blogs.
My current system setup before the process.
ubuntu64:~$ which node
/usr/local/bin/node
ubuntu64:~$ node -v
v0.4.9
Firstly make sure your system is upto date
ubuntu64:~$ sudo apt-get update
ubuntu64:~$ sudo apt-get install git-core curl build-essential
            openssl libssl-dev
Then clone the Node.js repository at git hub:
ubuntu64:~$ git clone https://github.com/joyent/node.git
ubuntu64:~$ cd node
I wanted the latest tagged version
ubuntu64:~/node$ git tag
....
....big list of all the tags
....
ubuntu64:~/node$ git checkout v0.9.2
Then I removed the old version of node
ubuntu64:~$ which node
/usr/local/bin/node
ubuntu64:~$ cd /usr/local/bin
ubuntu64:/usr/local/bin$ sudo rm node
Now to install the desired version, in may case v0.9.2
ubuntu64:/usr/local/bin$ cd ~/node
ubuntu64:~/node$ ./configure
....
ubuntu64:~/node$ make
....
ubuntu64:~/node$ sudo make install
....
Then I had to run the following to update the profile
ubuntu64:~/node$ . ~/.profile
Finally confirm that node is in fact upgraded, and npm has magically been installed too :-) bonus
ubuntu64:~/node$ which node
/usr/local/bin/node
ubuntu64:~/node$ node -v
v0.9.2
ubuntu64:~/node$ which npm
/usr/local/bin/npm
ubuntu64:~/node$ npm -v
1.1.61

Installing node.js on windows 7 machine

Ive recently been looking for instructions on how to install node.js (the javascript powered web server) on windows, and it received some quite confusing answers involving cygwin, building things from source etc.
And yes it is possible to install node on windows 7, in fact.... Its really really easy actually.

1. download node.exe from http://nodejs.org/dist/v0.6.0/node.exe (link avaliable from http://nodejs.org)

2. create a sample node app

var http = require('http');
http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('Hello World\n');
}).listen(1337, "127.0.0.1");
console.log('Server running at http://127.0.0.1:1337/');


save in a file 'example.js' and place in the same folder as node.exe (anywhere on your system)
open a command prompt in the folder with node.exe and your .js file

run > node example.js
and thats it, now open your browser and goto: http://127.0.0.1:1337/

Hello World

how easy was that?

JQuery and ASP.Net

Ive been playing with JQuery alot lately and am loving the ease of use and the feature rich toolset, you can get some realy good effects with it.

i know sometimes .net developers get, i dont know scared about trying new things that arnt born of microsoft, but JQuery is now being actively endorsed by microsoft and is actually shipped with the latest versions of the framework, so how do you use it.

1. If you dont have it get a copy of jquery.js its quite small (im currently using 'jquery-.3.2.min.js' which is 56Kb)
2. Include it in your web app
3. There are 2 ways to use it in a page
3a Add a script reference into your head section of the aspx page
<script src="Js cript/jquery -1.3.2.min.js" type="text/javascript"></script>
3b If you are using asp.net AJAX you can let the script manager handle it by:
3b.1 Adding a script manager to the form in the body of the aspx page
3b.2 Adding a scripts element
3b.3 Adding a script reference to the jquery file
3b.4 The result would look like this (remember it needs to go in side the pages form element
<asp:scriptmanager runat="server">
<scripts>
<asp:scriptreference path="~/Jscript/jquery-1.3.2.min.js">
</asp:scriptreference>
</scripts>

You are all set to use Jquery on the page

4. The most common way to initialise things is through the body onload, but this executes when the page is fully loaded, images and all, JQuery uses a special piece of script that executes when the page DOM is ready.

$(document).ready(function()
{
// do stuff when DOM is ready
});

This is where you can set styles, set up events to occur on different user actions all sorts of good stuff.
1st a couple of notes on the above code block:
1 it needs to be inside a sscript element
2 it needs to be located AFTER the scrip ref we declared in step 3 so if using the script manager it needs to be located inside the form after the script manager elements.
You can search the net for all the things you can do with JQuery, i wont repeat them all here, google is most definatly your friend.

But for now a simple example.

say you want to set up a click event to a group of radio buttons so that when clicked a javascript event is fired off to show or hide related DIV elements

<script type="text/javascript">
$(document).ready(function(){
$(":radio").click(function(event){RBEvent(this)});
$(".radioClass").hide();
});

function RBEvent(obj)
{
$(".radioClass").hide("slow");
$("#Div" + obj.id).show("slow");
}
</script>

And here is the aspx elements that go along with it

< asp:RadioButton ID="RBBus" runat="server" Text="Bus" GroupName="RB" />
< asp:RadioButton ID="RBTube" runat="server" Text="Tube" GroupName="RB" />
< div id="DivRBBus" class="radioClass">
Bus details< br />
< /div>
< div id="DivRBTube" class="radioClass">
Tube details< br />
< /div>

The neat thing that JQuery gives us, as in this example, is the ability to put all your page script into one place, and to have JQuery attach the events dynamically at run time making your javascript much easier to maintain. You could of course put all your script into a separate file as well which can be cached on the browser making the pages even quicker to load up.