Tuesday, 17 November 2015

Splunk alerts to slack using powershell on windows

We use Splunk to aggregate all the logs across all our services and APIs on many different machines. It gives us an invaluable way to report on the interactions of our customers through new business creation on the back-end servers running on NServiceBus, to the day to day client interactions on the websites and mobile apps.

We have been investing in more monitoring recently as the number of services (I hesitate to use the buzzword micro, but yes they are small) is increasing. At present pace I'd say there is a new service or API created almost each week. Keeping on top of all these services and ensuring smooth running is turning into a challenge, which splunk is helping us to meet. When you add service control, pulse and insight from particular (makers of NServiceBus) we have all bases covered.

We have recently added alerts to splunk to give us notifications in slack when we get errors.

The Setup

We are sending alerts from splunk to slack using batch scripts and powershell.

Splunk Alerts

First set up an alert in splunk, This splunk video tells you how to create an alert from a search results. We are using a custom script which uses arguments as documented here. Our script consists of 2 steps a bat file and a powershell file. The batch file calls the powershell passing on the arguments.

SplunkSlackAlert.bat script in C:\Program Files\Splunk\bin\scripts
@echo off
powershell "C:\Program` Files\Splunk\bin\scripts\SplunkSlackAlert.ps1 -ScriptName '%SPLUNK_ARG_0%' -NEvents '%SPLUNK_ARG_1%' -TriggerReason '%SPLUNK_ARG_5%' -BrowserUrl '%SPLUNK_ARG_6%' -ReportName '%SPLUNK_ARG_4%'"

SplunkSlackAlert.ps1 lives alongside
param (
   [string]$ScriptName = "No script specified",
   [string]$NEvents = 0,
   [string]$TriggerReason = "No reason specified",
   [string]$BrowserUrl = "https://localhost:8000/",
   [string]$ReportName = "No name of report specified"

$body = @{
   text = "Test for a parameterized script `"$ScriptName`" `r`n This script retuned $NEvents and was triggered because $TriggerReason `r`n The Url to Splunk is $BrowserUrl `r`n The Report Name is $ReportName"

#Invoke-RestMethod -Uri https://hooks.slack.com/services/AAAAAAAAA/BBBBBBBBB/CCCCCCCCC -Method Post -Body (ConvertTo-Json $body)

Slack Integration

You can see the call to the slack API in the invoke-restmethod, the slack documentation for using the incoming web hook is here. there is quite a rich amount of customization that can be performed in the json payload, have a play.

Before you can actually use this you must first setup slack integration as documented here which requires you to have a slack account.

The fruits of our labor:

All the script code is given in my gist here.


Thanks to my pair Ruben for helping on this, good work.

Tuesday, 3 November 2015

Developer podcasts v2

A couple of years ago i wrote a blog post about podcasts for developers, this is a follow up as I've now got substantially more. That and a couple of my colleges have asked for my list recently.


.NET Rocks! : Feed Url http://www.pwop.com/feed.aspx?show=dotnetrocks&filetype=master
Adventures in Angular : Feed Url http://feeds.feedwrench.com/AdventuresInAngular.rss
All Chariot Podcasts : Feed Url http://chariotsolutions.com/podcasts/show/all-shows/feed/
All Things Pivotal : Feed Url http://pivotalsoftwarepodcast.libsyn.com/rss
Azure Friday - Channel 9 : Feed Url https://channel9.msdn.com/Shows/Azure-Friday/RSS
CodeChat (Audio) - Channel 9 : Feed Url https://channel9.msdn.com/Shows/codechat/feed/mp3
Coding Blocks | Software and Web Programming / Security / Best Practices / Microsoft .NET : Feed Url http://www.codingblocks.net/podcast-feed.xml
Debug : Feed Url http://feeds.feedburner.com/debugshow
Devnology Podcast : Feed Url http://feeds.devnology.nl/DevnologyPodcast
DevRadio - Channel 9 : Feed Url https://channel9.msdn.com/Blogs/DevRadio/RSS
Full Stack Radio : Feed Url https://simplecast.fm/podcasts/279/rss
Functional Geekery : Feed Url http://www.functionalgeekery.com/feed/mp3/
Hack && Heckle : Feed Url http://feeds.feedburner.com/HackAndHeckle
Hanselminutes : Feed Url http://feeds.feedburner.com/Hanselminutes
Herding Code : Feed Url http://herdingcode.com/feed
Javascript Jabber : Feed Url http://feeds.feedwrench.com/JavaScriptJabber.rss
Jesse Liberty - Silverlight Geek : Feed Url http://feeds.feedburner.com/JesseLiberty-SilverlightGeek
MS Dev Show : Feed Url http://msdevshow.libsyn.com/rss
NodeUp : Feed Url http://feeds.feedburner.com/NodeUp
PowerScripting Podcast : Feed Url http://feeds.feedburner.com/PowerScripting
Radio TFS : Feed Url http://feeds.feedburner.com/radiotfs
Ruby Rogues : Feed Url http://rubyrogues.com/podcast.rss
RunAs Radio : Feed Url http://www.pwop.com/feed.aspx?show=runasradio&filetype=master
Simple Programmer Podcast : Feed Url http://simpleprogrammer.libsyn.com/rss
Software Engineering Radio - the podcast for professional software developers : Feed Url http://www.se-radio.net/rss
STLTechTalk Podcast : Feed Url http://stltechtalk.libsyn.com/rss
The Azure Podcast : Feed Url http://feeds.feedburner.com/TheAzurePodcast
The Cognicast - Cognitect Blog : Feed Url http://feeds.feedburner.com/cognicast
The Java Posse : Feed Url http://feeds.feedburner.com/javaposse
The Static Void Podcast : Feed Url http://www.staticvoidpodcast.com/feed/podcast/
This Week On Channel 9 (MP4) - Channel 9 : Feed Url http://s.ch9.ms/shows/This+Week+On+Channel+9/feed/ipod
ThoughtWorks : Feed Url http://feeds.soundcloud.com/users/soundcloud:users:94605026/sounds.rss
WebDevRadio : Feed Url http://webdevradio.com/feed/
Windows Weekly (MP3) : Feed Url http://feeds.twit.tv/ww.xml
YAPP: Yet Another Programming Podcast : Feed Url http://yapp.audio/feed/yapp


Arrested DevOps : Feed Url http://feeds.podtrac.com/VGAulpN7MY1U
DevOps Cafe Podcast : Feed Url http://devopscafe.libsyn.com/rss
Devops Mastery : Feed Url http://feeds.soundcloud.com/users/soundcloud:users:79143337/sounds.rss
DevOps.com : Feed Url http://feeds.soundcloud.com/users/soundcloud:users:87329161/sounds.rss
Ops All The Things! : Feed Url http://opsallthethings.s3-website-us-east-1.amazonaws.com/podcast.xml
The Food Fight Show : Feed Url http://feeds.feedburner.com/TheFoodFightShow
The Ship Show : Feed Url http://theshipshow.com/podcast.xml

Developer related

Developer On Fire : Feed Url http://feeds.feedburner.com/developeronfire
Get up and CODE! : Feed Url http://feeds.feedblitz.com/getupandcode
Mastering Business Analysis : Feed Url http://masteringbusinessanalysis.com/feed/podcast
Programmer Vs World : Feed Url https://programmervsworld.wordpress.com/category/the-podcast/feed/
Startups For the Rest of Us » Episodes : Feed Url http://www.startupsfortherestofus.com/category/episodes/feed
The Security Influencer's Channel : Feed Url http://contrastsecurity.libsyn.com//rss


Agile Instructor - Coaching for Agile Methodologies such as Scrum and Kanban : Feed Url http://feeds.feedburner.com/AllThingsAgile
Agile NYC : Feed Url http://feeds.feedburner.com/AgileFM
Agile Weekly Podcast : Feed Url http://integrumtech.com/feed/podcast/
The Agile Coffee Podcast : Feed Url http://agilecoffee.com/feed/podcast/
This Agile Life : Feed Url http://feeds.feedburner.com/thisagilelife/podcast

Non tech

60-Second Mind : Feed Url http://www.scientificamerican.com/podcast/sciam_podcast_i_psych.xml
99% Invisible : Feed Url http://feeds.99percentinvisible.org/99percentinvisible
All items | LSE Public lectures and events | Audio : Feed Url http://www.lse.ac.uk/assets/richmedia/webFeeds/publicLecturesAndEvents_iTunesRssAudioOnlyAllitems.xml
Freakonomics Radio : Feed Url http://feeds.feedburner.com/freakonomicsradio
Friday Night Comedy from BBC Radio 4 : Feed Url http://www.bbc.co.uk/programmes/p02pc9pj/episodes/downloads.rss
Haute Couture Podcast - Claudia Cazacu : Feed Url http://claudiacazacu.podomatic.com/rss2.xml
Monstercat Podcast : Feed Url https://www.monstercat.com/podcast/feed.xml
NPR: Invisibilia Podcast : Feed Url http://www.npr.org/rss/podcast.php?id=510307
NPR: TED Radio Hour Podcast : Feed Url http://www.npr.org/rss/podcast.php?id=510298
Planet Money : NPR : Feed Url http://www.npr.org/templates/rss/podlayer.php?id=93559255
Radiolab from WNYC : Feed Url http://feeds.wnyc.org/radiolab
RI Blog : Feed Url http://www.rigb.org/blog.ajax
TEDTalks (audio) : Feed Url http://feeds2.feedburner.com/tedtalks_audio
TEDTalks (video) : Feed Url http://feeds.feedburner.com/tedtalks_videos


Bear in mind that some of these podcasts are no longer active. I've kept them in my list because i find past episodes very relevant to the here and now. you can search back to find relevent episodes on anything you care to think of, super useful.

Incidentally my current podcast player of choice is podcastaddict, variable speed playback and great control of how to download files, brilliant search functionality for local podcasts and its very easy to search for and add new podcasts. use it :-)

My opml file extracted from podcast addict is located here: https://gist.github.com/DamianStanger/9606fcf3fb09cc0bd87f

Tuesday, 16 June 2015

Advanced filtering and navigation on Thoughtworks Go Cd with tamper monkey

Our problem

We use Go from Thoughtworks to manage the build and deployment of all our software. Attached below is a screen shot of the current pipelines.

As you can see we have quite a lot going on. In fact we have 390 pipelines (55 services and api builds, 40 nuget package build/publish, and the rest are deployments to our 5 environments from test through preprod and live). There are 730 stages in total (Build, test, deploy, ect). And we have 20 go agents running on different servers.

So you can imagine finding the pipeline you are looking for is tough. We have adopted naming conventions which help but its really quite difficult to find what you are after with the supplied search capabilities.

Also its very tricky to see if anything is broken (red), there is just no way you will notice it in all the pipelines.

As an aside we are using cradiator to give us an information radiator of all our build and deployment pipelines (I blogged about this a year ago) but with over 700 stages its really in need of an overhaul but that's for another blog post.

The solution

We have created a tamper monkey script (found here) that enhances the functionality of the Go pipelines view, allowing you to filter the visible pipelines by status or by keyword.

There is also the issue of navigating between the settings and the pipeline history, and visa versa. Often you are investigating a failure, you find the problem in the history and then want to change the settings. Well there is no link, so we created one for you to easily navigate between the 2 parts of the UI easily.

Notice the "Settings" link above and the "History" link below both in the header

The script can be found here: https://gist.github.com/DamianStanger/d1db873e9297b34160ce#file-go-cd-tampermoneyscript-js


Firstly you need to get tamper monkey installed in your browser, you can get it from the chrome web store or http://tampermonkey.net/

Then from the tamper monkey dashboard add the script and tell it what urls to attach itself to. The update url needs to be your instance of Go (our update URL is https://goserver:8154/go/* ). On the settings tab I also add in a couple of user includes to https://goserver:8154/go/home and https://goserver:8154/go/pipelines.

That's it, instantly enjoy better productivity. I imagine this is a good enhancement even if you only have a tenth of the pipelines we have.

Wednesday, 15 October 2014

Blue green web deployment with powershell and IIS

I wanted to follow up my earlier post (about our current CD process) with a more technically focussed one, one that can describe the nuts and bolts of the actual BlueGreenDeployment.


Powershell, powershell and powershell, oh and windows, IIS and Go (build server)


As I described in my earlier post the blue green web deploy consists of these steps:

1. Deploy
1.1 Fetch artifact
1.2 Select the config for this deployment
1.3 Delete the other configs
1.4 Deploy to staging (delete then copy)
1.5 Backup live

2. Switch blue green
2.2 Point live to new code
2.3 Point staging to old code

Blue Green Deployment

Before diving in to the details I should firstly convey what blue green deployments are, and what they are not.
There are a few different ways to implement blue green deployments but they all have the same goals:
1. Allow testing on live without actually being live.
2. Enable deployments to have the smallest possible impact on the live service as possible.
3. Give you an easy roll-back path.

This can be accomplished in many ways. Techniques include DNS switching, directory moving, or virtual path redirecting. 
We have chosen to do IIS physical path redirecting. This allows us to do the same technique on all our environments from test to live, same scripts, same code, and doesn't cost as much as requiring multiple servers which DNS switching would require.

Commands used for this demo are

PS> .\Create-Websites.ps1 -topLevelDomain co.uk
PS> Deploy-Staging -source c:\tmp -websiteName foobarapi -domainName foobar.co.uk
PS> Backup-Live -WebsiteName foobarapi -DomainName foobar.co.uk
PS> Switch-BlueGreen -WebsiteName foobarapi -DomainName foobar.co.uk

The code I'm going to talk through is all located here: https://github.com/DamianStanger/Powershell

Conventions used:

All websites are named name.domain and name-staging.domain
All backing folders are in c:\virtual and are named name.domain.green and name.domain.blue
You don't know if blue or green is currently serving live traffic.
Backups are taken to c:\virtual-backups\name.domain
Log files always live in c:\logs\name.domain
There is always a version.txt and bluegreen.txt in the root of every website/api

In this example I'm using name=foobarapi and domain=foobar.co.uk

The technical detail

This is the meaty stuff, it consists mainly of powershell, and should work no matter what CI software you are using. I can heartily recommend Go by Thoughtworks. It has a built in artifact repository and brilliant dependency tracking through its value stream map functionality.

Setup IIS and backing folders

To test my deployment scripts you will firstly need to set up the dummy/test folders and IIS websites. For this you can use this script: Create-Websites.ps1. I'm not going to go into detail of the script as its not the focus of this post but it creates your app pool and website.

The code is exercised with the following:
setupWebsite "foobarui" "foobarui-test" $true "green"
applyCert("*.foobar.*") <<optional if you want the sites to have an ssl cert applying>>

This will create 2 websites on IIS pointing to the green and blue folders as per the conventions outlined further above. Finally apply an SSL certification using powershell, this command will apply the SSL cert to all the websites in this instance of IIS.
To remove the created items from IIS issue commands similar to this:
PS> dir IIS:\AppPools | where-object{$_.Name -like "*.foobar.co*"} | Remove-Item
PS> dir IIS:\Sites | where-object{$_.Name -like "*.foobar.co*"} | remove-item
PS> dir IIS:\SslBindings | remove-item

Once you have the websites correctly set up you can then utilise the deploy blue green scripts :-)


The Blue Green deployment module is located here: BlueGreenDeployment.psm1 and will need importing into your powershell session with the following command:
PS> Import-module BlueGreenDeployment.psm1
Once you have the module imported you can issue the following commands:
PS> Deploy-Staging -source c:\tmp -websiteName foobarapi -domainName foobar.co.uk
PS> Backup-Live -WebsiteName foobarapi -DomainName foobar.co.uk
PS> Switch-BlueGreen -WebsiteName foobarapi -DomainName foobar.co.uk

Lets dig into these one by one.

1. Deploy-Staging
This is quite straight forward. Find the folder that is currently serving staging and copy the new version there. The interesting bit of code is the method of determining which folder to replace with the new version. IsLiveOnBlue and GetPhysicalPath work together to determine the folder in use on staging. Notice the retries inside GetPhysicalPath I found that sometimes IIS just doesn't want to play, but if you ask it a second time it will?? Don't ask..
The code that actually determines the physical path is:
$website = "IIS:\Sites\$WebsiteName.$domainName"
$websiteProperties = Get-ItemProperty $website
$physicalPath = $websiteProperties.PhysicalPath

The rest of the powershell is relatively straight forward

2. Backup-Live
Backing up live is again pretty standard powershell. Again determine the folder that is serving live then do a copy. Done.

3. Switch-BlueGreen
Performing the switch is actually really easy when it comes to it. Firstly determine which folder (blue or green) is serving live (same code as the deploy step) and then switch it with the staging website.
Set-ItemProperty $liveSite -Name physicalPath -Value $greenWebsitePath -ErrorAction Stop
The only added complication is the rewriting of the log file location in the web.config. Log4net only really works well if one process (web site) uses one log file. Again you can look this up yourselves as this is an aside to the main purpose of this post.


The interwebs in general are full of articles/opinions/tales of how bad windows is to automate, it actually winds me up. Maybe it used to be true but I've been finding that with powershell and Go I've been able to automate anything I need. It's so powerful. Don't let the microsoft haters stop you from doing what needs to be done.

The blue green deployment technique outlined here is working really well for us at the moment and has helped us to take our projects live sooner/quicker and with more confidence.

Automation for the win.

Sunday, 12 October 2014

Adventures in continuous delivery, our build/deployment pipeline


We have been undergoing a bit of a dev-ops revolution at work. We have been on a mission to automate everything, well as much is as possible. Exciting times but we are still only just setting out on this adventure, I wanted to document where we are currently at.

First a brief overview of what we have. We have many many small windows services, websites and apis each belonging to a service and performing a specific role. I must quickly add we are a microsoft shop. More and more are our services moving towards a proper service orientated architecture. I hesitate to use the term micro services as it's so hard to pin a definition on the term but let's just say they are quite small, focused on a single responsibility.

We have 5 or 6 SPA apps mainly written with durandal and angular. 7 or 8 different APIs serving data to these apps and to external parties. 10 to 15 windows services which mostly publish and subscribe to N service bus queues.

We currently have 8 environments that we need to deploy to (going to be difficult to do this by hand, me thinks) including CI, QA*, Test*, Pre-prod* and live* (* the last 4 are doubled as we deploy into 2 different regions which both operate slightly differently and have different config and testing). This list is growing with every month that passes. We really really needed some automation, when it was just 3 environments in the UK region we just about got by with manual deployments.

I'm going to outline how the build pipeline integrates with the deployment pipelines and the steps that we take in each stage. But I'm not really going to concentrate on the actual technical details, this is more of a process document. 

1.0 The build pipeline

We operate on a trunk based development model (most of the time) and every time you check in we run a build that will produce a build artifact, push that in to an artifact repository and then run unit and integration tests on the artifact. 

Fig 1. The build pipeline


1. Run a transform on the assembly info so that the resultant dll has build information inside the details. This aids us determine what version of a service is running on any environment, just look at the dlls properties.
2. Create a version.txt file that lives in the root of the service. This is easily looked at on an API or website as well as in the folder containing a service.
3. We check in all the versions of the config files for all the environments that we will be deploying to and use a transform to replace the specific parts of a common config file with environment specific details (e.g connection strings). Every environment's config is now part of the built artifact.
4. Build the solution, usually with msbuild, or for the SPA apps, gulp
5. If all this is successful upload the built artifact to the artifact repo (the go server)


6. Fetch the built artifact
7. Run unit tests
8. Run integration tests

The test stage is separate so that we can run tests on a different machine if necessary. It also allows us to parallelise the tests running them on many machines at once if required.

Not shown on this diagram are the acceptance tests, these are run in another pipeline. Firstly we need to do a web deploy (as below) then setup some data in different databases and finally run the tests.

2.0 The web deploy pipeline

So far so good, everything is automated on the success of the previous stage. We then have the deployment pipelines of which only the one to CI is fully automated so that acceptance tests can be run on the fully deployed code. All the other environments are push button deploys using Go.
The deployment of all our websites/APIs/SPAs are very similar to each other and the same across all the environments so we have confidence that it will work when finally run against live.

Fig 2. The web deploy pipeline


1. Fetch the build artifact
2. Select the desired config for this environment and discard the rest so there is no confusion later
3. Deploy to staging (I've written a separate article on this detailing how it works with IIS powershell and windows)
a. Delete the contents of the staging websites physical path
b. Copy the new code and config into the staging path

Switch blue green

We are using the BueGreenDeployment model for our deployments. Basically you deploy to a staging environment then when you are happy with any manual testing you switch it over to live with the use of powershell to switch the physical folders in IIS of staging and live. This gives a quick and easy rollback (just switch again) and minimises any down time for the website in question.

3.0 The service deployment pipeline

Much the same as the deployment of websites except for the fact the there is no blue green. The Services mainly read from queues and so this makes it difficult to run a staging version at the same time as a live version (not impossible but a bit advanced for us at the moment)

Fig 3. The service deploy pipeline


The install step again utilises powershell heavily, firstly to stop the services, then back things up and deploy the new code before starting the service up again.

There is no blue green style of rollback here as there are complications to doing this with windows services and with reading off the production queues. There is probably room for improvement here but we should be confident that things work by the time we deploy live as we have proved it out in 2 or 3 environments before live.


I'm really impressed with Go as our CI/CD platform it gives some great tooling around the value stream map, promotion of builds to the other environments, pipeline templates and flexibility. We haven't just arrived at this setup of course, its been an evolution which we are still undergoing. But we are in a great position moving forward as we need to stand up more and more environments both on prem and in the cloud.

Fig 4. The whole deployment pipeline

Room for improvement

There is plenty of room for improvement in all of this though

* Config checked into source control and built into the artifact
Checking the config into the code base is great for our current team, we all know where the config is, its easy to change or add new things to it. But for a larger team or where we didn't want the entire team to know secret connection string to live DBs it wouldn't work. Thank goodness we don't have any paranoid DBAs here. Also there is a problem if we want to tweak some config in an environment. we need to produce an entire new build artifact from source code, which might now have other changes in it that we don't want to go live. We can handle this using feature toggles and a branch by abstraction mode of working but it requires good discipline which we as a team are only just getting our heads around. Basically if the code is always in a releasable state this is not an issue.

* Staging and live both have the same config
When you do blue green deployments as we are doing, both staging and live always point to the live resources and databases, so it's hard to test that the new UI in staging works with the new API, also in staging as both the staging and current live UI will be pointing to the live API. Likewise the live and staging API will both be pointing to the live DB or other resources. Blue green deployments are not designed for integration testing like this, that's what the lower environments are for.
On a very similar vein, logging will go to the same log files which can be a problem if your logging framework takes out locks on files, we use log4net a lot which does. There are options to work in a lock when required mode with log4net but it can really hit performance. We have solved this by rewriting the path to the log file on blue green switch.

* No blue green style deployments of windows services
The lack of blue green deployment of services means that we have a longer period of disruption when deploying and a slower rollback strategy. Added to this you can't test the service on the production server before you actually put it live. There are options here but it gets quite complicated to do, and by the time the service is going live you should have finished all your testing by now.

* Database upgrades are not part of deployment
At the time of writing we are still doing database deployments by hand, this is slowly changing and some of our DBs do now have automated deployments, mainly using the redgate SQL tool set, but we are still getting better at this. It's my hope that we will get to the fully automated deployments of data schemas at some point, but we are still concentrating on the deployment of the code base

* Snowflake servers
All our servers both on prem and in the cloud are built, installed and configured manually. I've started to use chocolaty and powershell to automate what I can around set-up and configuration, but the fact still remains that its a manual process to get a new server up and running. The consequence of this is that each server has small differences to other servers that "should" be the same. This means that we could introduce bugs in different environments due to accidental differences in the server itself.

* Ability to spin up environments set up as needed for further growth 
Related to the above point, as a way to move away from the problem of snowflake servers we need to look at technologies like puppet, chef, Desired state configuration etc. If we had this automation we could spin up test servers, deploy to other regions/markets, or scale up the architecture by creating more machines. 

Relevant Technology Stack (for this article)

• Windows
• Powershell
• SVN and Git
• Msbuild and gulp

Next >>

Ive written a follow up article to this which details the nuts and bolts of the blue green deployment techniques we are currently using. blue-green-web-deployment-with-IIS-and-powershell.
The code for which can be found on my git hub here: https://github.com/DamianStanger/Powershell/

Sunday, 31 August 2014

Using Powershell to create local users on windows

We are setting up a server farm for a new environment consisting on many servers and we want to create many users with admin rights on each one, including the remote desktop user group.

We could have spent an hour or so and used the GUI on each server but we thought that a script would be quicker, not to mention more fun to write.

The latest version is here: https://github.com/DamianStanger/Powershell/blob/master/Add-LocalAdminUserAccount

The version at time of writing is below:

Function Add-LocalUserAdminAccount{
  param (

  foreach ($computer in $ComputerNames){
    foreach ($userName in $UserNames){
      Write-Host "setting up user $userName on $computer"

      $user.Put("Description","Scripted admin user for $userName")

      $flag=$User.UserFlags.value -bor 0x10000


      [ADSI]$group = “WinNT://$computer/Administrators,group”
      write-host "Adding" $user.path "to " $group.path

      [ADSI]$group = “WinNT://$computer/Remote Desktop Users,group”
      write-host "Adding" $user.path "to " $group.path

[string[]]$computerNames = "computer1", "computer2"
[string[]]$accountNames = "ops", "buildagent"

Add-LocalUserAccount -ComputerNames $computerNames -UserNames $accountNames -Password mysecurepassword

The lines that do the damage are 14 to 24 to create and save the user, then 26 to 32 to add the user to the required groups on the machine.

It would be trivial to change this script so it was a powershell module but the script as it stands serves my current needs. Just add more computer names and account names to suit your needs we have around 10 of each in the version of the scripts I'm running.