Bohdan Szymanik

IronPython and Numpy/Scipy

2011-10-20T08:50:00.000+13:00

Should I ever forget this in the future...

If you're trying to install numpy/scipy on ironpython and like me you're behind a corporate firewall, and like me you can't get proxying to work, and like me you can't figure ironpkg out, then don't give up. Download the eggs locally and use ironegg eg: ironegg nose-1.0.0-1.zip

Launching the Black Pearl!

2011-05-23T12:22:00.001+12:00

It’s not quite finished – I need to give a final rub down and coat of epoxy, and put the deck lines on, but with great weather this weekend I just had to launch it.

Couple of building notes:

I used thick minicell foam for the bulkheads. Expensive stuff in NZ but I got a slab of it while on a work trip to the US. I figure it makes for light bulkheads that will allow the body of the boat to flex.
The foot pegs are plastic – screwed into tee nuts embedded in a strip on each side. Not sure how adjustable these are going to be – with the small cockpit I can’t seem to reach in to change the settings!

I used Maroske’s internal tube fittings but the longest of these just wouldn’t let me extract the PVC tube… no problem, just slice em in half! They’re very easy to re-glue.

I doubled the high density fibreglass over 2/3rds of the length for the hull and deck interiors with an extra overlapping strip just behind the cockpit. Feels very robust while getting into/out of the cockpit.

What’s was it like to use?

Much lighter build than the Night Heron – on the NH I used 200g cloth doubled over most of the interior. The smaller surface area of the BP plus the lighter building materials (mostly Paulownia strips) means it’s about 14kgs. This makes it much easier to carry and a bit more responsive to my body movement on the water.
Very tippy! But secondary stability seems good – I just need to get used to the experience. Very different from anything I’ve been in so far and I’ll take my time to practice rolls before I get into rougher water.
Very responsive to my movements and to the water.
Lots of fun –which is what I wanted.
I was slightly concerned about the ocean cockpit but in practice it’s easy for me to get into and out of so no problem there.
Legs are quite straight. Again was a bit worried I’d be too uncomfortable, but actually for me – it’s fine.

So frankly, I’m utterly delighted!

Cockpit coaming step 1: Wood strips

2011-04-22T20:15:00.001+12:00

I’m building the cockpit coaming by first building a wood strip riser. I'll fibreglass it into place then put foam around the outside and use that as a form to create the coaming lip – at least that’s the plan.

First step is building the riser. To do this I followed Nick Schade’s instructional video and it’s seemed a pretty quick and easy exercise.

So far, the riser is in place, a fillet is on and when it dries I’ll wet sand and follow up with 2 layers of fibreglass.

It’s an ocean cockpit so fairly short and only 38cm interior width.

Current pic (plastic to protect the interior):

Black Pearl Update–Deck Fibreglassed

2011-04-16T18:24:00.001+12:00

The deck’s been glued, sanded and fibreglassed – at least on the outside.

Mostly Paulownia but 3 strips on each side of Cedar as I was running a bit low of the lighter wood. I’ve decided to leave the deck natural colour so that I can forever see how it was built. The cockpit’s been cut out but I’m leaving the hatches till after the fibreglass sets up. It seemed a little flimsy to be cutting too many holes in the deck.

Not many staple holes, it was easy to glue most strips in place without staples.

I kept the surface to an even height and used the odd bit of chopped up credit card under a couple of thinner strips.

Quick pic: of the cockpit:

Wood Density

2011-03-15T17:51:00.001+13:00

My fascination with the weight of the kayak I’m building is growing with my surprise at just how light in weight the hull is. I thought I’d do a quick check on a couple of samples of Paulownia and Cedar.

For the wood I’ve bought it works out that Cedar is about 0.40g/cm³ (about 17 lbs/ft³), and the Paulownia is about 0.27g/cm³ (about 25lbs/ft³). So the Paulownia is 2/3rds the density of the Cedar!

Now that has me thinking – I’m mostly building the deck in Cedar – the extra wood weight plus the coming/deck deckplates etc – hmmm, will it be tippy without me in it? I reckon from looking at it that the deck is about 2/3rds the size of the hull so it might will be balanced at about the same weight top half and bottom half!

Black Pearl Hull Completed

2011-03-14T14:55:00.001+13:00

Some quick notes on this.

Firstly, must remember to take care during strip gluing to get the glue all the way through – I didn’t and I found that after fibreglassing the outside and removing the forms… the bottom of the hull started to bend out in a couple of places. Remedy… add epoxy on the inside and glue the strips up properly and weigh down with a few clay bricks (sitting on plastic of course).

Secondly, I tried to reinforce the interior chines with a triangular strip of wood height about 5mm and base about 10mm. What a waste of time! Quick calculation after the fact shows that using wood came in at about 100 grams, but I’ve just checked the density of thickened epoxy – I’m getting about 10 to 20% lighter than straight epoxy – and if I’d just used a thickened epoxy bead it would’ve weighed about 200g. The extra 100g would have been worth not having to cut triangular strips!

And finally, I hate fibreglassing the interior – much more difficult than the outside. Fortunately, relatively easy to fix up any ugly bits a day or two later – in my case 3 patches that needed sanding back and redoing along the chines.

Also, I’ve added up the costs and just out of interest it’s looking like about $NZD1100 total for this one – that’s plans, fibreglass, epoxy, tints/dyes/stains/deck ports and loads of sandpaper. If I’d made some smarter choices along the way I certainly could’ve brought the cost down below $1000.

I’ll do a more thorough break down at the end.

And the weight! Before I forget, the hull with one layer of 175g high density fibreglass on the inside, one on the outside and an extra strip on the keel, it totals about 5kgs on my scales. Incredible really! The Paulownia wood is very light and the high density fibreglass certainly takes much less epoxy than the 200g normal fibreglass I used on the last kayak.

Multiple Monitoring Performance Data Collections

2011-02-21T23:16:00.001+13:00

Ah yes – that code from the last couple of posts has a bug…

The retrieval of data from SCOM has proven very useful in the last few days but it seems most commonly I’m getting multiple monitoring performance data collections - so the code I wrote in the last couple of posts on how to retrieve that data using F# and Sho needs a little adjusting to concatenate the collections into one sequence – yield! works great in F# eg

let perfData (start:DateTime) (finish:DateTime) : seq<DateTime * float> = 
    seq {
        for mpdsItem in mpds do
        yield! (mpdsItem.GetValues(start, finish)
                    |> Seq.map (fun(mpdv) -> (mpdv.TimeAdded, mpdv.SampleValue))
                    |> Seq.filter ( fun(d,v) -> v.HasValue)
                    |> Seq.map ( fun(d,v) -> (d,box v) )
                    |> Seq.map ( fun(d,v) -> (d, unbox v) )
            )
        }

Not sure about python but it must be similar.

SCOM data into F#

2011-02-14T18:45:00.001+13:00

Having just discovered how to get SCOM data from Microsoft Sho - a dynamic language analysis environment - I thought I’d try from F#. Pretty much the same but you need to think about what to do with the graphing, and you need to handle the Nullable<float> values coming back from SCOM.

I used fschart to do the plotting and a sequence filter and boxing/unboxing to handle the Nullable data.

The histogram function wouldn’t be hard to make but while stumbling across fschart I also stumbled across this.

So my contribution was do this:

#r @"System.Windows.Forms.DataVisualization.dll"
#r @"C:\extras\FSChart10\FSChart\bin\debug\FSChart.dll"

open FSChart

open System.IO
open System.Drawing
open System.Windows.Forms
open System.Windows.Forms.DataVisualization.Charting

#r "c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll"
open Microsoft.EnterpriseManagement.Monitoring

let mg = Microsoft.EnterpriseManagement.ManagementGroup("someManagementServer")
let mpdc = MonitoringPerformanceDataCriteria("ObjectName = 'ASP.NET' and CounterName like 'Request Execution%' and MonitoringObjectPath like 'someMonitoringObjectPathFilter%'")
let mpds = mg.GetMonitoringPerformanceData(mpdc)

open System

let (perfData:seq<DateTime * float>) = 
    mpds.[0].GetValues(DateTime.Today.AddDays(-1.), DateTime.Today)
    |> Seq.map (fun(mpdv) -> (mpdv.TimeAdded, mpdv.SampleValue))
    |> Seq.filter ( fun(d,v) -> v.HasValue)
    |> Seq.map ( fun(d,v) -> (d,box v) )
    |> Seq.map ( fun(d,v) -> (d, unbox v) )


hist 0.0 200. 50 (perfData |> Seq.map (fun (d,v) -> v) )

The result looks like this.

So the question is… what was the more efficient approach? Well when I did the sho/python code I actually had some c# code open in the background to help me through the classes - without that it would have been really hard. The F# approach made me scratch my head a few times wondering how to deal with charting and nullables but all the way through (much like the c#) I had the advantage of the richer type information to help me figure out what to do.

SCOM and Microsoft Sho–A better histogram example…

2011-02-14T09:29:00.001+13:00

The last blog post had a naff histogram as a few exceptionally long queries squashed the remainder of the data in the first bin, also I wasn’t checking for None so here’s a better, filtered example.

ShoLoadAssembly("c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll")
ShoLoadAssembly("c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll")

from Microsoft.EnterpriseManagement.Monitoring import *
from System import *

mg = Microsoft.EnterpriseManagement.ManagementGroup("someManagementServer")

mpdc = MonitoringPerformanceDataCriteria("ObjectName = 'ASP.NET' and CounterName like 'Request Execution%' and MonitoringObjectPath like 'someMonitoringPathFilter%'")

mpds = mg.GetMonitoringPerformanceData(mpdc)
mpds.Count

hist([mpdv.SampleValue for mpdv in mpds[0].GetValues(DateTime.Today.AddDays(-1), DateTime.Today) if ((mpdv.SampleValue < 200) and (mpdv.SampleValue is not None))])

Retrieving Performance Data from System Center Operations Manager

2011-02-11T22:59:00.001+13:00

System Center Operations Manager’s agent model means it can collect performance metrics from across a diverse technology environment. You use Rules to define the data collection and typically you use the Console to view the data in graphical format.

But, the Console by itself doesn’t allow any numerical analysis. If you want to do that you need to get the data out. You can copy to the clipboard within the Console Action menu but that’s very manual. A better approach is obviously programmatic access and this is where things get interesting.

For some reason there’s little information on the net on how you get data out of SCOM. What is there usuall refers to querying the SCOM database directly. This doesn’t seem right to me. I’d far prefer an API that stood a chance of staying intact in future versions. So here’s what I’ve done.

Firstly, use the Powershell integration to SCOM to get the data. (This is easiest if you setup the Powershell ISE to connect to SCOM.) You can then use the get-performancecounter and get-performancecountervalue cmdlets to retrieve data, for example:

$starttime = [datetime]::today.adddays(-20)
$endtime = [datetime]::today.adddays(1)

get-performancecounter |
    ? {$_.ObjectName -eq 'Web Service' -and
       $_.CounterName -eq 'Connection Attempts/sec' -and
       $_.InstanceName -eq '_Total' -and
       $_.MonitoringObjectPath -like 'usuallyAServerNameFilter*' } |
    get-performancecountervalue -starttime $starttime -endtime $endtime | 
    select SampleValue, TimeSampled | 
    export-csv "c:\temp\ConnectionAttemptsPerSec.csv"

You can also use the .Net framework and languages to get the data – the documentation isn’t great but this example works (I’ve reduced the font size to try and make it fit in this blog template):

/// <summary> 
/// Gather performance data
/// </summary>
using System;
using System.Collections.ObjectModel;
using Microsoft.EnterpriseManagement;
using Microsoft.EnterpriseManagement.Common;
using Microsoft.EnterpriseManagement.Configuration;
using Microsoft.EnterpriseManagement.Monitoring;

namespace MySamples
{
    class Program
    {
        static void Main(string[] args)
        {
            ManagementGroup mg = new ManagementGroup("someManagementServer");

            MonitoringPerformanceDataCriteria mpdc = 
                new MonitoringPerformanceDataCriteria(@"ObjectName = 'ASP.NET' and CounterName like 'Request Execution%' and MonitoringObjectPath like 'usuallyAServerNameFilter%'");
            ReadOnlyCollection<MonitoringPerformanceData> mpds = 
                mg.GetMonitoringPerformanceData(mpdc);

            if (mpds.Count > 0) {
                ReadOnlyCollection<MonitoringPerformanceDataValue> mpdvs = mpds[0].GetValues(DateTime.Today.AddDays(-1), DateTime.Today);

                foreach (MonitoringPerformanceDataValue mpdv in mpdvs) {
                    Console.WriteLine("TimeSampled = " + mpdv.TimeSampled + 
                        ",  SampleValue = " + mpdv.SampleValue);
                }
            }
            Console.ReadLine();
        }
    }
}

Now I typically take the data out to F# and/or Excel and graph/model it to help figure my way through a problem. However, Microsoft have just released a nicely packaged analysis tool in Microsoft Sho – basically IronPython wrapped up with some plotting/graphing/math libraries – much like many of the other Python packages out there.

I thought this would be a fun way to interactively deal with performance data collection and analysis so I tried the following as an experiment (put in appropriate management server/monitoring object filters for you – I just used ASP.NET request execution times from a test environment as in the programs above).  The hist command takes a list, or I guess anything it can enumerate over and get numbers from, and returns a histogram dialog and plot.

ShoLoadAssembly("c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll")
ShoLoadAssembly("c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll")

import Microsoft.EnterpriseManagement
import Microsoft.EnterpriseManagement.Configuration
from Microsoft.EnterpriseManagement.Monitoring import *
from System import *

mg = Microsoft.EnterpriseManagement.ManagementGroup("aManagementServer")

mpdc = MonitoringPerformanceDataCriteria("ObjectName = 'ASP.NET' and CounterName like 'Request Execution%' and MonitoringObjectPath like 'someMonitoringObjectFilter%'")

mpds = mg.GetMonitoringPerformanceData(mpdc)
mpds.Count

hist([mpdv.SampleValue for mpdv in mpds[0].GetValues(DateTime.Today.AddDays(-1), DateTime.Today)]
)

And here’s the output:

Cool huh!?

Engineering tricks with WolframAlpha

2011-02-09T22:29:00.001+13:00

I’ve found Google great for simple problems like this:

20 pounds per cubic foot in kilograms per cubic meter= 320.36926 kilograms per cubic meter

But wolframalpha also lets you combine unit conversions with calculations:

(2 * 175 grams) per square meter * (2*3 meters squared) + ((20 pounds per cubic foot in kilograms per cubic meter) * 3 square meters * 5 millimeters)

Or just click on this link: http://www.wolframalpha.com/input/?i=(2+*+175+grams)+per+square+meter+*+(2*3+meters+squared)+%2B+((20+pounds+per+cubic+foot+in+kilograms+per+cubic+meter)+*+3+square+meters+*+5+millimeters)

What’s the above tell me? It’s the theoretical weight of the kayak I’m currently building – about 7kg. In reality when hand laying fibreglass you get a much higher weight of epoxy rather than the one to one ratio that’s possible under vacuum application and which I’m using above.

Hull fibreglassing

2011-02-09T22:16:00.001+13:00

It’s taken awhile to get here, but the hull is now fibreglassed.

After struggling with the Resene colorwood stain affecting epoxy seal coats I found that a wash with isopropyl alcohol made for a nice clean bonding surface.
I rediscovered how to apply epoxy, everyone seems to have their own method but for me it’s plastic cards (old atm cards/loyalty cards etc are ideal) and short foam (usually yellow) rollers.
A hot air gun does absolute wonders at getting rid of air bubbles/foam.
I used high density 170g/sq m fibreglass that’s often used for model making. That equates to 5 ounce high density fibreglass which should be pretty strong (easy way to do the conversion… just use google – type “175 grams per square meter in ounces per square yard” in the search box).
Wet sanding within 48 hours is more effectively at smoothing the epoxy finish than waiting for it to set really hard.

Hull ready to be fibreglassed

2011-02-03T08:32:00.001+13:00

Finally completed staining/fairing/sanding the hull. Long process with lots of waiting for epoxy to dry. Here’s a photo of how the hull looks wetted down. The staining in combination with a kind of matching fairing compound does look patchy but I’ll roll on a very thin layer of epoxy to seal the wood before the fibreglass goes on and that will darken it down and make it look a lot more consistent.

Stems added and fairing the hull

2011-01-26T22:32:00.001+13:00

Slow progress on the Black Pearl (yet to give it my own name yet) kayak. I’ve added stems (just the white paulownia so not so tough but shows up nicely against the stained hull), and I’ve been applying a fairing compound made of epoxy, lightweight filler, wood dust etc.

Just waiting 7 days now for it to thoroughly set before I sand back.

Here’s a pic:

Biggest lesson from all this… get the strips flat before staining! Basically the better the stripping the easier the rest will be.

Staining has been interesting… painful but interesting. The Resene Colorwood stain does work but you definitely need only a light rag application and maybe a wash with water after it’s set up. For the fairing I’ve managed to replicate the color using epoxy colours (black, blue and red – I watched them making it at Resene so I knew the constituent colours and proportions – it was on a their computer screen ).

Hard to get a good photo while it sits in the garage but this boat looks real sleek.

Hull Stripping and Attempted Staining

2010-12-27T22:26:00.001+13:00

Stripping was relatively easy and very quick. The wood reminded me of making balsa wood models when I was a kid:)

Main points were:

Make sure strips are even thickness – if I do this again I’ll invest in a planner thicknesser.

I mostly just stapled strips only occasionally using a bead of glue – this had one advantage – towards the end as I looked at the forms and the shape of the hull I realised I had one form on slightly wrong angle – not sure how this happened as I was sure I had it right at the beginning. Nevermind, I unscrewed it and re-adjusted just a tiny bit and suddenly it looked great.
I thought I’d stain the strips before using epoxy to glue them all thogether – unfortunately, Resene interior stain turns out to not be as epoxy compatible as the Resene staff thought it would be. It beaded a bit but wasn’t irrecoverable – I might even put more stain on after I rub it all back in preparation for fibreglassing – but if I do I’ll wash with acetone and use a much thinner application of stain. Check the image below…

Hull looks great but I can’t get a good photo showing it the way I see it in real life. Perhaps for the next posting…

Spatial Notes–Getting Started

2010-12-08T13:08:00.001+13:00

Thought this would be harder than it turned out to be.

Objective: chart some metrics across geographic regions (in my case looking at customer metrics over New Zealand).

To get started I took advantage of SQL Server Spatial. Initially appearances are a bit daunting until you try it… and it’s fairly easy.

Finding the GIS data
- for NZ Statistics NZ publish digital boundaries – versioned by publication year (2007, 2001, 1996 etc: http://www.stats.govt.nz/browse_for_stats/people_and_communities/geographic-areas/download-digital-boundaries.aspx
- The data is available in both ESRI Shapefile format and MapInfo format. You can use shape2sql to load this data into sql server spatial. Unfortunately… it has a bug, it won’t create a table from the shp file inside shape2sql – I had to use SQL Profiler to capture the create table instruction and run it interactively – after that the data load works fine - not sure why.
- The data has a hierarchy of increasingly larger geometrical areas made up of units of the smallest size (the meshblock). There’s Area Units, Urban Areas, Regional Councils etc.
Connecting people to areas
- Either pay for the GeoPAF file from NZ Post – this gives the meshblock for a delivery address and a NZ Map Grid coordinate. (Not so useful NZ Post… NZMG has been superceeded by NZTM which conforms with WGS84.)
- Or, use the freely obtainable streetlink data from Stats NZ: http://www.stats.govt.nz/browse_for_stats/people_and_communities/geographic-areas/streetlink.aspx
Aggregating
- Use SQL Server Analysis Services, or much faster
- Use PowerPivot – for me this allowed me to very quickly aggregate customer metrics from an internal database across areas loaded from the Stats NZ files
Reporting
- It’d be nice if Excel had spatial reporting support, but it doesn’t – big thumbs down Microsoft…
- I used SQL Server Reporting Services and the Map control. Problem is that I need to have the map data in SQL/SSAS. It’d be nice to load from PowerPivot but I can’t do that unless I store the Excel PowerPivot doc in Sharepoint 2010. Jumping through hoops to do this though… Microsoft – just get the spatial support into Excel please!

Strip cutting

2010-12-05T21:12:00.001+13:00

I followed Bjorn’s instructions and cut the strips with a circular saw. The planks were pinned to a folding stand at each end. The paulownia was extremely stable. It didn’t split when I pinned it at the edge of the plank. Not one of the strips split or snapped. In the end it was much easier than I expected. Suspect it would’ve been even easier if I had a decent saw…

Ebonizing Paulownia

2010-12-05T21:01:00.001+13:00

Vinegar and steel wool, left for a few days, filtered and then applied to paulownia strip – thinking of using this on my kayak. Top of the strip has been painted twice.

Black Pearl Forms

2010-11-28T22:07:00.001+13:00

So terribly slow but here’s a snippet of the forms.

Lessons learned:

Level everything first – makes it much easier to align cross sections
Don’t start screwing cross sections to wooden cross bars until they all line up right. Use clamps first.
Drill a hole about 5mm on the center of the waterline and shine a bright light from one end – you should be able to see the light all the way down
I pre-cut the forms for the deck and re-attached them – supposedly to make it easier for later. Not sure it was worth it.

Here’s that light shining through a couple of the holes.

New Kayak Build: Black Pearl

2010-10-05T10:05:00.001+13:00

Two years on from building a Guillemot Kayaks Night Heron stitch and glue I’ve just purchased plans for a Bjorn Thomasson Black Pearl. The Night Heron is a great kayak that’s very quick in the water but I just enjoyed the building side too much so I’ve decided to try building a strip kayak. Rather than go with something closer to the Night Heron I decided to go down a slightly different path:

As light as possible – means small or low volume
Doesn’t need speed – the Night Heron fulfils that requirement for me
Play boat – I’ve taken up canoe polo so I want the most manuverable sea kayak experience I can get – something that will respond to hip/torso movement.

I think the Black Pearl will do this but the proof will be in the pudding. You don’t just order this off the book – the plans are custom printed to your height, arm span and hip measurements.

I’ll update the blog as I go. It will take awhile – no rush as I have an existing kayak already.

Populating a Word Document from a Sharepoint List

2010-10-04T13:47:00.001+13:00

I thought this would be simpler since Sharepoint and Word are so closely related. A mail merge based on list data perhaps? Maybe it’s possible with an office data connection file but I couldn’t figure it out.

Well, it was useful to me so here’s how I did it with Powershell and Word automation. The example here is a list of standards and the output will be a list of those standards formatted on one Word document. The soap call from powershell comes from somewhere on the net with an added soapaction header. The list has been created with folder items so it needs to list the items recursively – this wouldn’t apply to a normal list. You’ll figure it out as you start looking at the xml attributes. Best way to make progress with these sharepoint soap calls is to use Wireshark in combination with Stramit Caml Viewer.

Here we go…

function Execute-SOAPRequest 
( 
        [Xml]    $SOAPRequest,
        [String] $SOAPAction,
        [String] $URL 
) 
{ 
        write-host "Sending SOAP Request To Server: $URL" 
        $soapWebRequest = [System.Net.WebRequest]::Create($URL) 
        $soapWebRequest.Headers.Add("SOAPAction","`"" + $SOAPAction + "`"") 

        $soapWebRequest.ContentType = "text/xml;charset=`"utf-8`"" 
        $soapWebRequest.Accept      = "text/xml" 
        $soapWebRequest.Method      = "POST" 
        
        write-host "Authenticating"
        $soapWebRequest.Credentials = [System.Net.CredentialCache]::DefaultCredentials
        if($soapWebRequest.Proxy -ne $null)     {
            $soapWebRequest.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
        }
    
        write-host "Initiating Send." 
        $requestStream = $soapWebRequest.GetRequestStream() 
        $SOAPRequest.Save($requestStream) 
        $requestStream.Close() 
        
        write-host "Send Complete, Waiting For Response." 
        $resp = $soapWebRequest.GetResponse() 
        $responseStream = $resp.GetResponseStream() 
        $soapReader = [System.IO.StreamReader]($responseStream) 
        $ReturnXml = [Xml] $soapReader.ReadToEnd() 
        $responseStream.Close() 
        
        write-host "Response Received." 

        return $ReturnXml 
} 

function Add-Standard
(
        [String] $Standard,
        [String] $Justification,
        $Doc
)
{
$p = $Doc.Paragraphs.Add()
$p.Range.Text = $Standard
$p.Format.Style = "Heading 1"
$p.Range.InsertParagraphAfter()

$p = $Doc.Paragraphs.Add()
$p.Range.Text = $Justification
$p.Range.InsertParagraphAfter()
}

$d4 = [xml] @"
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" 
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
                xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <soap:Body>
        <GetListItems xmlns="http://schemas.microsoft.com/sharepoint/soap/">
            <listName>{CF6E48BA-650A-489E-83AB-8EF1E545388A}</listName>
            <queryOptions>
                <QueryOptions>
                    <ViewAttributes Scope="Recursive"/>
                    <IncludeMandatoryColumns>False</IncludeMandatoryColumns>
                </QueryOptions>
            </queryOptions>
        </GetListItems>
    </soap:Body>
</soap:Envelope>
"@


# create a document from list data!
$w = new-object -com Word.Application
$w.Visible = $true
$d = $w.Documents.Add()

Execute-SOAPRequest -SOAPRequest $d4 -SOAPAction "http://schemas.microsoft.com/sharepoint/soap/GetListItems" -URL "http://<some sharepoint site>/sites/CTO/_vti_bin/lists.asmx" |
foreach {$_.Envelope.Body.GetListItemsResponse.GetListItemsResult.listitems.data.row |
         foreach {Add-Standard -Standard $_.ows_Standard -Justification $_.ows_Justification  -d $d }
}

$d.SaveAs([ref]"c:\temp\standards.docx")
$d.Close()
$w.Quit()

FST Auckland Conference Presentation

2010-08-15T11:01:00.001+12:00

You can find the presentation I did for the Financial Services Technology conference in Auckland last week here.

Brief synopsis: systems/operational management provides a useful analogue to the problems faced by implementers of business intelligence environments: huge volumes of data streaming in for which you need to provide health indicators and predictions of the future.

In the case of Microsoft’s System Center Operations Manager tool this is done in a manner that is very different from conventional BI efforts that rely on staging large quantities of data, building relational and dimensional models and producing KPIs. The solutions developed for operations management can be usefully applied to BI efforts, particularly when not dealing with financial data.

CIO Conference–Enterprise 2.0 Presentation

2010-08-09T21:21:00.001+12:00

I’ve put the presentation here: http://cid-0e5ff53cd7f11485.office.live.com/browse.aspx/Public/CIO%20Conference%20July%2010

Questions from the audience about my criticism of data focussed BI got me thinking about what might be a better model. I’ve changed the slide deck to cover more of a BI story for presentation at the FST conference in Auckland this Wednesday.

Seq.unfold

2010-08-09T21:12:00.001+12:00

Don’t know why this had escaped me for so long!

let PascalsTriangle =
        let nextLine aLine =
            let partialLine = 
                aLine
                |> Seq.pairwise
                |> Seq.map (fun (x,y) -> x+y)
                |> Seq.toList
            List.append (1::partialLine) [1]
        [1] |> Seq.unfold (fun s -> Some(s, nextLine s))

Compare to: Pascal’s Triangle in F#

Heat/Image Mapping with .Net/F#

2010-07-30T10:15:00.001+12:00

As part of a demonstration to the Wellington .net user group this week I put together a demo showing a heat mapping of the performance of a transactional system as a function of time of day and arrival rate. My objective was to provide a simple eye balling test of whether performance was being impacted more by daytime customer loading or by competing overnight batch processing. I’ve tried a similar task before with Python and MatPlotLib but this time I wanted to see how easy it was to accomplish in .Net/F#.

The answer was ‘very easy’.

The interactive F# environment in VS2010 gives you JIT compiled code so it’s very fast to work with. The data structures/collections available within F# make importing and manipulating data simple – once it’s in memory you spend most of the time using filter, map and fold operations on elements – explicit recursion or iteration is rarely necessary.

I found I started with sequences then moved to arrays as data volumes increased and performance slowed. Array.Parallel.map/mapi seemed to make a significant difference whenever the operation being performed was of order a few milliseconds. It’s pointless parallising when the operation performed is short – the overhead of managing threads must just be too great.

Most of the effort was spent aggregating and binning the data, the actual image prep was trivial. The following assumes you have an array of data to display.

open System.Drawing
open System.Drawing.Imaging
open System.Windows.Forms
// adapted some code here from http://thecodedecanter.wordpress.com/2010/04/30/modelling-the-2d-heat-equation-in-f-using-100-lines-of-code/
let toBitmap (arr:Color[,]) =  
    let image = new Bitmap(arr.GetLength(0),arr.GetLength(1),Imaging.PixelFormat.Format24bppRgb)  
    for i=0 to image.Width-1 do 
        for j=0 to image.Height-1 do 
            image.SetPixel(i, j, (arr.[i,j]))  
        done 
    done 
    image 

let maxInArray2D (ar:'a[,]) =
    seq{ for r in 0..Array2D.length1 ar-1 do
            yield Seq.max (ar.[r..r,*] |> Seq.cast<'a>)
    } |> Seq.max

let intensityMap intensity = Color.FromArgb((int (intensity * 255.0)),0,0) 

let bitmap imageArray =
    let max = imageArray |> maxInArray2D
    imageArray 
    |> Array2D.map (fun f -> f/max)
    |> Array2D.map (fun f -> intensityMap f)
    |> toBitmap

let ShowForm (f : Form) =
#if INTERACTIVE
    f.Show()
#endif
#if COMPILED
    Application.Run(f)
#endif 

let BitmapForm bitmap = 
    let picBox = new PictureBox(BorderStyle = BorderStyle.Fixed3D, Image = bitmap, Size = bitmap.Size, 
                                Dock = DockStyle.Fill, SizeMode = PictureBoxSizeMode.StretchImage)
    let form = new Form(Text = "F# Connection Performance versus Connection Rate and Time", Size = bitmap.Size, AutoSize = true)
    form.Controls.Add(picBox)
    form

bitmap surfaceArray |> BitmapForm |> ShowForm

If you don’t have many points that’s going to generate a small image which you can expand by dragging the corner and Windows will magically interpolate for you. I’m sure there’s a way to explicitly do this with the BitmapForm API as well – just didn’t have time to figure that out.

If you want to interpolate yourself, you could try something simple like a bilinear approach:

let bilinearInterpolation f00 f01 f10 f11 x y =
    let boundingBox = Array2D.zeroCreate<float> 2 2     
    boundingBox.[0,0] <- f00
    boundingBox.[0,1] <- f01
    boundingBox.[1,0] <- f10
    boundingBox.[1,1] <- f11
    let A = RowVector.ofList [1.-x; x]
    let B = Matrix.ofArray2D boundingBox
    let C = Vector.ofList [1.-y; y]
    A * B * C

Or why not just double the resolution?

// take an array m x n and return (2m-1) x (2n-1)
let interpolate (A:float[,]) =
    let B = Array2D.zeroCreate<float> (2*(A.GetLength 0) - 1) (2*(A.GetLength 1) - 1)
    for row in 0..((A.GetLength 0) - 1) do
        for col in 0..((A.GetLength 1) - 1) do
            B.[2*row, 2*col] <- A.[row,col]
        done
    done
    // 
    for row in 0..((B.GetLength 0) - 1) do
        if row % 2 = 0 then
            for col in 0..((B.GetLength 1) - 1) do
                if col % 2 = 1 then
                    B.[row, col] <- (B.[row,col-1] + B.[row,col+1]) / 2.
            done
        else
            for col in 0..((B.GetLength 1) - 1) do
                if col % 2 = 0 then
                    B.[row, col] <- (B.[row-1,col] + B.[row+1,col]) / 2.
                else
                    B.[row, col] <- (B.[row-1,col-1] + B.[row+1,col-1] + B.[row-1,col+1] + B.[row+1,col+1]) / 4.
            done
    done
    B
 
// now I can do this...
bitmap (surfaceArray |> interpolate |> interpolate |> interpolate |> interpolate)|> BitmapForm |> ShowForm

And generate results like this (where connection rate increases down, time goes across, and redness indicates duration of calls):

F# for Analysis: Wellington .Net User Group Meeting Wed 28th July

2010-07-26T16:33:00.001+12:00

Some brief notes for anyone attending this session.

There’s a brief slide deck here.

There’s sample code working through the basics of the language here.

Sample code for the cross correlation of two time series in F# here.

I’ll put a separate post together for image generation of a heat map showing web performance data.

System Center Operations Manager and BI

2010-07-26T12:17:00.001+12:00

Last week as part of a presentation to the CIO summit in Auckland I had a bit of fun criticising conventional corporate BI efforts as being too focussed on data management. Basically, the argument went that companies spend too much of the budget worrying about getting the data right and not about enabling people to use it – and that ‘easy to use’ BI front ends are a pointless exercise. It’s not a hard argument to make – you can expend a great deal of time and effort on staging, transforming, cubing, KPI development, and reconciliation quality checks – usually driven by the fact that corporate BI is focussed on financial measures which drives people to want the data correct to the cent.

The natural question that comes back in response is what should be done instead?

Operations management using a model driven tool like System Center Operations Manager provides some useful pointers.

Think about it – what’s operations management?

= Data + Model + Health logic + Alerting for response

What’s BI?

= Data + Model + Business Health Logic + “Strategy execution” for response

What’s smart about model driven operations management?

The tools (like SCOM) monitor systems that can produce huge volumes of data; they give that data context with hierarchical models (à la cubes) that can even be graphical (LiveMaps/Visio 2010 diagrams); they provide health indicators (cf KPIs) and they empower end users by making the information available for more detailed analysis at the desktop.

So how would you put this kind of model into action for BI?

How about mirrored true to source data sources using cheap databases (maybe SQL Express/MySQL). You’d need a grooming schedule to stay on top of the data volumes – perhaps and agent model just like the operations management tools have.

To collect the data from multiple sources, and give some context a cube model is attractive, but what about implementing with PowerPivot. It puts the end user in charge. They can create health indications in an Excel like environment; and they can submit back into a shared (Sharepoint) repository.

I’ve got to present at the longwinded FST 3rd Annual Technology & Innovation – the Future of Banking & Financial Services New Zealand conference in August. I’ll be exploring this in more detail and hopefully have a few examples to test out.

Dynamically changing format statements in the F# ISE

2010-07-18T13:39:00.001+12:00

This had me stuck yesterday – how to change the width of some formatted printf output. In the end with a bit of searching I realised the TextWriterFormat member gives you a way to define the format such that you can change it in your program. An example is a better viewer for the Pascals Triangle sequence generator I made recently – all related to a school math project. Also – where does it tell you that to print a % you need to double it? Why didn’t they do as with speech marks and use a leading \?

let rec PascalsTriangle = seq {
    yield [1];
    for aLine in PascalsTriangle ->
        let newLine =
            aLine
            |> Seq.pairwise
            |> Seq.map (fun (x,y) -> x+y)
            |> Seq.toList
        List.append (1::newLine) [1]
    }

PascalsTriangle |> Seq.take 10
//|> Seq.iter (fun i -> printfn "%A" i)
|> Seq.iter (fun v -> 
                        v |> Seq.iter (fun w -> printf "%3i" w)
                        printfn ""
            )

// smarten up the printing...
let samplePT = PascalsTriangle |> Seq.take 12
let maxDigits = (Seq.concat samplePT |> Seq.max).ToString().Length
let numberFormatter digits = Printf.TextWriterFormat<int->unit>(sprintf "%%%dd" digits)
let spaceFormatter digits = Printf.TextWriterFormat<string->unit>(sprintf "%%%ds" digits)
let rows = Seq.length samplePT
samplePT |> Seq.iteri   ( fun u v ->
                                    for i in 1..((rows / 1) - u) do
                                        printf (spaceFormatter ((int)((maxDigits+1)/2))) " "
                                    v |> Seq.iter (fun w -> printf (numberFormatter (maxDigits+1)) w)
                                    printfn ""
                        )

The output is like this:

                      1
  
                    1   1

  
                  1   2   1

  
                1   3   3   1

  
              1   4   6   4   1

  
            1   5  10  10   5   1

  
          1   6  15  20  15   6   1

  
        1   7  21  35  35  21   7   1

  
      1   8  28  56  70  56  28   8   1

  
    1   9  36  84 126 126  84  36   9   1

  
  1  10  45 120 210 252 210 120  45  10   1

  
1  11  55 165 330 462 462 330 165  55  11   1

Wish I knew how to do this in WPF…

Wish I knew a better way to post into the blog…

IP Address to Country Map

2010-06-04T13:55:00.001+12:00

I’m learning how to use the F# HashMultiMap and I thought up a sample exercise – do a lookup in memory of IP address to a country code. This is a task I’ve seen embedded in a relational table before but that seemed daft – it’s on disk so it will be slow. It seems smarter to do this all in memory and just serve out a result.

It’s relatively easy to get hold of IP address country/region/ISP lookups in the form of a single file, for example http://software77.net/geo-ip/ provide one for country lookups under a GNU license.

First I load up the text file:

open System
open System.IO
open System.Text.RegularExpressions

let (|ActiveRegex|_|) regex str =
   let ms = Regex(regex).Matches(str)
   if ms.Count > 0
   then Some ((Seq.cast ms : Match seq))
   else None

let matches s re =
    match s with
    | ActiveRegex re results -> results
    | _ -> Seq.empty

let capturesSeq s p =
    seq{
        for m in matches s p ->
            Seq.skip 1 (seq{for g in m.Groups -> g.Value})
        }
    |> Seq.concat


let csvRegex = "\"([\w\s:;'`~!@#$%\^&\*_<>,\.\\\/\|\[\]\{\}\(\)\-\+\?]*)(?:\",|\"$)"

let isInt64 i =
    let v,_ = Int64.TryParse(i)
    v

let parseIpToCountryLine lineNo (line:String) =
    try
        let values = 
            capturesSeq line csvRegex
            |> Seq.toArray

        isInt64 values.[0] |> fun test -> if not test then failwith (sprintf "Bad IP FROM on line %i" lineNo)
        isInt64 values.[1] |> fun test -> if not test then failwith (sprintf "Bad IP TO on line %i" lineNo)
        int64 values.[0], int64 values.[1], string values.[6]
    with
        | :? System.IndexOutOfRangeException -> failwithf "Failed on line %A, contents: %A" lineNo line

let IpToCountryLines = File.ReadAllLines(@"c:\temp\IpToCountry.csv", Text.Encoding.Default) 

(*
File format like this... detail in the header comments of the file
# IP FROM      IP TO        REGISTRY  ASSIGNED   CTRY CNTRY COUNTRY
# "1346797568","1346801663","ripencc","20010601","il","isr","Israel"

there is a tricky line in there, it contains an 'Å' character which seems to require that I explicitly define the text encoding. Don't see why since it's just default...
*)

let getIpToCountry (lines:string []) =
    lines 
    |> Seq.filter (fun i -> not(i.StartsWith("#")))
    |> Seq.mapi parseIpToCountryLine 

let Ip2Country = getIpToCountry IpToCountryLines

Then a function to create a numeric form of an IP address:

let numIP (a:int) (b:int) (c:int) (d:int) = (int64 d) + ((int64 c)*256L) + ((int64 b)*256L*256L) + ((int64 a)*256L*256L*256L)

Then the hash and a quick test:

#r "FSharp.PowerPack.dll"
// count down until a key is found - could be speed up by creating extra entries when to-from is a large gap
let Ip2Country3 = HashMultiMap<_,_>(
                                    Ip2Country
                                    |> Seq.map (fun (ipFrom,ipTo,country) -> ipFrom, country )
                                , HashIdentity.Structural)

let rec countryFromIP (ip:int64) =
    if Ip2Country3.ContainsKey(ip) then 
        Ip2Country3.[ip]
    else
        countryFromIP (ip-1L)


// test with 10 addresses taken from our public site web logs
let testIPs = [… in the interests of confidentiality stick some of your own in here… as (a,b,c,d) tuples…]

let time f x =
    let timer = System.Diagnostics.Stopwatch.StartNew()
    try f x finally
    printf "Execution duration: %gms\n" timer.Elapsed.TotalMilliseconds;;

testIPs |> List.map (fun (a,b,c,d) -> numIP a b c d) |> time List.map (fun nIP -> countryFromIP nIP)

I’ve found it takes about 1 millisecond for 10 random lookups on my virtual workstation instance (running under Hyper-V with 3GB allocated on a desktop Dell Optiplex 760).

My Regex Lessons (Probably 1 of many…)

2010-05-26T14:23:00.003+12:00

OK, that previous post had a couple of minor problems.

First that regular expression – terribly unwieldy. I haven’t managed to shorten it fully yet, but this is marginally better:

"([\w\s:;… add whatever characters are valid here…]*)(?:\",|\"$)

Secondly, I wasn’t fully thinking through the matches/groups/captures hierarchy. I kept getting empty captures being reported back without understanding why I was getting them.

Updated code here:

open System
open System.Text.RegularExpressions

let (|ActiveRegex|_|) regex str =
   let ms = Regex(regex).Matches(str)
   if ms.Count > 0
   then Some ((Seq.cast ms : Match seq))
   else None

let matches s re =
    match s with
    | ActiveRegex re results -> results
    | _ -> Seq.empty

let capturesSeq s p =
    seq{
        for m in matches s p ->
            Seq.skip 1 (seq{for g in m.Groups -> g.Value})
        }
    |> Seq.concat

let csvRegex = "\"([\w\s:;~!@#$%\^&\*_<>,\.\\\/\|\[\]\{\}\(\)\-\+\?]*)(?:\",|\"$)"

let testLine = "\"31\",\"a 1\",\"b-2\",\"c+3\",\",.;~!@#$%^&*()\/?><,.|{}[]_+-\",\"\",\"14/05/2010 12:12:20 a.m.\",\"1: 2; 3. 4? 5[ 6] 7& 8*\",\"a,b\""

capturesSeq testLine csvRegex
|> Seq.iter (fun x -> printfn "%A" x)

Parsing CSV Escaped with Speech Marks

2010-05-25T11:51:00.001+12:00

I just know this isn’t the right way to do this but what the heck it seems to work for me.

The simple match is "([\w,]*)",+|"([\w,]*)"$ but I needed to account for all the other commonly occurring characters such as %&* etc. ( I would’ve thought I could just use .* but I can’t seem to get it to work.)

I’m using the F# Active Pattern approach to divvy up the matches – and returning the Match objects into the seq rather than breaking out the capture in the active pattern. This was more useful to me when I was using the matches later on in the code.

open System.Text.RegularExpressions

let (|ActiveRegex|_|) regex str =
   let ms = Regex(regex).Matches(str)
   if ms.Count > 0
   then Some ([ for m in ms -> m ])
   else None

let matches s re =
    match s with
    | ActiveRegex re results -> results
    | _ -> []

let testLine = "\"31\",\"a 1\",\"b-2\",\"c+3\",\",.;~!@#$%^&*()\/?><,.|{}[]_+-\",\"\",\"14/05/2010 12:12:20 a.m.\",\"1: 2; 3. 4? 5[ 6] 7& 8*\",\"a,b\""

matches testLine "\"([\w\s:;~!@#$%\^&\*_<>,\.\\\/\|\[\]\{\}\(\)\-\+\?]*)\",+|\"([\w\s:;~!@#$%\^&\*_<>,\.\\\/\|\[\]\{\}\(\)\-\+\?]*)\"$"

let printMatches s p =
    for m in matches s p do
        seq{for g in m.Groups -> g} 
        |> Seq.skip 1 
        |> Seq.iter (fun x -> printfn "%A" x)

printMatches testLine "\"([\w\s:;~!@#$%\^&\*_<>,\.\\\/\|\[\]\{\}\(\)\-\+\?]*)\",+|\"([\w\s:;~!@#$%\^&\*_<>,\.\\\/\|\[\]\{\}\(\)\-\+\?]*)\"$"

System Center Operations Manager Health Explorer Context

2010-05-20T14:39:00.001+12:00

I’ve been wondering for a while why the Health Explorer doesn’t always display the context for a state change. I found one recent link through both Bing and Google that explains what’s going on: http://social.technet.microsoft.com/Forums/en-US/systemcenteressentials/thread/df725076-e307-4d63-9be9-5d875c6924b5/.

Basically, at Entity level if there are multiple sources for state change information then the UI doesn’t know how to represent that information so it just says “No context was available for this state change event” and it displays a bunch of state changes with no text… something like this:

What you need to do is drill down into Availability/Configuration/Performance etc and as you do you’ll get the state change context appearing.

Pascal’s Triangle in F#

2010-05-19T12:35:00.001+12:00

Fun with F# :)

let rec PascalsTriangle = seq { 
    yield [1];
    for aLine in PascalsTriangle -> 
            List.append (1::Seq.toList (Seq.map (fun (x,y) -> x+y) (Seq.pairwise aLine))) ([1])
                    }

> PascalsTriangle |> Seq.take 10 |> Seq.toList;;
val it : int list list =
  [[1]; [1; 1]; [1; 2; 1]; [1; 3; 3; 1]; [1; 4; 6; 4; 1]; [1; 5; 10; 10; 5; 1];
   [1; 6; 15; 20; 15; 6; 1]; [1; 7; 21; 35; 35; 21; 7; 1];
   [1; 8; 28; 56; 70; 56; 28; 8; 1]; [1; 9; 36; 84; 126; 126; 84; 36; 9; 1]]
>

I actually started this with a pipe forward approach, but ( at least with my knowledge of the syntax) it seemed longer to write down.

let rec PascalsTriangle1 = seq { 
    yield [1];
    for aLine in PascalsTriangle1 -> 
            let newLine = 
                aLine
                |> Seq.pairwise
                |> Seq.map (fun (x,y) -> x+y)
                |> Seq.toList
            List.append (1::newLine) [1] 
     }

Discovering the presence of a database in a System Center Powershell script

2010-05-14T20:41:00.001+12:00

I’m pretty sure there’s a better way to do this using the native SQL Server management pack but this worked for me (peppered with debugging statements). It looks for a database called staging and if it finds it, it returns the discovery information back.

      <Discovery ID="B.Staging.DiscoverProcessingComponent" Enabled="true" Target="B.Staging.ComputerRole" ConfirmDelivery="true" Remotable="true" Priority="Normal">
        <Category>Discovery</Category>
        <DiscoveryTypes>
          <DiscoveryClass TypeID="B.Staging.ProcessingComponent" />
        </DiscoveryTypes>
        <DataSource ID="PSScript" TypeID="Windows!Microsoft.Windows.TimedPowerShell.DiscoveryProvider">
          <IntervalSeconds>180</IntervalSeconds>
          <SyncTime />
          <ScriptName>DiscoverBStagingProcessingComponent.ps1</ScriptName>
          <ScriptBody><![CDATA[
      param($sourceId, $managedEntityId, $computerName)
      
      $api = New-Object -comObject 'MOM.ScriptAPI'
      $api.LogScriptEvent("Processing discovery",101,2,"Created MOM.ScriptAPI with param sourceId = $sourceId , managedEntityId = $managedEntityId , computerName = $computerName ")
      
      $discoveryData = $api.CreateDiscoveryData(0, $sourceId, $managedEntityId) 
      $api.LogScriptEvent("Processing discovery",101,2,"Executed CreateDiscoveryData")
      
      [system.reflection.assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO') | out-null

      $s=new-object('Microsoft.SqlServer.Management.Smo.Server') $computerName
      $dbs=$s.databases
      $dbs | get-member -membertype property
      $staging = ($dbs | select name | where {$_.name -eq 'staging'})."Name"
      $api.LogScriptEvent("Processing discovery",101,2,"Found anything: $staging")
      
      $instance = $discoveryData.CreateClassInstance("$MPElement[Name='B.Staging.ProcessingComponent']$")
      $api.LogScriptEvent("Processing discovery",101,2,"Executed CreateClassInstance")

      if ($staging -eq 'staging') {
        $instance.AddProperty("$MPElement[Name='Windows!Microsoft.Windows.Computer']/PrincipalName$", $computerName)
        $instance.AddProperty("$MPElement[Name='B.Staging.ProcessingComponent']/B.Staging.ProcessingComponentKey$", "staging")
        
        $api.LogScriptEvent("Processing discovery",101,2,"In the staging branch")

      } else {
          $instance.AddProperty("$MPElement[Name='Windows!Microsoft.Windows.Computer']/PrincipalName$", "")
        $instance.AddProperty("$MPElement[Name='B.Staging.ProcessingComponent']/B.Staging.ProcessingComponentKey$", "")
        
        $api.LogScriptEvent("Processing discovery",101,2,"In the not-staging branch")

      }
      
      $api.LogScriptEvent("Processing discovery",101,2,"About to AddInstance")

      $discoveryData.AddInstance($instance)
      
      $api.LogScriptEvent("Processing discovery",101,2,"About to return discoveryData")

      $discoveryData
      
      ]]></ScriptBody>
          <Parameters>
            <Parameter>
              <Name>sourceID</Name>
              <Value>$MPElement$</Value>
            </Parameter>
            <Parameter>
              <Name>managedEntityID</Name>
              <Value>$Target/Id$</Value>
            </Parameter>
            <Parameter>
              <Name>computerName</Name>
              <Value>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/PrincipalName$</Value>
            </Parameter>
          </Parameters>
          <TimeoutSeconds>300</TimeoutSeconds>
        </DataSource>
      </Discovery>

Lessons in System Center Operations Manager Discovery

2010-05-14T20:27:00.001+12:00

Some quick notes before I forget them.

General Hints

Run through the Management Pack Authoring Guide first. It’s useful but I feel it didn’t give me enough of an explanation of what’s actually going on under the covers. Much of the inner workings of system center remain a mystery to me.
Setup a single all in one dev environment to allow yourself the chance to really play around when you’re not sure what’s going on. Then load on an agent onto your normal desktop workstation/laptop and test discoveries out on that machine at the same time as you begin deploying your management packs into the production environment.
Get a copy of Savision Live Maps – a 5 diagram copy is freely available. It has a simple, easy to use interface for finding your class instances and testing out your discoveries. And, I’ve just discovered – there’s a new version out – v5 is RTM - gotta try it.

Registry Discovery

Use the filtered registry discovery not the unfiltered discovery. This ensures you can get the Build Event Expression page which enables you to test for the existence of a registry key.

WMI Discovery

Use wbemtest.exe to try out queries and drill down into the CIM classes.
Documentation to help you with WMI is again weak. Classic example – at time of writing this there’s a community comment to highlight the fact that the SQL Server 2008 ComputerManagement namespace is incorrectly labelled (should be root\Microsoft\SqlServer\ComputerManagement10 – you can verify by using select * from __namespace under root\Microsoft\SqlServer).

Powershell Script Discovery

This one is still a bit of a mystery: I copied from the example in the Authoring Guide in combination with this example showing debugging.

Adding SCOM Class Properties to the DGML

2010-05-06T13:18:00.001+12:00

Updated transform for SCOM/OpsMgr management pack diagramming with DGML. This time I added in the SCOM class attributes as properties on DGML nodes – perhaps I should’ve used categories?? The legend in the visual studio viewer then lets you highlight parts of the class model based upon properties.

Next thing would be to include attributes about relationships as properties on the links. Also, what about referencing other system center management packs?

<?xmlversion="1.0" encoding="utf-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns="http://schemas.microsoft.com/vs/2009/dgml" version="1.0">
    <xsl:outputmethod="xml" indent="yes"/>
    <xsl:templatematch="/">
        <DirectedGraph>
            <Nodes>
                <xsl:apply-templatesselect="ManagementPack/TypeDefinitions/EntityTypes/ClassTypes/ClassType" />
            </Nodes>
            <Links>
                <xsl:apply-templatesselect="ManagementPack/TypeDefinitions/EntityTypes/RelationshipTypes/RelationshipType" />
            </Links>
            <Categories>
            </Categories>
            <Properties>
                <PropertyId="Accessibility" Label="Accessibility" Description="Accessibility" DataType="System.String" />
                <PropertyId="Abstract" Label="Abstract" Description="Whether this is an abstract class" DataType="System.Boolean" />
                <PropertyId="Base" Label="Base" Description="Whether this is a base class" DataType="System.String"/>
                <PropertyId="Hosted" Label="Hosted" Description="Whether this is hosted" DataType="System.Boolean"/>
                <PropertyId="Singleton" Label="Singleton" Description="Whether this is singleton" DataType="System.Boolean"/>
            </Properties>
        </DirectedGraph>
    </xsl:template>
    <xsl:templatematch="ManagementPack/TypeDefinitions/EntityTypes/ClassTypes/ClassType">
        <xsl:elementname="Node">
            <xsl:attributename="Id">
                <xsl:value-ofselect="@ID"/>
            </xsl:attribute>
            <xsl:attributename="Accessibility">
                <xsl:value-ofselect="@Accessibility"/>
            </xsl:attribute>
            <xsl:attributename="Abstract">
                <xsl:value-ofselect="@Abstract"/>
            </xsl:attribute>
            <xsl:attributename="Base">
                <xsl:value-ofselect="@Base"/>
            </xsl:attribute>
            <xsl:attributename="Hosted">
                <xsl:value-ofselect="@Hosted"/>
            </xsl:attribute>
            <xsl:attributename="Singleton">
                <xsl:value-ofselect="@Singleton"/>
            </xsl:attribute>
        </xsl:element>
    </xsl:template>

    <xsl:templatematch="ManagementPack/TypeDefinitions/EntityTypes/RelationshipTypes/RelationshipType">
        <Link>
            <xsl:attributename="Source">
                <xsl:value-ofselect="Source"/>
            </xsl:attribute>
            <xsl:attributename="Target">
                <xsl:value-ofselect="Target"/>
            </xsl:attribute>
        </Link>
    </xsl:template>
</xsl:stylesheet>

WMI Web Site Discovery with System Center Operations Manager

2010-05-03T14:20:00.001+12:00

Learning how to perform a discovery with WMI under SCOM/OpsMgr isn’t easy – documentation is appalling and there’s few examples on the net. Here’s my, eventually, successful attempt with some lessons learned along the way.

First off a quick description of the original problem – I want to discover multiple instances of web services running on servers.

After much, much time spent building a single server SCOM/OpsMgr development environment (just exactly what do people call this product anyway?) and figuring my way tentatively through SCOM and the SCOM Authoring Console I defined a class based on Microsoft.Windows.LocalApplication. If you ever have any issues with this stuff a good place to go is the Operations Manager Authoring discussion board – one of my questions and the helpful response from Elizabeth 1978 is here.

Then I created a discovery based upon a wmi query against the root\webadministration namespace to retrieve a list of sites. (Try it out in Powershell first: get-wmiobject –query “select * from site” –namespace root\webadministration.)

Now try to simulate this (learning to use the simulate tool as an experiment in itself – remember to load the MP into SCOM then click the arrow heads to attach to your development RMS so expressions can be resolved)… and it doesn’t work.

Strange. Then by trial and error I find that the asterisk doesn’t allow SCOM to pick up the data element for the name from the WMI query. Basically it seems to need you to identify each value you want returned in the WMI query to be sure you can map it in the Authoring Console mapper screen.

So this works:

You can check by running a simulation.

And to be doubly sure I imported the management pack into the SCOM environment with a monitor that targeted the class and checked when the ASP.NET Apps v4.0.30319 Requests/Sec counter was in excess of 10 requests/second.

As a web service I used a wcf service – just a simple hello world affair (in F# – why is there no template in visual studio??) and executed from a powershell console.

1..1000 | foreach-object {(new-object system.net.webclient).downloadstring(“http://localhost/service.svc”) }

“Ah Ha” Moment

2010-04-22T09:32:00.002+12:00

So if I can convert a system definition model from OpsMgr/SCOM to directed graph markup language… then presumably I could use VS2010 to create my architecture model and then generate the framework for my OpsMgr/SCOM management pack!

Update... it seems to me that none fo the vs2010 model types make sense with scom. Guess that it might be better waiting till the system center team publish a modelling tool.

SCOM Class Hierarchy to Directed Graph Markup

2010-04-21T22:14:00.003+12:00

As part of a struggle to get to grips with management pack authoring for Systems Center Operations Manager I wanted a way to diagram the relationships between classes. You can do it with the OpsMgr Authoring kit and Visio… but I don’t have it on my machine. However, with the management pack being xml, it seemed a simple job to transform to directed graph markup language to view inside Visual Studio 2010. Here’s a sample xsl transform.

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:template match="/">
    <DirectedGraph xmlns="http://schemas.microsoft.com/vs/2009/dgml">
      <Nodes>
      </Nodes>
      <Links>
        <xsl:for-each select="ManagementPack/TypeDefinitions/EntityTypes/RelationshipTypes/RelationshipType">
        <Link>
          <xsl:attribute name="Source"><xsl:value-of select="Source"/></xsl:attribute>
          <xsl:attribute name="Target"><xsl:value-of select="Target"/></xsl:attribute>
        </Link>
        </xsl:for-each>
      </Links>
      <Categories>
      </Categories>
    </DirectedGraph>
  </xsl:template>
</xsl:stylesheet>

This seems to work with the sample management pack created on Brian Wren’s blog for an article about embedded Powershell scripts. Using the transform and either the Altova xslt1 processor or the native transform in VS2010 I got a dgml file that displays like this.

PowerShell Data Munging

2010-01-18T14:02:00.002+13:00

There’s a data munging exercise on Rosetta Code which was missing a PowerShell solution. Since I spend a lot of time importing data to graph and present in my day job – often using PowerShell, I thought I’d add a couple of PowerShell options for the import of the data in the Data Munging 2 problem.

First just using iteration alone to look up good values.

$dateHash = @{}
$goodLineCount = 0
get-content c:\temp\readings.txt
    ForEach-Object {
        $line = $_.split(" `t",2)
        if ($dateHash.containskey($line[0])) {
            $line[0] + " is duplicated"
        } else {
            $dateHash.add($line[0], $line[1])
        }
        # split up the 24 instrument values and count the total number of entries with flag >=1
        $readings = $line[1].split()
        $goodLine = $true
        if ($readings.count -ne 48) { $goodLine = $false; "incorrect line length : $line[0]"  }
        for ($i=0; $i -lt $readings.count; $i++) {
            if ($i % 2 -ne 0) {                               
                if ([int]$readings[$i] -lt 1) {
                    $goodLine = $false
                }
            }
        }
        if ($goodLine) { $goodLineCount++ }
    }
$goodLineCount

And secondly taking advantage of the regular expression syntax.

$dateHash = @{}
$goodLineCount = 0
ForEach ($rawLine in ( get-content c:\temp\readings.txt) ){
    $line = $rawLine.split(" `t",2)
    if ($dateHash.containskey($line[0])) {
        $line[0] + " is duplicated"
    } else {
        $dateHash.add($line[0], $line[1])
    }
    $readings = [regex]::matches($line[1],"\d+\.\d+\s-?\d")
    if ($readings.count -ne 24) { "incorrect number of readings for date " + $line[0] }
    $goodLine = $true
    foreach ($flagMatch in [regex]::matches($line[1],"\d\.\d*\s(?<flag>-?\d)")) {
        if ([int][string]$flagMatch.groups["flag"].value -lt 1) {
            $goodLine = $false
        }
    }
    if ($goodLine) { $goodLineCount++}
}
[string]$goodLineCount + " good lines"

The F# solution on the site includes use of seq.forall – I thought maybe this would be useful to implement for the PowerShell solution as well – but I couldn't figure it out. Fortunately the good folks at stackoverflow helped out on that...

Connecting to SCOM from the Powershell ISE

2009-11-04T15:01:00.003+13:00

Temporary interlude, figuring out how to get some numbers from our IT environment to create some models for complexity in our technology environment. I thought System Center Operations Manager should be useful here but running powershell from the SCOM shell isn't that clever. It'd be much nicer in a decent editor... but how? I couldn't find this anywhere else on the net... so here's how I managed to get SCOM (R2) accessible from the Powershell ISE...

import-module "C:\Program Files\System Center Operations Manager 2007\Microsoft.EnterpriseManagement.OperationsManager.ClientShell.dll"

&"C:\Program Files\System Center Operations Manager 2007\Microsoft.EnterpriseManagement.OperationsManager.ClientShell.Functions.ps1"

Start-OperationsManagerClientShell -ManagementServerName: somemachinename -PersistConnection: $true -Interactive: $True

S&G Night Heron – Fibreglassing the Outside

2009-03-18T12:58:00.001+13:00

Quick notes from here on as the exterior was considerably easier than the interior.

I bias cut the fibrglass for the exterior to get it to drape better. The epoxy application was much simpler than inside although I did find a few areas that ended up a bit on the dry side. Main reason for this was I ended up working into the evenings and the lighting was awful. This also resulted in a couple of unnecessary floating bubbles and drips – but all easily fixed. So much easier after having done the interior first.

One tip (I think) was to fibreglass across the sheer line; that is, when I fibreglassed the deck, I went to about an inch beyond the deck-hull join, and likewise when I fibreglassed the hull, so that the join between deck and hull was secured with interior tape and a doubling of the exterior glass.

I doubled the glass on the exterior hull from the sheer line to the keel so that under my body I ended up with 4 layers of 200g (6oz) fibreglass.

Having read about the difficulties of making a lightweight boat I spent a lot of time checking the weight as I progressed. One very useful thing here (especially to avoid wastage) was a cheap digital scale. I bought this from a supermarket for $16 and used it to successfully mix epoxy down to 1g hardener and 5g epoxy. I recorded the weight of each batch of epoxy and I weighed the boat after each major application so I could clearly see the increase from 12kg prior to exterior glassing to 19kg after the fill coats were complete.

I found the fill coats a little painful. I seemed to never quite get the right amount on. I’d put a coat on, sand some off, then try again; many times over. What was particularly irritating here was that to avoid excessive exposure to the epoxy while curing I would make myself wait a few days before any sanding. The time just seemed to drag!

If I was doing it again I’d make two changes.

Firstly, I’d use a slow hardener and warm epoxy/conditions. I found that on a few of my (autumn) evenings the temperature had dropped to maybe 15C and the epoxy had thickened and would spread easily. You want it to spread really easily so a really hot day with a slow hardener would be a better combination.

Secondly, I’d use a random orbital sander more extensively. I spent far too much time continuing on with hand sanding when I should have just used the ROS. The pads clog up quick but crikey it does a hell of a job!

S&G Night Heron – Varnishing and Outfitting

2009-03-18T12:57:00.001+13:00

I was initially unsure what to use for the final coat: varnish or polyurethane. The documentation on the net seemed a little confusing. In the end I’ve gone with a varnish from Altex. Not sure if this was the best choice, I suspect it’s more expensive than the equivalent products I could have just bought at Mitre 10 or Placemakers. Nevermind.

The can said 6 to 10 coats. I think I’ve run up 7 on the top and 6 on the bottom. Hard to remember!

I used the ROS with 180 grit pads and this seems to produce a good surface. I’ve still got a fair 250mls left so I’ll put that on sometime in winter.

I enjoyed this part much more than the epoxy. No special safety gear required and it could be done at an enjoyable pace.

I put the kayak in the garage – there was some dust but it pails into insignificance when viewed from a few feet. Runs were definitely a problem. Sponge applicators helped but invariably I’d come back the next day and find a few runs that had formed.

For the outfitting I’ve kept to a bare minimum, at least for the moment. I used window weather stripping for the hatches – seems a bit leaky but I’m sure I’ll be able to tune it better.

I used plastic foot pegs; very light but they also seem a bit flimsy. In retrospect I’d go aluminium.

The back rest provided in the Night Heron plans seems quite effective without any padding. In fact, I spent $80 on closed cell foam and so far haven’t put any in yet, but it doesn’t seem to be especially uncomfortable so I’m in no great rush to finish it.

For the spray deck I ended up, after a few false starts, with getting one made by someone (Gabi) in Nelson. You can get hold of them on TradeMe and I’d recommend the product. Other options include Rasdex and BackOfBeyond.

And the final result? Well, here I am on the inaugural launch in the Waikanae River at the Otaihanga domain. Beautiful spot; you can head out at high tide down the 2kms to the river mouth, play in the sea, then come back with the salt water washed off in the river.

In the end I guess I spent 10 hours / week from the middle of November 2008 until the first week of March this year. So that’s about 160 hours. The cost is approximately $1000. It’s easy to spend more if you don’t hunt around for good prices on ply, fibreglass, epoxy and varnish.

And performance? Well, I’m very familiar with the plastic sea kayaks that you can hire. My Night Heron is faster, feels more manoeuvrable (and it doesn’t have a rudder) and it’s much lighter. So all in all – I LOVE IT!

Functionally complete kayak

2009-02-27T19:10:00.003+13:00

Second to last day of February and a quick note to remind me in the future: kayak is functionally complete. No photograph, camera batteries are flat, but the rubber seals are on hatch rims, the foot pegs are in, the back rest is in. That makes it about 15 weeks from when I started in mid november. Phew. One thing left to do - varnish, maybe wait a couple of weeks to be sure the epoxy has set. Weather this weekend looks shocking. Maybe Sunday to try on the sea? Write some more detail on the last steps of the build after that. For those interested, final weight about 19kgs. Hard to tell, scales difficult to read, maybe 18.5. That's with 6oz everywhere, 2 layers on the bottom and 2 layers most of the inside hull.

S&G Night Heron – Attaching the Deck to the Hull

2009-02-02T22:13:00.001+13:00

I found I couldn’t do this with just tape. The hull and deck needed more force to get the positioning of them together so I ended up wiring them together. This was the second time I did this. I did it first time round when wiring together the deck on top of the hull. I probably should’ve put more effort in then with the stern and it would’ve made life easier for me now.

The stern of the Night Heron has to be squashed together with some force. I’m not the only one that seems to have to do this. In the pictures supplied with Nick’s instructions you can see he has it happen to him and I’ve seen pictures from another builder with the same problem:

http://outdoors.webshots.com/photo/1165926632056308877loLvMK – this builder’s comments mirrored my experience exactly

The bow was easy. The last two feet of the stern were stubborn and took a bit of technique. There I used a paint spatula to force out the hull and then pressed hard and fast on the deck to squish the two beveled edges together for taping. One side of the stern just didnt want to play nice...

Maybe an extra form or two would be useful when creating the stern of the boat?

To help with getting the stern to fit I ended up putting in extra stitches along the rear of the boat. Getting them in at the back is easy as long you use long wires and get them through both holes while the deck and hull are still some distance apart. Then, only after all wires are in do you push the deck down onto the hull and tighten the wires.

I then glued up between the stitches – wish I’d done a cleaner job as I’m still sanding the rough bits up - and then waited a few days to be sure the two pieces wouldn’t crack or pop apart at any point.

With the stitches gone I then put the kayak up on it’s side and taped the inside seam. Notice I cut the hatches before attaching the deck. This made the taping quite easy.

I taped up to maybe 50cms from each end but in retrospect I could have gone further. The secret seemed to be lots of light!

Here’s one bit that went a bit wobbly:)

(Notice all the sanding dust still left inside!)

And then I just had to put it into the pool and proved it worked!

S&G Night Heron – Extra Cockpit Glass

2009-02-02T22:11:00.001+13:00

Since I did such a terrible job of glassing the inside of the hull on my first attempt I decided to put a second, large piece of glass on the inside of the boat.

I guess I covered maybe 40% of the inside of the hull with a second layer of 200g glass (equivalent to 6oz), so it should be very strong!

Here’s the finished product. Much better! This time I took extra care to wipe off excess epoxy during the initial curing process and I used a tape edge to stop the frayed glass fibres sticking.

S&G Night Heron – Grab Loops

2009-02-02T22:10:00.001+13:00

Rather than a big end pour I’ve gone with a couple of small plastic tubes (from old felt pens). I drilled the holes using the stitching holes in the hull as a guide for positioning them on each side of the hull.

To glue them into place I used the chopped up glass, epoxy mix. I wish I’d discovered this earlier. It looks very useful.

Notice the digital scales in the picture. I bought these from a supermarket for $16. They’ve been extremely useful in ensuring I get the epoxy mix right. I use small mixes of typically 36g, 48g, or 60g total weight (the mix ratio for Nuplex R180 is 1:5…).

S&G Night Heron – Deck Line Fittings

2009-02-02T22:09:00.001+13:00

For these I used the Maroske style fitting. I used the instructions on Bjorn Thomasson’s site but in retrospect would do things a little differently.

The first couple that I did I left the pvc tubes in place for more than 24 hours and had a tough time pulling the tube out. In the end I found a hair dryer would blow in hot air and soften the tube sufficiently they came out. For the rest I found if I just pulled the tube out within 12 hours I was fine.

If I was doing it again I’d put a few larger pieces under the pvc loop then cover the pvc with epoxy mixed with lots of chopped up fibreglass then place a couple of larger pieces of glass over the top. I’m sure it would make a smaller, better fitting.

S&G Night Heron – Hip Plates

2009-02-02T22:08:00.001+13:00

These were quite easy to position prior to gluing with a few lengths of masking tape.

I glued them into position with a microfibre additive then created a thick fillet and applied fibreglass up each side then a couple more shorter fibreglass pieces to strengthen the joint on the inside (where I’ll never be able to reach again once the deck is attached to the hull).

S&G Night Heron Build Notes – Fibreglassing The Coaming

2009-02-02T22:07:00.001+13:00

Quick post. This wasn’t especially easy. Two layers of bias cut cloth. Putting them on wasn’t hard until it came time to push the cloth under the deck. You have to cut darts and fold the cloth under but it’s a tight corner and fibreglass doesn’t like tight corners. If I was doing this again I’d make a rounder corner to fold under. Unfortunately this is neigh on impossible with the thigh braces as you’ve only got the thickness of the ply to work with. Also, I flipped the deck upside down and the glass on the top of the coaming wanted to come off. Maybe I should have waited for the epoxy to gel before turning the deck upside down?

S&G Night Heron Build Notes – Gluing the Coaming

2009-01-11T20:09:00.001+13:00

Well this was a bit tricky! For the life of me I couldn’t get the cutout coaming risers to bend round the curve of the cockpit cutout. I tried kerfing every inch, I tried kerfing much less than an inch (bad idea), I tried a heat gun (good idea). In the end the result was a lot of fine line cracks at the points that I’d cut kerfs. I’m not too fussed. I can easily adjust the final strength by changing the amount of fibreglass I lay on; and now I’ve had success with fibreglassing the inside of the deck that doesn’t bother me one bit.

One thing I was a bit surprised about – how come I couldn’t get the two coaming risers to meet neatly? There was a 5mm gap that I had to plug with a bit of ply.

Hmm, just looking at that photograph I can see I should’ve sanded the insert down a mm or two. Nevermind.

The surprising thing for me looking from the top down is how come the top of the coaming still fits neatly. Obviously the risers must have been cut out ok. No problem, it was easy to plug and actually I quite like the way it’s on an angle. I wonder what a riser would look like made of angled strips?

As for the final product see below. Obviously I’ve yet to fibreglass it. Currently I’ve just glued the pieces. I’m going to fillet tomorrow the fibreglass within the next day or so.

A quick note here about the use of cyano-acrylate glue. I didn’t use it but I’m thinking perhaps I should’ve. I liked the idea that I could use the epoxy and it would give me time to set it up right whereas I was worried that with CA glue I’d fix into the wrong place too quickly. In retrospect it probably would’ve made the stitching and this coaming much faster.

S&G Night Heron Build Notes – Fibreglassing

2009-01-08T21:25:00.001+13:00

Well this was exciting. What amazing stuff fibreglass is. I read and re-read numerous fibreglassing guides from across the net; use a spatula, use a paintbrush, use a roller. I just came away quite confused.

So I’m going to recount my experiences for you, and probably it’ll just help to confuse you too. Probably main suggestion I have is to find your own way of doing it.

Anyhow, here’s my fibreglassing lessons gleaned from doing the inside of my hull and deck.

Firstly, I think it does make it easier to seal the wood with a thin coat of epoxy first. I ended up deciding that a bit of rag was as good as anything else at applying the epoxy. Wear two layers of gloves to be sure they don’t rip, then dab the rag in the epoxy and rub it on the wood. Worked fine for me and a lot cheaper than any other tool. Certainly means you can apply a very thin layer.

Secondly, for me, rollers suck. I’ve decided that a flat plastic spatula made from the lids of takeaway plastic food containers work great. They’re sufficiently flexible that they can get into funny internal corners easily and they don’t grab at the cloth. They especially good and removing excess epoxy and spreading it away from where it might otherwise pool.

Thirdly, and perhaps most importantly for me, on the internal sides of the deck and hull you should cut the cloth on the bias. This makes a huge difference. The pain with which I watched my first attempts to lay the fibreglass just pull away from the internal corners whenever I ever so slightly stretched the fabric was truly irritating.

Cutting the fabric on the bias (at an angle – 45 degrees seemed good to me) gives it much more stretch on those internal angles.

Here’s the result of my first attempt:

Compare that with this:

S&G Night Heron Build Notes – Attaching the Panels

2009-01-08T21:05:00.001+13:00

This was an interesting exercise. Handling the long hull panels is a difficult task and getting them to fit well is not trivial.

There were a few lessons here.

Alignment

Firstly, get a properly flat work surface to ensure the stations are positioned correctly.

Use a pencil to mark out a long straight edge down one side of the flat surface and mark out the perpendicular lines where the stations should sit.

In the S&G Night Heron instructions Nick suggests holding the stations vertical using a couple of quick clamps. I found that was painful as they’d move out of place whenever I pushed the kayak eg trying to knock an edge off; sand a bump off an edge or generally just touch it...

Instead, lock the stations into place using some scrap wood that’s wide enough to clamp securely into place. In the photo below you can see the deck on the stations and how they’re clamped to the table. (I wish I’d done this while building the hull:))

Bevelling

Next thing to remember is, bevel to the edge. Assuming you’ve cut the panels correctly this should ensure your boat stitches up as it is supposed to. Compare figure A to figure B… I naively would’ve thought that B was preferred to keep all fillets on the inside of the kayak, however, this might lead you to think differently.

Nick Schade’s instructions said to bevel so I worked on the idea it was figure B. Checking a station plan against the hull bottom confirms this, so, yes, you need to bevel completely to the outer edge (but not beyond).

The best way to ensure you don’t overdo it is to bevel most of the way with a plane or sanding block; loosely tie the panels together; then run a bit of sandpaper up and down the join until the panels close up together. Check the picture below.

Stitching

What sort of wire do you use??? I tried copper wire and galvanised steel tie wire. The tie wire worked fine. After reading various messages on the different kayak building boards on the net I was quite confused about what to use. Once you try it you realise you can use just about anything. The important this is that you’ve cut the panels correctly and there is a consistent internal bevel.

Gluing

I found this a bit confusing. Do you use plain epoxy or do you add an adhesive additive like West Systems 403. My current take on it is this: quickly wipe over plain epoxy because it will soak into the wood easily; then add a bit of the adhesive additive (made up of microfibres and colloidal silica I believe) and paint into the joint. You only want a minimal amount to actually glue the joint together. Don’t worry about forming a fillet during gluing.

If you have trouble removing stitches after the glue has set I found that the commonly repeated advice to use a soldering iron works great.

Filleting

For the filleting I used a lightweight filler (from Epiglass I think) and masking tape. Masking tape is great.

I tried lots of methods to apply the fillets: a plastic spoon worked great, a wooden stirring stick shaped like a doctor’s tongue depressor worked great and I think I could easily have cut any bit of plastic to a round shape to form an applicator. So all in all, it wasn’t very hard.

Stitch and Glue Night Heron Build Notes – Cutting the Panels

2009-01-05T22:11:00.001+13:00

There are a few important points here which I learn’t from the process.

Scarfing.

I used the jig described in the One Ocean Kayaking building manual. It worked but my implementation of the jig wasn’t that great. In retrospect I think I might in future either do it by hand using a sharp plane, or use a router jig I’ve seen on the Kayak Forum.

Holding onto a circular saw and ensuring it cuts cleanly and evenly across the ply requires a fair bit of attention and a good horizontal guide with a clean, straight edge. The first piece I used was too thin and had a slight waiver in the middle. Better to make it nice and wide so it remains a true straight edge.

The ply sheets are thin with external veneers and an internal core; the saw is rough, so use a bit of waste material as the bottom and top layers. I ended up with the top and bottom sheets having some tatty edges (because of course I didn’t use the waste material top and bottom).

Clamp or weigh down the center of the sheets otherwise the middle has a habit of lifting.

Finally, check before you’ve finshed that the scarf is accurately square to the long edge of ply. Use a large builders square. If I had I would’ve seen I had a waiver in the middle of my scarfs.

Scarf Gluing

This would’ve been a whole lot easier if I didn’t have that waiver in the scarf!

Having said that, the output hasn’t been that bad. There was some badly overlapping ply, and I really should have taken more time with the weighting down of the joint, and actually, in retrospect I went too light with the epoxy, but in the end it appears to have worked fine.

Just one note on the use of the epoxy. This was the first time I’d used the epoxy and if there’s one lesson with doing anything new – it’s give yourself time and opportunity to learn from your errors. I’ve tried a few different methods of application in the last 2 weeks and my current favourite is the nylon brush topped foam pads that are sold in the hardware stores as speed brushes. I can buy the pads for just a few dollars and they stick onto a plastic holder which, at least for me, is easier to work with than a roller.

I think the best approach is to mix up some epoxy, run a quick line of it up and down the scarf edges to allow it to soak in; then add an adhesive filler, give it a swirl and apply it to the edges before joining. This will help fill voids better than just epoxy itself.

Cutting the Panels

First discovery: positioning the paper patterns on the ply with spray on glue was a breeze. I was worried that I’d permanently stick the patterns on but it was surprisingly easy to apply a thin coat that would allow me to reposition a pattern. Having said that, I did end up using a couple of different spray glue brands. The 3M glue seemed to be far better to use.

I also used a string line for the main pattern that included the hull pieces. This is such a long paper pattern that I wanted to be sure that a center line drawn on the pattern actually ran straight down the table. Getting this wrong would be annoying as you’re cutting through two ply panels not just one.

I drilled the tie wire holes next and put little bits of wire in the lot of them to keep the panels from moving; then just for overkill I decided I’d hot glue round the edges of the two panels. I used a sharp hobby knife to cut round the pattern edges – surprisingly easy to do. Then I used a jigsaw to cut out each piece. The jig saw did a good job (I think) but the best method appeared to be to stay a mm or two away and use a block plane or sanding block to finish. That gave a very good finish. Just to give myself time to improve I purposefully cut the stations first.

Since cutting the main panels I’ve actually moved to using a fine point panel saw. It’s just less noisy, less dusty and more accurate for the straighter lines.

Lifting the cut pieces was alarming, They wobble a lot and by experimenting with offcuts I could tell the scarf joints could break at the point the ply starts to thin. There was often no epoxy at that point and when combined with a thinning (or complete lack) of external veneer, the ply was substantially weakened. My advice: don’t go too light on the epoxy round the scarf joint, and make sure in the positioning of the pieces that the inner ply layer isn’t left substantially exposed anywhere along the joint.

If you do get a dodgy joint just try a wood backing block. In the case of the photograph below I’d been a bit aggressive and planning away some badly overlapping ply on a scarf. The result was a weak point so I then thinned a piece of scrap ply down and glued that onto the joint. I later sanded this down so just a thin veneer was left covering the low points. It appears to have fixed my problem nicely. (Note that I decided to put a thin epoxy coat on panels but at the time I took the photo I hadn’t epoxied under the clamp.)

Stitch and Glue Night Heron

2009-01-05T21:48:00.003+13:00

Over recent weeks I’ve been building a stitch and glue Night Heron kayak designed by Nick Schade from Guillemot Kayaks. There’s a lot I’ve learnt along the way so I thought it useful to post up some notes on the blog.

As a quick background to the project, the Night Heron family of kayaks are 5.5m ocean going sea kayaks available in plywood hard chin form or strip form. Check out the web site as there’s a lot more information there. Of special use when building is the Kayak Building Bulletin Board hosted by Guillemot. This is a great resource.

Dr Joe Trodahl on Radio NZ with Veronika Meduna

2008-07-20T20:12:00.001+12:00

Here’s a voice from the past, my old thesis supervisor at Victoria University, Joe Trodahl on Radio NZ speaking about new materials that combine ferroelectric and ferromagnetic properties. Joe’s got a great accent with that wonderful Alaskan sound.

http://www.radionz.co.nz/audio/national/ocw/2008/07/17/new_materials

Label Cloud in Blogger

2008-07-15T12:48:00.002+12:00

Great, just found how to do it. Seems to work fine with Blogger labels. Go to http://phy3blog.googlepages.com/Beta-Blogger-Label-Cloud.html for details.

More Adventures with Innovation Blogging

2008-07-14T17:34:00.003+12:00

So, as I reported recently I’ve been working on an internal blog to report the clever stuff we do in the IT shared service line within the corporate where I work. The reason behind doing it is of course to raise awareness of the great stuff I see the broader team delivering each week. This might be grass roots incremental or possibly (hopefully!) ground breaking and disruptive.

Now I’m getting blog postings, I’ve managed to deliver a KPI on the Sharepoint list hosting the blog, I’ve worked through how to create a chart to make progress through the year obvious, but now I want to get a summary of the blog postings on pages elsewhere in the Sharepoint server.

Unfortunately, while I have a large team of developers, the reality is they’re all fully busy so I’ve had to work through this problem myself.

Here’s some guidance for others doing the same. First off quick note to say I was surprised at how hard it was. Still, I’ve got there.

So, to get the blog posts and re-display them elsewhere I used Sharepoint Designer. This gives you the ability to place a DataView in a webpart on any page. It turns out the DataView is the Swiss Army knife of Sharepoint programming (at least when you’re using Sharepoint Designer). This control allows you perform an XSL transform on a data source and display the results. The query is represented using CAML and the best reference to help me with my problem was a posting on the Sharepoint Designer blog. Look for the entry on the CAML query. My select command turned out like this:

selectcommand="<View><Query><OrderBy><FieldRef Name='PublishedDate' Ascending='FALSE'/></OrderBy><Where><Geq><FieldRef Name='PublishedDate'/><Value Type='DateTime'><Today OffsetDays='-30'/></Value></Geq></Where></Query></View>"

The next problem is formatting the output. If you don’t format the output you’ll get HTML tags that will disrupt the rest of the page.

So, in the end I did something like the following. Note that I used some XSL to remove the markup where it occurred. I got that from http://blogs.pointbridge.com/Blogs/morse_matt/Pages/Post.aspx?_ID=37.

<xsl:template name="RemoveHtmlTags">

    <xsl:param name="html"/>

        <xsl:choose>

            <xsl:when test="contains($html, '&lt;')">

            <xsl:value-of select="substring-before($html, '&lt;')"/>

           <!-- Recurse through HTML -->

           <xsl:call-template name="RemoveHtmlTags">

             <xsl:with-param name="html" select="substring-after($html, '&gt;')"/>

           </xsl:call-template>

         </xsl:when>

         <xsl:otherwise>

           <xsl:value-of select="$html"/>

         </xsl:otherwise>

       </xsl:choose>

</xsl:template>

    

<xsl:template name="dvt_1.rowview">

    <xsl:variable name="pureTextBody">

        <xsl:call-template name="RemoveHtmlTags">

              <xsl:with-param name="html" select="substring(@Body,0,250)" />

        </xsl:call-template>

    </xsl:variable>

 

    <tr>

        <xsl:if test="position() mod 2 = 1">

            <xsl:attribute name="class">ms-alternating</xsl:attribute>

        </xsl:if>

        <xsl:if test="$dvt_1_automode = '1'" ddwrt:cf_ignore="1">

            <td class="ms-vb" width="1%" nowrap="nowrap">

                <span ddwrt:amkeyfield="ID" ddwrt:amkeyvalue="ddwrt:EscapeDelims(string(@ID))" ddwrt:ammode="view"></span>

            </td>

        </xsl:if>

        <td class="ms-vb">

            <a href="/CleverStuff/Lists/Posts/Post.aspx?ID={@ID}"><xsl:value-of select="@Title"/></a>

        </td>

        <td class="ms-vb">

            <xsl:value-of select="ddwrt:FormatDate(string(@PublishedDate), 1033, 5)"/>

        </td>

        <td class="ms-vb">

            <xsl:value-of select="$pureTextBody" disable-output-escaping="yes" /></td></tr>

</xsl:template>

l:stylesheet>

Now, I still have a problem. I’m getting an error message at the bottom of the (successfully) displayed list. The message is “An error occurred processing the data view. The XslText property is empty.”. Any ideas???? This Microsoft KB article has some information but it’s for Sharepoint 2003. I might have to get one of the dev team onto it:)

Update: 16 July

Doh! Figured the problem out - I had an extra DataView webpartzone inserted. I think this is an outcome of the sluggish performance of Sharepoint Designer. You tend to click too many times when you're inserting controls...

Update on the List Activity Viewer

2008-07-09T21:37:00.001+12:00

Just a quick note regarding deployment of the activity grapher I described last week. This is now deployed and working a treat but I did have a minor problem getting it going. Turns out our current production implementation of sharepoing isn’t using the delegation model for Excel Services security. Instead we’re using the trusted subsystem approach (typical for a farm). In this instance to get the UDF going I just needed to remove the impersonation I was using for my dev/test environment.

Great Day

2008-07-04T21:30:00.001+12:00

What a great day! A month ago almost exactly, I spoke at the FSI Media Technology and Innovation Conference and mentioned the importance of internal selling in a corporate. As an example of the opportunities we often miss I mentioned that I have a development team sitting in an open plan pod next to the CEOs pod (a remarkable fact in itself for a corporate of our size!). David, the team lead, had recently shown me his real time graphs of call centre activity displayed using Microsoft’s new WPF technology. They looked fantastic and the obvious thought was to maximise the impact by making available a big screen that ensured CEO, GMs all and sundry could see the impressive output from these monitoring programs. All I had to do was make a big screen available.

Well, it took a month but finally we got a 40” screen up there and yes, they had the desired impact. Before leaving work today we had a cluster of people ranging from analysts through our EXT team admiring the graphs and watching them turn from green to red and back as the metrics progressively changed in real time.

Success and many thanks to David!

Graphing Sharepoint List Activity

2008-07-01T23:11:00.004+12:00

Here’s an interesting challenge that I’ve had over the last few days. At work we’re beginning to use sharepoint (MOSS/WSS) more and more as a tool within our office environment. It’s used for document/record management; it’s used for team workplaces; it’s used for hosting business processes; heck, it’s used for all sorts of things. In my case I’ve been using it lately to track the innovative ideas and deliverables that our IT shared service line are coming up with. I bug them for their clever stuff and either they post an entry into a blog, or I do it for them.

Now, here’s the problem. The blog looks great and I have a KPI web part showing blog items submitted over the last 2 weeks. I even have separate KPIs for different team’s submissions (identified by the category of the blog post), but what I really wanted was a graph of activity over the weeks. Do you think this would be easy? Well no!

You can easily add a web part to show an Excel report using Excel Services at the top of the blog. You can access sharepoint list data in an Excel workbook. The tricky part is that when you go to publish it into Excel Services you find an error message saying that retrieving data from sharepoint lists isn’t supported yet! A search on the net tells you that you need to create a UDF. I read somewhere that Excel Services is a ‘version 1’ product and apparently it doesn’t support everything you’d expect. Hmm, consuming a sharepoint list was one thing I definitely was expecting Excel Services to be capable of doing. Nevermind…

OK - back on track, so there is some sample code out there (eg check http://msdn.microsoft.com/en-us/library/bb267252.aspx) but explanations are fairly limited. So here’s a quick run down of what I’ve managed to do.

Firstly, borrow someone else’s code (eg from the link above):) Then iterate through items in the sharepoint list and accumulate across a number of previous week counters (or maybe put into a collection and try using linq?) eg:

// Create an object array to return to Excel and initialise values

toExcel = new object[weekCount, 2];

for (int i = 0; i < weekCount; i++)

{

    toExcel[i, 0] = i;

    toExcel[i, 1] = 0;

}

 

// two approaches to try

// 1: iterate through all list items and increment week activity counters when in range; or

// 2: use linq

 

CultureInfo ci = new CultureInfo(CultureInfo.CurrentCulture.Name, false);

int thisWeek = ci.Calendar.GetWeekOfYear(DateTime.Now, CalendarWeekRule.FirstDay, DayOfWeek.Monday);

 

// Iterate through SPListItems in the view.

foreach (SPListItem currVal in values)

{

    // get created date field

    // !** might need to set DateTimeFormatInfo appropriately 

    // use Calendar.GetWeekOfYear (from System.Globalization) to determine difference in weeks from today

    // or perhaps SPIntlCal.GetWeekNumber from Microsoft.Sharepoint.Utilities

 

    SPField field = currVal.Fields.GetFieldByInternalName("Created");

    DateTime created = DateTime.Parse(field.GetFieldValueAsText(currVal[field.Id]));

 

    // find week number for today

    // then find week number of list item

    // subtract and accumalate appropriate counter

    // if week difference < max number of weeks previous

    int createdWeek = ci.Calendar.GetWeekOfYear(created, CalendarWeekRule.FirstDay, DayOfWeek.Monday);

    int weeksPast = thisWeek-createdWeek;

 

    if (weeksPast < weekCount)

    {

        toExcel[weeksPast, 1] = (int)toExcel[weeksPast, 1] + 1;

    }

    else

    {

        // must be the first few weeks of a new year

 

    }

}

Then return to Excel, generate the chart and post back into Excel Services so you get something like the screenshot below (all test data of course).

In this case there were 8 posts in the previous week, 1 post in the current week, and none going back 2 weeks.

One thing to keep in mind with the code I put up above – I didn’t put in any logic to handle week intervals around the change of one year to another – in this instance I think you’ll need to add 52. You can grab some sample code from my public skydrive folder: ListActivityInWeeks.zip-download. Suggestion: grab a copy of Cum Grano Sails Professional Excel Services book.

Update 2 July 08: Code changed slightly from snippet above - check download for difference - basically just to add 52 if week difference goes negative...

Kayak Strongback Option 2

2008-06-14T22:40:00.002+12:00

Just seen in a recent mailer from Bunnings - 4m aluminium planks for around the $NZ150 mark. Surely this would provide a good rigid base for building a kayak.

What Are The Odds?

2008-06-11T20:44:00.002+12:00

You go walking along the Wellington waterfront and... end up on the 6 o'clock news on TV.

Apparently not so low. Check the last interviewee on http://tvnz.co.nz/view/video_popup_windows_skin/1842042 . Ah yes, that's me. Now I must admit this is at least my second performance on the biggish screen. I remember my primary school class being used as in the opening to Nice One Stu - a popular 1970s, kids after school TV programme in NZ.

Fortunately I suspect the video clip will be archived fairly soon. But it's good to know that in little 'ol NZ it's not that hard to end up in the media:)

Kayak Strongback Spacing Planner

2008-06-06T22:34:00.002+12:00

Now getting away from banking, and remembering that I’ve kept my blog as a personal blog first and foremost, here’s an article (hopefully the first of many) on the building of (hopefully) 3 kayaks.

For some time now I’ve been prepping myself for the construction of a fibreglass composite kayak and finally I feel I’m underway! The basic process is to construct a sandwich core with a wood (or other) substrate used to create the form for the kayak with fibreglass on each side providing tensile strength on the inside and abrasion resistance on the outside.

There are two common methods for creating the sandwich core: stitch and glue plywood; and wood strip. I’m starting off with stitch and glue plywood but I have it firmly in mind to go on and create 2 more strip form kayaks. Wife and daughter will surely be bemused as I gift them their own high performance kayak...

The initial kayak is a Night Heron from Nick Schade at Guillemot kayaks. Check the web site and especially the message board he administers. This is an invaluable resource. Once that is done I’ve got my eye on Ross Leidy’s Whiptail, and one of Bjorn Thomasson’s kayaks in particular the Njord, Hunter or maybe Isfjord. No special reason apart from the fact that they just all look so HOT!

So, what about the strongback spacing planner you ask!? Well, when you build your sandwich core form you need a rigid straight guide. This is the strongback and you can create it in a variety of ways. (And no, I don’t need one for the stitch and glue Night Heron but it should make a good flat surface for construction and will suit my needs for the later kayaks.)

The thing is how you go about creating a strongback. I’m very keen on the approach taken by Bjorn Thomasson and Dan Caouette but in building one for myself I used (like Dan) doubled up plywood sheet strips (although in my case I went 150mm wide). The problem is when I went to bend them they seemed in a lot of stress and I wasn’t sure how to even that curvature stress out over the full length. Well, this is where a bit of geometry came in useful and I thought it was worth reporting out for others.

First a diagram; let’s use H to represent the length from the centre to the long end of the strongback, W the distance from the centre to the edge at the widest point (ie half the width), r to represent the radius of the circles that would make up the two sides of the strongback, and h and w to represent the distance up and across at an arbitrary point on the strongback.

How do you get the radius of the circles? Easy thanks to Pythagoras.

So, solve for r to get:

Question is now how to get the width at some arbitrary distance along the centreline of the strongback. No worries,

This time we want to solve for w.

And remember that the width at distance h from the centre is 2w.

Using Excel for my case where I wanted the total length to be 5.00m with greatest width 0.6m I get a table such as this.

Overall Length	Greatest Width	Circle Radius	Offset from Centre	Width
5.00	0.60	10.57	0.00	0.60
			0.50	0.58
			1.00	0.51
			1.50	0.39
			2.00	0.22
			2.50	0.00

Applying this to my sample kayak strongback gives me the following drawing.

Update: 9th June 2008: Thanks to the wizardry of Google Docs you can view a spreadsheet:

http://spreadsheets.google.com/pub?key=pYTbzqIDg8t4ncGmXY2EBIw

Or just view the graph here:

Future of Banking Conference

2008-06-05T09:32:00.006+12:00

Or more accurately titled the 2nd Annual Technology and Innovation Conference - The Future of Banking and Financial Services.

Crikey, bit of a mouthful.

I attended this yesterday and spoke briefly just before lunch. Briefer than intended unfortunately because of the length of the previous presentation. Still, I finished on time and had a ball!

Main messages and comments:

The role of IT, knowing why you're there: operations plus business transformation. Understanding just why you're so critically a part of the business whenever change is necessary, especially disruptive change.
The need to sell and market internally. No innovation will get anywhere when you're not selling it. Just as a quick aside, I noted this come up a day ago in a post from James Gardner on BankerVision. It is hard but you *can* innovate in banks; just understand that the selling will be a great part of the effort.
Handling of conflicts of interest especially product management versus project management. We do this so poorly in banks - the approaches taken for managing the ongoing development of software products that have enormous impact on the customer service offered by the institution is really quite simplistic. Considering that these systems are extensive pieces of IP consisting of code, hardware, ideas in the heads of testers, developers, analysts, architects, and managers, well we really should approach it like a software product vendor. We need to keep the people together and we need to balance off the conflict to build right versus build the right product. It was comforting to hear Jeff Smith from Suncorp also spoke about this within the first session of the day. In fact, quick note hear to say I was very impressed with Jeff's talk. Suncorp have scored well there.
Finally I managed ever (ever) so briefly to talk about complexity and how to deal to it with services. Never had anywhere near enough time to talk about this one. I noticed that the IBM stand in the vendor area had a bunch of dice. I was tempted to grab a dozen, toss them on the floor and ask the audience how complex they thought that system was. Then split them into two groups of dice, put one lot on a desk and another on a different desk with a string between them representing an interface. Then I could show the transitioning of total complexity from multiplicative to additive. This would have given a good demonstration of the power of componentisation. As an aside I have done this before with a group building many sided solid objects from paper cutouts (tetrahedrons, cubes, dodecahedrons etc), I think it worked so there's a potential way of demonstrating complexity in a tactile manner.

A few general comments about the rest of the event...

Gosh what a lot of suits! and what a lot of people from ANZ! I guess their building was just right next door. Lot's of conversation about mobile technologies. Much overrated based on my experiences although I am very interested in the browser interfaces possible with the iPhone and Windows Touch phones. I can see that xhtml compliance with an understanding of modern phone browser capabilities could provide a far faster and more usable method of deploying mobile applications. Java midlets might let you control the buttons, but who cares when there are no buttons! Of all the presentations that I saw I found the most value in Jeff Smiths. Good commentary on people and approach with a good criticism of the prevalence of the waterfall approach to project managing complex system change in financial services companies. A real need for more maturity of project management in our sector.

So all in all a great time and now I'm looking forward to getting stuck into some real work back in the office.

Presentation Roadtrip

2008-03-27T08:39:00.001+13:00

It's been 3 weeks of presenting for me for two separate events; Microsoft Architect Councils in Christchurch, Melbourne, Canberra, Sydney, Auckland, and a BrightStar Enterprise Architecture conference also held in Auckland. The tripping is now over and it's time I put some background material up onto this blog.

For the BrightStar conference attendees I know you can get the presentation from the BrightStar site and you should have been provided with the link and credentials from Brightstar.

For the Microsoft Architect Council meetings, which were public events, I've got a copy of the presentation loaded on my public SkyDrive folder.

Looking back over the presentations it's now easier for me to see the main themes come through and to recognise which have raised the most interest from attendees.

In random order they appear to be.

The recognition of business value in the IT shared service line and the special function of architects to enable change, especially disruptive change in a company; something typically unrecognised in companies as typified by the statement fragment 'the business must make the decision...'.
The explicit recognition of the conflict of interest between building a product right and building the right product and why this leads you away from a single project manager role in your project delivery towards separate programme and product manager roles.
The discussion in the presentations about the normal but arguably short sighted pecking order of market facing teams over shared service lines and why this leads to the failure of brands and companies due to disruptive innovations.
Why companies fail to be creative because they deny their staff the time fully saturate themselves with all the information required to solve a problem, and the time for their subconscious mind to actually figure it out. You can't achieve innovation in a project, projects are defined once you know what you're planning to do.
The importance of addressing complexity and efforts to do this from modular programming through object orientation and service oriented architecture with lots of references to the work of Roger Sessions and his recent complexity papers.

All in all it was a great experience that benefited me probably a lot more than the attendees so a big thank you goes out to Microsoft and BrightStar for getting me along.

From a recent trip to India

2008-02-10T20:41:00.001+13:00

In Mumbai an elephant is stuck in a traffic jam while a young woman wearing dirt covered rags sits on a pile of rubbish and combs her hair, in another world she could be a model. In the far reaches of a rough shop in Agra sits a derelict life size wooden statue of a Hindu god, intricately carved, old and dirty, why isn't it out the front in place of the tacky tourist baubles? On the road to Delhi a camel pulls a cart in the early morning while just off the road men squat and presumably undertake morning ablutions. In Bangalore high technology campuses with green grounds and gates and walls and guards provide paradise away from the daily grime of the city. Contrasts run so great that senses are initially sent awray until a form of acceptance comes in and you now longer lift an eyelid at the sights in front of you. I liked it, a lot.

barcamp agile Wellington and the power of the individual

2007-12-07T13:04:00.001+13:00

http://barcamp.org/BarCampAgileWellington

...is underway and the it's proving more thought provoking than I'd assumed. The development process isn't something I'm directly focused on in my everyday work but it's certainly an important area for me. The biggest theme striking me today is the focus on the individual which I find fascinating.

There was a trend for awhile that worked hard to deliver results through process. Methodology x, prescriptive guidance y. The dehumanisation of the worker.

It just doesn't work that way - we're human after all. It really is all about individuals. Teams yeah, sure - but every member of that team is an individual with ideas and beliefs. You either recognise it and work with that or you ignore at your peril.

I love the blogging world today and I love the way guru characters have appeared that lead through the posting of their ideas. They're enormously important at driving company directions and in defining respect and appreciation from those that watch and listen. Companies such as Google, IBM, Microsoft and Sun all have these people, but there are plenty of companies that don't yet. Big companies, large consulting companies and product companies. I have no faith in their ideas, direction, or products not until I see a passionate, articulate voice from someone I trust.

Schneier on Security: More "War on the Unexpected"

2007-11-23T12:56:00.001+13:00

One can only hope that the increasing information transparency we see occurring on the Internet and in modern computing technology continues...

Schneier on Security: More "War on the Unexpected"

Wellington Weather...

2007-10-18T12:02:00.003+13:00

...is incomprehensible!

Yesterday morning I awoke to howling westerlies and heavy rain in Paraparaumu, then exited the train in Wellington (50km distance) to a warm, dry, windy day with the sun shining!

This morning I awake to incredibly heavy rain, take the train to Wellington and find it cold with a strong damp southerly.

What will it be tomorrow? Heck, what will it be tonight?

Living here you know that you just don't know. Looking on the bright side, at least we have something to talk about.

And as for tonight, best advice I can give is follow the MetVUW site (and keep looking out the window). Based upon the forecast for 7pm, I'm guessing it may have cleared up again!

Heat Mapping

2007-10-16T13:18:00.001+13:00

I've recently had the pleasure of uncovering some new data visualisation -visualization for you Americans- capabilities, in particular heat mapping a measure that's a function of two input variables.

Generating eye catching representations of information is crucial in the corporate world, and yet for the most part, business data visualisation is very poor. Back in the 90s I was busy with research for a physics PhD and I lived by gnuplot and a Windows package called Origin. In transitioning into the business world I also transitioned into Excel. I've enjoyed using Excel and I'm a really great fan of Excel 2007 and especially the data mining tools that Microsoft now provide with it but the problem with the graphs, is that everyone else at work generates graphs that look, well, exactly the same. So, I thought I'd go and take a look at what else is now available.

First off I discovered that Gnuplot and Origin still exist and their capabilities have advanced far beyond what I used when I produced a thesis on a 386 linux pc with.. was it 120MB of disk But, then I discovered Python, the scipy (scientific Python) and numpy (numeric python) libraries, and the MatPlotLib library. Now these looked especially useful because of the image mapping capability which could be used to generate a heat map.

Working with a range of people with different backgrounds in a corporate environment you're constantly trying to re-represent information in differing ways to achieve some connection with your audience. For some people an Excel bar chart works great, for others scatter plots, for others they'll want error bars etc. So you want a range of graphing tools available. Now Excel is good, especially Excel 2007, but there's still a range of visualisations that it can't perform, especially image mapping or heat mapping and some 3D plot types.

Now MatPlotLib can help to fill some of the void, particularly with the image mapping. Check the screenshots to see a range of examples of image (heat) maps and polar plots mixed along with more conventional 2D plot types.

In one example I looked at the pattern of customer usage through our Internet channel as a function of customer age and their total relationship with the company. To do this with MatPlotLib I first collected the data from our database environment, I then used Excel to manipulate the data into an array format (same format as an Excel surface plot), then saved it to a file, I then loaded it into a Python array and used the imshow() command to generate a heat map. I regenerated the graph over successive half year intervals keeping the color range set to run over a constant interval, then combined the images together into a movie with Windows Movie Maker (you can also do this with command line tools such as memcoder which part of the opensource mplayer suite).

The output of this exercise was a movie that showed customer activity through the Internet increasing over time across all age ranges but accentuated by the degree of total transactional relationship they had with the company.

One of the biggest problems you face with generating heat maps/image plots is getting a full range of data. If the value you want to plot is a function of x and y, and you're not actually taking measurements, but just relying instead on historical data you'll probably end up with a lot of missing data points. The imshow() command interpolates data points to smooth over the holes.

The easiest way to get MatPlotLib on your pc/laptop is to install the standard Python distribution and then over the top install the IPython shell. IPython provides you with a scipy item in your programs list which gives you a Python command prompt with all the necessary libraries pre-loaded.

If you're interested in other visualisation tools then I've since found a good background reference at IBM: http://www.ibm.com/developerworks/linux/library/l-datavistools/. It's say's it's for Linux but in reality most of this software runs on Windows (and in fact if you look at the numbers of downloads on some of the packages the majority of downloads are for the WinOS).

The Business Shutdown Statement

2007-09-06T16:56:00.001+12:00

http://www.andredurand.com/2007/05/10.html#a689

It's not a day goes by I get the very same. As if I'm not thinking about the business!

The Value Of An Employee

2007-08-24T21:18:00.001+12:00

Some recent project activity I've been involved with has reminded me of the all too commonly occurring scenario that plays out in corporate offices up and down the land each day; the relegation of the employee to support while the contractors populate the projects. (Disclaimer : I've been a contract resource, an employee, a project manager, and team manager so I've covered pretty much all the bases...)

Now, it never ceases to amaze me that organisations persist in driving project delivery with their contract staff in preference to their corporate staff. The usual argument is to better manage resources and budget for asset development enabling easy capitalisation of time, depreciation of asset cost, and flexible staffing levels with project load, but consider this.

How will your employees feel if they're forever stuck on support and bug fixing while the glory of new project delivery (and the happy celebrations on project completion) are forever the domain of your contractors?
How will you stop your people believing that your employees are second rate staff to the contractors you bring on for the big projects?
How do you successfully manage operation of the project deliverables when the operational staff were not involved in the project development?
How do you continue long term development and idea innovation on a system produced by one team and operated by another?
And just how many projects actually really do successfully deliver generating the actual value originally envisaged in the business cases? Wouldn't it be better to manage the cost up front knowing you can capitalise after the fact?

If you're building a valuable asset then you should be building a commensurately valuable human structure to continue development over time, not just hand over and dash off to the next engagement.

It's a naive view that sees a complex system developed, deployed and left alone. Again, anything valuable at a point in time needs to change to remain valuable in response to a changing environment. It's simply ridiculous to imagine a complex system can be simply bounded by the original project definition.

Relegating employees to operational activity because of concerns over managing budget is a strategic error for any company.

Richard Rumelt on making choices and Disruptive Innovation

2007-08-16T10:08:00.001+12:00

Thanks go to the McKinsey Quarterly Journal for this great interview. You'll need to register (it's free) to read the original article.

Now the title of this post isn't exactly what Rumelt wrote but the theme runs strongly through the interview.

While responding to questions on the nature of corporate strategy Rumelt gives the example of the resurgence of Apple (and really Steven Jobs) through the iPod:

"Jobs didn't give me a doorknob-polishing answer. He didn't say, 'We're cutting costs and we're making alliances.' He was waiting until the right moment for that predatory leap, which for him was Pixar and then, in an even bigger way, the iPod. That very predatory approach of leaping through the window of opportunity and staying focused on those big wins - not on maintenance activities - is what distinguishes a real entrepreneurial strategy."

And,

"Enter Jobs. He was perfectly positioned because he was a bit of an insider in the entertainment industry but didn't have any of those asset positions that were being threatened. He didn't need to make a fantastic leap of imagination into the far future. He found a set of ideas that needed to be quickly and decisively acted upon."

Two other good points in the interview are the power of writing down thoughts in sentences over bullet points and the concept of value denial.

Computerworld Article Response

2007-08-16T09:41:00.000+12:00

Well I was quoted in Computerworld yesterday(http://computerworld.co.nz/news.nsf/tech/DABFB2798D9A6E50CC257336007EAB04) as a result of statements made during my voice of the customer track presentation on Kiwibank at TechEd 07 in Auckland. During the presentation I made a point that platform infrastructure is an important enabling factor in exploiting future opportunities; and that at Kiwibank we were embarking on a desktop upgrade, initially to XP and at a later date to Vista.

During question time at the end of the presentation I was asked to explain in more detail why we were not going directly from Windows 2000 to Vista. The answer is simply that we need to complete our natural desktop replacement to ensure people get a reasonable performance. This is underway and from my perspective, being mostly interested in application deployment, it is not a great inhibitor to future progress. Windows XP does provide organisations with the ability to deploy the .net 3 components for presentation, workflow management, and communication and while I'm sure that from an infrastructure perspective there are many good reasons to deploy Vista for improved management, I'm satisfied with the operating system for application hosting.

The remainder of the article referred to some of the more interesting ways in which we're trying to take advantage of a range of new technologies within the bank; if I had had time in the presentation I could've shown many more.

New Web

2007-08-14T12:42:00.000+12:00

These notes come from Michael Platt's Web 2.0 presentation at TechEd07 Auckland.

Why TCP?
Why HTTP?

Why not UDP and BitTorrent...
This follows on from the SAF06 event and Bill Gates talk. Current web protocols were designed with low bandwidth environments in mind; now there is a high bandwidth environment and the time is right for a disruptive innovation. A new read/write peer to peer protocol could easily supplant HTTP with the PUT/POST/GET verbs.

If we do this then REST could provide a model for a new implementation.

Basic support available in WCF1.0 but it's native in Orcas.

Question could be what REST support exists in Silverlight?

Interesting point raised on process management and the Robotics SDK which contains graphical process designer. In a bidirectional web environment you need process support.

TechEd 2007 Auckland Presentation

2007-08-13T17:38:00.000+12:00

Just finished! I can never tell in advance how the presentation will go until it's performed in front of the obvious. It's only then that you can gauge the feed back and know which areas of the presentation were actually important to the audience.

In this particular case I think the following areas stood out:

Innovation is not the result of projects
My rant on incubators never succeeding
The requirement to sell internally your ideas
The need to think ahead two, three or more years and use that to build your strategy for your technology development

Big thanks go out to Lou Carbone for his keynote, best keynote speech I've seen, and I took a hint and used my time before the presentation to add some pictures - they speak a thousand words.

Main message from Lou's speech was that emotional clues are everywhere; and you're daft if you don't take heed.

Surfacing a new Internet

2007-06-01T10:31:00.001+12:00

The message has been clear for months; the internet has been limited through protocols designed in 1994 andwe're now on the verge of a new electronic world. I watched Bill Gates give this message last December at the Strategic Architects Forum in Seattle and it's now becoming very clear what he had in mind.

First the release of the Silverlight beta with vector graphics; then the Silverlight alpha with a downloadable sandbox; and now Surface. I've been telling people in my industry ever since that the world will be dramatically different in 5 years time. I don't think it's been well understood but maybe now it will be different.

When you think of a core Internet application such as Internet Banking you realise that a customer's expectations of what Internet banking is has only been based upon 10 years of experience at most. Five years from now this experience will be very different. Silverlight and Surface will enable whole new styles of user interaction. Interacting with the bank becomes a flick of the wrist!

Now, one last memory from that conference in Seattle: the statement that the Internet is increasingly becoming a mirror of reality. Seems more accurate every day...

Almost Forgotten

2007-05-28T20:15:00.001+12:00

A few weeks ago it was ANZAC day in NZ; a day of remembrance originally defined by the failed WW1 landings at Gallipoli, but now extended to encompass those Australians and New Zealanders injured or lost during all the wars since. A day also celebrated in Turkey for the creation of a secular state: for one world a disastrous invasion; for another the rise of a modern state.

Now, I usually find these days disconcerting; there seems too great a yearning for days past, as though something about those historic events made us better people, proved our metal, gave perhaps a reason to celebrate our forefather's heroic achievements. It seems a dangerous thing.

Now that might be true but on this occasion, I think I've found something else that warrents mentioning.

My family background on my father's side is Polish. That side of my family came to NZ as refugees in the 1940s and 1950s via the Soviet Union, Persia, and Palestine. Of 1.5 million Poles deported by Stalin in 1939/1940 to Siberian labour camps, only about 700 made there way to start new lives in NZ. The original arrivals consisted of orphaned kids, followed after the war, thanks to the Red Cross, by those close family members that managed to find their children or siblings. It was by this mechanism that what remained of my polish ancestry came together in NZ.

The impact upon families of the trip from labour camps was severe with many dying along the way including members of my father's family. The men of fighting age went into the army, navy or airforce and variously split into those fighting for the West and those fighting for the Soviets.

I recently came across a web site which attempts to document some of the Polish forces during the war, a force which included members of my family. I know that the internet is unreliable but the site seems to confirm what I heard from my family as I grew up.

The Polish forces were a major part of WW2 and yet also now largely forgotten.

The Soviet controlled Polish army fought through to the battle for Berlin and put a Polish flag temporarily on the Brandenburg gate next to the Soviet flags, albiet temporarily as it was rapidly taken down. They numbered 396,000 through the war with 23000 killed or missing in action.

The British controlled forces numbered 255,000 across airforce, army, and navy with 13,000 killed or missing in action.

This means that ignoring the initial invasion of Poland, and the 15,000 officers believed killed by the Soviets in camps in 1940, Poland still fielded a combined total of approximately 650,000 servicemen through WWII making it the 4th largest allied armed forces.

Comparing the casuality rate with that suffered by the US forces during the war is enlightening: 5% of those fighting in the West were killed or missing; 6% of those fighting in the East were killed or missing. As many Poles fought in WW2 under British and Soviet command as did US Marines but they suffered twice the casualty rate (http://www.usmm.org/casualty.html). And that's excluding the initial invasion of Poland by Germany and the Soviet Union in 1939.

Yet you'll never hear that mentioned on the History channel!

With Yalta and Stalin's desire to incorporate Poland into the Eastern Bloc the efforts of those 2 armies was effectively sealed. In 1946 the British command demobilized the Polish army. They were not allowed to join the London victory parade and according to an interview on the Polish Solder website they were given identity cards with "Enemy Alien" written on them.

In the case of my family 3 members were mobilized into the army, navy and cadets. All survived and ended up coming to live in NZ, a country that interestingly fought alongisde the Polish 2nd Corps throughout North Africa and Italy.

I think it's about time that the efforts of these forgotten soldiers were better brought to light.

SQL Server 2005 Data Mining Classification Matrix

2007-05-28T19:55:00.001+12:00

Finally figured it out, so easy, and yet so poorly described in the documentation; the classification model tab in SQL Server 2005 data mining shows you the accuracy of your model prediction. Figuring out how to use it had me confused for quite some time. If you're as daft as me then perhaps the following might help.

The easiest way to discover how to correctly read the classification matrix is to generate a table from your testing data set (you do split your original data into training and test data sets don't you...!?) with the ID, the actual outcome you're trying to predict, and the predicted value. You can do this in the Mining Model Prediction Tab. It's easier there because the option to save to table in your database is accessible from the little disk icon in the top left corner. You choose the ID, the actual and the predicted value in the bottom half of the screen. The first time I looked at this it was a bit of mystery how to drive that half of the screen, I'm guessing you've mastered that part.

Load up the table that is saved from your mining query into SQL Server's query window and count up the number of entries that are correctly predicted; and the number of actual values for each state of the actual attribute. You'll then see how it maps to the classification table. Basically, the succinct description at the top of the table is correct. Columns correspond to actuals; rows to predicted values. The bit they leave out that would've been useful to me is that the total number of cases of any one actual value you get from summing vertically; the total number of predicted cases of a value you get from summing the row.

Singleton DMX

2007-04-19T08:58:00.001+12:00

If you've used DMX to generate predictions from a data mining model you will be familiar with the singleton query. This is where you obtain a prediction for a particular set of input attributes; either within a case table or in a nested table. There was an example of one of these in a previous post.

Now the form of that query was:

select prediction
from data mining model
prediction join
(select (select something as nested row
union select somethingelse as nested row etc) as nestedtable)
as t

Now, what I wanted to do was using openquery() to return all the nested rows, but I did not have an explicit case. How do you make it work?

After bugging the team on the www.sqlserverdatamining.com web site in the newsgroup section the answer was obvious. Just go ahead and create a dummy case. A sample query that works for one of my models follows.

SELECT
    Predict([Model_NaiveBayes].[Bucket]),
    PredictProbability([Model_NaiveBayes].[Bucket])
From [Model_NaiveBayes]
NATURAL PREDICTION JOIN
SHAPE { OPENQUERY(Test, 'SELECT 1 as CaseKey') }
APPEND (
    { OPENQUERY(Test,'
    select 1 as ForeignKey, Term
    from Terms
    cross apply Matches('some long discourse containing many terms that I want to characterise'',''\b('' + Term + '')\b'')')
    } RELATE [CaseKey] TO [ForeignKey]
) AS [Msg Term Vectors]
AS T

In this case it uses the Matches TVF which I describe in a previous post to identify the terms in the text.

Exposing the Regular Expression Match Collection to SQL Server as a Table-Value Function

2007-04-18T08:28:00.001+12:00

For a recent pet project I've been attempting to create a text mining model in SQL Server 2005 to analyse incoming messages and automatically bucket them into one of a number of categories. This follows straight on from the text mining example contained in the SQL Server tutorials.

With the model implemented (using Decision Trees, Naive Bayes etc) it's easy to create a singleton query by hand that has the following form:

SELECT
[Model_NB].[Bucket],
TopCount(PredictHistogram([Bucket]), $AdjustedProbability, 3)
From
[Model_NB]
NATURAL PREDICTION JOIN
(SELECT (SELECT 'some defining term' AS [Term]
UNION SELECT 'another identifying noun or phrase' AS [Term]) AS [Msg Term Vectors]) AS t

But I still needed to extract the identifying noun phrases that make up the terms. Given a dictionary of terms and a length of freeform text how do you find all the term occurences?

Using the SQL Server string functions is painful so I thought I'd try the Match Collection object in the CLR. To expose this you need to perform the following operations.

Firstly, enable CLR integration with

EXEC sp_configure 'clr enabled' , '1'

Then create a SqlServer function in .net to expose the MatchCollection eg

using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using System.Text.RegularExpressions;
using System.Collections;
public partial class UserDefinedFunctions
{
     [Microsoft.SqlServer.Server.SqlFunction (FillRowMethodName="RowFiller", TableDefinition="Match NVARCHAR(MAX)")]
     public static IEnumerable Matches(String text, String pattern)
    {
        MatchCollection mc = Regex.Matches(text, pattern);
        return mc;
    }
    // this gets called once each time the framework calls the iterator on the underlying matchcollection
    public static void RowFiller(object row, out string MatchTerm) {
        Match m1 = (Match)row;
        MatchTerm = m1.Value;
    }
};

Then deploy using Visual Studio, or if you want to do it manually try:

CREATE ASSEMBLY MatchesAssembly FROM 'c:\somewhere\some.dll' WITH PERMISSION_SET = SAFE

CREATE FUNCTION Matches(@text NVARCHAR(MAX), @term NVARCHAR(MAX))
RETURNS TABLE
(Matches NVARCHAR(MAX))
AS
EXTERNAL NAME TextTools.UserDefinedFunctions.Matches

And voila, you can do the following...

select *
from terms cross apply Matches(@text,'\b(' + terms.term + ')\b')

Where terms is a table of noun phrases you're searching for in the variable @text that contains the message text.

Measuring how quickly we learn from our mistakes

2007-04-10T16:55:00.000+12:00

For 10 years I've heard how you need to learn from your mistakes and it seems quite a reasonable mantra. But the problem I've got with learning from mistakes is how quickly do you learn.

Some people and some organisations seem to learn really quickly, others, more like myself, take a while longer, others take the whole of their existence.

So the question I've got is how do measure the responsiveness of someone, some organisation, to learning from their mistakes?

What metric could we use to tune performance?

Not sure, but maybe we can think something up.

Whales off the Beach

2007-04-08T20:44:00.001+12:00

Wonderful sight - whales off the beach, maybe 300m from shore on a beautiful still, sunny day. Not sure what type, but I don't think they were Orcas - they didn't seem large enough. The shame of it is that if it weren't for the cold I've had for the last few days, my plan was to be out kayaking this morning. Wouldn't that have been amazing!

Oh, perhaps a bigger shame, especially from the point of view of the whales, was the idiot out there going round and round in circles on a jetski. Confirms every opinion I've ever had of them.

Update: Talking to someone this morning I found out they were indeed Orcas. Now I really wish I'd been up to going out on the kayak.

The value of a demonstrable prototype

2007-04-06T10:04:00.000+12:00

I've just been reading an interview by Ron Jacobs with Scott Guthrie; it's up on The Microsoft Architect Journal (http://msdn2.microsoft.com/en-us/library/bb266332.aspx).

What's caught my attention is the importance Scott put on having a demonstrable prototype of their ASP.Net technology early in the development. "There were three or four of us" and yet they managed to create one of the key product offerings of one of the world's largest companies. A truly remarkable outcome.

Now I'm sure that the team itself was extraordinary, but still I bet in many companies they would have been driven into the ground under the weight of the stakeholders. The more people involved, the more diluted the good ideas become and the greater effort needs to be put into selling ideas and managing the stakeholders. The processes companies typically put into place to manage new delivery put a stranglehold on true innovation.

'...That's one of the successful things that we did with ASP.NET. We said that we're
going to throw away every line of code we're going to write for the next couple
of months. Let's all agree on that. We're not going to say, "Oh let's take this
and adapt it; we can clean it up." No. We're going to throw it away. We're going
to "deltree" this subdirectory at some point, and that way we can be more
adventurous about trying new things. We don't have to worry about making sure
that everything's robust because it's going to be in the final version.

We actually did that for a few months and said, "We're done, delete it; let's start
over from scratch; now let's write the full production code and make sure we
bake in quality at the time." I think a lot of teams could benefit from that.
The hardest thing is making sure you delete the prototype code. Too often,
projects develop with "Well, it's kind of close." It's very difficult to start
with a prototype and make it robust. I'm a firm believer in starting with a
prototype phase and then deleting it.'

The typical corporate company approach would not have sustained this type of development.

It would be an incredibly gifted individual that could convince a steering committee to support expenditure on experimentation when little or no artifacts could be easily capitalised at the end of a 3 month development. Most companies want functional delivery early with a clear line of connection from end user requirements to final deliverable.

Scott talks specifically about the value of a prototype to help sell the idea.

"We certainly had to persuade a number of people along the way. One thing we
did early in the project was get running code in prototypes that we could show
people. Often when you're working on a project that's new or something that
hasn't been done before, it's easy to put together a bunch of PowerPoint slides
that sound good, but it's especially valuable to actually show code and walk
people through code that's running. Not only does the prototype prove that it's
real, but also you just learn a terrific amount by doing it."

Perhaps their greatest achievement was the successful marketing of the development from initial concept through to the current deliverables. The early prototyping and demonstrations were obviously a critical part of this achievement.

At the end of the day all of this makes great fodder for the creation of a myth list: the My Myth List. My first two myths are going to be:

End users know what they require
Product delivery is a linear process

Because it's apparent that people don't know what they want. It's the visualisation of what is possible that provides a basis for requirements. And product delivery is never linear.

I'm sure I'll think of more over the next few weeks.

21C in the pool on April 1st

2007-04-01T22:10:00.001+12:00

Global warming? Well it was a late start to summer...

Commotion at the beach

2007-04-01T22:09:00.001+12:00

One of the Kapiti island ferrys was caught out this afternoon. I joined in a small crowd to watch as the tractor that was supposed to haul it out of the water got stuck in the sea. The engine appeared to have failed and in a scene reminescent of the Little Digger, 3 tractors lined up in a row to try and tow the stranded tractor and ferry out, without success. They temporarily gave up on the ferry, hauled out the tractor before the tide came in further, then towed the ferry onto the shore to allow the passengers out. A bit of excitment while I was down at the playground with Lil!

ARCast video up for Kiwibank Case Study

2007-03-22T21:23:00.000+12:00

I've just found out that Ron Jacobs has posted a video interview of myself and two members of my team on his ARCast site.

The interview was based on the Kiwibank Case Study which we undertook last year with Microsoft. The interview takes a broad look at how Kiwibank has used technology to help it go from 0 to 12% of NZ's population in 5 years. I speak along with David Grahame and Sushil Kamanahalli. David is the client applications architect and Sushil is the service layer architect.

Special thanks go out to Mark Carroll for helping to organise this and to Ron Jacobs for the interview and the work to put it together.

Fashion at Government House!

2007-03-22T21:05:00.001+12:00

What a fantastic night. I had the absolute pleasure of seeing my wife, Miriam Gibson, present her winter collection womens fashion wear at a charity event held at Government House in Wellington on Tuesday night.

It was magnificent. The show was held in the Ballroom with the Governer General, His Excellency The Honorable Anand Satyanand, and his wife, Her Excellency Mrs Susan Satyanand, army staff in regalia, members of Rotary and the charitable organisation, Refugees as Survivors, and over 200 guests who had come to see the launch of the 2007 winter range and support a good cause.

And I was very, very, very impressed. Miriam, Victoria, Sue, Sarah, Veronica, and all the models - you all did a helluva job!

And standing up there following up the Governor General, the local Rotary head, and representatives of the charity with a speech on the podium with microphones, photographers, and press present. Crikey - my recent speaking engagements pale into comparison.

The range was fantastic so I definitely recommend checking it out: head to the stores in Margaret Rd, Raumati and Hunter St, Wellington, or check out the web site to find out more (pictures from the event are promised over the next few days). Nothing for the guys, this is ladies only. And don't forget the charity - they're a very worthwhile cause in this country which has been the fortunate recipient of many refugees in the past, including the Polish half of my ancestry...

Rod Drury on investment for IP

2007-03-17T17:44:00.000+13:00

http://www.drury.net.nz/2007/03/17/building-intellectual-property/

Notice the graph that Rod has obtained of normalised patents per country. See how NZ sits at about 0.5 in 22nd place. Finland rides at the head on 4.5 and the OECD average sits at just under 2 which means it's closer to Finland than NZ.

This is a graphic representation of the under investment in R&D in NZ compared to other countries. This isn't about centralised R&D run out of the government, but about the failing of companies to invest in product development. Great thing that Rod pointed this measure out.

The Business, You and Me; Get It Together!

2007-03-17T16:36:00.001+13:00

If you've read some of the previous posts you may have realised that I currently work in the IT area of a bank: the chief architect role at Kiwibank in fact (as those who attended the keynote at the recent Microsoft NZ Tech Briefings, or have read our Microsoft case study will know). Prior to Kiwibank I worked for a year at ENZA, then before that a few years at Deloitte Consulting, and prior to that I undertook a physics PhD from Victoria University in association with Industrial Research Ltd, a Crown Research Institute.

Each of these organisations has taught me a little bit more about how people work together and what makes us succeed in delivering. It has also highlighted the precarious and unappreciated position that the shared service line holds, especially the IT shared service line.

ENZA was an organisation that went during the year that I was there, from an external market focussed apple and pear exporter with producer board status and mandated export control, to a grower focussed commercial entity that retreated from the political environment of Wellington to the safety and security of the grower stronghold Hastings. As a company it had two strategic directions ahead of it, either strive to become a global category specialist based perhaps in one of the major trading hubs, or become a grower focussed organisation that showed it's value at the farm gate. Retreating was certainly the less risky option and it was that path that lead to the rationalisation with Turners and Growers in a 2003 merger.

At ENZA the (naive) question faced in 2000 was what part of the company represented the future of the business: the export facing arm or the grower facing arm. IT at this time was treated as a cost centre run under the finance group. And seeing as the organisation ran SAP it was certainly some cost.

In the middle of 2000 while working at ENZA I happened to come across a chap at Turners and Growers who explained that they were running a home grown software suite that was at the end of the tether. Myself and two others actually went and visited them in Auckland and it was apparent that they had problems. The obvious thought to the three of us was imagine combining the two organisations and taking advantage of the SAP implementation at ENZA. It would be a great asset right?

Well, that is indeed what happened. Someone out there saw the synergy: did Tony Gibbs think about this I wonder? Whoever it was they certainly knew a thing or two about SAP and IT in general. I note that there's now a customer success story about SAP and Turners and Growers on the SAP web site.

Prior to that experience I was at Deloitte Consulting which presented me with opportunities to work in companies and organisations across government, the health sector, and telecommunications. Now for all the minor gripes that many of us had there at the time - long hours etc etc - one thing definitely stands out: the value of good people. I did work with some very good people and while often thrown in at the deep end we did ok. A small group went on to do especially well, witness Trademe and AMR. Being a consulting group we didn't have much of an IT function. Information Technology was a core attribute of our service line and overlaid across the group was a matrix model representing sector and service advocacy. I think it worked well.

In comparison many of the companies we worked for had well defined structures with strong vertical focus on product delivery. You'd walk into these organisations and there were barriers everywhere. Internal development was hardly ever undertaken. Individual business units would occasionally issue RFIs, RFPs or succumb to the salesmanship of a clever vendor. Work would always proceed on the basis of a long chain approach that ensured the people that understood what was possible never had a chance to really influence the development of new ideas in the organisation. Certainly not outside of the immediate business unit.

In these environments you'd always here the catch phrase: "it's up to the business to decide", or often from the PMs/BAs, "we have to listen to the business", or the classic "the business wants...".

The depressing thing is that this is more often voiced by the staff of the IT department than the business units themselves. If people in an IT department don't think they're value contributing then they deserve to be treated as a cost centre and outsourced to the likes of EDS or IBM.

It's a personal mission of mine that my application delivery group does not come out with the same nonsense. It's the innovation that comes from those that know what's possible combined with the people that can advocate for a customer, and those that know the financial constraints and tools, and those that can market the products that creates value contribution in a company. Any organisation that forgets the value of the combined talent of all its resources deserves to lose market value.

Oh, and Industrial Research? A depressing environment of disillusioned scientists with ideas but no knowledge of how to commercialise them...

Wellington Microsoft Tech Briefing

2007-03-15T22:08:00.001+13:00

It was another great event yesterday and it's just fantastic to be seeing so many people. Wellington is my home town so there were plenty (plenty) of faces I recognised in the audience. A big thank you for opportunity goes out to Mark Carroll, Rebecca, Sean, Dean, Carol and all the others. Being the second time through the short speech I make in the keynote gave me a chance to think twice about the message I was trying to present.

And it's confirmed in my mind that the main message I want to get across to people is actively think about opportunities in their organisations, experiment with technologies and tools, and work on marketing any ideas they may come up with. It's how to make things happen and you know, life is too short to being dumb stuff when you could be doing cool stuff.

As a great way to finish off the day I got to attend the Microsoft Architects Council meeting at the hotel. I like to attend these events as it's a good chance to catch up with people I don't see every day. We have an active group of people up and down the country that attend these events and the chance to explore ideas is never something to pass up.

Next week is the final Tech Briefing in Christchurch. I can't wait for this as I know by then I'll be wanting to tune the message once more!

Prioritisation: Apples and Oranges

2007-03-15T21:15:00.001+13:00

Every company seems to share the ritual of the prioritisation session. It has a common format and a common process.

Each business head gets to voice their opinion on what's most important to them. These are dutifully collated into a master list and then a discussion takes place to rank one above the other based upon some number of criteria; typically financial, customer experience, and compliance. Finally the agreed list is circulated for action.

It's value is normally limited because importance is not a good measure for prioritisation.

Importance is a measure of emotional conviction. It is a broad term that can mean many things depending upon the subject. The importance of a programme of work is not the same thing as the importance of an immediate fix, or the importance of a process review, or the importance of addressing a particular risk. In each case the definition of the term, importance, differs and therefore it can not be used for comparison. It is accurately a measure of emotional response, but I doubt little else.

What is the alternative? Perhaps it's better to ask what's the point.

The purpose of the prioritisation session is to allocate scarce resources. Scarcity can only be resolved through a process of trade-off (this is text book economics). What complicates the task in an organisation are the differing time requirements, resource specialisations, and dependency effects.

We have a limited ability to weigh up the combination of time, resource, specialisation and dependency factors to determine how limited resources can be applied to a range of competing tasks. Our minds have to make best guess estimates and the wider the scope the greater the problem (I bet someone out there can prove this is a power law expansion).

Which drives us to smaller delivery teams to reduce the scope of the problem.

So what should the prioritisation session be?

Perhaps to set areas of focus and define the criteria for prioritisation. I doubt little more.

Innovation in the Corporate Environment

2007-03-13T18:32:00.001+13:00

This is a hot topic for me. I work in one of these environments and I'm involved in a fairly traditional (these days) role of enterprise architect: nominally responsible for the overall design of systems to ensure they meet business needs, and typically driven more from the perspective of policy and process than the introduction of new ideas. I'm afraid I'm not a very good enterprise architect.

Being on the back foot and not contributing to the ideas that form the basis of many of the commercial opportunities seems quite daft to me.

My aim in life is instead to communicate the opportunities of technology, or in fact, any thing that comes to mind actually. You know I went through university in a rather clueless manner and it's only now I see the possibilities of the methods taught to me at the time. There is just so much out there that can help give you an edge. (Wish I'd paid a bit more attention in the lecture rooms....)

Ensuring technology meets business needs is never going to see innovative solutions deployed, it's never going to see solutions applied when problems aren't yet realised to exist. How often have we looked around and seen only in retrospect that we missed the ball completely (trust me in my 4o years it's happened a helluva lot!).

So, yes, innovation is a hot topic for me.

Today I read an article syndicated from some offshute of The Economist called, innovatively enough, The Economist Newspaper, referring to the demise of traditional R&D and the rise of a new form of directed innovation concentrating on the D aspect. I'm not against this, a lot of the great ideas out there (that I've missed the boat on) have typically only been a couple of years ahead of everyone else's thinking. But the fact they were ahead proved a significant advantage.

The article began by looking at the output of Vannevar Bush, a gifted thinker in his own right, and an advisor to the Roosevelt and later administrations. It was Vannevar that spearheaded America's implementation of government and military funded R&D from the 1940s to the 1970s. "Industry is generally inhibited by preconceived goals, by its own clearly defined standards, and by the constant pressure of commercial necessity," he wrote in 1945. It still rings true today.

But, the days of the big labs are gone. Bell Labs has fallen apart, IBM's research labs are far more highly directed now, Microsoft Research nominally allow free reign but then look at the narrow range of papers on their site. Where's all the Research and what can a smaller company do?

It seems to me that in the mid-size corporate environment (with a few hundred staff) there is one classic failing: the creation of the product delivery chain. You know the one. It starts with the customer on the street, then there's marketing, then BAs, then project teams, and at the end of the line, the implementors.

Nothing driven down such a long chain will be innovative. The people at the end of the line act out a Dilbertesque cartoon living in perpetual frustration. The customers only get what they ask for; and no more. The nimble, smart companies out there create their own new niches and the slow ones are left to play catch up.

I think the secret to this is to keep team sizes down and allow small teams to experiment with ideas ensuring that at an overview level there is a process of nurturing and selection. Allowing failure to occur has to be an integral part. "Please fail very quickly - so that you can try again" says Eric Schmidt from Google.

Ground breaking products and processes are always due to the conceptual insights of individuals. So it should be the task of every innovative organisation to provide a mechanism to foster the intellectual output of their staff.

Response Time Distributions, IIS Log Files, and the question of the Missing Events

2007-03-13T17:10:00.000+13:00

Over the last year I've been involved in a number of investigations attempting to find bottlenecks in systems consisting of clients, web service hosts, and databases (usually containing application logic in addition to data). The details of each system's implementation is not especially important to this discussion because what I want to do here is just relate one of my recent experiences regarding measurement of response times. You might like to check if you get similar behaviour.

Firstly, let's just describe the typical situation I find myself looking it. It's very generic, I'm sure the same thing will apply to you. I usually have some client systems accessing a service layer hosted in IIS6 talking to a database server (usually with significant embedded application logic).

The Problem
The problem (or my lesson in this case) is how to interpret the numbers you get from the IIS log files on the web service hosts. These files give you HTTP request duration and the arrival time of the request. Now what happens when you naively plot a distribution of the the duration (ie request execution time)? You might expect a nice symmetrical peak centered on some value, or you might get something such as the following.

I found that I was consistently getting this sort of shape across different types of requests. This isn't really all that unexpected. Each URI in the log file corresponds to a web service against which a number of web methods may be called. The web methods may have significantly different response times so the graph of the service call is really just a summation of all the individual web method calls. And in fact we've now implemented duration timing on each individual web method call, and in fact we get a much simplified distribution centred round one peak.

Anyhow, while I was looking into the double peak I decided to look at the arrival rate and compare that to the duration of the call. Theoretically you should get a Poisson distribution for random, uncorrelated events and on those occasions when there were many simultaneous arrivals you'd also expect the response time to slow down (although whether this was linear/non-linear is another question).

So, I looked for a period of time during which we have fairly constant activity and chose 2 hours in the middle of the day.

You can see that there's a nice distribution with the expected shape centred on 4 arrivals/second. Of course the IIS log files only record data when a request actually arrives. Looking at the bar graph you'd therefore naively expect about 300 one second intervals over the two hour period during which no requests arrived.

And this is where I got a surprising result.

So, it appears that the IIS HTTP arrival time is not accurate. I think that what I'm seeing here is that IIS is already queuing the requests up for processing - presumably because processing downstream is taking too long.

If you get anything like this, have seen it before, or have a bit more knowledge of what is going on I'd love to hear about it.

Powershell versus Perl

2007-03-12T21:27:00.000+13:00

I've just been preparing some data for a post which I've been meaning to put up for a few months now. The data comes from an IIS log file and I need to pull out of it the time of the HTTP request, the HTTP service (uri-stem), and the duration of the request (time-taken). In the past I've always used Perl for this sort of task taking advantage of the regular expression syntax to extract my chosen data elements. For a 50MB sample log file (all logging options turned on) this takes approximately 12 seconds.

The Perl I've just used to test this follows:


use File::DosGlob 'glob';
use File::DosGlob 'GLOBAL_glob';

@logfiles = glob "ex*.log";

for my $logfile (@logfiles) {

  open(INFILE, "$logfile");
  $logfile =~ s/ex//g;

  open(OUTFILE, ">$logfile");

  while() {
      if (m"^(\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d).*/(.*\.asmx).*\s(\d+)$") {
          $file = lc($2);
          print OUTFILE "$1\t$file\t$3\n";
      }
  }
  close(INFILE);
  close(OUTFILE);
}

Now, since I've started using Powershell recently to extract and manipulate data for analysis I thought I'd also try the same thing with that. Note I'm just a beginner at this so I could be doing this the wrong way but here's what I tried:

Get-Content ex070223.log |

foreach-object { if ($_ -match "(?^\d\d\d\d-\d\d-\d\d\s\d\d:\d\d:\d\d).*/(?.*\.asmx).*\s(?\d+$)")

 { & { $matches["occured"] + "," + $matches["service"] + ", " + $matches["duration"] } } }

The regular expression syntax is very powerful, I like the named matches - I guess this is a straight .net runtime feature, but I'm easily impressed. However, the time it takes to complete is abominable! It took 20 minutes to complete where the Perl program took 12 seconds.

Still, the power available from the command line is impressive...

Gadgets

2007-03-07T20:51:00.000+13:00

I'm having a relaxing evening now after an exciting day mostly spent in Auckland at a Microsoft Tech Briefing. A great experience for me as I had the opportunity to be part of the key note speech. I happen to work for an up and coming NZ bank and I was able to present some of the experiences I've gained from my time in the bank - from being a member of a 20 something project team in 2001 - before the bank establishment - to the present day: an organisation of 700 with a 6 month profit of $11m.

The experience of being part of the key note was enlighting in itself. The nervous energy beforehand, the videos, sound, lighting, the 900+ people in the room, being part of a team that consisted of some much brighter people than I, it was great. And I still have Wellington and Christchurch to come over the next 2 weeks! What a buzz! I've never spoken to so many people before. For better or worse, the nearest thing to it in my memory is being 14 and being asked to recite a poem at my uncle's funeral. He was a very popular person and the church was packed. As sad as the event was I still remember the energy of the occasion clearly.

Of the event itself, and the reason for this posting, was the cementing in my mind of the importance of Vista's new gadgets. Jeremy Boyd of Mindscape demonstrated a remarkable gadget for the approval of Vista Dreamscene videos on a community site (can't find it - maybe they haven't put it live on the net?). This particular gadget used the capabilities of WCF to securely connect to a service and perform an operation. Working as enterprise architect at a bank you can imagine my interest. I've been harping on about this at work for a while now and recently noticed a blog post from someone I've met that used to work as a banking consultant at the Microsoft Sydney office, (James Gardner). It's just a matter of time... will we get there first?

The question of who get's there first was actually one of the main themes of my presentation. Looking at the way in which innovation can occur. Looking back on my historical work experiences I can now see how large corporates fail so often at delivering innovation. They start off so nimble and quick and then slow down to a near grinding halt. Achieving change becomes increasingly difficult because of the burden of process and competition of people. Does it have to be this way?

I'm sure I'm going to post more on this but it seems to me that there's a lot to be learnt from the field of R&D in traditional high risk, high reward industries and the application of R&D to service based industries. I know the potential margins haven't traditionally been seen as high enough to counter the cost but I think the time is right. I believe the risk reward matrix is increasingly favouring small experimental developments to highlight problem domains and visualise potential solutions. If we just manage to do those two things we'll be making downstream project delivery so much better (let alone considering the commercial benefits).

I strongly feel that the opportunities for applying technology are stronger today than ever before. It's the technologists who are currently creating the business models of tomorrow, not the business school graduates.

In the meantime, I recommend following JB, JD, and Andrew at Mindscape - they are a clever bunch of guys. I'm sure they'll go far.

SAF 2006 - A Retrospective View

2006-12-07T19:51:00.001+13:00

It's a few days since the event now, the small details are beginning to fade away, and I can look back now and think about the overriding themes. Of these, one strikes me more than any other: a statement from Garry Flake which went roughly like "the Internet is an increasingly complete reflection of reality".

Garry's presentation ("How I learned to stop worrying and love the imminent Internet singularity") took a look ahead at the future. He picked up on an idea published in 1993 by Vernor Vinge: the idea that the exponential speed of technological improvements will produce super-human capabilities, making the future unknowable, an idea that I think harks back to Arther C Clarke and the concept that any sufficiently advanced technology constitutes magic. Vernor coined this occurence a singularity because we have no knowledge of the future beyond that point. This is an interesting thought, especially with respect to the Internet. Garry highlighted (as had Charles Fitzgerald in his earlier presentation, "Software and Services") that information technology is most strongly advancing in two areas: memory and networking. Both of these are improving well ahead of the rate of improvement of processing power. (But as for other areas of technology, I personally doubt we're making such significant progress. Don't you think that living from 1900 to 1970 would have been just as remarkable as living from 1940 to the present?)

With a near future characterised by a massive increase in easily accessible storage and network bandwidth it shouldn't be surprising that more and more information about our daily lives and the world around us will be stored and catalogued for later use. The stored information will increasingly reflect the detail of the real world and the Internet will increasingly become a reflection of reality.

To me this is a wonderful concept because it means that we have a representation of reality that's accessible, can be queried, and is an enabler to learn about our world and improve it. To business it should also be wonderful because it opens up opportunities in many more ways than are possible now: mining, characterising, segmenting, predicting, combining, and in general just responding better to customer needs.

The reflected electronic world of information and the services that operate on that information will also provide value from the integration across organisation boundaries. If you can spot the opportunity that links a group of disparate services together you could be clipping the ticket to your first billion. This is the basis of the mashup and we're sure to see making unexpected outcomes from the smart combination of diverse services and information.

The feflected reality was not the only nugget to come out from the SAF, there were also several other ideas that came through the presentations and roundtable workshops I attended.

Bill Gates highlighted the limitations and short life ahead of the existing web browser model; Ray Lane emphasized the uneconomic application development environment present in the US and the likely impact of globalisation; Norm Judah gave a great overview of how Microsoft manage their own internal services; Garry Flake presented on the Innovators Dilemma, and there was a lot of discussion across all presentations on the long tail effect.

Bill Gates' brief discussion of the future of the browser came up during the Q&A session at the end of the SAF. He made the point that the web browser was designed in the 1990s to fulfill a need to deliver an interface to a system over a limited bandwidth link. That environment is not true now and we know it will be increasingly invalid going forward. Network bandwidth is increasing rapidily so you can envisage a future where you connect to a system on the Internet and a much more functional application interface is downloaded and run on your device using some sort of application hosting environment (perhaps based on a virtual machine created for the session, or a sandboxed machine such as you get with .NET and Java). The standards for this form of interactivity have a long way to go but the direction is clear. I wonder how long before this will impact the next generation of Internet sites?

Ray Lane's presentation of the economic problems of the application development market highlighted the large number of existing small development companies, the available venture funding, the commoditization occuring in the development process and as a result the market contraction we can expect (up to 70% of existing companies going). However, Ray also highlighted the new opportunities created by the long tail and networking effects hence the name of the presentation, "The Personal Enterprise".

Norm Judah's presentation of "The Last Architectural Mile" showed us how Microsoft run their own services. This eye opener showed them using their own toolsets to manage their data centres with millions of servers. The use of the Business Scorecard Manager to present their current status was great as was the simple idea that success of failure occurs in the last mile. Delivering into production is so important and yet, while complex IT systems are designed with great thought and effort, the human processes that operate those systems are often not.

Garry Flake's singularity presentation also introduced the innovator's dilemma (first coined by Clayton M. Christensen and described in his 1997 book The Innovator's Dilemma), that innovation comes from the bottom not the top. As companies become established new start ups innovate and then take over the entranched company's market. Companies that survive show an ability to sustain innovation by being prepared to make significant change (or destroy themselves as Garry put it). I wonder to what extent you can say that we've seen this with the likes of IBM and Microsoft? Certainly IBM have moved a long way from the days when the mainframe was their core deliverable. One thing that does mark both companies is their massive investment in R&D, billions of dollars per year. Garry is an example of the impact of Microsoft Labs, his team form Live Labs, a collaboration with MS Research that is developing the functionality you can now get now through www.live.com.

The concept of an increasingly accurate reality represented on the Internet was an overriding theme throughout the event. The impacts are diverse; the wealth of information, the potential outcomes from better analysis, providing richer choice, responding better to people, clustering, trending, making decisions at the edge, networked opportunities, the long tail. I've tried to put some order to these ideas in the bullet pointed list below and in the posts under this you'll see the notes I recorded at the time of the presentations. Some of these notes I posted on the Internet as soon as I'd finished writing them in the lecture rooms, and others I posted over the last couple of days from the notes I'd taken on my laptop. Hope it's helpful.

Change Drivers: Opportunity Creation from Changing Technology

Storage and networking improving much faster than local cpu
Parallel processing increasingly applied to overcome limitations in cpu speed increases
This is opening up new opportunities due to network effects
- The long tail
  - The affect of the many in the tail can be greater than the few in the head
- The internet is increasingly mirroring the physical world

The Internet as a Mirror of Reality

Responding to people
Understanding people
How?
- You can't manage what you can't measure
Analytics
- Application to numerous areas
  - Process improvement
  - Understanding and responding to human behaviours
  - Processing of different data types
    - Text
    - Numeric
    - Image data
    - Sound
  - The internet increasingly mirrors real life - the data on the net is becoming an increasingly complete representation of real life
    - Brute force statistical analysis
      - Trending
      - Clustering
Personal choice
- Make options available
- Natural selection and evolution
Beware human faults
- The Innovators Dilemma
  - Look down as well as up
Take advantage of the masses - the long tail
- Opportunities exist due to the accessibility of the many
Decision making at the edge
- Responding to your customers at the edge of your organisation versus centralised product development
- Responding at the edge is equivalent to greater customer focus
- Create networked opportunities
  - Connect individual to individual
- Great local example - Seattle traffic reporting via the internet