Heat/Image Mapping with .Net/F#

As part of a demonstration to the Wellington .net user group this week I put together a demo showing a heat mapping of the performance of a transactional system as a function of time of day and arrival rate. My objective was to provide a simple eye balling test of whether performance was being impacted more by daytime customer loading or by competing overnight batch processing. I’ve tried a similar task before with Python and MatPlotLib but this time I wanted to see how easy it was to accomplish in .Net/F#.

The answer was ‘very easy’.

The interactive F# environment in VS2010 gives you JIT compiled code so it’s very fast to work with. The data structures/collections available within F# make importing and manipulating data simple – once it’s in memory you spend most of the time using filter, map and fold operations on elements – explicit recursion or iteration is rarely necessary.

I found I started with sequences then moved to arrays as data volumes increased and performance slowed. Array.Parallel.map/mapi seemed to make a significant difference whenever the operation being performed was of order a few milliseconds. It’s pointless parallising when the operation performed is short – the overhead of managing threads must just be too great.

Most of the effort was spent aggregating and binning the data, the actual image prep was trivial. The following assumes you have an array of data to display.

open System.Drawing
open System.Drawing.Imaging
open System.Windows.Forms
// adapted some code here from http://thecodedecanter.wordpress.com/2010/04/30/modelling-the-2d-heat-equation-in-f-using-100-lines-of-code/
let toBitmap (arr:Color[,]) =  
    let image = new Bitmap(arr.GetLength(0),arr.GetLength(1),Imaging.PixelFormat.Format24bppRgb)  
    for i=0 to image.Width-1 do 
        for j=0 to image.Height-1 do 
            image.SetPixel(i, j, (arr.[i,j]))  
        done 
    done 
    image 

let maxInArray2D (ar:'a[,]) =
    seq{ for r in 0..Array2D.length1 ar-1 do
            yield Seq.max (ar.[r..r,*] |> Seq.cast<'a>)
    } |> Seq.max

let intensityMap intensity = Color.FromArgb((int (intensity * 255.0)),0,0) 

let bitmap imageArray =
    let max = imageArray |> maxInArray2D
    imageArray 
    |> Array2D.map (fun f -> f/max)
    |> Array2D.map (fun f -> intensityMap f)
    |> toBitmap

let ShowForm (f : Form) =
#if INTERACTIVE
    f.Show()
#endif
#if COMPILED
    Application.Run(f)
#endif 

let BitmapForm bitmap = 
    let picBox = new PictureBox(BorderStyle = BorderStyle.Fixed3D, Image = bitmap, Size = bitmap.Size, 
                                Dock = DockStyle.Fill, SizeMode = PictureBoxSizeMode.StretchImage)
    let form = new Form(Text = "F# Connection Performance versus Connection Rate and Time", Size = bitmap.Size, AutoSize = true)
    form.Controls.Add(picBox)
    form

bitmap surfaceArray |> BitmapForm |> ShowForm

If you don’t have many points that’s going to generate a small image which you can expand by dragging the corner and Windows will magically interpolate for you. I’m sure there’s a way to explicitly do this with the BitmapForm API as well – just didn’t have time to figure that out.

If you want to interpolate yourself, you could try something simple like a bilinear approach:

let bilinearInterpolation f00 f01 f10 f11 x y =
    let boundingBox = Array2D.zeroCreate<float> 2 2     
    boundingBox.[0,0] <- f00
    boundingBox.[0,1] <- f01
    boundingBox.[1,0] <- f10
    boundingBox.[1,1] <- f11
    let A = RowVector.ofList [1.-x; x]
    let B = Matrix.ofArray2D boundingBox
    let C = Vector.ofList [1.-y; y]
    A * B * C

Or why not just double the resolution?

// take an array m x n and return (2m-1) x (2n-1)
let interpolate (A:float[,]) =
    let B = Array2D.zeroCreate<float> (2*(A.GetLength 0) - 1) (2*(A.GetLength 1) - 1)
    for row in 0..((A.GetLength 0) - 1) do
        for col in 0..((A.GetLength 1) - 1) do
            B.[2*row, 2*col] <- A.[row,col]
        done
    done
    // 
    for row in 0..((B.GetLength 0) - 1) do
        if row % 2 = 0 then
            for col in 0..((B.GetLength 1) - 1) do
                if col % 2 = 1 then
                    B.[row, col] <- (B.[row,col-1] + B.[row,col+1]) / 2.
            done
        else
            for col in 0..((B.GetLength 1) - 1) do
                if col % 2 = 0 then
                    B.[row, col] <- (B.[row-1,col] + B.[row+1,col]) / 2.
                else
                    B.[row, col] <- (B.[row-1,col-1] + B.[row+1,col-1] + B.[row-1,col+1] + B.[row+1,col+1]) / 4.
            done
    done
    B
 
// now I can do this...
bitmap (surfaceArray |> interpolate |> interpolate |> interpolate |> interpolate)|> BitmapForm |> ShowForm

And generate results like this (where connection rate increases down, time goes across, and redness indicates duration of calls):

F# for Analysis: Wellington .Net User Group Meeting Wed 28th July

Some brief notes for anyone attending this session.

There’s a brief slide deck here.

There’s sample code working through the basics of the language here.

Sample code for the cross correlation of two time series in F# here.

I’ll put a separate post together for image generation of a heat map showing web performance data.

System Center Operations Manager and BI

Last week as part of a presentation to the CIO summit in Auckland I had a bit of fun criticising conventional corporate BI efforts as being too focussed on data management. Basically, the argument went that companies spend too much of the budget worrying about getting the data right and not about enabling people to use it – and that ‘easy to use’ BI front ends are a pointless exercise. It’s not a hard argument to make – you can expend a great deal of time and effort on staging, transforming, cubing, KPI development, and reconciliation quality checks – usually driven by the fact that corporate BI is focussed on financial measures which drives people to want the data correct to the cent.

The natural question that comes back in response is what should be done instead?

Operations management using a model driven tool like System Center Operations Manager provides some useful pointers.

Think about it – what’s operations management?

= Data + Model + Health logic + Alerting for response

What’s BI?

= Data + Model + Business Health Logic + “Strategy execution” for response

What’s smart about model driven operations management?

The tools (like SCOM) monitor systems that can produce huge volumes of data; they give that data context with hierarchical models (à la cubes) that can even be graphical (LiveMaps/Visio 2010 diagrams); they provide health indicators (cf KPIs) and they empower end users by making the information available for more detailed analysis at the desktop.

So how would you put this kind of model into action for BI?

How about mirrored true to source data sources using cheap databases (maybe SQL Express/MySQL). You’d need a grooming schedule to stay on top of the data volumes – perhaps and agent model just like the operations management tools have.

To collect the data from multiple sources, and give some context a cube model is attractive, but what about implementing with PowerPivot. It puts the end user in charge. They can create health indications in an Excel like environment; and they can submit back into a shared (Sharepoint) repository.

I’ve got to present at the longwinded FST 3rd Annual Technology & Innovation – the Future of Banking & Financial Services New Zealand conference in August. I’ll be exploring this in more detail and hopefully have a few examples to test out.

Dynamically changing format statements in the F# ISE

This had me stuck yesterday – how to change the width of  some formatted printf output. In the end with a bit of searching I realised the TextWriterFormat member gives you a way to define the format such that you can change it in your program. An example is a better viewer for the Pascals Triangle sequence generator I made recently – all related to a school math project. Also – where does it tell you that to print a % you need to double it? Why didn’t they do as with speech marks and use a leading \?

let rec PascalsTriangle = seq {
    yield [1];
    for aLine in PascalsTriangle ->
        let newLine =
            aLine
            |> Seq.pairwise
            |> Seq.map (fun (x,y) -> x+y)
            |> Seq.toList
        List.append (1::newLine) [1]
    }

PascalsTriangle |> Seq.take 10
//|> Seq.iter (fun i -> printfn "%A" i)
|> Seq.iter (fun v -> 
                        v |> Seq.iter (fun w -> printf "%3i" w)
                        printfn ""
            )

// smarten up the printing...
let samplePT = PascalsTriangle |> Seq.take 12
let maxDigits = (Seq.concat samplePT |> Seq.max).ToString().Length
let numberFormatter digits = Printf.TextWriterFormat<int->unit>(sprintf "%%%dd" digits)
let spaceFormatter digits = Printf.TextWriterFormat<string->unit>(sprintf "%%%ds" digits)
let rows = Seq.length samplePT
samplePT |> Seq.iteri   ( fun u v ->
                                    for i in 1..((rows / 1) - u) do
                                        printf (spaceFormatter ((int)((maxDigits+1)/2))) " "
                                    v |> Seq.iter (fun w -> printf (numberFormatter (maxDigits+1)) w)
                                    printfn ""
                        )

The output is like this:

                      1
  
                    1   1

  
                  1   2   1

  
                1   3   3   1

  
              1   4   6   4   1

  
            1   5  10  10   5   1

  
          1   6  15  20  15   6   1

  
        1   7  21  35  35  21   7   1

  
      1   8  28  56  70  56  28   8   1

  
    1   9  36  84 126 126  84  36   9   1

  
  1  10  45 120 210 252 210 120  45  10   1

  
1  11  55 165 330 462 462 330 165  55  11   1

Wish I knew how to do this in WPF…

Wish I knew a better way to post into the blog…