New Web

These notes come from Michael Platt's Web 2.0 presentation at TechEd07 Auckland.

Why TCP?
Why HTTP?

Why not UDP and BitTorrent...
This follows on from the SAF06 event and Bill Gates talk. Current web protocols were designed with low bandwidth environments in mind; now there is a high bandwidth environment and the time is right for a disruptive innovation. A new read/write peer to peer protocol could easily supplant HTTP with the PUT/POST/GET verbs.

If we do this then REST could provide a model for a new implementation.

Basic support available in WCF1.0 but it's native in Orcas.

Question could be what REST support exists in Silverlight?

Interesting point raised on process management and the Robotics SDK which contains graphical process designer. In a bidirectional web environment you need process support.

TechEd 2007 Auckland Presentation

Just finished! I can never tell in advance how the presentation will go until it's performed in front of the obvious. It's only then that you can gauge the feed back and know which areas of the presentation were actually important to the audience.

In this particular case I think the following areas stood out:

• Innovation is not the result of projects
• My rant on incubators never succeeding
• The requirement to sell internally your ideas
• The need to think ahead two, three or more years and use that to build your strategy for your technology development

Big thanks go out to Lou Carbone for his keynote, best keynote speech I've seen, and I took a hint and used my time before the presentation to add some pictures - they speak a thousand words.

Main message from Lou's speech was that emotional clues are everywhere; and you're daft if you don't take heed.

Surfacing a new Internet

The message has been clear for months; the internet has been limited through protocols designed in 1994 andwe're now on the verge of a new electronic world. I watched Bill Gates give this message last December at the Strategic Architects Forum in Seattle and it's now becoming very clear what he had in mind.

First the release of the Silverlight beta with vector graphics; then the Silverlight alpha with a downloadable sandbox; and now Surface. I've been telling people in my industry ever since that the world will be dramatically different in 5 years time. I don't think it's been well understood but maybe now it will be different.

When you think of a core Internet application such as Internet Banking you realise that a customer's expectations of what Internet banking is has only been based upon 10 years of experience at most. Five years from now this experience will be very different. Silverlight and Surface will enable whole new styles of user interaction. Interacting with the bank becomes a flick of the wrist!

Now, one last memory from that conference in Seattle: the statement that the Internet is increasingly becoming a mirror of reality. Seems more accurate every day...

Almost Forgotten

A few weeks ago it was ANZAC day in NZ; a day of remembrance originally defined by the failed WW1 landings at Gallipoli, but now extended to encompass those Australians and New Zealanders injured or lost during all the wars since. A day also celebrated in Turkey for the creation of a secular state: for one world a disastrous invasion; for another the rise of a modern state.

Now, I usually find these days disconcerting; there seems too great a yearning for days past, as though something about those historic events made us better people, proved our metal, gave perhaps a reason to celebrate our forefather's heroic achievements. It seems a dangerous thing.

Now that might be true but on this occasion, I think I've found something else that warrents mentioning.

My family background on my father's side is Polish. That side of my family came to NZ as refugees in the 1940s and 1950s via the Soviet Union, Persia, and Palestine. Of 1.5 million Poles deported by Stalin in 1939/1940 to Siberian labour camps, only about 700 made there way to start new lives in NZ. The original arrivals consisted of orphaned kids, followed after the war, thanks to the Red Cross, by those close family members that managed to find their children or siblings. It was by this mechanism that what remained of my polish ancestry came together in NZ.

The impact upon families of the trip from labour camps was severe with many dying along the way including members of my father's family. The men of fighting age went into the army, navy or airforce and variously split into those fighting for the West and those fighting for the Soviets.

I recently came across a web site which attempts to document some of the Polish forces during the war, a force which included members of my family. I know that the internet is unreliable but the site seems to confirm what I heard from my family as I grew up.

The Polish forces were a major part of WW2 and yet also now largely forgotten.

The Soviet controlled Polish army fought through to the battle for Berlin and put a Polish flag temporarily on the Brandenburg gate next to the Soviet flags, albiet temporarily as it was rapidly taken down. They numbered 396,000 through the war with 23000 killed or missing in action.

The British controlled forces numbered 255,000 across airforce, army, and navy with 13,000 killed or missing in action.

This means that ignoring the initial invasion of Poland, and the 15,000 officers believed killed by the Soviets in camps in 1940, Poland still fielded a combined total of approximately 650,000 servicemen through WWII making it the 4th largest allied armed forces.

Comparing the casuality rate with that suffered by the US forces during the war is enlightening: 5% of those fighting in the West were killed or missing; 6% of those fighting in the East were killed or missing. As many Poles fought in WW2 under British and Soviet command as did US Marines but they suffered twice  the casualty rate (http://www.usmm.org/casualty.html). And that's excluding the initial invasion of Poland by Germany and the Soviet Union in 1939.

Yet you'll never hear that mentioned on the History channel!

With Yalta and Stalin's desire to incorporate Poland into the Eastern Bloc the efforts of those 2 armies was effectively sealed. In 1946 the British command demobilized the Polish army. They were not allowed to join the London victory parade and according to an interview on the Polish Solder website they were given identity cards with "Enemy Alien" written on them.

In the case of my family 3 members were mobilized into the army, navy and cadets. All survived and ended up coming to live in NZ, a country that interestingly fought alongisde the Polish 2nd Corps throughout North Africa and Italy.

I think it's about time that the efforts of these forgotten soldiers were better brought to light.

SQL Server 2005 Data Mining Classification Matrix

Finally figured it out, so easy, and yet so poorly described in the documentation; the classification model tab in SQL Server 2005 data mining shows you the accuracy of your model prediction. Figuring out how to use it had me confused for quite some time. If you're as daft as me then perhaps the following might help.

The easiest way to discover how to correctly read the classification matrix is to generate a table from your testing data set (you do split your original data into training and test data sets don't you...!?) with the ID, the actual outcome you're trying to predict, and the predicted value. You can do this in the Mining Model Prediction Tab. It's easier there because the option to save to table in your database is accessible from the little disk icon in the top left corner. You choose the ID, the actual and the predicted value in the bottom half of the screen. The first time I looked at this it was a bit of mystery how to drive that half of the screen, I'm guessing you've mastered that part.

Load up the table that is saved from your mining query into SQL Server's query window and count up the number of entries that are correctly predicted; and the number of actual values for each state of the actual attribute. You'll then see how it maps to the classification table. Basically, the succinct description at the top of the table is correct. Columns correspond to actuals; rows to predicted values. The bit they leave out that would've been useful to me is that the total number of cases of any one actual value you get from summing vertically; the total number of predicted cases of a value you get from summing the row.

Singleton DMX

If you've used DMX to generate predictions from a data mining model you will be familiar with the singleton query. This is where you obtain a prediction for a particular set of input attributes; either within a case table or in a nested table. There was an example of one of these in a previous post.

Now the form of that query was:

select prediction
from data mining model
prediction join
(select (select something as nested row 
          union select somethingelse as nested row etc) as nestedtable)
as t

Now, what I wanted to do was using openquery() to return all the nested rows, but I did not have an explicit case. How do you make it work?

After bugging the team on the www.sqlserverdatamining.com web site in the newsgroup section the answer was obvious. Just go ahead and create a dummy case. A sample query that works for one of my models follows.

 

SELECT
    Predict([Model_NaiveBayes].[Bucket]),
    PredictProbability([Model_NaiveBayes].[Bucket])
From [Model_NaiveBayes]
NATURAL PREDICTION JOIN
SHAPE { OPENQUERY(Test, 'SELECT 1 as CaseKey') }
APPEND (
    { OPENQUERY(Test,'
    select 1 as ForeignKey, Term
    from Terms
    cross apply Matches('some long discourse containing many terms that I want to characterise'',''\b('' + Term + '')\b'')')
    } RELATE [CaseKey] TO [ForeignKey]
) AS [Msg Term Vectors]
AS T

In this case it uses the Matches TVF which I describe in a previous post to identify the terms in the text.

Exposing the Regular Expression Match Collection to SQL Server as a Table-Value Function

For a recent pet project I've been attempting to create a text mining model in SQL Server 2005 to analyse incoming messages and automatically bucket them into one of a number of categories. This follows straight on from the text mining example contained in the SQL Server tutorials.

With the model implemented (using Decision Trees, Naive Bayes etc) it's easy to create a singleton query by hand that has the following form:

 

SELECT
[Model_NB].[Bucket],
TopCount(PredictHistogram([Bucket]), $AdjustedProbability, 3)
From
[Model_NB]
NATURAL PREDICTION JOIN
(SELECT (SELECT 'some defining term' AS [Term]
UNION SELECT 'another identifying noun or phrase' AS [Term]) AS [Msg Term Vectors]) AS t

But I still needed to extract the identifying noun phrases that make up the terms. Given a dictionary of terms and a length of freeform text how do you find all the term occurences?

Using the SQL Server string functions is painful so I thought I'd try the Match Collection object in the CLR. To expose this you need to perform the following operations.

Firstly, enable CLR integration with

EXEC sp_configure 'clr enabled' , '1'

Then create a SqlServer function in .net to expose the MatchCollection eg

using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using System.Text.RegularExpressions;
using System.Collections;
public partial class UserDefinedFunctions
{
     [Microsoft.SqlServer.Server.SqlFunction (FillRowMethodName="RowFiller", TableDefinition="Match NVARCHAR(MAX)")]
     public static IEnumerable Matches(String text, String pattern)
    {
        MatchCollection mc = Regex.Matches(text, pattern);
        return mc;
    }
    // this gets called once each time the framework calls the iterator on the underlying matchcollection
    public static void RowFiller(object row, out string MatchTerm) {
        Match m1 = (Match)row;
        MatchTerm = m1.Value;
    }
};

Then deploy using Visual Studio, or if you want to do it manually try:

CREATE ASSEMBLY MatchesAssembly FROM 'c:\somewhere\some.dll' WITH PERMISSION_SET = SAFE

CREATE FUNCTION Matches(@text NVARCHAR(MAX), @term NVARCHAR(MAX))
RETURNS TABLE
(Matches NVARCHAR(MAX))
AS
EXTERNAL NAME TextTools.UserDefinedFunctions.Matches

And voila, you can do the following...

select *

from terms cross apply Matches(@text,'\b(' + terms.term + ')\b')

Where terms is a table of noun phrases you're searching for in the variable @text that contains the message text.

Measuring how quickly we learn from our mistakes

For 10 years I've heard how you need to learn from your mistakes and it seems quite a reasonable mantra. But the problem I've got with learning from mistakes is how quickly do you learn.

Some people and some organisations seem to learn really quickly, others, more like myself, take a while longer, others take the whole of their existence.

So the question I've got is how do measure the responsiveness of someone, some organisation, to learning from their mistakes?

What metric could we use to tune performance?

Not sure, but maybe we can think something up.

Whales off the Beach

Wonderful sight - whales off the beach, maybe 300m from shore on a beautiful still, sunny day. Not sure what type, but I don't think they were Orcas - they didn't seem large enough. The shame of it is that if it weren't for the cold I've had for the last few days, my plan was to be out kayaking this morning. Wouldn't that have been amazing!

Oh, perhaps a bigger shame, especially from the point of view of the whales, was the idiot out there going round and round in circles on a jetski. Confirms every opinion I've ever had of them.

Update: Talking to someone this morning I found out they were indeed Orcas. Now I really wish I'd been up to going out on the kayak.

The value of a demonstrable prototype

I've just been reading an interview by Ron Jacobs with Scott Guthrie; it's up on The Microsoft Architect Journal (http://msdn2.microsoft.com/en-us/library/bb266332.aspx).

What's caught my attention is the importance Scott put on having a demonstrable prototype of their ASP.Net technology early in the development. "There were three or four of us" and yet they managed to create one of the key product offerings of one of the world's largest companies. A truly remarkable outcome.

Now I'm sure that the team itself was extraordinary, but still I bet in many companies they would have been driven into the ground under the weight of the stakeholders. The more people involved, the more diluted the good ideas become and the greater effort needs to be put into selling ideas and managing the stakeholders. The processes companies typically put into place to manage new delivery put a stranglehold on true innovation.

'...That's one of the successful things that we did with ASP.NET. We said that we're
going to throw away every line of code we're going to write for the next couple
of months. Let's all agree on that. We're not going to say, "Oh let's take this
and adapt it; we can clean it up." No. We're going to throw it away. We're going
to "deltree" this subdirectory at some point, and that way we can be more
adventurous about trying new things. We don't have to worry about making sure
that everything's robust because it's going to be in the final version.

We actually did that for a few months and said, "We're done, delete it; let's start
over from scratch; now let's write the full production code and make sure we
bake in quality at the time." I think a lot of teams could benefit from that.
The hardest thing is making sure you delete the prototype code. Too often,
projects develop with "Well, it's kind of close." It's very difficult to start
with a prototype and make it robust. I'm a firm believer in starting with a
prototype phase and then deleting it.'

The typical corporate company approach would not have sustained this type of development.

It would be an incredibly gifted individual that could convince a steering committee to support expenditure on experimentation when little or no artifacts could be easily capitalised at the end of a 3 month development. Most companies want functional delivery early with a clear line of connection from end user requirements to final deliverable.

Scott talks specifically about the value of a prototype to help sell the idea.

"We certainly had to persuade a number of people along the way. One thing we
did early in the project was get running code in prototypes that we could show
people. Often when you're working on a project that's new or something that
hasn't been done before, it's easy to put together a bunch of PowerPoint slides
that sound good, but it's especially valuable to actually show code and walk
people through code that's running. Not only does the prototype prove that it's
real, but also you just learn a terrific amount by doing it."

Perhaps their greatest achievement was the successful marketing of the development from initial concept through to the current deliverables. The early prototyping and demonstrations were obviously a critical part of this achievement.

At the end of the day all of this makes great fodder for the creation of a myth list: the My Myth List. My first two myths are going to be:

1. End users know what they require
2. Product delivery is a linear process

Because it's apparent that people don't know what they want. It's the visualisation of what is possible that provides a basis for requirements. And product delivery is never linear.

I'm sure I'll think of more over the next few weeks.

21C in the pool on April 1st

Global warming? Well it was a late start to summer...

Commotion at the beach

One of the Kapiti island ferrys was caught out this afternoon. I joined in a small crowd to watch as the tractor that was supposed to haul it out of the water got stuck in the sea. The engine appeared to have failed and in a scene reminescent of the Little Digger, 3 tractors lined up in a row to try and tow the stranded tractor and ferry out, without success. They temporarily gave up on the ferry, hauled out the tractor before the tide came in further, then towed the ferry onto the shore to allow the passengers out. A bit of excitment while I was down at the playground with Lil!

ARCast video up for Kiwibank Case Study

I've just found out that Ron Jacobs has posted a video interview of myself and two members of my team on his ARCast site.

The interview was based on the Kiwibank Case Study which we undertook last year with Microsoft. The interview takes a broad look at how Kiwibank has used technology to help it go from 0 to 12% of NZ's population in 5 years. I speak along with David Grahame and Sushil Kamanahalli. David is the client applications architect and Sushil is the service layer architect.

Special thanks go out to Mark Carroll for helping to organise this and to Ron Jacobs for the interview and the work to put it together.

Fashion at Government House!

What a fantastic night. I had the absolute pleasure of seeing my wife, Miriam Gibson, present her winter collection womens fashion wear at a charity event held at Government House in Wellington on Tuesday night.

It was magnificent. The show was held in the Ballroom with the Governer General, His Excellency The Honorable Anand Satyanand, and his wife, Her Excellency Mrs Susan Satyanand, army staff in regalia, members of Rotary and the charitable organisation, Refugees as Survivors, and over 200 guests who had come to see the launch of the 2007 winter range and support a good cause.

And I was very, very, very impressed. Miriam, Victoria, Sue, Sarah, Veronica, and all the models - you all did a helluva job!

And standing up there following up the Governor General, the local Rotary head, and representatives of the charity with a speech on the podium with microphones, photographers, and press present. Crikey - my recent speaking engagements pale into comparison.

The range was fantastic so I definitely recommend checking it out: head to the stores in Margaret Rd, Raumati and Hunter St, Wellington, or check out the web site to find out more (pictures from the event are promised over the next few days). Nothing for the guys, this is ladies only. And don't forget the charity - they're a very worthwhile cause in this country which has been the fortunate recipient of many refugees in the past, including the Polish half of my ancestry...

Rod Drury on investment for IP

http://www.drury.net.nz/2007/03/17/building-intellectual-property/

Notice the graph that Rod has obtained of normalised patents per country. See how NZ sits at about 0.5 in 22nd place. Finland rides at the head on 4.5 and the OECD average sits at just under 2 which means it's closer to Finland than NZ.

This is a graphic representation of the under investment in R&D in NZ compared to other countries. This isn't about centralised R&D run out of the government, but about the failing of companies to invest in product development. Great thing that Rod pointed this measure out.