tag:blogger.com,1999:blog-176720342024-03-05T19:41:28.433+13:00Bohdan SzymanikPostings on topics that will hopefully be of use to someone...Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.comBlogger131125tag:blogger.com,1999:blog-17672034.post-51837849645954031382015-11-30T08:50:00.001+13:002015-11-30T08:50:37.608+13:00Sharpen up your legacy app(s) performance with a bit of F#First off, thanks go to <a href="https://twitter.com/sergey_tihon" target="_blank">Sergey</a> for organising another <a href="https://sergeytihon.wordpress.com/2015/10/25/f-advent-calendar-in-english-2015/" target="_blank">F# Advent Calendar event</a>. Normally, I'd just watch and learn, but this year I thought I'd try and contribute back something of my own experiences from using F# in a corporate/enterprise environment.<br />
<br />
The problem I've faced this last 6 months has been to try and get a better understanding of the performance characteristics of the systems where I work. This isn't really a technical post - although I do have some links to sample code on my Github repository, it's mostly a background perspective on what I've found helpful and why - and it needs to be taken from the perspective of someone that doesn't code for a job - I'm mostly pushing out powerpoints and word docs but when the opportunity allows I love the chance to get into a bit of the data and make something of it.<br />
<br />
To get a bit more specific in this case, my co-workers and I need to get a good handle on the performance characteristics of a large legacy technology system that's getting replaced; it's at the heart of a customer facing system with upwards of a million customers so performance is very important. The profiling is essential for defining non-functional requirements (terrible term - qualities of service is so much better but the industry can't seem to get away from it), and it's those NFRs which are often misrepresented, misunderstood and yet become crucial to successful delivery.<br />
<br />
So what do we do and why has F# been helpful? Okay there’s probably more steps but for me I'm thinking of 3 things.<br />
<ol>
<li>We have to baseline existing performance which means lots of analysis. </li>
<li>Then we set our target requirements based on the baseline and maybe introduce some tougher targets, because why not, it’s a new and hopefully better thing you’re developing/configuring/deploying. </li>
<li>Then we need to verify through testing under load.</li>
</ol>
F# and the broader .NET application environment has been incredibly useful to this exercise for the data analysis side, and it's looking very promising for the testing.<br />
<br />
So why is that?<br />
<br />
Well let's looking at profiling, what that means is collecting lots of data; it’s coming from log files and databases, and it’s got shape, it can all be described as being of some type and F# is really, really good at dealing with data that's got some form of shape, some constraints or rules.<br />
<br />
I've previously used Perl, Python, R, Excel and even VBA; they're all popular choices, but that ability to type the data starts to become really useful when trying to get to grips with the edge cases that you trip up on later, or when you start moving into larger volumes of data. The dynamic typing approach you get with Perl, Python and R is great for getting you started but then you so often keep hitting the edge cases as you start processing larger volumes of data and it's sometimes really hard to figure out why because it's basically just happening at runtime. The strong upfront typing in F# (and other statically typed languages) is very, very handy in these circumstances. So that's reason number one for me: static inferred data typing which tells you what's going on right up front.<br />
<br />
OK, so how about another reason. Data manipulation. Let's say you get the data into memory and you're working on it. The REPL in combination with the inferred typing, functional and non-functional data structures (Arrays, Seqs, <a href="http://bluemountaincapital.github.io/Deedle/" target="_blank">Deedle</a> data frames etc) and functional approaches like Maps makes data transformation a whiz. OK, Microsoft's SQL Server is pretty good - I/we use it a lot but while it's a truly wonderful database, Transact-SQL is just not the best approach to dealing with many parsing or analysis problems, you rapidly end up playing tricks like <a href="http://blogs.msdn.com/b/mrorke/archive/2005/08/02/446810.aspx" target="_blank">constructing XML</a> and doing <a href="https://www.mssqltips.com/sqlservertip/1771/splitting-delimited-strings-using-xml-in-sql-server/" target="_blank">this</a>. Nope, F# rocks for interactive data transformation, it's definitely reason number two.<br />
<br />
The third big reason for me is the broader .NET ecosystem. It just works well together. Plugging libraries in isn't hard. Things tend to work and the last few years have seen packages on Nuget expand dramatically. It's a wonderful thing to pull up the <a href="http://accord-framework.net/" target="_blank">Accord</a> libraries, or Deedle, or the many type providers, or <a href="http://fslab.org/" target="_blank">FsLab</a> and just get it all working straight away.<br />
<br />
The last reason has to be brought up. This is an awesome <a href="http://fsharp.org/" target="_blank">community</a>. People help and they're positive. That's a helluva a plus when you're just trying to learn.<br />
<br />
Ok, so those are the reasons why I find F# works so well for me, but let's go into a little more detail and go through a typical workflow in trying to do this performance profiling.<br />
<br />
Step 1 is reading log data. Most often it's going to be with File.ReadAllLines, or SQL data via one of the SQL type providers. Most enterprise environments will also have a variety of monitoring systems to query - we happen to have Microsoft’s System Center Operations Manager installed with a data warehouse that contains multiple years worth of performance metrics. The SQL type providers make reading this easy. Here's a question - has anyone considered a type provider for IntelliTrace files? I can't seem to find any information on how to query these outside of Visual Studio but would love to know.<br />
<br />
I like regular expressions and active patterns to parse text data – it helps tease out edge cases but if there’s a simple structure then basic string operations like split can be fast and simple. In practice I haven’t had much luck with the CSV type provider – too many oddities in the files I seem to come up against - far too many of them effectively spread a record across multiple lines, and I very rarely come across simple csv data. In the example code I've got a typical example of a multi-line file that needs to be <a href="https://github.com/bohdanszymanik/2015Advent/blob/master/Demo.fsx#L21-L32" target="_blank">pre-parsed</a> before handing off to the ReadCsv method in Deedle.<br />
<br />
So the data gets into local memory and that's good enough for most of the problems we face, in fact you can make a pretty good argument that <a href="http://datascience.la/big-ram-is-eating-big-data-size-of-datasets-used-for-analytics/" target="_blank">big RAM is beating big data</a>. Representing it in memory is usually a case of an Array, or a Deedle data frame. Data frames are very handy but there's a bit of a learning curve due to the increased flexibility. (Fortunately I see a new book has just been <a href="http://www.oreilly.com/programming/free/analyzing-visualizing-data-f-sharp.csp" target="_blank">published</a> - I need to read it!)<br />
<br />
Next step's usually a series of transformations. Using Array.mapi means if something goes wrong I can get a line number. Array.Parallel.mapi is nice when an individual task can be particularly slow but often running interactively it doesn't provide a simple performance boost. Using the Some/None option types makes missing values explicit which turns out to be incredibly useful – missing values are everywhere in real-world data. It makes you explicitly handle the missing data case and then progressively unwrap like <a href="https://github.com/bohdanszymanik/VariousImports/blob/master/PINVerify.fsx#L41" target="_blank">this</a>.<br />
<br />
Timing a series of piped-forward transformations can be useful to know where the lengthy steps are which can be easily done using <a href="https://gist.github.com/bohdanszymanik/eeb2cba00061ec282017" target="_blank">a custom operator</a>. (btw I'd love to know if there was a way to reflect back something readable about the nature of the line of the code being executed in the pipeline.)<br />
<br />
So, let’s say we’ve transformed the data and we now have something like durations at certain timestamps. What now?<br />
<br />
Time to use FSLab and Accord.NET. Graphing distributions of response times or numbers of requests per time interval provides a great way of understanding the system you’re dealing with. The shapes can give some insight on the underlying processes that generate the distributions.Classic examples are random arrival events resulting in a <a href="https://en.wikipedia.org/wiki/Poisson_binomial_distribution" target="_blank">Poisson distribution</a>, measurement errors resulting in a normal Gaussian distribution or failure events resulting in a <a href="https://en.wikipedia.org/wiki/Weibull_distribution" target="_blank">Weibull distribution</a>. The Accord.NET author, <a href="https://twitter.com/cesarsouza" target="_blank">César de Souza</a>, has produced a very useful <a href="http://www.codeproject.com/Articles/835786/Statistics-Workbench" target="_blank">tool and covering article</a> to getting a handle on different types of distributions with documentation to understand the underlying nature of the distribution and sample code for interactively working with them. In practice with the real system data you're going to find lots of skewing and probably multiple peaks due to competing processes - use your eyes to figure out what's happening.<br />
<br />
I used FsLab for that visualisation, it incorporates both <a href="http://fslab.org/FSharp.Charting/" target="_blank">FSharp.Charting</a> and <a href="https://tahahachana.github.io/XPlot/" target="_blank">XPlot.GoogleCharts</a>. They both work well.<br />
<br />
The Accord Framework is very broad - I've used it here for modelling, fitting and sampling distributions but it goes far further - machine learning, image processing etc.<br />
<br />
With a good understanding of the baseline performance characteristics you’re in a position to set service level expectations for response times – basically, the QoS requirements - and with requirements then it means you'll want to be testing against them. So you'll either be sampling against real historical data or against a fitted distribution which can allow you push into the long tails. That's important because unless you take a really long historical sample you'll probably never get to test the extreme values, after all they're only rarely occurring but they're the ones that might bring your application to a grinding halt.<br />
<br />
Calling into libraries, legacy systems etc to execute your tests is a piece of cake with F#. If it's a web service endpoint the <a href="https://msdn.microsoft.com/en-us/library/hh362328.aspx" target="_blank">WSDL type provider</a> makes calling a service easy. The Async MailboxProcessor is a very simple way to run up parallel workers for stress testing.<br />
<br />
I've put a sample <a href="https://github.com/bohdanszymanik/2015Advent" target="_blank">fsx script</a> into GitHub which shows a typical scenario that I might go through in the office. It starts with the retrieval and parsing of data using Deedle and the use of Accord.NET to find the best fit across common distribution types. There's a number of transformations involved and some sample questions, approaches to answering them and graphing tasks. The data in this case doesn't have enough input variables to be useful for a machine learning demo but it is very typical of what gets written from an older enterprise application log. (In retrospect it would have been interesting to include measures of the batch sizes and relative complexity of the processing steps. The we could have considered how the output processing times relate to multiple input variables.)<br />
<br />
The last big part of this overall process is the presentation of the data back and the incorporation into documents and presentations. For me, Microsoft Office still rules the roost and the entry point for data is Excel. My default way (<a href="https://gist.github.com/bohdanszymanik/6b7966a7658e5b2fefc6" target="_blank">for example</a>) to get data into it now is using Deedle's Frame.SaveCsv. This works well as part of a file processing pipeline:<br />
<br />
File data -> import process -> transformation and analysis -> representation as an array of records -> Frame.ofRecords -> .SaveCsv. -> Excel and Word/PowerPoint drudgery.<br />
<br />
So there you have it. An F# Advent posting simply because it deserves it. A very useful programming language, community and technology toolkit that's made many a problem go away this past year.<br />
<br />
Bring on 2016.Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com1tag:blogger.com,1999:blog-17672034.post-62668123695487510932013-07-23T19:28:00.000+12:002013-07-23T19:28:05.625+12:00Simple example of reading Azure Table Storage within F# scriptIn case it's useful...<br />
<br />
<script src="https://gist.github.com/bohdanszymanik/6060464.js"></script><br />
<br />Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com1tag:blogger.com,1999:blog-17672034.post-79983695289871829602013-07-11T22:46:00.000+12:002013-07-12T08:34:03.503+12:00Azure Table Storage F# most basic exampleIn case anyone else was as confused as I was.... while trying an experiment out with WindowsAzure Table Storage. Searching the net just keeps highlighting old documentation and with the API changing it had me going no where for quite awhile. Basic run through here:<div>
<br /></div>
<div>
<script src="https://gist.github.com/bohdanszymanik/5974459.js"></script></div>
<div>
<br /></div>
Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-57703986776212739082011-10-20T08:50:00.000+13:002011-10-20T08:50:12.967+13:00IronPython and Numpy/Scipy<div>
Should I ever forget this in the future...<br />
<br />
If you're trying to install numpy/scipy on ironpython and like me you're behind a corporate firewall, and like me you can't get proxying to work, and like me you can't figure ironpkg out, then don't give up. Download the eggs locally and use ironegg eg: ironegg nose-1.0.0-1.zip</div>
<div>
</div>Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-86174942371314825942011-05-23T12:22:00.001+12:002011-05-23T12:22:45.759+12:00Launching the Black Pearl!<p>It’s not quite finished – I need to give a final rub down and coat of epoxy, and put the deck lines on, but with great weather this weekend I just had to launch it.</p> <p>Couple of building notes:</p> <ul> <li>I used thick minicell foam for the bulkheads. Expensive stuff in NZ but I got a slab of it while on a work trip to the US. I figure it makes for light bulkheads that will allow the body of the boat to flex.</li> <li>The foot pegs are plastic – screwed into tee nuts embedded in a strip on each side. Not sure how adjustable these are going to be – with the small cockpit I can’t seem to reach in to change the settings!</li> </ul> <p><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="IMG_1346" border="0" alt="IMG_1346" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHAEinQ6sN9eZX5hebAURdE4mBiE9_GOqycKWXzzC9P-3RnvIfvidlSiL3AM5Z36GCTyMluZ1ep6iQQHfZhplZDkq_zpCG3mpjsGRTPN-4FvyP6uT_v94cMXkPvIonucWGqX-_ZA/?imgmax=800" width="324" height="484" /></p> <ul> <li>I used Maroske’s internal tube fittings but the longest of these just wouldn’t let me extract the PVC tube… no problem, just slice em in half! They’re very easy to re-glue.</li> </ul> <p><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="IMG_1345" border="0" alt="IMG_1345" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9WWzmd9Lu9rknzcUXsRMgEeQMPaVSuGkcgnZxhQKq7VD0ocizHc8MxSMPLIJdiHIdh_locm1QS7qU4-amML4NlvOen75cPB_TTnbmqOUu19j0MMPaqzRH97wIeYU9Q3c_WyRGMg/?imgmax=800" width="644" height="431" /></p> <ul> <li>I doubled the high density fibreglass over 2/3rds of the length for the hull and deck interiors with an extra overlapping strip just behind the cockpit. Feels very robust while getting into/out of the cockpit.</li> </ul> <p>What’s was it like to use?</p> <ul> <li>Much lighter build than the Night Heron – on the NH I used 200g cloth doubled over most of the interior. The smaller surface area of the BP plus the lighter building materials (mostly Paulownia strips) means it’s about 14kgs. This makes it much easier to carry and a bit more responsive to my body movement on the water.</li> <li>Very tippy! But secondary stability seems good – I just need to get used to the experience. Very different from anything I’ve been in so far and I’ll take my time to practice rolls before I get into rougher water.</li> <li>Very responsive to my movements and to the water.</li> <li>Lots of fun –which is what I wanted.</li> <li>I was slightly concerned about the ocean cockpit but in practice it’s easy for me to get into and out of so no problem there.</li> <li>Legs are quite straight. Again was a bit worried I’d be too uncomfortable, but actually for me – it’s fine.</li> </ul> <p>So frankly, I’m utterly delighted!</p> <p><a href="http://lh4.ggpht.com/_cWvIIiKNA4g/Tdmo0KpuxDI/AAAAAAAAAOc/PtUnwePqiDA/s1600-h/IMG_1364%5B6%5D.jpg"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="IMG_1364" border="0" alt="IMG_1364" src="http://lh3.ggpht.com/_cWvIIiKNA4g/Tdmo1Isto-I/AAAAAAAAAOg/uJZyWmCnlMg/IMG_1364_thumb%5B7%5D.jpg?imgmax=800" width="644" height="412" /></a></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-2532374697156251572011-04-22T20:15:00.001+12:002011-10-20T08:51:04.236+13:00Cockpit coaming step 1: Wood strips<p>I’m building the cockpit coaming by first building a wood strip riser. I'll fibreglass it into place then put foam around the outside and use that as a form to create the coaming lip – at least that’s the plan.</p> <p>First step is building the riser. To do this I followed <a href="http://www.guillemot-kayaks.com/guillemot/blog/nick/overview_making_cockpit_coaming">Nick Schade’s instructional video</a> and it’s seemed a pretty quick and easy exercise.</p> <p>So far, the riser is in place, a fillet is on and when it dries I’ll wet sand and follow up with 2 layers of fibreglass.</p> <p>It’s an ocean cockpit so fairly short and only 38cm interior width.</p> <p>Current pic (plastic to protect the interior): </p> <p><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="IMG_1327" border="0" alt="IMG_1327" src="http://lh3.ggpht.com/_cWvIIiKNA4g/TbE5K4rubLI/AAAAAAAAAOQ/dU4HOB2B4LY/IMG_1327%5B6%5D.jpg?imgmax=800" width="581" height="484" /></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-41161078006338033232011-04-16T18:24:00.001+12:002011-10-20T08:51:04.241+13:00Black Pearl Update–Deck Fibreglassed<p>The deck’s been glued, sanded and fibreglassed – at least on the outside.</p> <p>Mostly Paulownia but 3 strips on each side of Cedar as I was running a bit low of the lighter wood. I’ve decided to leave the deck natural colour so that I can forever see how it was built. The cockpit’s been cut out but I’m leaving the hatches till after the fibreglass sets up. It seemed a little flimsy to be cutting too many holes in the deck.</p> <p>Not many staple holes, it was easy to glue most strips in place without staples.</p> <p>I kept the surface to an even height and used the odd bit of chopped up credit card under a couple of thinner strips.</p> <p>Quick pic: of the cockpit:</p> <p><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="IMG_1324" border="0" alt="IMG_1324" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpQ-lPmkTx1PWjOShW0yoAIgdv9EOxWgWS7HoZH3RLA_JlNBoByqJSrExlCS74CZmQbpa6t2G2bz65pUcvO_VjDQA1JSeb9bA5ngAMq0e7zKbR90r4czpQ3kvvVRJSqQjHV4VWeQ/?imgmax=800" width="244" height="154" /></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-48932716162927655802011-03-15T17:51:00.001+13:002011-03-15T17:51:27.034+13:00Wood Density<p>My fascination with the weight of the kayak I’m building is growing with my surprise at just how light in weight the hull is. I thought I’d do a quick check on a couple of samples of Paulownia and Cedar.</p> <p>For the wood I’ve bought it works out that Cedar is about 0.40g/cm<sup>3</sup> (about 17 lbs/ft<sup>3</sup>), and the Paulownia is about 0.27g/cm<sup>3</sup> (about 25lbs/ft<sup>3</sup>). So the Paulownia is 2/3rds the density of the Cedar!</p> <p>Now that has me thinking – I’m mostly building the deck in Cedar – the extra wood weight plus the coming/deck deckplates etc – hmmm, will it be tippy without me in it? I reckon from looking at it that the deck is about 2/3rds the size of the hull so it might will be balanced at about the same weight top half and bottom half!</p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com2tag:blogger.com,1999:blog-17672034.post-39633016130801255792011-03-14T14:55:00.001+13:002011-03-30T21:40:32.400+13:00Black Pearl Hull Completed<p>Some quick notes on this.</p> <p>Firstly, must remember to take care during strip gluing to get the glue all the way through – I didn’t and I found that after fibreglassing the outside and removing the forms… the bottom of the hull started to bend out in a couple of places. Remedy… add epoxy on the inside and glue the strips up properly and weigh down with a few clay bricks (sitting on plastic of course).</p> <p>Secondly, I tried to reinforce the interior chines with a triangular strip of wood height about 5mm and base about 10mm. What a waste of time! Quick calculation after the fact shows that using wood came in at about 100 grams, but I’ve just checked the density of thickened epoxy – I’m getting about 10 to 20% lighter than straight epoxy – and if I’d just used a thickened epoxy bead it would’ve weighed about 200g. The extra 100g would have been worth not having to cut triangular strips!</p> <p>And finally, I hate fibreglassing the interior – much more difficult than the outside. Fortunately, relatively easy to fix up any ugly bits a day or two later – in my case 3 patches that needed sanding back and redoing along the chines.</p> <p>Also, I’ve added up the costs and just out of interest it’s looking like about $NZD1100 total for this one – that’s plans, fibreglass, epoxy, tints/dyes/stains/deck ports and loads of sandpaper. If I’d made some smarter choices along the way I certainly could’ve brought the cost down below $1000.</p> <p>I’ll do a more thorough break down at the end.</p> <p>And the weight! Before I forget, the hull with one layer of 175g high density fibreglass on the inside, one on the outside and an extra strip on the keel, it totals about 5kgs on my scales. Incredible really! The Paulownia wood is very light and the high density fibreglass certainly takes much less epoxy than the 200g normal fibreglass I used on the last kayak.</p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-63711828175548961642011-02-21T23:16:00.001+13:002011-02-21T23:16:39.958+13:00Multiple Monitoring Performance Data Collections<p>Ah yes – that code from the last couple of posts has a bug… </p> <p>The retrieval of data from SCOM has proven very useful in the last few days but it seems most commonly I’m getting multiple monitoring performance data collections - so the code I wrote in the last couple of posts on how to retrieve that data using F# and Sho needs a little adjusting to concatenate the collections into one sequence – yield! works great in F# eg</p> <pre class="csharpcode"><font size="1">let perfData (start:DateTime) (finish:DateTime) : seq<DateTime * <span class="kwrd">float</span>> = <br /> seq {<br /> <span class="kwrd">for</span> mpdsItem <span class="kwrd">in</span> mpds <span class="kwrd">do</span><br /> <span class="kwrd">yield</span>! (mpdsItem.GetValues(start, finish)<br /> |> Seq.map (fun(mpdv) -> (mpdv.TimeAdded, mpdv.SampleValue))<br /> |> Seq.filter ( fun(d,v) -> v.HasValue)<br /> |> Seq.map ( fun(d,v) -> (d,box v) )<br /> |> Seq.map ( fun(d,v) -> (d, unbox v) )<br /> )<br /> }<br /></font></pre><br /><br /><p><style type="text/css"><br />.csharpcode, .csharpcode pre<br />{<br /> font-size: small;<br /> color: black;<br /> font-family: consolas, "Courier New", courier, monospace;<br /> background-color: #ffffff;<br /> /*white-space: pre;*/<br />}<br />.csharpcode pre { margin: 0em; }<br />.csharpcode .rem { color: #008000; }<br />.csharpcode .kwrd { color: #0000ff; }<br />.csharpcode .str { color: #006080; }<br />.csharpcode .op { color: #0000c0; }<br />.csharpcode .preproc { color: #cc6633; }<br />.csharpcode .asp { background-color: #ffff00; }<br />.csharpcode .html { color: #800000; }<br />.csharpcode .attr { color: #ff0000; }<br />.csharpcode .alt <br />{<br /> background-color: #f4f4f4;<br /> width: 100%;<br /> margin: 0em;<br />}<br />.csharpcode .lnum { color: #606060; }</style></p><br /><br /><p>Not sure about python but it must be similar.</p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-23603226572539272332011-02-14T18:45:00.001+13:002011-02-14T18:45:58.520+13:00SCOM data into F#<p>Having just <a href="http://bohdanszymanik.blogspot.com/2011/02/scom-and-microsoft-shoa-better.html" target="_blank">discovered how to get SCOM data from Microsoft Sho</a> - a dynamic language analysis environment - I thought I’d try from F#. Pretty much the same but you need to think about what to do with the graphing, and you need to handle the Nullable<float> values coming back from SCOM.</p> <p>I used <a href="http://code.msdn.microsoft.com/fschart" target="_blank">fschart</a> to do the plotting and a sequence filter and boxing/unboxing to handle the Nullable data.</p> <p>The histogram function wouldn’t be hard to make but while stumbling across fschart I also stumbled across <a href="http://cs.hubfs.net/forums/permalink/15951/15934/ShowThread.aspx#15934" target="_blank">this</a>.</p> <p>So my contribution was do this:</p> <pre class="csharpcode">#r <span class="str">@"System.Windows.Forms.DataVisualization.dll"</span><br />#r <span class="str">@"C:\extras\FSChart10\FSChart\bin\debug\FSChart.dll"</span><br /><br />open FSChart<br /><br />open System.IO<br />open System.Drawing<br />open System.Windows.Forms<br />open System.Windows.Forms.DataVisualization.Charting<br /><br />#r <span class="str">"c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll"</span><br />open Microsoft.EnterpriseManagement.Monitoring<br /><br />let mg = Microsoft.EnterpriseManagement.ManagementGroup(<span class="str">"someManagementServer"</span>)<br />let mpdc = MonitoringPerformanceDataCriteria(<span class="str">"ObjectName = 'ASP.NET' and CounterName like 'Request Execution%' and MonitoringObjectPath like 'someMonitoringObjectPathFilter%'"</span>)<br />let mpds = mg.GetMonitoringPerformanceData(mpdc)<br /><br />open System<br /><br />let (perfData:seq<DateTime * <span class="kwrd">float</span>>) = <br /> mpds.[0].GetValues(DateTime.Today.AddDays(-1.), DateTime.Today)<br /> |> Seq.map (fun(mpdv) -> (mpdv.TimeAdded, mpdv.SampleValue))<br /> |> Seq.filter ( fun(d,v) -> v.HasValue)<br /> |> Seq.map ( fun(d,v) -> (d,box v) )<br /> |> Seq.map ( fun(d,v) -> (d, unbox v) )<br /><br /><br />hist 0.0 200. 50 (perfData |> Seq.map (fun (d,v) -> v) )</pre><br /><br /><pre class="csharpcode"><font face="Georgia">The result looks like this.</font></pre><br /><br /><pre class="csharpcode"><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh3.ggpht.com/_cWvIIiKNA4g/TVjBlbyJBWI/AAAAAAAAAN8/vAUR26JhXFI/image%5B3%5D.png?imgmax=800" width="244" height="240" /></pre><br /><br /><pre class="csharpcode"><font face="Georgia">So the question is… what was the more efficient approach? Well when I did the sho/python code I actually had some c# code open in the background to help me through the classes - without that it would have been really hard. The F# approach made me scratch my head a few times wondering how to deal with charting and nullables but all the way through (much like the c#) I had the advantage of the richer type information to help me figure out what to do. </font></pre> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-26094969688003310962011-02-14T09:29:00.001+13:002011-02-14T18:46:59.284+13:00SCOM and Microsoft Sho–A better histogram example…<p>The last blog post had a naff histogram as a few exceptionally long queries squashed the remainder of the data in the first bin, also I wasn’t checking for None so here’s a better, filtered example.</p> <pre class="csharpcode">ShoLoadAssembly(<span class="str">"c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll"</span>)<br />ShoLoadAssembly(<span class="str">"c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll"</span>)<br /><br />from Microsoft.EnterpriseManagement.Monitoring import *<br />from System import *<br /><br />mg = Microsoft.EnterpriseManagement.ManagementGroup(<span class="str">"someManagementServer"</span>)<br /><br />mpdc = MonitoringPerformanceDataCriteria(<span class="str">"ObjectName = 'ASP.NET' and CounterName like 'Request Execution%' and MonitoringObjectPath like 'someMonitoringPathFilter%'"</span>)<br /><br />mpds = mg.GetMonitoringPerformanceData(mpdc)<br />mpds.Count<br /><br />hist([mpdv.SampleValue <span class="kwrd">for</span> mpdv <span class="kwrd">in</span> mpds[0].GetValues(DateTime.Today.AddDays(-1), DateTime.Today) <span class="kwrd">if</span> ((mpdv.SampleValue < 200) and (mpdv.SampleValue <span class="kwrd">is</span> not None))])</pre><br /><style type="text/css"><br />.csharpcode, .csharpcode pre<br />{<br /> font-size: small;<br /> color: black;<br /> font-family: consolas, "Courier New", courier, monospace;<br /> background-color: #ffffff;<br /> /*white-space: pre;*/<br />}<br />.csharpcode pre { margin: 0em; }<br />.csharpcode .rem { color: #008000; }<br />.csharpcode .kwrd { color: #0000ff; }<br />.csharpcode .str { color: #006080; }<br />.csharpcode .op { color: #0000c0; }<br />.csharpcode .preproc { color: #cc6633; }<br />.csharpcode .asp { background-color: #ffff00; }<br />.csharpcode .html { color: #800000; }<br />.csharpcode .attr { color: #ff0000; }<br />.csharpcode .alt <br />{<br /> background-color: #f4f4f4;<br /> width: 100%;<br /> margin: 0em;<br />}<br />.csharpcode .lnum { color: #606060; }</style><style type="text/css"><br />.csharpcode, .csharpcode pre<br />{<br /> font-size: small;<br /> color: black;<br /> font-family: consolas, "Courier New", courier, monospace;<br /> background-color: #ffffff;<br /> /*white-space: pre;*/<br />}<br />.csharpcode pre { margin: 0em; }<br />.csharpcode .rem { color: #008000; }<br />.csharpcode .kwrd { color: #0000ff; }<br />.csharpcode .str { color: #006080; }<br />.csharpcode .op { color: #0000c0; }<br />.csharpcode .preproc { color: #cc6633; }<br />.csharpcode .asp { background-color: #ffff00; }<br />.csharpcode .html { color: #800000; }<br />.csharpcode .attr { color: #ff0000; }<br />.csharpcode .alt <br />{<br /> background-color: #f4f4f4;<br /> width: 100%;<br /> margin: 0em;<br />}<br />.csharpcode .lnum { color: #606060; }</style><br /><br /><p> </p><br /><br /><p><img style="background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="image" border="0" alt="image" src="http://lh6.ggpht.com/_cWvIIiKNA4g/TVg_Mcp8SyI/AAAAAAAAAN4/Ic0dw66dKrY/image%5B3%5D.png?imgmax=800" width="244" height="187" /></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-43763045416997393702011-02-11T22:59:00.001+13:002011-02-14T18:46:59.285+13:00Retrieving Performance Data from System Center Operations Manager<p>System Center Operations Manager’s agent model means it can collect performance metrics from across a diverse technology environment. You use Rules to define the data collection and typically you use the Console to view the data in graphical format.</p> <p>But, the Console by itself doesn’t allow any numerical analysis. If you want to do that you need to get the data out. You can copy to the clipboard within the Console Action menu but that’s very manual. A better approach is obviously programmatic access and this is  where things get interesting.</p> <p>For some reason there’s little information on the net on how you get data out of SCOM. What is there usuall refers to querying the SCOM database directly. This doesn’t seem right to me. I’d far prefer an API that stood a chance of staying intact in future versions. So here’s what I’ve done.</p> <p>Firstly, use the Powershell integration to SCOM to get the data. (This is easiest if you setup the <a href="http://bohdanszymanik.blogspot.com/2009/11/connecting-to-scom-from-powershell-ise.html" target="_blank">Powershell ISE to connect to SCOM</a>.) You can then use the get-performancecounter and get-performancecountervalue cmdlets to retrieve data, for example:</p> <pre class="csharpcode">$starttime = [datetime]::today.adddays(-20)<br />$endtime = [datetime]::today.adddays(1)<br /><br />get-performancecounter |<br /> ? {$_.ObjectName -eq <span class="str">'Web Service'</span> -and<br /> $_.CounterName -eq <span class="str">'Connection Attempts/sec'</span> -and<br /> $_.InstanceName -eq <span class="str">'_Total'</span> -and<br /> $_.MonitoringObjectPath -like <span class="str">'usuallyAServerNameFilter*'</span> } |<br /> get-performancecountervalue -starttime $starttime -endtime $endtime | <br /> select SampleValue, TimeSampled | <br /> export-csv <span class="str">"c:\temp\ConnectionAttemptsPerSec.csv"</span></pre><br /><br /><pre class="csharpcode"><font face="Georgia">You can also use the .Net framework and languages to get the data – the documentation isn’t great but this example works (I’ve reduced the font size to try and make it fit in this blog template):</font></pre><br /><br /><pre class="csharpcode"><font size="1"><span class="rem">/// <summary> </span><br /><span class="rem">/// Gather performance data</span><br /><span class="rem">/// </summary></span><br /><span class="kwrd">using</span> System;<br /><span class="kwrd">using</span> System.Collections.ObjectModel;<br /><span class="kwrd">using</span> Microsoft.EnterpriseManagement;<br /><span class="kwrd">using</span> Microsoft.EnterpriseManagement.Common;<br /><span class="kwrd">using</span> Microsoft.EnterpriseManagement.Configuration;<br /><span class="kwrd">using</span> Microsoft.EnterpriseManagement.Monitoring;<br /><br /><span class="kwrd">namespace</span> MySamples<br />{<br /> <span class="kwrd">class</span> Program<br /> {<br /> <span class="kwrd">static</span> <span class="kwrd">void</span> Main(<span class="kwrd">string</span>[] args)<br /> {<br /> ManagementGroup mg = <span class="kwrd">new</span> ManagementGroup(<span class="str">"someManagementServer"</span>);<br /><br /> MonitoringPerformanceDataCriteria mpdc = <br /> <span class="kwrd">new</span> MonitoringPerformanceDataCriteria(</font><font size="1"><span class="str">@"ObjectName = 'ASP.NET' and CounterName like 'Request Execution%' and MonitoringObjectPath like 'usuallyAServerNameFilter%'"</span>);<br /> ReadOnlyCollection<MonitoringPerformanceData> mpds = <br /> mg.GetMonitoringPerformanceData(mpdc);<br /><br /> <span class="kwrd">if</span> (mpds.Count > 0) {<br /> ReadOnlyCollection<MonitoringPerformanceDataValue> mpdvs = mpds[0].GetValues(DateTime.Today.AddDays(-1), DateTime.Today);<br /><br /> <span class="kwrd">foreach</span> (MonitoringPerformanceDataValue mpdv <span class="kwrd">in</span> mpdvs) {<br /> Console.WriteLine(<span class="str">"TimeSampled = "</span> + mpdv.TimeSampled + <br /> <span class="str">", SampleValue = "</span> + mpdv.SampleValue);<br /> }<br /> }<br /> Console.ReadLine();<br /> }<br /> }<br />}</font></pre><br /><br /><pre class="csharpcode"><font face="Georgia">Now I typically take the data out to F# and/or Excel and graph/model it to help figure my way through a problem. However, Microsoft have just released a nicely packaged analysis tool in Microsoft Sho – basically IronPython wrapped up with some plotting/graphing/math libraries – much like many of the other Python packages out there.</font></pre><br /><br /><pre class="csharpcode"><font face="Georgia">I thought this would be a fun way to interactively deal with performance data collection and analysis so I tried the following as an experiment (put in appropriate management server/monitoring object filters for you – I just used ASP.NET request execution times from a test environment as in the programs above). The hist command takes a list, or I guess anything it can enumerate over and get numbers from, and returns a histogram dialog and plot.</font></pre><br /><br /><pre class="csharpcode">ShoLoadAssembly(<span class="str">"c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll"</span>)<br />ShoLoadAssembly(<span class="str">"c:\Program Files\System Center Operations Manager 2007\SDK Binaries\Microsoft.EnterpriseManagement.OperationsManager.dll"</span>)<br /><br />import Microsoft.EnterpriseManagement<br />import Microsoft.EnterpriseManagement.Configuration<br />from Microsoft.EnterpriseManagement.Monitoring import *<br />from System import *<br /><br />mg = Microsoft.EnterpriseManagement.ManagementGroup(<span class="str">"aManagementServer"</span>)<br /><br />mpdc = MonitoringPerformanceDataCriteria(<span class="str">"ObjectName = 'ASP.NET' and CounterName like 'Request Execution%' and MonitoringObjectPath like 'someMonitoringObjectFilter%'"</span>)<br /><br />mpds = mg.GetMonitoringPerformanceData(mpdc)<br />mpds.Count<br /><br />hist([mpdv.SampleValue <span class="kwrd">for</span> mpdv <span class="kwrd">in</span> mpds[0].GetValues(DateTime.Today.AddDays(-1), DateTime.Today)]<br />)</pre><br /><style type="text/css"><br /><br />.csharpcode, .csharpcode pre<br />{<br /> font-size: small;<br /> color: black;<br /> font-family: consolas, "Courier New", courier, monospace;<br /> background-color: #ffffff;<br /> /*white-space: pre;*/<br />}<br />.csharpcode pre { margin: 0em; }<br />.csharpcode .rem { color: #008000; }<br />.csharpcode .kwrd { color: #0000ff; }<br />.csharpcode .str { color: #006080; }<br />.csharpcode .op { color: #0000c0; }<br />.csharpcode .preproc { color: #cc6633; }<br />.csharpcode .asp { background-color: #ffff00; }<br />.csharpcode .html { color: #800000; }<br />.csharpcode .attr { color: #ff0000; }<br />.csharpcode .alt <br />{<br /> background-color: #f4f4f4;<br /> width: 100%;<br /> margin: 0em;<br />}<br />.csharpcode .lnum { color: #606060; }</style><br /><br /><pre class="csharpcode"><font face="Georgia">And here’s the output:</font></pre><br /><br /><pre class="csharpcode"><a href="http://lh4.ggpht.com/_cWvIIiKNA4g/TVUIedaVGLI/AAAAAAAAANs/8FhbciEHoVQ/s1600-h/image2.png"><img style="background-image: none; border-right-width: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px; padding-top: 0px" title="image" border="0" alt="image" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIRMmxJVX5mTvSVZPvPunrNK0d4pmFUrEWMdMY-5X928T4y48u6-5egyLD1xCs_AkE-vWpCQspMdIQ8umE397FpFPBgm6hrVYggkl5DUkjr1x76NvHIQoR88taGkUaRrunQ-dx4w/?imgmax=800" width="244" height="183" /></a></pre><br /><br /><pre class="csharpcode"><font face="Georgia">Cool huh!?</font></pre><br /><style type="text/css"><br /><br />.csharpcode, .csharpcode pre<br />{<br /> font-size: small;<br /> color: black;<br /> font-family: consolas, "Courier New", courier, monospace;<br /> background-color: #ffffff;<br /> /*white-space: pre;*/<br />}<br />.csharpcode pre { margin: 0em; }<br />.csharpcode .rem { color: #008000; }<br />.csharpcode .kwrd { color: #0000ff; }<br />.csharpcode .str { color: #006080; }<br />.csharpcode .op { color: #0000c0; }<br />.csharpcode .preproc { color: #cc6633; }<br />.csharpcode .asp { background-color: #ffff00; }<br />.csharpcode .html { color: #800000; }<br />.csharpcode .attr { color: #ff0000; }<br />.csharpcode .alt <br />{<br /> background-color: #f4f4f4;<br /> width: 100%;<br /> margin: 0em;<br />}<br />.csharpcode .lnum { color: #606060; }</style><style type="text/css"><br /><br />.csharpcode, .csharpcode pre<br />{<br /> font-size: small;<br /> color: black;<br /> font-family: consolas, "Courier New", courier, monospace;<br /> background-color: #ffffff;<br /> /*white-space: pre;*/<br />}<br />.csharpcode pre { margin: 0em; }<br />.csharpcode .rem { color: #008000; }<br />.csharpcode .kwrd { color: #0000ff; }<br />.csharpcode .str { color: #006080; }<br />.csharpcode .op { color: #0000c0; }<br />.csharpcode .preproc { color: #cc6633; }<br />.csharpcode .asp { background-color: #ffff00; }<br />.csharpcode .html { color: #800000; }<br />.csharpcode .attr { color: #ff0000; }<br />.csharpcode .alt <br />{<br /> background-color: #f4f4f4;<br /> width: 100%;<br /> margin: 0em;<br />}<br />.csharpcode .lnum { color: #606060; }</style> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-382927389777459332011-02-09T22:29:00.001+13:002011-02-09T22:29:36.651+13:00Engineering tricks with WolframAlpha<p>I’ve found Google great for simple problems like this:</p> <p>20 pounds per cubic foot in kilograms per cubic meter= 320.36926 kilograms per cubic meter</p> <p>But wolframalpha also lets you combine unit conversions with calculations:</p> <p>(2 * 175 grams) per square meter * (2*3 meters squared) + ((20 pounds per cubic foot in kilograms per cubic meter) * 3 square meters * 5 millimeters)</p> <p>Or just click on this link: <a href="http://www.wolframalpha.com/input/?i=(2+*+175+grams)+per+square+meter+*+(2*3+meters+squared)+%2B+((20+pounds+per+cubic+foot+in+kilograms+per+cubic+meter)+*+3+square+meters+*+5+millimeters)">http://www.wolframalpha.com/input/?i=(2+*+175+grams)+per+square+meter+*+(2*3+meters+squared)+%2B+((20+pounds+per+cubic+foot+in+kilograms+per+cubic+meter)+*+3+square+meters+*+5+millimeters)</a></p> <p>What’s the above tell me? It’s the theoretical weight of the kayak I’m currently building – about 7kg. In reality when hand laying fibreglass you get a much higher weight of epoxy rather than the one to one ratio that’s possible under vacuum application and which I’m using above.</p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-27972751915434233052011-02-09T22:16:00.001+13:002011-02-09T22:16:24.448+13:00Hull fibreglassing<p>It’s taken awhile to get here, but the hull is now fibreglassed.</p> <ul> <li>After struggling with the Resene colorwood stain affecting epoxy seal coats I found that a wash with isopropyl alcohol made for a nice clean bonding surface.</li> <li>I rediscovered how to apply epoxy, everyone seems to have their own method but for me it’s plastic cards (old atm cards/loyalty cards etc are ideal) and short foam (usually yellow) rollers.</li> <li>A hot air gun does absolute wonders at getting rid of air bubbles/foam.</li> <li>I used high density 170g/sq m fibreglass that’s often used for model making. That equates to 5 ounce high density fibreglass which should be pretty strong (easy way to do the conversion… just use google – type “175 grams per square meter in ounces per square yard” in the search box).</li> <li>Wet sanding within 48 hours is more effectively at smoothing the epoxy finish than waiting for it to set really hard.</li> </ul> <p><a href="http://lh4.ggpht.com/_cWvIIiKNA4g/TVJbZcL8BDI/AAAAAAAAANc/CSPoB74eejw/s1600-h/second%20hull%20fill%20coat%5B2%5D.jpg"><img style="background-image: none; border-bottom: 0px; border-left: 0px; margin: ; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="second hull fill coat" border="0" alt="second hull fill coat" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhFt9D1a_UIIKM50LLeQ3liEhJLZHymVia6dkKPwuXMddCMlTw7amkRJHFETWII2q4rszFvvbV3X667RRXTxnW8wfAwio427c9an0t5hQEBh1dAPMzBn80i1ny_k3qDTSCyGy_Ppw/?imgmax=800" width="93" height="244" /></a></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-60235921925623671892011-02-03T08:32:00.001+13:002011-02-09T22:36:37.373+13:00Hull ready to be fibreglassed<p>Finally completed staining/fairing/sanding the hull. Long process with lots of waiting for epoxy to dry. Here’s a photo of how the hull looks wetted down. The staining in combination with a kind of matching fairing compound does look patchy but I’ll roll on a very thin layer of epoxy to seal the wood before the fibreglass goes on and that will darken it down and make it look a lot more consistent. </p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhy5r2AqHM77mBpvBf1ZGn2Bq9KDXV8GDGRFHla5AuSr33ROwpcaLbPRZaNA3UBlsfG25-h28EGr9GmsgEPsrV5hD7DXL8YOXGjc7k4wVjbxhZTYSik_Ya_3nUH6sGvBPssSW_Uhg/s1600-h/WP_000022%5B4%5D.jpg"><img style="background-image: none; border-bottom: 0px; border-left: 0px; margin: ; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="WP_000022" border="0" alt="WP_000022" src="http://lh3.ggpht.com/_cWvIIiKNA4g/TUmxagabmZI/AAAAAAAAANU/QCipttLLr6k/WP_000022_thumb%5B5%5D.jpg?imgmax=800" width="244" height="209" /></a></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-51294222367635856972011-01-26T22:32:00.001+13:002011-01-26T22:32:52.762+13:00Stems added and fairing the hull<p>Slow progress on the Black Pearl (yet to give it my own name yet) kayak. I’ve added stems (just the white paulownia so not so tough but shows up nicely against the stained hull), and I’ve been applying a fairing compound made of epoxy, lightweight filler, wood dust etc.</p> <p>Just waiting 7 days now for it to thoroughly set before I sand back.</p> <p>Here’s a pic:</p> <p> </p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheB5P0M5oZjNmzMMl0tM0RhUtnN96ndLHM-QCDxTs9BPhuQ6VHJ7J6L2bRxuOZS6iIXFKxv64LQwLlEUCH_3veSpCjByVqzMlz7QeIPTYn2O9uTVdmGoGaGU1r5NfhDBMEamjI_A/s1600-h/20%5B3%5D.jpg"><img style="background-image: none; border-bottom: 0px; border-left: 0px; margin: ; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="20" border="0" alt="20" src="http://lh3.ggpht.com/_cWvIIiKNA4g/TT_qQotVAUI/AAAAAAAAANI/QYycxeYW-6s/20_thumb.jpg?imgmax=800" width="244" height="184" /></a></p> <p>Biggest lesson from all this… get the strips flat before staining! Basically the better the stripping the easier the rest will be.</p> <p>Staining has been interesting… painful but interesting. The Resene Colorwood stain does work but you definitely need only a light rag application and maybe a wash with water after it’s set up. For the fairing I’ve managed to replicate the color using epoxy colours (black, blue and red – I watched them making it at Resene so I knew the constituent colours and proportions – it was on a their computer screen <img style="border-bottom-style: none; border-left-style: none; border-top-style: none; border-right-style: none" class="wlEmoticon wlEmoticon-smile" alt="Smile" src="http://lh6.ggpht.com/_cWvIIiKNA4g/TT_qQ6cfqgI/AAAAAAAAANM/41nG8qWfrkA/wlEmoticon-smile%5B2%5D.png?imgmax=800" />).</p> <p>Hard to get a good photo while it sits in the garage but this boat looks real sleek.</p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-29237299805313163802010-12-27T22:26:00.001+13:002011-02-09T22:36:37.374+13:00Hull Stripping and Attempted Staining<p>Stripping was relatively easy and very quick. The wood reminded me of making balsa wood models when I was a kid:)</p> <p>Main points were:</p> <ul> <li>Make sure strips are even thickness – if I do this again I’ll invest in a planner thicknesser.</li> </ul> <p><a href="http://lh4.ggpht.com/_cWvIIiKNA4g/TRhb3HKnRYI/AAAAAAAAAM0/jjXqpLvslB0/s1600-h/WP_000011%5B3%5D.jpg"><img style="background-image: none; border-bottom: 0px; border-left: 0px; margin: ; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="WP_000011" border="0" alt="WP_000011" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnBX6b8yN9maWCuKcAHeQTI6EHAxvpQu7UGfVll1rCBv9Dzqmej2mRgG7P4WrnfIRGG_bt-3W5b3vOgaJbB8YWug4c74s_pKLCTDRbUWhV-rqpHX4IRJyCuBJZj1utKXHuoZ3bOg/?imgmax=800" width="244" height="184" /></a></p> <ul> <li>I mostly just stapled strips only occasionally using a bead of glue – this had one advantage – towards the end as I looked at the forms and the shape of the hull I realised I had one form on  slightly wrong angle – not sure how this happened as I was sure I had it right at the beginning. Nevermind, I unscrewed it and re-adjusted just a tiny bit and suddenly it looked great. </li> <li>I thought I’d stain the strips before using epoxy to glue them all thogether – unfortunately, Resene interior stain turns out to not be as epoxy compatible as the Resene staff thought it would be. It beaded a bit but wasn’t irrecoverable – I might even put more stain on after I rub it all back in preparation for fibreglassing – but if I do I’ll wash with acetone and use a much thinner application of stain. Check the image below…</li> </ul> <p><a href="http://lh3.ggpht.com/_cWvIIiKNA4g/TRhb4Cu-AHI/AAAAAAAAAM8/GI6yCz9bDDg/s1600-h/WP_000024%5B3%5D.jpg"><img style="background-image: none; border-bottom: 0px; border-left: 0px; margin: ; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="WP_000024" border="0" alt="WP_000024" src="http://lh5.ggpht.com/_cWvIIiKNA4g/TRhb4VWPFEI/AAAAAAAAANA/wT-fZppyiRI/WP_000024_thumb.jpg?imgmax=800" width="244" height="184" /></a></p> <p> </p> <p>Hull looks great but I can’t get a good photo showing it the way I see it in real life. Perhaps for the next posting…</p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-31059796497508787442010-12-08T13:08:00.001+13:002011-02-09T22:37:17.406+13:00Spatial Notes–Getting Started<p>Thought this would be harder than it turned out to be.</p> <p>Objective: chart some metrics across geographic regions (in my case looking at customer metrics over New Zealand).</p> <p>To get started I took advantage of SQL Server Spatial. Initially appearances are a  bit daunting until you try it… and it’s fairly easy.</p> <ul> <li>Finding the GIS data <ul> <li>for NZ  Statistics NZ publish digital boundaries – versioned by publication year (2007, 2001, 1996 etc: <a href="http://www.stats.govt.nz/browse_for_stats/people_and_communities/geographic-areas/download-digital-boundaries.aspx">http://www.stats.govt.nz/browse_for_stats/people_and_communities/geographic-areas/download-digital-boundaries.aspx</a> </li> <li>The data is available in both ESRI Shapefile format and MapInfo format. You can use <a href="http://sharpgis.net/page/Shape2SQL.aspx" target="_blank">shape2sql</a> to load this data into sql server spatial. Unfortunately… it has a bug, it won’t create a table from the shp file inside shape2sql – I had to use SQL Profiler to capture the create table instruction and run it interactively – after that the data load works fine - not sure why. </li> <li>The data has a hierarchy of increasingly larger geometrical areas made up of units of the smallest size (the meshblock). There’s Area Units, Urban Areas, Regional Councils etc. <br /></li> </ul> </li> <li>Connecting people to areas <ul> <li>Either pay for the GeoPAF file from NZ Post – this gives the meshblock for a delivery address and a NZ Map Grid coordinate. (Not so useful NZ Post… NZMG has been superceeded by NZTM which conforms with WGS84.) </li> <li>Or, use the freely obtainable streetlink data from Stats NZ:  <a title="http://www.stats.govt.nz/browse_for_stats/people_and_communities/geographic-areas/streetlink.aspx" href="http://www.stats.govt.nz/browse_for_stats/people_and_communities/geographic-areas/streetlink.aspx">http://www.stats.govt.nz/browse_for_stats/people_and_communities/geographic-areas/streetlink.aspx</a> </li> </ul> </li> <li>Aggregating <ul> <li>Use SQL Server Analysis Services, or much faster </li> <li>Use PowerPivot – for me this allowed me to very quickly aggregate customer metrics from an internal database across areas loaded from the Stats NZ files </li> </ul> </li> <li>Reporting <ul> <li>It’d be nice if Excel had spatial reporting support, but it doesn’t – big thumbs down Microsoft… </li> <li>I used SQL Server Reporting Services and the Map control. Problem is that I need to have the map data in SQL/SSAS. It’d be nice to load from PowerPivot but I can’t do that unless I store the Excel PowerPivot doc in Sharepoint 2010. Jumping through hoops to do this though… Microsoft – just get the spatial support into Excel please! </li> </ul> </li> </ul> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-47832331695036559122010-12-05T21:12:00.001+13:002010-12-06T18:45:30.260+13:00Strip cutting<p>I followed <a href="http://www.thomassondesign.com/building/building_manual/preparations.aspx">Bjorn’s instructions</a> and cut the strips with a circular saw. The planks were pinned to a folding stand at each end. The paulownia was extremely stable. It didn’t split when I pinned it at the edge of the plank. Not one of the strips split or snapped. In the end it was much easier than I expected. Suspect it would’ve been even easier if I had a decent saw…</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivIQ2C30mKgVU3e8j59rsOX8fHaNS8uBLFm-kmNVNpf_vAdQrIgVqnBy0jkoizdvLHm65BsaJzzlAqmb8npc7FWwwotv67gKIylnTp9Ry211xp8t9LjHeN9COZAB9ezO92Kz4HzQ/s1600-h/Photo0045%5B4%5D.jpg"><img style="background-image: none; border-bottom: 0px; border-left: 0px; margin: ; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="Photo0045" border="0" alt="Photo0045" src="http://lh5.ggpht.com/_cWvIIiKNA4g/TPtJb1oO72I/AAAAAAAAAMU/DR5RIBrjRtU/Photo0045_thumb%5B6%5D.jpg?imgmax=800" width="118" height="244" /></a></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com2tag:blogger.com,1999:blog-17672034.post-8075981266172555372010-12-05T21:01:00.001+13:002010-12-06T18:45:30.261+13:00Ebonizing Paulownia<p>Vinegar and steel wool, left for a few days, filtered and then applied to paulownia strip – thinking of using this on my kayak. Top of the strip has been painted twice.</p> <p><a href="http://lh5.ggpht.com/_cWvIIiKNA4g/TPtGyxXWwOI/AAAAAAAAAMA/rsxprQGD_Sk/s1600-h/Photo0047%5B3%5D.jpg"><img style="background-image: none; border-bottom: 0px; border-left: 0px; margin: ; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px" title="Photo0047" border="0" alt="Photo0047" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisRY1x70IqZyicA7AYDBoYGHvI8Yz_mWwR6p5fjFFjhDjVQ6nIS7ntX8hq5pzJxmKfQSEW0so5BPcN0LzZyu_i8UKUuMJzr4B_FN8Hjss0W4wMKQhSnJo190e30UUlEVil_Gqj6A/?imgmax=800" width="184" height="244" /></a></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com3tag:blogger.com,1999:blog-17672034.post-10383112017135766202010-11-28T22:07:00.001+13:002010-11-29T09:34:36.596+13:00Black Pearl Forms<p>So terribly slow but here’s a snippet of the forms. </p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwvu59TXuZOp7pO3kTVe2Q5H6T-79BiV9X5F6ghXfJI1RmM6VrRms2oHH3u2THiNiLrHD7EumwM-CWtY0ZzeLmmL9z2wD9yK6bZgDi66gXC4VR-pdin8wPmaMnpMU0Scvb7-ICkg/s1600-h/IMG_1180%5B3%5D.jpg"><img style="background-image: none; border-right-width: 0px; margin: ; padding-left: 0px; padding-right: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px; padding-top: 0px" title="IMG_1180" border="0" alt="IMG_1180" src="http://lh5.ggpht.com/_cWvIIiKNA4g/TPIb5-u3HKI/AAAAAAAAAL0/Go0MO4iHBlg/IMG_1180_thumb.jpg?imgmax=800" width="164" height="244" /></a></p> <p> </p> <p>Lessons learned:</p> <ul> <li>Level everything first – makes it much easier to align cross sections </li> <li>Don’t start screwing cross sections to wooden cross bars until they all line up right. Use clamps first. </li> <li>Drill a hole about 5mm on the center of the waterline and shine a bright light from one end – you should be able to see the light all the way down </li> <li>I pre-cut the forms for the deck and re-attached them – supposedly to make it easier for later. Not sure it was worth it. </li> </ul> <p>Here’s that light shining through a couple of the holes.</p> <p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2EF4D0NANcngwwXVQC9R3zZ0gqRepEVdka5zFbeGDfP2Z8E5zTFw-EsCpgpXkdB2KvzT9YaDUhIZQfGx9E24XNJpKKFvlnZSLTAKEA98BM6ALexkzQ4gS-qi_vpj7McTo2m_p1A/s1600-h/IMG_1182%5B3%5D.jpg"><img style="background-image: none; border-right-width: 0px; margin: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px; padding-top: 0px" title="IMG_1182" border="0" alt="IMG_1182" src="http://lh3.ggpht.com/_cWvIIiKNA4g/TPIb7BnUwbI/AAAAAAAAAL8/cffog7GLsrc/IMG_1182_thumb.jpg?imgmax=800" width="244" height="164" /></a></p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com2tag:blogger.com,1999:blog-17672034.post-9646606533668011482010-10-05T10:05:00.001+13:002010-10-05T10:05:32.686+13:00New Kayak Build: Black Pearl<p>Two years on from building a Guillemot Kayaks <a href="http://bohdanszymanik.blogspot.com/2009/03/s-night-heron-varnishing-and-outfitting.html" target="_blank">Night Heron stitch and glue</a> I’ve just purchased plans for a <a href="http://www.thomassondesign.com/choosing/my_kayaks/black_pearl.aspx" target="_blank">Bjorn Thomasson Black Pearl</a>. The Night Heron is a great kayak that’s very quick in the water but I just enjoyed the building side too much so I’ve decided to try building a strip kayak. Rather than go with something closer to the Night Heron I decided to go down a slightly different path:</p> <ul> <li>As light as possible – means small or low volume</li> <li>Doesn’t need speed – the Night Heron fulfils that requirement for me</li> <li>Play boat – I’ve taken up canoe polo so I want the most manuverable sea kayak experience I can get – something that will respond to hip/torso movement.</li> </ul> <p>I think the Black Pearl will do this but the proof will be in the pudding. You don’t just order this off the book – the plans are custom printed to your height, arm span and hip measurements.</p> <p>I’ll update the blog as I go. It will take awhile – no rush as I have an existing kayak already.</p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com2tag:blogger.com,1999:blog-17672034.post-90461224598533844082010-10-04T13:47:00.001+13:002010-10-04T13:52:24.621+13:00Populating a Word Document from a Sharepoint List<p>I thought this would be simpler since Sharepoint and Word are so closely related. A mail merge based on list data perhaps? Maybe it’s possible with an office data connection file but I couldn’t figure it out.</p> <p>Well, it was useful to me so here’s how I did it with Powershell and Word automation. The example here is a list of standards and the output will be a list of those standards formatted on one Word document. The soap call from powershell comes from somewhere on the net with an added soapaction header. The list has been created with folder items so it needs to list the items recursively – this wouldn’t apply to a normal list. You’ll figure it out as you start looking at the xml attributes. Best way to make progress with these sharepoint soap calls is to use Wireshark in combination with Stramit Caml Viewer.</p> <p>Here we go…</p> <pre class="csharpcode">function Execute-SOAPRequest <br />( <br /> [Xml] $SOAPRequest,<br /> [String] $SOAPAction,<br /> [String] $URL <br />) <br />{ <br /> write-host <span class="str">"Sending SOAP Request To Server: $URL"</span> <br /> $soapWebRequest = [System.Net.WebRequest]::Create($URL) <br /> $soapWebRequest.Headers.Add(<span class="str">"SOAPAction"</span>,<span class="str">"`"</span><span class="str">" + $SOAPAction + "</span>`<span class="str">""</span>) <br /><br /> $soapWebRequest.ContentType = <span class="str">"text/xml;charset=`"</span>utf-8`<span class="str">""</span> <br /> $soapWebRequest.Accept = <span class="str">"text/xml"</span> <br /> $soapWebRequest.Method = <span class="str">"POST"</span> <br /> <br /> write-host <span class="str">"Authenticating"</span><br /> $soapWebRequest.Credentials = [System.Net.CredentialCache]::DefaultCredentials<br /> <span class="kwrd">if</span>($soapWebRequest.Proxy -ne $<span class="kwrd">null</span>) {<br /> $soapWebRequest.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials<br /> }<br /> <br /> write-host <span class="str">"Initiating Send."</span> <br /> $requestStream = $soapWebRequest.GetRequestStream() <br /> $SOAPRequest.Save($requestStream) <br /> $requestStream.Close() <br /> <br /> write-host <span class="str">"Send Complete, Waiting For Response."</span> <br /> $resp = $soapWebRequest.GetResponse() <br /> $responseStream = $resp.GetResponseStream() <br /> $soapReader = [System.IO.StreamReader]($responseStream) <br /> $ReturnXml = [Xml] $soapReader.ReadToEnd() <br /> $responseStream.Close() <br /> <br /> write-host <span class="str">"Response Received."</span> <br /><br /> <span class="kwrd">return</span> $ReturnXml <br />} <br /><br />function Add-Standard<br />(<br /> [String] $Standard,<br /> [String] $Justification,<br /> $Doc<br />)<br />{<br />$p = $Doc.Paragraphs.Add()<br />$p.Range.Text = $Standard<br />$p.Format.Style = <span class="str">"Heading 1"</span><br />$p.Range.InsertParagraphAfter()<br /><br />$p = $Doc.Paragraphs.Add()<br />$p.Range.Text = $Justification<br />$p.Range.InsertParagraphAfter()<br />}<br /><br />$d4 = [xml] <span class="str">@"<br /><?xml version="</span>1.0<span class="str">" encoding="</span>utf-8<span class="str">"?><br /><soap:Envelope xmlns:soap="</span>http:<span class="rem">//schemas.xmlsoap.org/soap/envelope/" </span><br /> xmlns:xsi=<span class="str">"http://www.w3.org/2001/XMLSchema-instance"</span> <br /> xmlns:xsd=<span class="str">"http://www.w3.org/2001/XMLSchema"</span>><br /> <soap:Body><br /> <GetListItems xmlns=<span class="str">"http://schemas.microsoft.com/sharepoint/soap/"</span>><br /> <listName>{CF6E48BA-650A-489E-83AB-8EF1E545388A}</listName><br /> <queryOptions><br /> <QueryOptions><br /> <ViewAttributes Scope=<span class="str">"Recursive"</span>/><br /> <IncludeMandatoryColumns>False</IncludeMandatoryColumns><br /> </QueryOptions><br /> </queryOptions><br /> </GetListItems><br /> </soap:Body><br /></soap:Envelope><br /><span class="str">"@<br /><br /><br /># create a document from list data!<br />$w = new-object -com Word.Application<br />$w.Visible = $true<br />$d = $w.Documents.Add()<br /><br />Execute-SOAPRequest -SOAPRequest $d4 -SOAPAction "</span>http:<span class="rem">//schemas.microsoft.com/sharepoint/soap/GetListItems" -URL "http://<some sharepoint site>/sites/CTO/_vti_bin/lists.asmx" |</span><br /><span class="kwrd">foreach</span> {$_.Envelope.Body.GetListItemsResponse.GetListItemsResult.listitems.data.row |<br /> <span class="kwrd">foreach</span> {Add-Standard -Standard $_.ows_Standard -Justification $_.ows_Justification -d $d }<br />}<br /><br />$d.SaveAs([<span class="kwrd">ref</span>]<span class="str">"c:\temp\standards.docx"</span>)<br />$d.Close()<br />$w.Quit()</pre><br /><style type="text/css"><br />.csharpcode, .csharpcode pre<br />{<br /> font-size: small;<br /> color: black;<br /> font-family: consolas, "Courier New", courier, monospace;<br /> background-color: #ffffff;<br /> /*white-space: pre;*/<br />}<br />.csharpcode pre { margin: 0em; }<br />.csharpcode .rem { color: #008000; }<br />.csharpcode .kwrd { color: #0000ff; }<br />.csharpcode .str { color: #006080; }<br />.csharpcode .op { color: #0000c0; }<br />.csharpcode .preproc { color: #cc6633; }<br />.csharpcode .asp { background-color: #ffff00; }<br />.csharpcode .html { color: #800000; }<br />.csharpcode .attr { color: #ff0000; }<br />.csharpcode .alt <br />{<br /> background-color: #f4f4f4;<br /> width: 100%;<br /> margin: 0em;<br />}<br />.csharpcode .lnum { color: #606060; }</style> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0tag:blogger.com,1999:blog-17672034.post-14182967043282828662010-08-15T11:01:00.001+12:002010-08-15T11:01:12.756+12:00FST Auckland Conference Presentation<p>You can find the presentation I did for the Financial Services Technology conference in Auckland last week <a href="http://cid-0e5ff53cd7f11485.office.live.com/view.aspx/Public/FST%20Auckland%20-%20Aug%202010%20-%20Future%20of%20Banking/Creating%20a%20Smarter%20Bank%20-%20FST%20Aug%202010.pptx">here</a>. </p> <p>Brief synopsis: systems/operational management provides a useful analogue to the problems faced by implementers of business intelligence environments: huge volumes of data streaming in for which you need to provide health indicators and predictions of the future. </p> <p>In the case of Microsoft’s System Center Operations Manager tool this is done in a manner that is very different from conventional BI efforts that rely on staging large quantities of data, building relational and dimensional models and producing KPIs. The solutions developed for operations management can be usefully applied to BI efforts, particularly when not dealing with financial data.</p> Bohdan Szymanikhttp://www.blogger.com/profile/13123728021973733938noreply@blogger.com0