Pages

September 11, 2013

Configuration Pattern

Easy configuration pattern with typed, stateful, and easy to use manage settings.

Settings are represented by a custom type. Since the type is owned by the user it can be fully tailored with little constraint. For example, in WPF it can support INotifyPropertyChanged and be completely bound to your view. It can have in memory default values, be inherited, and even encapsulated logic.

public class MySettings {
 int Host { get; set;}
 int Port { get; set;}
}

The settings library itself is a static container that manages the saving (serialization) and loading (deserialization) of your in-memory settings object. It also provides a consistent way of accessing the object as well as maintaining its lifetime.

public static class Config<T> where T : new(){
 private T _instance = new T(); //Can be made lazy
 
 public T Instance { get { return _instance; } }

  //Instance is created via configurator which 
  //implements the deserialization logic
  public static void Load(IConfigurator conf, string location){
  _instance= conf.Load(location);
 }

 //Instance is persisted via configurator which 
 //implements the serialization logic
 public static void Save(string location){
  conf.Save(Instance , location);
 }
}

A simple interface allows you to create your own configurators that define how serialization and deserialization will be performed. The configurators can be fed to the static config class, or the static config class can be written with pre-canned configurators.

interface IConfigurator
{
 T Load<T>(string location);
 void Save<T>(T settings, string location);
}


Using the configuration class is simple. Since the actual settings object is manged by the encapsulating static config class, all you need to do is just call it. Accessing the settings properties without loading will always yield the default memory values. Calling the Load method will update the internal instance of the settings object.

Config<MySettings>.Load(new XmlConfigurator(), "settings.xml");

Once loaded, the Instance property returns a shared instance of in memory settings.

var host = Config<MySettings>.Instance.Host;
var port = Config<MySettings>.Instance.Port;

The settings can be easily manipulated in memory.
Config<MySettings>.Instance.Host = host;
Config<MySettings>.Instance.Port = port;

The settings can also be saved just as easily.
Config<MySettings>.Save(new DBConfigurator(), sqlConnectionString);

September 6, 2013

Zip to Get Element to Element Delta

Often when manipulating time series, you may want to get difference between adjacent elements in a list.

Take a look at the example below:

"Original" is the original time series list of Date/Value, with the Delta row illustrating the expected dela we are trying to calculate.

Linq can handle this in a pretty simple way (this can be pasted into linqpad):

var original = new [] { 
  Tuple.Create(new DateTime(2010, 1, 1), 100),
  Tuple.Create(new DateTime(2010, 2, 1), 50),
  Tuple.Create(new DateTime(2010, 3, 1), 75)
};

var anchor = Tuple.Create(new DateTime(2009, 12, 1), 0);

var B = (new []{ anchor }).Concat(original);
var A = original;

var deltas = A.Zip(B, (a, b) => Tuple.Create(a.Item1, a.Item2 - b.Item2));

deltas.Dump(1);

Note that if you don't really care about the first diff, you can simply Skip and Zip like so:
var B = new [] { 
  Tuple.Create(new DateTime(2010, 1, 1), 100),
  Tuple.Create(new DateTime(2010, 2, 1), 50),
  Tuple.Create(new DateTime(2010, 3, 1), 75)
};

var A = B.Skip(1);

var deltas = A.Zip(B, (a, b) => Tuple.Create(a.Item1, a.Item2 - b.Item2));

deltas.Dump(1);

July 3, 2012

Introducing C# Toolkit for loading adhoc csv files into a database: Dfloader

Have you ever needed a simple programmatic API in C# for loading islands of data from csv files into a SQL database? Well we have been asked to do this sort of thing on a couple of different projects. So we wrote a little toolkit to allows us to manage the process of uploading adhoc csv files into a set of highly normalized tables in SQL Server.

The toolkit is open source - please do check it out at http://dfloader.codeplex.com

June 27, 2012

Sql Server TRY..CATCH

The other day had to do a giant update script on Microsoft SQL Server (this works in 2005+). And was wondering if something happens in the middle, what would happen to the transaction wrapping the sea of inserts/updates/deletes. Would all of them get rolled back, or only some?

It turns out that if you naively (although it makes sense) try to do:

BEGIN TRANSACTION
insert(1)...
insert(2)... -- Error here!
insert(3)...
COMMIT TRANSACTION

contrary to intuition, although insert(2) fails, and you would think the transaction gets aborted at this point, insert(3) still goes through. in order to rollback inserts 1 through 3, you would typically need to check for @@ERROR after each of the inserts, and do some sort of GOTO, which will let you jump to some error handling code section, which will perform ROLLBACK TRANSACTION.

Obviously this gets to be pretty, if not excruciatingly painful, to insert the check for @@ERROR after every single statement!

Thankfully there is a much cleaner way of doing this: TRY..CATCH

BEGIN TRANSACTION
BEGIN TRY
     insert(1)...
     insert(2)... -- Error here!
     insert(3)... -- This never happens.
     insert(4)...
     etc...
     -- If we got here there were no errors above.
     print 'Committing transaction...'
     COMMIT TRANSACTION
END TRY
BEGIN CATCH
     -- On insert(2) we drop through here. Nice!
     print 'Rolling back transaction...'
    ROLLBACK TRANSACTION
END CATCH;

Hope you find this useful... sure better than copy/pasting IF @@ERROR <> 0 a hundred times!

Excel Substitute function

2 small things that make fairly regular data munging tasks a little easier:

Excel

When dealing with lots of adhoc data and relational databases, you want to be able to generate insert/update SQL statements in excel, based on some data in the spreadsheet.

Typically you will have some template SQL statement with placeholders for data that varies from row to row. Here is one approach to accomplishing this in Excel with relative ease.

Put your SQL template in cell A1. Lets say my template looks like this:

insert into some_table (one, two) values (@some_value, '@some_string_value')

The funny looking  @ denote placeholder where we want to insert our variable content.
This turns out to be fairly easy to do in Excel, if you use SUBSTITUTE() function.

=SUBSTITUTE(SUBSTITUTE($A$1, "@some_value", B1), "@some_string_value", B2)

November 2, 2011

Web conference software

Once in a while we have a need to do web conferences. This need does not come up very often; for most purposes we are quite happy with Skype. However if you have a real conference going on (ie multiple people), or you need to share your desktop with someone who does not have Skype installed a tool like Webex is quite handy, however it costs money. Being so very thrifty, we did a little searching and found this little nugget: http://vyew.com; this little guy has a quite capable free version of web conferencing. We were able to share a desktop quite nicely. The only thing you need installed on host/client PC is java; which most of us should have by now. Let's just hope it does not get bought and killed like http://dimdim.com

June 1, 2011

nuget

Discovery of the week dependency and package management for .net: NUGET

Actually using it on one of our WPF .NET 4 project, after 2 hours of using it so far looks looks pretty solid, with pretty good tooling support in VS 2010.

One caveat, the Package Manager Console is kind of neat, but seems to be inferior to nuget.exe (especially when creating new packages).

A very solid, newer (may 2011) in depth video on use and features of nuget get is on channel 9

January 13, 2011

Rx.net ObserveOn performance

I been playing with benchmarking performance of pushing small messages via Rx where message producer and message consumer are on different threads. With one producer and possibly many consumers.

Rx has a useful operator called ObserveOn, which let's you process messages on the threadpool. However it seems to have a couple of shortcomings.

1. It does not process several messages concurrently, although it is using a thread pool, it will process messages serially. So if message post processing done by the consumer is time consuming, you may want fork and join type of semantics (spin off N tasks, where N is the size of the thread pool), and then assemble the results in the right order and forward (right order is very important for correct Rx operation).

2. Rx seems to be slow! I wrote a little benchmark that pushes small messages as fast as it can (messages are preallocated, so little memory allocator overhead in the test timing), then I implemented a synchronized queue (user ReaderWriterLockSlim and Queue) to do same produce/consume logic, and the results are dramatically different, tests we done on a 2 core, 3 Ghz machine.

With Rx I am getting around 220k msg/sec, with my hand coded queue, I am getting a wopping (compared to Rx), 4m msg/sec, that is 20x faster than Rx.

Source code for the benchmark is here (note to self, clean up):
https://gist.github.com/90983196d1a45e91d11e

January 3, 2011

Erlang Ring Benchmark Part 3

In Ring Benchmark Part 2 I presented a more compact solution to the ring benchmark. It's performance is about same as the original solution that was using pure recursion to setup the ring. Both part 1 and part 2 were waiting for a message to go around the ring completely, before injecting the next message. In this version we will inject messages 1 through M as soon as the ring is setup, this considerably improves throughput. Line 10 below is the part where we spawn off a process to inject M..0 messages into the ring.

Timings Across Part 1-3

  • Part 1: N=1000, M=1000, Run Time = 600 ms
  • Part 2: N=1000, M=1000, Run Time = 600 ms
  • Part 3: N=1000, M=1000, Run Time = 400 ms

Code


-module(ring3_2).
-include("debug.hrl").
-export([start/2]).
%% ring benchmark using lists methods
start(N, M) ->
    io:format("Starting ring benchmark on pid=~p, N = ~p, M = ~p.~n", [self(), N, M]),
    statistics(wall_clock),
    statistics(runtime),
    Last = lists:foldl(
    fun (X, Pid) -> 
        spawn(fun () -> loop(X, Pid) end) end, self(), lists:seq(N-1, 1, -1)),    
    spawn(fun () -> 
        lists:foreach(fun (X) -> Last ! X end, lists:seq(M-1, 0, -1)) end),
    wait(),
    {_, WC} = statistics(wall_clock),
    {_, RT} = statistics(runtime),
    io:format("Done running ring benchmark in Wall Clock = ~p, Runtime = ~p.~n", [WC, RT]).

wait() ->
    receive
 Xm when Xm =:= 0 -> 
     ?DEBUG("main node done waiting for x = ~p.~n", [Xm]),
     void;
 Xm -> ?DEBUG("main node done waiting for x = ~p.~n", [Xm]), wait()       
    end.

loop(Nth, Pid) ->
    receive
 X when X =:= 0 -> 
     ?DEBUG("~p dying.~n", [Nth]),
     Pid ! X;
 X -> 
     ?DEBUG("~p got message ~p.~n", [Nth, X]), 
     Pid ! X,
     loop(Nth, Pid)
    end.

Erlang Ring Benchmark Part 2

Using list comprehensions - results in more compact code.

-module(ring3).
-include("debug.hrl").
-export([start/2]).
%% ring benchmark using lists methods
start(N, M) ->
    io:format("Starting ring benchmark on pid=~p, N = ~p, M = ~p.~n", [self(), N, M]),
    statistics(wall_clock),
    statistics(runtime),
    Last = lists:foldl(fun (X, Pid) -> 
        spawn(fun () -> loop(X, Pid) end) end, self(), lists:seq(N-1, 1, -1)),    
    lists:foreach(fun (X) -> send(X, Last) end, lists:seq(M-1, 0, -1)),
    {_, WC} = statistics(wall_clock),
    {_, RT} = statistics(runtime),
    io:format("Done running ring benchmark in Wall Clock = ~p, Runtime = ~p.~n", [WC, RT]).

send(X, Pid) ->
    ?DEBUG("sending X=~p to ring.~n", [X]),
    Pid ! X,
    receive
 Xm -> ?DEBUG("main node done waiting for x = ~p.~n", [Xm])
    end.

loop(Nth, Pid) ->
    receive
 X when X =:= 0 -> 
     ?DEBUG("~p dying.~n", [Nth]),
     Pid ! X;
 X -> 
     ?DEBUG("~p got message.~n", [Nth]), 
     Pid ! X,
     loop(Nth, Pid)
    end.
           

December 28, 2010

Erlang Ring Benchmark

I am reading Joe Armstrong's Programming Erlang: Software for a Concurrent World, in chapter 8 he poses a little challenge to create a ring benchmark:
Write a ring benchmark. Create N processes in a ring. Send a mes- sage round the ring M times so that a total of N * M messages get sent. Time how long this takes for different values of N and M.
My solution is below... It was fun.

Usage from erlang command line

1> c(ring2).
2> ring2:start(100, 1000). % ring size = 100, send msg 1000 times round the ring
... sample below ...
42> ring2:start(300, 2000).
Starting ring benchmark on pid=<0.30.0>.
Done initializing nodes.
Got X(600000) >= M(600000) in pid=<0.30.0>.
Done running ring benchmark in Wall Clock = 437, Runtime = 391.
ok

The results above were obtained on a Core 2 Duo E8400 @ 3.00GHz

ring2.erl


-module(ring2).
-include("debug.hrl").
-export([start/2]).

%% ring benchmark

start(N, M) ->
    io:format("Starting ring benchmark on pid=~p.~n", [self()]),
    statistics(wall_clock),
    statistics(runtime),
    NextPid = node(N-1, N*M, self(), self()),
    NextPid ! 1,
    io:format("Done initializing nodes.~n"),
    wait(N*M, NextPid, self()),
    NextPid ! die,
    {_, WC} = statistics(wall_clock),
    {_, RT} = statistics(runtime),
    io:format("Done running ring benchmark in Wall Clock = ~p, Runtime = ~p.~n", [WC, RT]).
     
node(1, M, NextPid, DonePid) ->
    Pid = spawn(fun () -> wait(M, NextPid, DonePid) end),
    ?DEBUG("Creating final node: ~p.~n", [Pid]),
    Pid;
node(N, M, NextPid, DonePid) -> 
    Pid = spawn(fun () -> wait(M, NextPid, DonePid) end),
    ?DEBUG("Creating node ~p = ~p.~n", [N, Pid]),
    node(N-1, M, Pid, DonePid).

wait(M, NextPid, DonePid) ->
    receive
 die ->
     ?DEBUG("~p dying.~n", [self()]),
     if NextPid =/= DonePid -> NextPid ! die;
        true -> void
     end,
     void;
 X when X >= M -> 
     io:format("Got X(~p) >= M(~p) in pid=~p.~n", [X, M, self()]),
     DonePid ! die,
     wait(M, NextPid, DonePid);
 X   ->
     ?DEBUG("Got X = ~p in pid=~p, fwd to ~p.~n", [X, self(), NextPid]),
     NextPid ! X+1,
     wait(M, NextPid, DonePid)
    end.     


I am using a little debugging macro to turn logging on/off. At erlang shell: c(ring2, {d, debug}). to turn debugging trace on.

debug.hrl


-ifdef(debug).
-define(DEBUG(Format, Args), io:format("~s.~w: DEBUG: " ++ Format, [ ?MODULE, ?LINE | Args])).
-else.
-define(DEBUG(Format, Args), true).
-endif.

here are some solutions from other people: http://goo.gl/kE42w

December 27, 2010

Book Review: Beautiful Architecture


Beautiful Architecture: Leading Thinkers Reveal the Hidden Beauty in Software Design by Diomidis Spinellis


I got a chance to review one of the books in OReilly's "Beautiful" series, in this case it was Beautiful Architecture.
One of the attractions of the books in this series, is the premise that you get to hear first-hand accounts, in an essay format, from the people behind some of the more interesting projects out there. So the book can be viewed as an opportunity to ask a bunch of innovative technologists the question: how did you build it?

On the back cover the book claims that the contributors cover notable and innovative software architectures. At first glance the table of contents looked pretty promising, for example “The architecture of the Facebook platform”. However, after spending some time with the book and reading through 5 random chapters, it feels like there is just not enough substance for the book to achieve what most people are probably hoping to get out of the book: concrete blueprints or solid design patterns to build systems demonstrating desired characteristics.

For the first 45 or pages of the book, the essays tended to meander around the topics of general definition of architecture vs software design etc, and general experience of people working on some non-trivial projects. But somehow the conversation always feels more academic than practical. So you kind of feel like you are sitting on a plane waiting for it to disengage from the gate and move towards the tarmac. And given the fact that this book is supposed to be about “beautiful” architectures, I am not at all sure why we need to spend time reading about “The Messy Metropolis”. Given the title and general back cover description of the book I was expecting the narratives to be more specific and more parallels drawn with real-world systems that people interested in this book could be building.

The chapter “Architecting for Scale” by Jim Waldo, was also a bit underwhelming. We basically get a pretty high level overview of a system that was never really tested in the field, was not thoroughly bench-marked, and is addressing a fairly narrow problem domain of online gaming. Although some interesting ideas are presented, you have a hard time agreeing that this is necessarily beautiful because it does not seem that the system is proven to successfully solve their scalability goals. The chapter about facebook platform also failed to describe the thinking process and design trade-offs, you get a description of general building blocks but you don't really get any aha moments.

I did enjoy “Software Architecture: Object-Oriented versus Functional” by Bertrand Meyer, however one might argue that the book might have been better off focusing on systems architecture and concrete examples of real-world projects as opposed to discussion of type systems and other programming language esoterica, especially when the Meyer makes it clear that he has not actually worked in finance nor released production systems dealing with financial contracts.

I think this book is decent read on your e-book reader, but if you have limited bookshelf space and want to keep it filled with true classics, this book is not it.

3/5 stars.

December 18, 2010

Big Cities Make People More Efficient

From a NY Times article: A Physicist Solves the City
The first data set they analyzed was on the economic productivity of American cities, and it quickly became clear that their working hypothesis — like elephants, cities become more efficient as they get bigger — was profoundly incomplete. According to the data, whenever a city doubles in size, every measure of economic activity, from construction spending to the amount of bank deposits, increases by approximately 15 percent per capita. It doesn’t matter how big the city is; the law remains the same. “This remarkable equation is why people move to the big city,” West says. “Because you can take the same person, and if you just move them to a city that’s twice as big, then all of a sudden they’ll do 15 percent more of everything that we can measure.” While Jacobs could only speculate on the value of our urban interactions, West insists that he has found a way to “scientifically confirm” her conjectures. “One of my favorite compliments is when people come up to me and say, ‘You have done what Jane Jacobs would have done, if only she could do mathematics,’ ” West says. “What the data clearly shows, and what she was clever enough to anticipate, is that when people come together, they become much more productive.”

December 16, 2010

Rx.NET Subject useful for connecting decoupled components

Rx has a nifty abstraction called Subject shipped as part of the System.Reactive assembly.

It's basically a buffer with channel like capabilities rolling up an Obsever/Observable pattern into a single class.

In cases where you have two or more classes that don't really know about each other but sometimes need to be connected (ie output of one should go into input of another), you can can inject an instance of a Subject class into both components, and then have them send messages. Subject provides a standard Rx.NET friendly API to push/pull messages, this makes it quite powerful as you can use the full expressive power of Rx.NET to consume message from Subjects in different scenarios (one-to-one, one-to-many, filtering, Join, etc) .





November 17, 2010

Direct Edge stock exchange using .NET

Wow a major exchange, Direct Edge, is using .NET to run it's exchange.

http://blogs.technet.com/b/windowsserver/archive/2010/07/22/direct-edge-stock-exchange-launches-on-windows-server-sql-server.aspx

C++/CLI

Having spent a lot of time programming in C++ over the last few months, I have come to appreciate some of the features of the language, such as the powerful templating mechanism, which is quite a bit more powerful than the .NET generics. There are seem to be a few cases when one would have to resort to tricky reflection techniques in C#/VB.NET, which costs both in terms of runtime overhead and also in code mantainability.

I have only recently became more acutely aware of the C++/CLI. Seems like it has a number of cool features that a worth a look:
  1. template system similar to C++
  2. much cheaper mechanism to invoke native code (compared to PInvoke)
  3. possibility to mix and match managed and unmanaged memory pools for performance optimization.
There is a pretty solid article from a solid C++ guy about C++/CLI.

Here is an excerpt:

C++/CLI represents a tuple. C++ refers, of course, to the C++ programming language invented by Bjarne Stroustrup at Bell Laboratories. It supports a static object model that is optimized for the speed and size of its executables. However, it doesn't support run-time modification of the program other than heap allocation. It allows unlimited access to the underlying machine, but very little access to the types active in the running program and no real access to the associated infrastructure of that program. Herb Sutter, a former colleague of mine at Microsoft and the chief architect of C++/CLI, refers to C++ as a concrete language.


CLI refers to the Common Language Infrastructure, a multitiered architecture supporting a dynamic component programming model. In many ways, this represents a complete reversal of the C++ object model. A runtime software layer, the virtual execution system, runs between the program and the underlying operating system. Access to the underlying machine is fairly constrained. Access to the types active in the executing program and the associated program infrastructure—both as discovery and construction—is supported. The slash (/) represents a binding between C++ and the CLI. The details surrounding this binding make up the general topic of this column.


So, a first approximation of an answer to what is C++/CLI is that it is a binding of the static C++ object model to the dynamic component object model of the CLI. In short, it is how you do .NET programming using C++ rather than C# or Visual Basic®. Like C# and the CLI itself, C++/CLI is undergoing standardization under the European Computer Manufacturers Association (ECMA) and eventually under ISO.


The common language runtime (CLR) is the Microsoft version of the CLI that is specific to the Windows® operating system. Similarly, Visual C++® 2005 is the implementation of C++/CLI.


As a second approximation of an answer, I would say that C++/CLI integrates the .NET programming model within C++ in the same way as, back at Bell Laboratories, we integrated generic programming using templates within the then existing C++. In both of these cases your investment in an existing C++ codebase and in your existing C++ expertise are preserved. This was an essential baseline requirement of the design of C++/CLI.

Memory Tour De Force

What Every Programmer Should Know About Memory by Ulrich Drepper

Fascinating, enlightening, extremely useful (memory speeds have not kept up with progress in raw computing power in modern computers see, beware of cpu stalling!). Expect it to take a couple of weeks to get through it and really digest all of it, it's a self-contained mini-course in electrical engineering...

here is a diagram comparing memory access speed vs cpu speed evolution - notice the growing gap.
from another excellent read showing how proper memory access can dramatically speed up your program.

November 8, 2010

Earth Equatorial Radius == 1.0 * Length Of Amazon River

Creepy coincidence or as-designed by aliens?

Earth Equator Radius = Amazon River Length = 6400 km

Note, equator radius is not same as circumference (which is around 2*pi*r = 40,000 km)