Privosoft LLC Blog

Web conference software

2011-11-02T17:31:00.004-05:00

Once in a while we have a need to do web conferences. This need does not come up very often; for most purposes we are quite happy with Skype. However if you have a real conference going on (ie multiple people), or you need to share your desktop with someone who does not have Skype installed a tool like Webex is quite handy, however it costs money. Being so very thrifty, we did a little searching and found this little nugget: http://vyew.com; this little guy has a quite capable free version of web conferencing. We were able to share a desktop quite nicely. The only thing you need installed on host/client PC is java; which most of us should have by now. Let's just hope it does not get bought and killed like http://dimdim.com

nuget

2011-06-01T18:46:00.001-05:00

Discovery of the week dependency and package management for .net: NUGET

Actually using it on one of our WPF .NET 4 project, after 2 hours of using it so far looks looks pretty solid, with pretty good tooling support in VS 2010.

One caveat, the Package Manager Console is kind of neat, but seems to be inferior to nuget.exe (especially when creating new packages).

A very solid, newer (may 2011) in depth video on use and features of nuget get is on channel 9

Rx.net ObserveOn performance

2011-01-13T11:18:00.002-05:00

I been playing with benchmarking performance of pushing small messages via Rx where message producer and message consumer are on different threads. With one producer and possibly many consumers.

Rx has a useful operator called ObserveOn, which let's you process messages on the threadpool. However it seems to have a couple of shortcomings.

1. It does not process several messages concurrently, although it is using a thread pool, it will process messages serially. So if message post processing done by the consumer is time consuming, you may want fork and join type of semantics (spin off N tasks, where N is the size of the thread pool), and then assemble the results in the right order and forward (right order is very important for correct Rx operation).

2. Rx seems to be slow! I wrote a little benchmark that pushes small messages as fast as it can (messages are preallocated, so little memory allocator overhead in the test timing), then I implemented a synchronized queue (user ReaderWriterLockSlim and Queue) to do same produce/consume logic, and the results are dramatically different, tests we done on a 2 core, 3 Ghz machine.

With Rx I am getting around 220k msg/sec, with my hand coded queue, I am getting a wopping (compared to Rx), 4m msg/sec, that is 20x faster than Rx.

Source code for the benchmark is here (note to self, clean up):
https://gist.github.com/90983196d1a45e91d11e

Erlang Ring Benchmark Part 3

2011-01-03T14:10:00.003-05:00

In Ring Benchmark Part 2 I presented a more compact solution to the ring benchmark. It's performance is about same as the original solution that was using pure recursion to setup the ring. Both part 1 and part 2 were waiting for a message to go around the ring completely, before injecting the next message. In this version we will inject messages 1 through M as soon as the ring is setup, this considerably improves throughput. Line 10 below is the part where we spawn off a process to inject M..0 messages into the ring.

Timings Across Part 1-3

Part 1: N=1000, M=1000, Run Time = 600 ms
Part 2: N=1000, M=1000, Run Time = 600 ms
Part 3: N=1000, M=1000, Run Time = 400 ms

Code

-module(ring3_2).
-include("debug.hrl").
-export([start/2]).
%% ring benchmark using lists methods
start(N, M) ->
    io:format("Starting ring benchmark on pid=~p, N = ~p, M = ~p.~n", [self(), N, M]),
    statistics(wall_clock),
    statistics(runtime),
    Last = lists:foldl(fun (X, Pid) -> spawn(fun () -> loop(X, Pid) end) end, self(), lists:seq(N-1, 1, -1)),    
    spawn(fun () -> lists:foreach(fun (X) -> Last ! X end, lists:seq(M-1, 0, -1)) end),
    wait(),
    {_, WC} = statistics(wall_clock),
    {_, RT} = statistics(runtime),
    io:format("Done running ring benchmark in Wall Clock = ~p, Runtime = ~p.~n", [WC, RT]).

wait() ->
    receive
 Xm when Xm =:= 0 -> 
     ?DEBUG("main node done waiting for x = ~p.~n", [Xm]),
     void;
 Xm -> ?DEBUG("main node done waiting for x = ~p.~n", [Xm]), wait()       
    end.

loop(Nth, Pid) ->
    receive
 X when X =:= 0 -> 
     ?DEBUG("~p dying.~n", [Nth]),
     Pid ! X;
 X -> 
     ?DEBUG("~p got message ~p.~n", [Nth, X]), 
     Pid ! X,
     loop(Nth, Pid)
    end.

Erlang Ring Benchmark Part 2

2011-01-03T14:02:00.002-05:00

Using list comprehensions - results in more compact code.

-module(ring3).
-include("debug.hrl").
-export([start/2]).
%% ring benchmark using lists methods
start(N, M) ->
    io:format("Starting ring benchmark on pid=~p, N = ~p, M = ~p.~n", [self(), N, M]),
    statistics(wall_clock),
    statistics(runtime),
    Last = lists:foldl(fun (X, Pid) -> spawn(fun () -> loop(X, Pid) end) end, self(), lists:seq(N-1, 1, -1)),    
    lists:foreach(fun (X) -> send(X, Last) end, lists:seq(M-1, 0, -1)),
    {_, WC} = statistics(wall_clock),
    {_, RT} = statistics(runtime),
    io:format("Done running ring benchmark in Wall Clock = ~p, Runtime = ~p.~n", [WC, RT]).

send(X, Pid) ->
    ?DEBUG("sending X=~p to ring.~n", [X]),
    Pid ! X,
    receive
 Xm -> ?DEBUG("main node done waiting for x = ~p.~n", [Xm])
    end.

loop(Nth, Pid) ->
    receive
 X when X =:= 0 -> 
     ?DEBUG("~p dying.~n", [Nth]),
     Pid ! X;
 X -> 
     ?DEBUG("~p got message.~n", [Nth]), 
     Pid ! X,
     loop(Nth, Pid)
    end.

Erlang Ring Benchmark

2010-12-28T14:46:00.004-05:00

I am reading Joe Armstrong's Programming Erlang: Software for a Concurrent World, in chapter 8 he poses a little challenge to create a ring benchmark:

Write a ring benchmark. Create N processes in a ring. Send a mes- sage round the ring M times so that a total of N * M messages get sent. Time how long this takes for different values of N and M.

My solution is below... It was fun.

Usage from erlang command line

1> c(ring2).
2> ring2:start(100, 1000). % ring size = 100, send msg 1000 times round the ring
... sample below ...
42> ring2:start(300, 2000).
Starting ring benchmark on pid=<0.30.0>.
Done initializing nodes.
Got X(600000) >= M(600000) in pid=<0.30.0>.
Done running ring benchmark in Wall Clock = 437, Runtime = 391.
ok

The results above were obtained on a Core 2 Duo E8400 @ 3.00GHz

ring2.erl

-module(ring2).
-include("debug.hrl").
-export([start/2]).

%% ring benchmark

start(N, M) ->
    io:format("Starting ring benchmark on pid=~p.~n", [self()]),
    statistics(wall_clock),
    statistics(runtime),
    NextPid = node(N-1, N*M, self(), self()),
    NextPid ! 1,
    io:format("Done initializing nodes.~n"),
    wait(N*M, NextPid, self()),
    NextPid ! die,
    {_, WC} = statistics(wall_clock),
    {_, RT} = statistics(runtime),
    io:format("Done running ring benchmark in Wall Clock = ~p, Runtime = ~p.~n", [WC, RT]).
     
node(1, M, NextPid, DonePid) ->
    Pid = spawn(fun () -> wait(M, NextPid, DonePid) end),
    ?DEBUG("Creating final node: ~p.~n", [Pid]),
    Pid;
node(N, M, NextPid, DonePid) -> 
    Pid = spawn(fun () -> wait(M, NextPid, DonePid) end),
    ?DEBUG("Creating node ~p = ~p.~n", [N, Pid]),
    node(N-1, M, Pid, DonePid).

wait(M, NextPid, DonePid) ->
    receive
 die ->
     ?DEBUG("~p dying.~n", [self()]),
     if NextPid =/= DonePid -> NextPid ! die;
        true -> void
     end,
     void;
 X when X >= M -> 
     io:format("Got X(~p) >= M(~p) in pid=~p.~n", [X, M, self()]),
     DonePid ! die,
     wait(M, NextPid, DonePid);
 X   ->
     ?DEBUG("Got X = ~p in pid=~p, fwd to ~p.~n", [X, self(), NextPid]),
     NextPid ! X+1,
     wait(M, NextPid, DonePid)
    end.

I am using a little debugging macro to turn logging on/off. At erlang shell: c(ring2, {d, debug}). to turn debugging trace on.

debug.hrl

-ifdef(debug).
-define(DEBUG(Format, Args), io:format("~s.~w: DEBUG: " ++ Format, [ ?MODULE, ?LINE | Args])).
-else.
-define(DEBUG(Format, Args), true).
-endif.

here are some solutions from other people: http://goo.gl/kE42w

Book Review: Beautiful Architecture

2010-12-27T12:25:00.002-05:00

Beautiful Architecture: Leading Thinkers Reveal the Hidden Beauty in Software Design by Diomidis Spinellis

I got a chance to review one of the books in OReilly's "Beautiful" series, in this case it was Beautiful Architecture.

One of the attractions of the books in this series, is the premise that you get to hear first-hand accounts, in an essay format, from the people behind some of the more interesting projects out there. So the book can be viewed as an opportunity to ask a bunch of innovative technologists the question: how did you build it?

On the back cover the book claims that the contributors cover notable and innovative software architectures. At first glance the table of contents looked pretty promising, for example “The architecture of the Facebook platform”. However, after spending some time with the book and reading through 5 random chapters, it feels like there is just not enough substance for the book to achieve what most people are probably hoping to get out of the book: concrete blueprints or solid design patterns to build systems demonstrating desired characteristics.

For the first 45 or pages of the book, the essays tended to meander around the topics of general definition of architecture vs software design etc, and general experience of people working on some non-trivial projects. But somehow the conversation always feels more academic than practical. So you kind of feel like you are sitting on a plane waiting for it to disengage from the gate and move towards the tarmac. And given the fact that this book is supposed to be about “beautiful” architectures, I am not at all sure why we need to spend time reading about “The Messy Metropolis”. Given the title and general back cover description of the book I was expecting the narratives to be more specific and more parallels drawn with real-world systems that people interested in this book could be building.

The chapter “Architecting for Scale” by Jim Waldo, was also a bit underwhelming. We basically get a pretty high level overview of a system that was never really tested in the field, was not thoroughly bench-marked, and is addressing a fairly narrow problem domain of online gaming. Although some interesting ideas are presented, you have a hard time agreeing that this is necessarily beautiful because it does not seem that the system is proven to successfully solve their scalability goals. The chapter about facebook platform also failed to describe the thinking process and design trade-offs, you get a description of general building blocks but you don't really get any aha moments.

I did enjoy “Software Architecture: Object-Oriented versus Functional” by Bertrand Meyer, however one might argue that the book might have been better off focusing on systems architecture and concrete examples of real-world projects as opposed to discussion of type systems and other programming language esoterica, especially when the Meyer makes it clear that he has not actually worked in finance nor released production systems dealing with financial contracts.

I think this book is decent read on your e-book reader, but if you have limited bookshelf space and want to keep it filled with true classics, this book is not it.

3/5 stars.

Big Cities Make People More Efficient

2010-12-18T15:15:00.000-05:00

From a NY Times article: A Physicist Solves the City

The first data set they analyzed was on the economic productivity of American cities, and it quickly became clear that their working hypothesis — like elephants, cities become more efficient as they get bigger — was profoundly incomplete. According to the data, whenever a city doubles in size, every measure of economic activity, from construction spending to the amount of bank deposits, increases by approximately 15 percent per capita. It doesn’t matter how big the city is; the law remains the same. “This remarkable equation is why people move to the big city,” West says. “Because you can take the same person, and if you just move them to a city that’s twice as big, then all of a sudden they’ll do 15 percent more of everything that we can measure.” While Jacobs could only speculate on the value of our urban interactions, West insists that he has found a way to “scientifically confirm” her conjectures. “One of my favorite compliments is when people come up to me and say, ‘You have done what Jane Jacobs would have done, if only she could do mathematics,’ ” West says. “What the data clearly shows, and what she was clever enough to anticipate, is that when people come together, they become much more productive.”

Rx.NET Subject useful for connecting decoupled components

2010-12-16T13:29:00.002-05:00

Rx has a nifty abstraction called Subject shipped as part of the System.Reactive assembly.

It's basically a buffer with channel like capabilities rolling up an Obsever/Observable pattern into a single class.

In cases where you have two or more classes that don't really know about each other but sometimes need to be connected (ie output of one should go into input of another), you can can inject an instance of a Subject class into both components, and then have them send messages. Subject provides a standard Rx.NET friendly API to push/pull messages, this makes it quite powerful as you can use the full expressive power of Rx.NET to consume message from Subjects in different scenarios (one-to-one, one-to-many, filtering, Join, etc) .

Direct Edge stock exchange using .NET

2010-11-17T13:04:00.000-05:00

Wow a major exchange, Direct Edge, is using .NET to run it's exchange.

http://blogs.technet.com/b/windowsserver/archive/2010/07/22/direct-edge-stock-exchange-launches-on-windows-server-sql-server.aspx

C++/CLI

2010-11-17T12:50:00.003-05:00

Having spent a lot of time programming in C++ over the last few months, I have come to appreciate some of the features of the language, such as the powerful templating mechanism, which is quite a bit more powerful than the .NET generics. There are seem to be a few cases when one would have to resort to tricky reflection techniques in C#/VB.NET, which costs both in terms of runtime overhead and also in code mantainability.

I have only recently became more acutely aware of the C++/CLI. Seems like it has a number of cool features that a worth a look:

template system similar to C++
much cheaper mechanism to invoke native code (compared to PInvoke)
possibility to mix and match managed and unmanaged memory pools for performance optimization.

There is a pretty solid article from a solid C++ guy about C++/CLI.

Here is an excerpt:

C++/CLI represents a tuple. C++ refers, of course, to the C++ programming language invented by Bjarne Stroustrup at Bell Laboratories. It supports a static object model that is optimized for the speed and size of its executables. However, it doesn't support run-time modification of the program other than heap allocation. It allows unlimited access to the underlying machine, but very little access to the types active in the running program and no real access to the associated infrastructure of that program. Herb Sutter, a former colleague of mine at Microsoft and the chief architect of C++/CLI, refers to C++ as a concrete language.

CLI refers to the Common Language Infrastructure, a multitiered architecture supporting a dynamic component programming model. In many ways, this represents a complete reversal of the C++ object model. A runtime software layer, the virtual execution system, runs between the program and the underlying operating system. Access to the underlying machine is fairly constrained. Access to the types active in the executing program and the associated program infrastructure—both as discovery and construction—is supported. The slash (/) represents a binding between C++ and the CLI. The details surrounding this binding make up the general topic of this column.

So, a first approximation of an answer to what is C++/CLI is that it is a binding of the static C++ object model to the dynamic component object model of the CLI. In short, it is how you do .NET programming using C++ rather than C# or Visual Basic®. Like C# and the CLI itself, C++/CLI is undergoing standardization under the European Computer Manufacturers Association (ECMA) and eventually under ISO.

The common language runtime (CLR) is the Microsoft version of the CLI that is specific to the Windows® operating system. Similarly, Visual C++® 2005 is the implementation of C++/CLI.

As a second approximation of an answer, I would say that C++/CLI integrates the .NET programming model within C++ in the same way as, back at Bell Laboratories, we integrated generic programming using templates within the then existing C++. In both of these cases your investment in an existing C++ codebase and in your existing C++ expertise are preserved. This was an essential baseline requirement of the design of C++/CLI.

Memory Tour De Force

2010-11-17T09:58:00.002-05:00

What Every Programmer Should Know About Memory by Ulrich Drepper

Fascinating, enlightening, extremely useful (memory speeds have not kept up with progress in raw computing power in modern computers see, beware of cpu stalling!). Expect it to take a couple of weeks to get through it and really digest all of it, it's a self-contained mini-course in electrical engineering...

here is a diagram comparing memory access speed vs cpu speed evolution - notice the growing gap.
from another excellent read showing how proper memory access can dramatically speed up your program.

Earth Equatorial Radius == 1.0 * Length Of Amazon River

2010-11-08T17:24:00.000-05:00

Creepy coincidence or as-designed by aliens?

Earth Equator Radius = Amazon River Length = 6400 km

Note, equator radius is not same as circumference (which is around 2*pi*r = 40,000 km)

And the best use of ASCII in a comment goes to ...

2010-11-02T15:23:00.004-05:00

Undo Tree Package For Emacs.

The documentation comment is fantastic. One of the best uses of ascii I have seen in a while. http://www.dr-qubit.org/undo-tree/undo-tree.el

The package is actually of great use for people working with version control system allowing non-linear change history, such as the increasingly popular git and mercurial.

Here is a sample:

Rx Guidelines

2010-11-02T12:51:00.001-05:00

Rx (Reactive Extensions for .NET) is very cool. However there is fairly limited documentation. Rx Team recently released a more structured effort to document the functionality (as opposed to trying to string together a complete picture of Rx functionality from many different blog entries). It is available here: http://blogs.msdn.com/b/rxteam/archive/2010/10/28/rx-design-guidelines.aspx

Side Effect Of Type Constraints in .NET

2010-10-26T14:53:00.005-05:00

Experimenting With Generic Constraints

As I was doing some experiments as exercise in reminding myself the difference between virtual and non-virtual method overloading behavior in C#, and stumbled on an unexpected side-effect of the constraint mechanism in Generics.

I have class B inherit from A, and wanted to write a generic test harness that would invoke methods f1 and f2 on an instance passed in, but with a generic capability to have the runtime cast the instance to the most specific type. So if I passed in B, the variable would be manipulated as B, etc. So I thought a generic method would deal with this quite nicely: void Do<T>(T arg) { arg.f1(); arg.f2; }. Now, the declaration will not compile as is, the compiler insists to know the type of T so that it can figure out whether the class supports f1(), f2() functions, fine, i go ahead and add where T : A to the signature, which turns out to introduce the side effect which is the subject of this blog entry.

When I implemented this scheme (see code below) I was surprised that even when invoking Do<B>(bInstance); , bInstance was treating as if it was A.

So the generic behavior is very different from what one would thing to be logical, and in fact C++ template mechanism does do what one expects if you were to implement a similar method, bInstance will be treated as a B.

using System;
using System.Text;

namespace SimpleMethodInheritanceTest
{
    class A
    {
        internal virtual void f1() { System.Console.WriteLine("f1 in A"); }
        internal void f2() { System.Console.WriteLine("f2 in A"); }
    }

    class B : A
    {
        internal override void f1() { System.Console.WriteLine("f1 in B"); }
        internal void f2() { System.Console.WriteLine("f2 in B"); }
    }

    class Program
    {
        static void DoBase(A a)
        {
            System.Console.WriteLine("======================== DoBase ========================");
            a.f1();
            a.f2();
        }

        static void DoBBB(B a)
        {
            System.Console.WriteLine("======================== DoBBB ========================");
            a.f1();
            a.f2();
        }


        static void Do<T>(T a) where T : A
        {
            System.Console.WriteLine("======================== Do<T> ========================");
            var specificClass = a.GetType().Name;
            System.Console.WriteLine("======================== Do<T>: T = {0}", specificClass);
            a.f1();
            a.f2();
        }
        static void Main(string[] args)
        {
            var a = new A();
            DoBase(a);
            Do<A>(a);
            var b = new B();
            DoBase(b);
            Do<B>(b);
            DoBBB(b);
            System.Console.ReadKey();
        }
    }
}

Output

I expected line 14 to say f2 in B, since T = B, so I was expecting the argument to Do<T>() to be treated as a B. But instead I get f2 in A!!!

======================== DoBase ========================
f1 in A
f2 in A
======================== Do<T> ========================
======================== Do<T>: T = A
f1 in A
f2 in A
======================== DoBase ========================
f1 in B
f2 in A
======================== Do<T> ========================
======================== Do<T>: T = B
f1 in B
f2 in A
======================== DoBBB ========================
f1 in B
f2 in B

Discussion

So we would expect D: T = B to have f2 in B, but we got f2 in A instead. So to investigate as to what might be going on under the covers, I decompiled the executable into IL using redgate's reflector.

Let's take a look at the results below.

Do<T> Dissassembled

If we dissasemble D<T> we see that on Line 25 f2 is explicitly invoked on class A. This is causes the unexpected results of f2 in A.

.method private hidebysig static void Do<(SimpleMethodInheritanceTest.A) T>(!!T a) cil managed
{
    .maxstack 2
    .locals init (
        [0] string specificClass)
    L_0000: nop 
    L_0001: ldstr "======================== Do<T> ========================"
    L_0006: call void [mscorlib]System.Console::WriteLine(string)
    L_000b: nop 
    L_000c: ldarga.s a
    L_000e: constrained !!T
    L_0014: callvirt instance class [mscorlib]System.Type [mscorlib]System.Object::GetType()
    L_0019: callvirt instance string [mscorlib]System.Reflection.MemberInfo::get_Name()
    L_001e: stloc.0 
    L_001f: ldstr "======================== Do<T>: T = {0}"
    L_0024: ldloc.0 
    L_0025: call void [mscorlib]System.Console::WriteLine(string, object)
    L_002a: nop 
    L_002b: ldarga.s a
    L_002d: constrained !!T
    L_0033: callvirt instance void SimpleMethodInheritanceTest.A::f1()
    L_0038: nop 
    L_0039: ldarga.s a
    L_003b: constrained !!T
    L_0041: callvirt instance void SimpleMethodInheritanceTest.A::f2()
    L_0046: nop 
    L_0047: ret 
}

DoBBB Disassembled

The helper method that specifies class B as the explicit type of argument, works as expected (calling B::f2()). As we can see below in line 12, compiler emits IL that specifies B::f2 explicitly.

.method private hidebysig static void DoBBB(class SimpleMethodInheritanceTest.B a) cil managed
{
    .maxstack 8
    L_0000: nop 
    L_0001: ldstr "======================== DoBBB ========================"
    L_0006: call void [mscorlib]System.Console::WriteLine(string)
    L_000b: nop 
    L_000c: ldarg.0 
    L_000d: callvirt instance void SimpleMethodInheritanceTest.A::f1()
    L_0012: nop 
    L_0013: ldarg.0 
    L_0014: callvirt instance void SimpleMethodInheritanceTest.B::f2()
    L_0019: nop 
    L_001a: ret 
}

CloudCamp NYC 2010

2010-10-15T08:36:00.001-05:00

Cloud computing, and more specifically effecient virtualization is a very interesting topic.

Here is an opportunity to mingle and learn more if you are in NYC, on October 21st, 2010.

It's downtown, near Wall St.

http://www.cloudcamp.org/ny/2010-10-21

App Engine For The Enterprise?

2010-10-15T08:33:00.000-05:00

I like writing software, not the process of deploying it. The idea of abstracting away configuration, hosting and complex network setup has always been very appealing to me. This is one of the big promises of cloud computing; so I have been looking at and playing various cloud technologies as they have been popping up over the last 3-4 years.

Recently I have started using Google App Engine for non-trivial web apps. My experience so far has been a pleasure! You may say: What!? that's not a very precise description, true, but this is not meant to be a thorough analysis or comparison of every feature that one can think of or use in a real-world applications. What I can say, is that it's been painless to do all the regular things that you have to during a standard application development and deployment process. The API's are elegantly simple to learn and to use, and make it easy for you to start handling requests, persisting data, doing background processing, all on a scalable platform that can handle from 1 to thousands concurrent requests (at least in theory).

At the same time, having worked at a number of large corporations I have seen a number of good and bad attempts at streamlining the deployment processes and abstracting away architecture complexities. Again, I have mostly been in the end user role, not the implementor of the abstraction/automation frameworks and systems, so my view may be biased/limited/etc, but at least it represents an experience of a typical developer at a large organization. Typically it's been a rather arduous experience. Some firms are better than others, but in almost all cases you have to pay lots of attention to fairly low level architecture of your application. You can't just write a simple script and deploy it into the cloud.

Given my recent experiments with the App Engine, and extensive experience with large enterprise IT environments, I have starting thinking about how cloud computing can apply to the enterprise.

Low Level Infrastructure ------> High Level API

At the low level, you have to worry about how you are going to scale your application and the exact technology stack needed for your particular application. This is essentially what is being offered by various virtual machine cloud providers: Amazon EC2, Rackspace Cloud, and others. You have to figure out what VM you want to run (ie pick OS: linux flavor, windows), and what goes inside the VM (mysql/postgres/couchdb/other db, apache, lighthttpd, j2ee, IIS), how many nodes you want to have for failover and scalability, how many tiers you need to scale, etc. This gives you ultimate flexibility, but requires a LOT of extra architecture work. This comes at the expense of agility.

At the mid level of the spectrum, you have something like Windows Azure. You get to use familiar development tools and components (SQL, asp.net). But you still have to think about the architecture: how many worker/web/db nodes you want to have, some design of the schema.

At the highest level of architecture abstraction you will find Google App Engine. As a developer on this platform you don't have to worry about configuring VMs, figuring out the number and size of logical tiers. You are able to focus directly on solving your business problem. To Be Continued...

MSBuild Environment Sandboxing

2010-10-14T07:08:00.015-05:00

Sandboxing 101

In any non trivial, team oriented project you have multiple configurations:

team-member-one-sandbox, team-member-two-sandbox
QA
UAT
PROD-Americas, PROD-Europe, PROD-Asia
Etc

For each environment you want to configure database connection strings, web services, various limits and runtime settings; all possibly located across multiple configuration files. It is also convinient to reuse some sane defaults for a lot of the settings, and have the option to override some settings for a specific environment. Conceptually you want to have the following hierarchy of settings, where the most nested overrides or adds to the properties defined at the previous level:

Defaults
- PROD-Defaults
  - Americas
  - Europe
  - Asia

.NET/MSBuild Solution

There are a couple of choices available:

T4
MSBuild
NAnt
Custom PostBuild event scripts in Visual Studio
An AdHoc combination of the above

When we were designing the solution described in this post, we wanted to keep the process as simple and as integrated with Visual Studio as possible, while at the same time making it simple to do automated builds from the command line. In the end, we were able to combine the Configuration functionality within with Visual Studio, with targeting of specific environments in a fairly natural way; if you follow this approach you will be able to select a target environment within VS via the configuration drop-down. At a very high level you will have the following: 1. One or more templated configuration files with ${setting1} for various configurable settings. For example you will have a bunch of settings in your app.config. 2. A set of configuration files containing the settings that need to be replaced in your templated files: The proposed solution will allow you to define your settings in the following configuration files:

defaults.cfg
prod-americas.cfg
prod-europe.cfg
prod-asia.cfg

3. A couple of custom MSBuild tasks in your CoolApp.csproj file. We will utilize open source build task library: http://msbuildtasks.tigris.org/ 4. A couple of custom Configurations set up within Visual Studion (these go into CoolApp.sln file)

App.config

<configuration>
  <appSettings>
    <add key="setting_one" value="${setting_one}"/>
    ${custom_settings}
  </appSettings>
</configuration>

defaults.cfg

<settings>
  <x key="setting_one">Hello World From DEFAULTS!</x>
</settings>

debug.cfg (ie dev/qa/uat/etc)

<settings>
  <x key="custom_settings">
    <add key="custom_setting_1" value="This is DEBUG custom setting ONE."/>
    <add key="custom_setting_2" value="This is DEBUG custom setting TWO."/>
  </x>
</settings>

release.cfg

<tokens>
<settings>
  <x key="setting_one">Hello World From RELEASE!</x>
  <x key="custom_settings">
    <add key="custom_setting_1" value="This is RELEASE custom setting ONE."/>
    <add key="custom_setting_2" value="This is RELEASE custom setting TWO."/>
  </x>
</settings>

MSBuild Task Detail

<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />
  <PropertyGroup>
    <ConfigOutDir>$(OutputPath)\cfg</ConfigOutDir>
  </PropertyGroup>
  <Target Name="ProcessConfig">
    <Message Text="==&gt;&gt;&gt;&gt; CurrentEnvironment = $(EnvId)" />
    <!-- BEGIN default (ie common) settings -->
    <XmlQuery XmlFileName="$(MSBuildProjectDirectory)\cfg\defaults.cfg" XPath="//settings/*">
      <Output TaskParameter="Values" ItemName="DefaultSettingValues" />
    </XmlQuery>
    <ItemGroup>
      <Tokens Include="%(DefaultSettingValues.key)">
        <ReplacementValue>%(DefaultSettingValues._innerxml)</ReplacementValue>
      </Tokens>
    </ItemGroup>
    <!-- END defaults.config -->
    <!-- specific environment -->
    <XmlQuery XmlFileName="$(MSBuildProjectDirectory)\cfg\$(EnvId).cfg" XPath="//settings/*">
      <Output TaskParameter="Values" ItemName="SettingValues" />
    </XmlQuery>
    <ItemGroup>
      <Tokens Include="EnvId">
        <ReplacementValue>$(EnvId)</ReplacementValue>
      </Tokens>
    </ItemGroup>
    <!-- Remove duplicates - this is important for "overrides" -->
    <ItemGroup>
      <Tokens Remove="%(SettingValues.key)" />
    </ItemGroup>
    <ItemGroup>
      <Tokens Include="%(SettingValues.key)">
        <ReplacementValue>%(SettingValues._innerxml)</ReplacementValue>
      </Tokens>
    </ItemGroup>
    <Message Text="Parsed Tokens =&gt; %(Tokens.Identity): %(Tokens.ReplacementValue)" />
    <MakeDir Directories="$(ConfigOutDir)" />
    <TemplateFile Template="$(MSBuildProjectDirectory)\app.config" OutputFilename="$(ConfigOutDir)\$(EnvId).app.config" Tokens="@(Tokens)" />
  </Target>
  <Target Name="BeginProcessConfig">
    <ItemGroup>
      <EnvConfigFiles Include="$(MSBuildProjectDirectory)\cfg\*.cfg" Exclude="$(MSBuildProjectDirectory)\cfg\defaults.cfg" />
    </ItemGroup>
    <MSBuild Targets="ProcessConfig" Properties="EnvId=%(EnvConfigFiles.Filename)" Projects="$(MsBuildProjectFile)" />
    <!-- Copy over the correct file based on configuration in visual studio or command line -->
    <PropertyGroup>
      <EnvConfigIn>$(ConfigOutDir)\$(Configuration).app.config</EnvConfigIn>
    </PropertyGroup>
    <Message Text="EnvConfigIn = $(EnvConfigIn)" />
    <Copy SourceFiles="$(EnvConfigIn)" DestinationFiles="$(OutputPath)$(AssemblyName).exe.config" />
  </Target>
  <Target Name="AfterBuild" DependsOnTargets="BeginProcessConfig">
    <Message Text="This is the AfterBuild target: $(MSBuildToolsPath)" />
  </Target>

Demo Project

1. Install MSbuild Common Tasks

http://msbuildtasks.tigris.org/

Note: the easy way to experiment is to just install via .msi installer, alternatively you can download the .zip file with all the binaries, but then you will have to configure the Import of msbuild tasks in the .csproj file - not difficult, and definitely something you want to do for self contain projects (typically you would include the binaries in some libs sub-folder of your project and reference it via relative paths from msbuild).

2. Download Source

View envcfgdemo-zip or if you behind an evil firewall: Download Demo Visual Studio 2008 Project

When opening the project you will be prompted with a dialog telling you that the project file is using some external extensions and whether you want to trust it. This is normal and you have to do it just once.

Try selecting different Configurations in Visual Studio, or run from command line with

msbuild /p:Configuration=CustomEnv

or
msbuild /p:Configuration=Debug

Note: all of the configurations are built and placed into bin\$(Configuration)\cfg folder, you can examine the files and sometimes it's useful for further automatic deployments (ie you can copy over the right config file via a script at the time you deploy vs build time).

Language Semantics

2010-09-21T08:53:00.000-05:00

Some 50 years ago, the renowned linguist Roman Jakobson pointed out a crucial fact about differences between languages in a pithy maxim: “Languages differ essentially in what they must convey and not in what they may convey.” This maxim offers us the key to unlocking the real force of the mother tongue: if different languages influence our minds in different ways, this is not because of what our language allows us to think but rather because of what it habitually obliges us to think about.

...

When your language routinely obliges you to specify certain types of information, it forces you to be attentive to certain details in the world and to certain aspects of experience that speakers of other languages may not be required to think about all the time. And since such habits of speech are cultivated from the earliest age, it is only natural that they can settle into habits of mind that go beyond language itself, affecting your experiences, perceptions, associations, feelings, memories and orientation in the world.

From http://lambda-the-ultimate.org/node/4062

ZUI Presentations

2010-09-10T07:52:00.000-05:00

Zoomable UI, definitely feels cool on first encounter - not convinced though that it's much better than regular slides. Nice implementation though from prezi.com

Academy: Prezi Workflow in 15 minutes on Prezi

Polyglot Programming

2010-09-09T22:32:00.007-05:00

A typical week (and sometimes day) consists of a little bit of the following: C++, C#, Python, Javascript.

C#

Asset-Backed Position Management UI in WinForms.
Trade And PnL UI in WPF for a high frequency trading engine. Using RX for processing streams of incoming messages, hoping it can scale to > 1k ticks/sec.

C++

high frequency trading engine: position management, pnl calculations, high-performance messaging. Advanced use of templates, custom memory pooling, concurrency. Ahh, it's been a while since I cared so much about bits and bytes. But as Alan Kay once said "if a company is really serious about software they need to build their own hardware", so I got myself a new book on top of the stack on my desk: Bebop to the Boolean Boogie - it's a fun book, even if at the expense of accuracy (for example the author claims that Perl is one of the first scripting languages, I am no expert on these matters but looks like sh predates perl by a decade, while I was fact checking I was shocked to discover that perl is now 23 years old! and python is almost 20).

Python

Building a web app on Google AppEngine. (using django templates, meta programming with decorators, metaclasses, and other nifty python stuff that make it a very elegant language. Python is FUN; glad that the author of a book i am currently reading - programming collective intelligence - opted to use py for the examples).
Curses UI in a Unix environment (initially it started out as a somewhat specialized file grep+tail, but, bam, a couple hundred of python lines later it's a multithreaded, curses-based powertool).

Javascript

Mostly using JQuery.

Curses

2010-08-24T01:14:00.005-05:00

No I am not particularly angry - I am talking about ncurses text user interface programming. Surprisingly curses is a very nice solution when you are constrained by complex network paths from where you have data and where you need to display it; with curses all you need is an ssh connection.

Curses provides a simple abstraction of the terminal's coordinates, colorization, and some other special effects - yes in the world of GPU accelerated toolkits, calling blinking text a special effect makes one smile, but some simple things are quite effective in real use.

Python has excellent support for curses (it comes with a module that is a pretty complete wrapper of the curses api). I was able to whip up a multithreaded curses based position tracking script in a few evenings, works like a charm.

syntax highlighting

2009-11-25T19:32:00.015-05:00

Here is some C-Sharp code

public static void Main() {
    Console.Out.WriteLine("test")
}

And some F-Sharp code

let xs = seq { for i in 1..10 yield (i, i*i) }
let result = 
   xs 
     |> Seq.iter (fun item -> printfn "%A" item)

the wonderful world of .net

2009-11-25T19:23:00.002-05:00

love the innovation in programming language design happening in the world of .net.

with powershell, now even have a very respectable command line.

with windows7 now finally have a modern OS? may be, the jury is still out, i have been using it on my home install, so far so good.

i have been slowly but surely moving towards doing more and more hacking in F#.

have even been using it to get real work done - but so far these have been scripts in the 30 - 100 line range. even did a presentation at work about it!

blue brain

2008-01-04T12:12:00.000-05:00

a group of researchers in switzerland have been growing an artifical brain via simulated (virtual) neurons on a petaflop supercomputer. very cool stuff... yes indeed the cingularity described by ray kurzweil may be coming sooner than we think.

very cool .net 3 based mind mapper

2008-01-04T12:09:00.000-05:00

a very cool and free mind mapping tool. playing with it now; first impression is very nice http://cayra.net/screenshots

progamming language book sales

2007-05-21T09:17:00.001-05:00

interesting analysis of book sales by programming language. in a nutshell scripting languages are spiking up in popularity. given that most of the scripting languages encourages functional style (list comprehensions, map, apply) of programming this bodes well for encouraging people to look at more robust (type enforcement), functional languages (F#, boo, nemerle, scala, ocaml, haskell). gut feeling is that functional and domain specific languages are gaining momentum. may be eventually DSL's will replace the multitude of framework API's. http://radar.oreilly.com/archives/2007/05/state_of_the_co_10.html

new york functional programmers meeting

2007-05-04T08:34:00.000-05:00

met a bunch of very good people. great to see that functional languages are used in real projects at leading financial services institutions. haskell and ocaml appears to be very popular. people seem to like the idea and implementation of F#, but a quite concerned about licensing issues. The main fear is that microsoft will stop development related to F# and won't release the implementation to public domain, so the language will be stillborn. i think such worries are a little premature. Microsoft also has a fairly decent record with regard to using public domain for it's CLR and C# specification (enabling success of projects like mono). Given enough interest and requests to F# team, surely it will be in everyone's interest to make the language implementation available for the public.

microsoft financal services tech conference 2007

2007-04-26T17:44:00.000-05:00

spent 2 days at the conference. this is my first time going to a microsoft-centric technology conference, having spent a good 6-7 years in the java camp. here are some takeaways for me personally, in no particular order:

although this was a financial services specific technology conference, there was not a whole lot of technology being discussed that would be terribly useful for an enterprise type of application. there was only one session covering microsoft financial services related application block (unfortunately i did not make it to this one - oops).
ms is trying real hard to push wpf (winfx, xaml, etc). there were a couple of session focused almost exclusively on use of this technology for building thick clients.
given the poor tooling in visual studio 2005 for xaml based development i am surprised ms is pushing this technology to financial services audience. right now, it seems the only audience for this would be software tools builders and hackers (ahhh, fond memories of mozilla XUL development on one of the past projects we did).
apperantly microsoft HPC addition is cheap to deploy (about $400 per cpu). need to follow up on this, i have been itching to start utilizing grid computing.
sadly, there are still plenty of people who tend to think that writing hundreds of lines of convoluted xslt is a much nicer option to QA's c# code. apperantly there is not need to qa a 50 line block of inlined/recursive called to insane string manipulation routines and just release it right into production. just because a particular technology does not need to be compiled into an EXE does not mean it has to go through a different release process. Even configuration changes should be heavily QA'd before making changes in production.

wisdom of many to rate my credit

2007-03-29T06:27:00.000-05:00

prosper.com is an peer-to-peer lending site. fantastic concept. wave of the future.