A picture is worth 1000 words

Two to three weeks ago if you ran this battery of garbage collection related tests in moonlight, they'd mostly be red. How things have changed :)

Writing a profiler for mono

As some of you may know, my day job is working on Moonlight, the FOSS implementation of Silverlight. This is a fairly complex piece of code as we have C++ interoperating with C# and Javascript, so managing the lifecycle of objects can be difficult. C++ refcounts, C# has an automatic garbage collector and Javascript... well, let's just say that it complicates things a *lot* when combined with the other two languages.

One of the tricks we use to ensure that objects do not die early is to use a (normal) GCHandle on our C# objects to ensure the garbage collector doesn't mark them as junk before they're definitely finished with in C++. However, we were having issues where some of our objects appeared to never ever get garbage collected even though they should be eligible. We suspected that excess GCHandles were causing it but we had no easy way of tracking this. Without detailed information on when and where the GCHandles were allocated, it would be next to impossible to track down where we were going wrong. After several aborted attempts to track this with Console.WriteLine I figured there must be a better way.

How about a custom profiler for mono?

Documentation on this is fairly sparse currently, so hopefully this will help. Thanks go to Paolo Molaro for helping me when I got stuck! This code is mostly copied/pasted from moonlight so it's C++ ish and also may contain some moonlight specific code.

Step 1:
Open up the "mono/metadata/profiler.h" file and check out all the profiler hooks that are available. They are all documented in profiler.c, so open that if you want a description of how they all work. Take note of the ones you'll need. All I needed to do was track GCHandle allocations, so I just wanted this hook:

void mono_profiler_install_gc_roots    (MonoProfileGCHandleFunc handle_callback, MonoProfileGCRootFunc roots_callback);

Step 2:
I now know what function I need to provide and the data I want to gather, so it's time to create a struct to store my data. This is an opaque struct which will be passed to the mono runtime and passed back in every profiler hook, so you can stick whatever you want in it. I know I want to store stacktraces, allocated gchandles and a typename, so I'll start with just those. I'll also declare my prototype for my profiler hook here.

struct _MonoProfiler {
const char *type_name;
GPtrArray *gchandles;
GPtrArray *stacktraces;
Moonlight::Mutex locker; /* used to ensure only one thread accesses the arrays */

static void track_gchandle (_MonoProfiler *prof, int op, int type, uintptr_t handle, MonoObject *obj);
};

typedef _MonoProfiler MonoProfiler;

Step 3:
Implementation time! First the constructor for the MonoProfiler. I use an environment variable to limit which types we store stacktraces for as they're expensive to generate and consume a lot of memory to store when you have 10,000s of them. The important functions are the last three. The first registers our profiler with the runtime. The second installs the hook so we can get notifications for gchandle allocations. The final call enables the gc_handle events in the profiler which makes the runtime invoke the hook we just registered.

_MonoProfiler::_MonoProfiler ()
{
   type_name = g_getenv ("GCHANDLES_FOR_TYPE");

   gchandles = g_ptr_array_new ();
   stacktraces = g_ptr_array_new_with_free_func (g_free);
   /* Register the profiler with mono */
   mono_profiler_install (this, NULL);
   /* Supply a function for the gc_roots hook */
   mono_profiler_install_gc_roots (track_gchandle, NULL);
   /* Enable gc_roots tracking in the profiler so our hook is invoked */
   mono_profiler_set_events (MONO_PROFILE_GC_ROOTS);
}

Now all I need is the implementation for track_gchandle and I'm done!

void
MonoProfiler::track_gchandle (MonoProfiler *prof, int op, int type, uintptr_t handle, MonoObject *obj)
{
   // Ignore anything that isn't a strong GC handle (docs say type == 2 is a strong gchandle)
   if (type != 2)
       return;

   prof->locker.Lock ();

   GPtrArray *gchandles = prof->gchandles;
   GPtrArray *stacktraces = prof->stacktraces;

   if (op == MONO_PROFILER_GC_HANDLE_CREATED) {
       // Add the GCHandle to this array
       g_ptr_array_add (gchandles, (gpointer) handle);
       // If the target of the gchandle is of the correct type, store its stack trace
       // Otherwise store NULL so that we can keep the index of the gchandle and corresponding
       // stack trace in sync.
       if (prof->type_name && !strcmp (prof->type_name, mono_class_get_name (mono_object_get_class(obj))))
           g_ptr_array_add (stacktraces, get_stack_trace ());
       else
           g_ptr_array_add (stacktraces, NULL);
   } else if (op == MONO_PROFILER_GC_HANDLE_DESTROYED) {
       // Walk our list of gchandles and when we find the index of the destroyed handle
       // remove the handle and corresponding stacktrace
       for (int i = 0; i < (int)gchandles->len; i++) {
           if (g_ptr_array_index (gchandles, i) == (gpointer) handle) {
               g_ptr_array_remove_index_fast (gchandles, i);
               g_ptr_array_remove_index_fast (stacktraces, i);
               break;
           }
       }
   }

   prof->locker.Unlock ();
}

Step 4:
Profit! When you're ready, you can iterate over the gchandle array and pull out whatever statistics you want. For example here is the code I use to iterate the gchandles array and work out how many instances of each object type there are.

void
accumulate_g_ptr_array_by_type (gpointer data, gpointer user_data)
{
   // The hashtable I passed in as the user_data in g_ptr_array_foreach
   // which i am using to link a type name to a count
   GHashTable *by_type = (GHashTable*) user_data;

   // Get the MonoObject from the gchandle.
   MonoObject *ob = mono_gchandle_get_target (GPOINTER_TO_INT (data));

   // Get the type name from the mono_object
   const char *name = mono_class_get_name (mono_object_get_class(ob));

   // Find out how many instances we have already
   int count = GPOINTER_TO_INT (g_hash_table_lookup (by_type, name)) + 1;

   // Update the hashtable with an incremented count.
   g_hash_table_insert (by_type, name, GINT_TO_POINTER (count));
}

I run that hashtable through another function to order to sort the list by the number of each type of object allocated so I can have a nicely formatted list as follows:

1 instances GCHandled of type Surface
1 instances GCHandled of type Deployment
4 instances GCHandled of type NameScope
20 instances GCHandled of type ControlTemplate
218 instances GCHandled of type MonoType
3985 instances GCHandled of type Uri

Needless to say, it's now immediately obvious to me that Uris are being GChandled and never freed, so i just have to enable tracing of their stacktraces and I can see exactly where we allocated the handle and from there figure out why we haven't freed it.

MonoTorrent Lives! (on github)

The mono project, who've been hosting MonoTorrent since day 1, recently decided to host all their code at github. As such, MonoTorrent now lives in github! I took this opportunity to rename the module from 'bitsharp' to 'monotorrent' to avoid confusing people.

So what does this mean? Well, it means it's trivial for you to commit patches now! All you do is set up a github account, fork monotorrent, commit your changes to your fork and issue a pull request. I'll then automagically be informed of your changes and can click a button to either merge them directly into monotorrent or reject them (with an explanation). No more emailing patches or attaching them to forum posts. Pretty nifty, eh?

So get forking and lets see how great we can make MonoTorrent!

Hackwee Day IV - The final showdown

Yesterday was a bit of an anti-climax codewise. Pretty much everything I had planned on doing I had completed by midday. I then had a nice long chat with lamalex about what I had to do to get ipod support into his udev branch and hashed out a rough idea about how to upstream these changes. I believe he's going to do a bit of work over the next week or so and once that's complete I'll be able to upstream the remainder of my patches.

So at the end of the 4 days the state of things is:

Device support:
1) Hotplugging works
2) Syncing music to and from the device works
3) Removing music from the device works
4) Basic properties can be read about the device and displayed in banshee
5) Playlists aren't supported yet but should be fairly simple to do

Code written:
1) I've upstreamed my patches for libgpod-sharp
2) I spoke with lamalex about upstreaming the rest of my work there.
3) There are now packages available for openSUSE 11.2 courtesy of FunkyM which can be used to provide most of the required packages.
4) There are still some bleeding edge libraries required which aren't packaged anywhere. These will have to be packaged.
5) I think the best way to support the libgpod based iDevice would be to create a new addin so that you can run both the old addin and the new addin in parallel as some of the required libraries for libgpod based iDevice support are very new and not widely available as of yet. This will take a little bit of time.

I'm off on holidays next week so when I get back I'll start pushing my remaining work upstream. Once that's done it's just the boring process of streamlining the build to remove the dozen manual steps which are currently required :) All in all, it's been a successful hackweek. But as with all good hacks, the 5:1 rule applies. For every day spent adding functionality, I'll need 5 to make it stable and usable for everyone.

I have to thank teuf again for all the help over the last few days (and for libgpod itself!), Nathaniel McCallum for the awesome start to the .NET bindings for libgpod, Alex Launi for the banshee gio/udev work which saved me a lot of time and also for sparing the time yesterday to talk about how I can get my work upstreamed.

If I've forgotten anyone or anything, sorry! But it's been a hectic week and it was hard enough keeping track of all the packages and patches I had floating around, never mind everyone I was talking to :)

I'll post again when I get everything upstreamed and I'll have a nice set of instructions for anyone wishing to test out their iDevice with banshee.

Hackweek Day III - The coffee shortage

Today was a slow but interesting day (again). I took all my banshee work and rebased it on top of lamalexs work to bring udev/gio support to banshee. This saved me having to write my own backend, but it meant that once again I had to start upgrading core parts of my OS. Luckily nothing blew up this time!

So after a few hours hacking and pulling my hair out because seemingly simple commands were failing (they still are failing, i just worked around it) I now have proper detection implemented. There are no more hardcoded horrible hacks :) Anyone with the right pre-requisites and the udev branch of banshee with my patch can have basic iDevice support (tracks only).

This is awesome!

This will not be pushed to the main banshee repository as it depends on udev/gio to be useful. I will instead push my patches upstream to lamalex and his udev branch. All that has to be done then is to push the udev work to mainline banshee and everyone can enjoy full iDevice support.

Tomorrow I hope to get in touch with lamalex and iron out a few bugs I've experienced and then upstream my work to him. My bugfixes to the libgpod-sharp bindings have already been upstreamed and should be merged with the main codebase very soon.

Hackweek Day II - Attack of the code

Today has been reasonably productive. I found some bugs in the libgpod bindings which were fairly major and implemented some more features in banshee.

First the fun bug. The libgpod bindings did the normal thing of defining a managed struct which mirrored the native struct that libgpod uses. This means we can do a trivial byte copy of the unmanaged memory into one of our managed structs and trivially access the information in a safe way. This is all well and good right up until you realise that when you update a property on that struct, you are not actually changing the value of the unmanaged memory. Essentially you have two copies of the same data which are disconnected - one in managed memory and one in native memory. Updating one copy is not reflected in the other and there's no easy way to keep them synced without the risk of losing data.

After a bit of brainstorming with Jeremie, we came up with a rather neat and nifty solution. Take this struct as an example:

struct NativeDate {
    public int Day;
    public int Month;
    public int Year;
}

If native code allocates one of those and passes it to .NET as an IntPtr, there is a trivial way to map this to managed code without requiring a copy of the data.

unsafe class Date {
    IntPtr Native {
        get; set;
    }

    public int Day {
        get { return ((NativeDate *)Native)->Day; }
        set { ((NativeDate *) Native)->Day = value; }
    }
    public int Month {
        get { return ((NativeDate *) Native)->Month; }
        set { ((NativeDate *) Native)->Month = value; }
    }

    public int Year {
        get { return ((NativeDate *) Native)->Year; }
        set { ((NativeDate *) Native)->Year = value; }
    }

    public Date (IntPtr native)
    {
        Native = native;
    }
}

All we do is use a bit of unsafe code to take the pointer that we got from native code and cast it to a NativeStruct* and read or write the data as appropriate. Now both managed land and unmanaged land are working off the same hunk of memory so they can never get out of sync.

Now for the features! Banshee can now sync music to my iPhone and remove it aswell! This means all the basic stuff that's required for basic syncing is supported now. It took a lot longer than it should've to get this working because my iPhone wasn't actually set up right. There was a command (i can't remember the name of it now) which should've been run automatically to populate iTunes_Control/Device with some required information. This was never run for some bizarre reason (yay for bleeding edge) which meant nothing I did could ever actually update the database.

However once I got that setup, things progressed pretty rapidly. There's one major feature left which is automatic detection when the device is mounted. If i can get this done tomorrow i'll try to upstream everything and get people testing this by the weekend (or friday!).

Hackweek V - Day 1

So at the end of Day 1, things have progressed well enough. I managed to get all my requirements installed (just about!) though to do it I needed to install older versions of some of the ipod stack. Hopefully this won't bite me in the ass as the week progresses ;)

So a brief synopsis of the state of the world as I found it after one day:

1) Things are still very bleeding edge. On opensuse you still need just the right packages and svn checkouts to even begin to get things working. Nothing you need is in the standard repositories, I had to use the awesome opensuse build service to provide some packages (gvfs >= 1.5.1, libplist, umuxd) and had to compile just the right version of libimobiledevice manually. Not a pleasant experience.

2) Once all that horribleness is done, actually detecting the ipod in banshee is difficult. HAL is useless. You need either gio# or udev to be able to detect the device when it's plugged in. This puts further requirements on your distro to ship the latest goodies. I worked around this by hardcoding my iphones mount point for now ;)

3) Banshee is good to work with. Integrating DAPs with banshee is easier than it used to be. Hopefully things will simplify the more I delve into things and learn what's available. Some things in the existing code I hope are only there for historical reasons and can be removed with this rewrite ;)

4) So far I have two big checkpoints: Banshee can load the tracks on my iphone and it can also import them to its collection. I'm in the process of allowing you to copy tracks to the device, but I hit a few issues with libgpod which have yet to be investigated.

So things are looking good for the first day. Hopefully day 2 will be full of interesting developments. Maybe I'll get playlist syncronisation working, maybe I'll get track uploading working. Who knows!

Hackweek V

Next week is Hackweek V in Novell, the week in which we can work on whatever we want. This year, my hackweek proposal is to give banshee a new ipod addin based on libgpod. This should fix some of banshees bugs.

While working on this I'll try live tweet as I hack so if you're interested, do tune in from tuesday morning onwards (GMT). Mondays a bank holiday so I'm going to have to complete my hack in 4 days.

If you can, do give the proposal a +1. I've no idea what affect it'll have but do it anyway! ;)

EDIT: Link the correct hackweek proposal. Whoopsie :)

Performance benchmarks... or lack thereof

A friend of mine was over the other day and I was showing him this cool website I had recently heard of, grooveshark. So after admiring it for a little while he came out with "pity it's written in flash". So of course I asked the obvious question "Why?" and got the obvious answer "Because it's inefficient and sucking up all your CPU". He popped open 'top' and sure enough it was using 35% of my cpu, his point was proven.

However, that is completely and utterly irrelevant. This does not make flash a bad technology nor does it make grooveshark a badly written application. Maybe 35% cpu usage is just the cost of running in the browser. It's like the old complaint against firefox, "Firefox uses too much memory". By itself that means absolutely nothing. It's nonsense. For that to have any real meaning you'd have to compare firefox to another browser doing the same task. A legitimate complaint might be "Firefox uses 3 times more memory than opera when I view www.example.com, 300 mb versus 100mb". An arbitrary decision as to what amounts to be "too much" is useless without a reference. This is exactly what far too many people are doing to flash.

Sure, grooveshark uses more CPU and memory than banshee, but banshee isn't running inside firefox. It annoys me that people make such arbitrary decisions and try to pass off as fact that flash is inherently bad and HTML5 is inherently so much better and yet have no reference to compare against. I haven't seen anything even close to the complexity of grooveshark written using html5. So before you get on your high horse decrying all plugins, show me a comparable application which uses $COMPETING_TECHNOLOGY_OF_CHOICE and performs measurably better.

Zencomic - increasing happiness since nineteen digity two

Today I had some spare time and decided to hack together a quick addin for Zencomic. I choose garfield minus garfield as I feel it reflects my life pretty closely in some aspects (not the manic depressive aspects though ;) ).

One of the downsides of working with a distributed team is that everyone starts work at a different time. On some of the quieter days when people are away on holidays, I'm sometimes the only hacker actively working on Moonlight in the wee small hours of 9am UTC. By the time the Americans wake up, I've sometimes left a nice little monologue in the IRC channel detailing my frustrations and successes during the morning.

Anyone, now for the obligatory screenshot


The addin was hacked together over the course of 30 minutes. It should be able to open any garfield-garfield comic, but if you find a particular one it fails on do let me know. Also, please don't suggest I switch my beautiful string splitting code into a Regex unless you plan on writing it yourself ;)

UPDATE: I just fixed the last known issue. The addin now detects exactly how many pages of g-g goodness there are rather than being hardcoded to assume 101 pages (the current number).

hapy birfday 2 me

My confusion this morning was evident

FOSDEM 2010

So Ruben has been working on getting a bunch of talks organised for the Mono room in FOSDEM and I've been lucky enough to get a slot. I have a 30 minute slot, so I figured that would be long enough to talk about 2 things properly and then take questions.

I've decided that I want to talk about two things.

1. The piece picker algorithm
2. How i handle threading

The piece picking code handles which segment of the file will be downloaded next. It has gone through several iterations, each time adding new features. Eventaully the whole class became so hard to maintain and test that I just had to rewrite things from the ground up using a new approach. This might give some people ideas on how to tackle problems in their own projects but even if it doesn't, it should still be interesting ;)

MonoTorrent handles a lot of concurrent connections. Using one thread per connection just wouldn't scale yet using plain asynchronous sockets and locking has huge potential for deadlocks and race conditions. Getting this right took well over a year - probably due to my lack of experience at this - but right now I think I've hit the winning threading solution. This part of the talk should be useful to anyone who's worked on an app which does a lot of socket operations. Maybe I'll wake up in a few months time and see a dozen apps using that technique.

Anyway, feel free to drop in for my talk (and all the others!). If you have any questions about any other aspects of MonoTorrent, or programming in general, feel free to find me and ask.

Expression Trees - serializing your data

Update: Just to clarify - the code snippets below are under the MIT/X11 license.

I spent a few hours over the weekend writing a binary serializer using expression trees. I wanted to see how things would look using the new features available in .NET 4.0. My requirements were pretty simple:

1) Serialize all public properties in a type or a subset of them
2) Control the order in which they're serialized - sometimes you need to interop with an existing and you must write your data in a specific order
3) Control how a primitive is converted - Do you need to write value types in big endian, little endian, middle endian?
4) Easy to use API.

So lets start with the API. This is what I was hoping to use:

public class Secondary
{
public int First { get; set; }
public int Second { get; set; }
public int Third { get { return First + Second; } }
}

public class MyClass
{
public byte ByteProp { get; set; }
public short ShortProp { get; set; }
public int IntProp { get; set; }
public long LongProp { get; set; }
public string StringProp { get; set; }
}

static void Main(string[] args)
{
// Register a message so that all public fields will be serialized
Message.Register<MyClass>();

// Register a message so that only some fields are serialized and
// they are serialized in the specified order
Message.Register<Secondary>(
   d => d.Second,
   d => d.First
);

// Create a stream to serialize the data to
Stream s = new MemoryStream();
var message = new MyClass {
   IntProp = 1,
   LongProp= 2,
   ByteProp= 3,
   ShortProp = 4,
   StringProp = "Hello World"
};

// Encode the message to the stream
MessageEncoder.Encode(message, s);

// Rewind the stream and then decode the message
s.Position = 0;
var decoded = MessageDecoder.Decode<MyClass>(s);
}

It's pretty standard stuff. You can work with the standard serializer logic (serialize properties alphabetically) by registering an object without specifying any specific properties or you can customise which properties are serialized. This could also be done using attributes, but using attributes to control the order in which properties are serialized would be more error prone than the above.

Firstly, sometimes you need to write your data in big endian, others you need little endian. Sometimes you won't care. What you need is to be able to control this:

MessageEncoder.RegisterPrimitiveEncoder<int>((value, stream) => {
  stream.Write(BitConverter.GetBytes(value));
});

It's simple. Any type which can be directly converted to an array of bytes is classified as a 'primitive'. Each primitive can have an encoder/decoder pair registered as above.

public static class MessageEncoder
{
static Dictionary<Type, Delegate> encoders;
static Dictionary<Type, Delegate> primitives;

static MessageEncoder()
{
   encoders = new Dictionary<Type, Delegate>();
   primitives = new Dictionary<Type, Delegate>();
   RegisterPrimitiveEncoders();
}

static void RegisterPrimitiveEncoders()
{
   RegisterPrimitiveEncoder<byte>((value, stream) =>
       stream.WriteByte(value)
   );

   RegisterPrimitiveEncoder<short>((value, stream) =>
       stream.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(value)))
   );

   RegisterPrimitiveEncoder<int>((value, stream) =>
       stream.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(value)))
   );

   RegisterPrimitiveEncoder<long>((value, stream) =>
       stream.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(value)))
   );

   var intWriter = (Action<int, Stream>)primitives[typeof (int)];
   RegisterPrimitiveEncoder<string>((value, stream) => {
       var buffer = Encoding.UTF8.GetBytes(value);
       intWriter(buffer.Length, stream);
       stream.Write(buffer);
   });
}

public static void RegisterPrimitiveEncoder<T>(Action<T, Stream> encoder)
{
   primitives [typeof (T)] = encoder;
}

public static void RegisterMessage<T>(params Expression<Func<T, object>>[] properties)
{
   RegisterMessage<T>(properties.Select(p => p.AsPropertyInfo ()));
}

public static void RegisterMessage<T>(IEnumerable<PropertyInfo> properties)
{
   var propertyEncoders = new List<Expression>();

   // The encode function takes an instance of the class we're decoding and the Stream
   // which we should write the data to.
   ParameterExpression source = Expression.Parameter(typeof(T), "source_param");
   ParameterExpression stream = Expression.Parameter(typeof(Stream), "stream");

   // For each property, get the encoder which will convert the value of the property to a byte[]
   // which can be written to the stream.
   foreach (var property in properties) {
       // Get the encoder for this property type
       var action = primitives[property.PropertyType];
       // Create a var which holds the Action <T, Stream> which encodes the data to the stream
       Expression converter = Expression.Constant(action, action.GetType ());
       // Invoke the encoder passing the value of the property and the 'stream'
       Expression invoker = Expression.Invoke(converter, Expression.Property(source, property), stream);
       // Add the encoder for this property to the list.
       propertyEncoders.Add(invoker);
   }

   // Create an expression block which will execute each of the encoders one by one
   Expression block = Expression.Block(propertyEncoders);
   encoders.Add(typeof(T), Expression.Lambda<Action<T, Stream>>(
       block,
       source,
       stream
   ).Compile());
}

public static void Encode<T>(T message, Stream s)
{
   var encoder = (Action<T, Stream>)encoders[typeof (T)];
   encoder (message, s);
}
}

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Linq.Expressions;
using System.Reflection;
using System.Net;
using System.IO;

namespace Encoder
{
public static class MessageDecoder
{
   static Dictionary<Type, Delegate> decoders;
   static Dictionary<Type, Delegate> primitives;

   static MessageDecoder()
   {
       decoders = new Dictionary<Type, Delegate>();
       primitives = new Dictionary<Type, Delegate>();
       RegisterDefaultDecoders();
   }

   static void RegisterDefaultDecoders()
   {
       RegisterPrimitiveDecoder<byte>((s) => {
           var val = s.ReadByte();
           if (val == -1)
               throw new EndOfStreamException();
           return (byte)val;
       });

       RegisterPrimitiveDecoder<short>((s) => IPAddress.NetworkToHostOrder (s.ReadShort()));
       RegisterPrimitiveDecoder<int>(s => IPAddress.NetworkToHostOrder (s.ReadInt()));
       RegisterPrimitiveDecoder<long>(s => IPAddress.NetworkToHostOrder (s.ReadLong()));

       var intDecoder = (Func<Stream, int>)primitives[typeof(int)];
       RegisterPrimitiveDecoder<string>(s => {
           var length = intDecoder(s);
           var buffer = new byte[length];
           s.Read(buffer, 0, buffer.Length);
           return Encoding.UTF8.GetString(buffer);
       });
   }

   public static void RegisterPrimitiveDecoder<T>(Func<Stream, T> decoder)
   {
       primitives.Add(typeof(T), decoder);
   }

   public static void RegisterMessage<T>(params Expression<Func<T, object>>[] properties)
   {
       RegisterMessage<T>(properties.Select(d => d.AsPropertyInfo()));
   }

   public static void RegisterMessage<T>(IEnumerable<PropertyInfo> properties)
   {
       var propertyDecoders = new List<Expression>();

       // The decode function takes an instance of the class we're decoding and the Stream
       // containing the data to decode.
       ParameterExpression source = Expression.Parameter(typeof(T), "source_param");
       ParameterExpression stream = Expression.Parameter(typeof(Stream), "stream");

       // For each property, get the primitive decoder which will read data from the stream and
       // return a value of the correct type.
       foreach (var property in properties) {
           var action = primitives[property.PropertyType];
           // Create a var which holds the Func <Stream, T> which decodes the data from the stream
           Expression decoder = Expression.Constant(action, action.GetType());
           // Invoke the decoder passing 'stream' as the parameter
           Expression invoker = Expression.Invoke(decoder, stream);
           // Store the return value of the decoder in the property.
           Expression setter = Expression.Call(source, property.GetSetMethod(), invoker);
           // Add the decoder for this property to the list.
           propertyDecoders.Add(setter);
       }

       // Create a block which will execute the decoders for all the fields one after another.
       Expression block = Expression.Block(propertyDecoders);
       decoders.Add (typeof (T), Expression.Lambda<Action<T, Stream>>(
           block,
           source,
           stream
       ).Compile ());
   }

   public static T Decode<T>(Stream s) where T : class, new()
   {
       T t = new T();
       var decoder = (Action<T, Stream>)decoders[typeof(T)];
       decoder(t, s);
       return t;
   }
}
}

The idea is quite simple. For each class we can generate an ideal serializer using expression trees which doesn't require boxing or casting. This way we can avoid the use of reflection when serializing objects and so avoid the performance penalties incurred that. The code above only handles the simple case where a class consists of primitive types (int, long, string) , though it'd be easy enough to extend it to support more complex scenarios.

The serializer as you see it could not have been written with .NET 3.0. Some of the key components like BlockExpression were only introduced with .NET 4.0. If your object contains an array which needs to be serialized, you'll need the new IndexExpression too. Sure, it's possible to fake these using some anonymous delegates and Actions, but that's not pretty :)

The total implementation is less than 170 LOC. I'd be willing to bet that with another 100 LOC you could support most constructs. If you're currently a heavy user of reflection to provide object serialization, it's time to update ;)

New years resolutions

It's tradition in quite a lot of countries to make a new years resolution on the 1st of January. Most people forget about them within a few days or weeks. This year, I'll be making one I'm going to keep!

I want to take part in a dancing [0] flash mob whether it's in this country or another.



What ideas do you have? Anything strange, interesting, unusual? Leave a comment and let me know, maybe you have a better idea than being part of a flash mob.

[0] Me and dancing don't get on particularly well, so it'll be an interesting challenge ;)

Yet another INotifyPropertyChanged with Expression Trees - Part 2

In my last post, I described a method whereby you can implement INotifyPropertyChanged with zero performance overhead and near-zero boilerplate code. The only boilerplate left was the delegate you had to create to invoke the event:

public Book()
{
    // Boilerplate - eugh!
    Action<string> notify = (propertyName) => {
        var h = PropertyChanged;
        if (h != null)
            h(this, new PropertyChangedEventArgs(propertyName));
    };

    author = new ChangeNotifier<string> (() => Author, notify);
    price = new ChangeNotifier<decimal> (() => Price, notify);
    quantity = new ChangeNotifier<int> (() => Quantity, notify);
    title = new ChangeNotifier<string> (() => Title, notify);
}

The entire point of my implementation was to avoid writing boilerplate, so this was slightly irritating. Unfortunately, there's no trivial way around the problem as the .NET framework really limits what you can do with events. The first thing you'd think of is "pass the actual object into the ChangeNotifier constructor and just raise the event that way". For example my constructors would change to:

new ChangeNotifier<string>(() => Author, this);

That's well and good, right up until you realise that it's impossible for one object to raise an event that's declared on another object.

public class A
{
    public event EventHandler MyEvent;
}

public class B
{
    public void AccessEvent (A a)
    {
        // Invalid - you can't raise an event which is declared in another class
        a.MyEvent(this, EventArgs.Empty);

        // Invalid - you can't copy the event either
        EventHandler h = a.MyEvent;
        h(this, EventArgs.Empty);
    }
}

Another alternative would be to pass the event itself into the ChangeNotifier object:

new ChangeNotifier<string> (() => Author, PropertyChanged);

But this won't work because a copy of the delegate list is created. That means if anyone adds event handlers later on, they won't be invoked when the property changes. So with that stuck firmly in my mind, I never gave much thought to removing that last remaining bit of boilerplate. That's about to change!

What I really want is for my final implementation to look more like this:

public class Book : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    ChangeNotifier<string> author;

    public string Author
    {
        get { return author.Value; }
        set { author.Value = value; }
    }

    public Book()
    {
        author = ChangeNotifier.Create(() => Author, ????);
    }
}

That's short and sweet . The generic types should be automatically inferred, you shouldn't have to create the delegate to raise the event, it's beautiful! The only problem is to figure out what I should replace the question marks with. I need something that will allow me to get at the current list of event handlers from outside of the Book object, i.e. something along the lines of this:

Func<PropertyChangedEventHandler> getter = delegate { return PropertyChanged; };

Prettying it up a little, this is how my Book class looks:

public class Book : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    ChangeNotifier<string> author;

    public string Author {
        get { return author.Value; }
        set { author.Value = value; }
    }

    public Book()
    {
        author = ChangeNotifier.Create (() => Author, () => PropertyChanged);
    }
}

Beautiful! The more astute readers might notice a problem at this stage. Fine, the ChangeNotifier object can get the event list and raise the event, but it can't fill in the 'sender' - it has no reference to the 'book' object! Have no fear, it's already taken care of! The getter delegate has a reference to the book object (Delegate.Target), so we can fill everything in perfectly! The final implementation of the ChangeNotifier class is this:

public static class ChangeNotifier
{
    public static ChangeNotifier<TValue> Create<TValue>(Expression<Func<TValue>> expression, Func<PropertyChangedEventHandler> notifier)
    {
        return new ChangeNotifier<TValue>(expression, notifier);
    }
}

public class ChangeNotifier<TValue>
{
    Func<PropertyChangedEventHandler> notifier;
    string propertyName;
    TValue value;

    public TValue Value {
        get { return value; }
        set {
            if (!EqualityComparer<TValue>.Default.Equals(this.value, value)) {
                this.value = value;
                // Get the current list of registered event handlers
                // then invoke them with the correct 'sender' and event args
                PropertyChangedEventHandler h = notifier();
                if (h != null)
                    h(notifier.Target, new PropertyChangedEventArgs(propertyName));
            }
        }
    }

    public ChangeNotifier(Expression<Func<TValue>> expression, Func<PropertyChangedEventHandler> notifier)
    {
        if (expression.NodeType != ExpressionType.Lambda)
            throw new ArgumentException("Value must be a lamda expression", "expression");
        if (!(expression.Body is MemberExpression))
            throw new ArgumentException("The body of the expression must be a memberref", "expression");

        MemberExpression m = (MemberExpression)expression.Body;
        this.notifier = notifier;
        this.propertyName = m.Member.Name;
    }
}

I have one final trick up my sleeve. Suppose you have a field (Progress) whose value is calculated based on other values (CurrentStep, TotalSteps) and you want to get Notifications whenever any of those fields changes, well, that's easy!

public class Worker : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    ChangeNotifier<int> currentStep;
    ChangeNotifier<int> totalSteps;

    public int CurrentStep {
        get { return currentStep.Value; }
        set { currentStep.Value = value; }
    }
    public int TotalSteps {
        get { return totalSteps.Value; }
        set { totalSteps.Value = value; }
    }
    public double Progress
    {
        get { return (double)CurrentStep / TotalSteps; }
    }

    public Worker()
    {
        Func<PropertyChangedEventHandler> notifier = () => PropertyChanged;

        currentStep = ChangeNotifier.Create(() => CurrentStep, notifier);
        totalSteps = ChangeNotifier.Create(() => TotalSteps, notifier);

        // A PropertyChanged notification will be created for Progress every time
        // either the CurrentStep *or* TotalSteps changes.
        ChangeNotifier.CreateDependent(
            () => Progress,
            notifier,
            () => CurrentStep,
            () => TotalSteps
        );
    }
}

And the new helper methods are:

public static class ChangeNotifier
{
    static string GetPropertyName(Expression expression)
    {
        while (!(expression is MemberExpression)) {
            if (expression is LambdaExpression)
                expression = ((LambdaExpression)expression).Body;
            else if (expression is UnaryExpression)
                expression = ((UnaryExpression)expression).Operand;
        }
       
        return ((MemberExpression)expression).Member.Name;
    }

    public static void CreateDependent<TValue>(Expression<Func<TValue>> property, Func<PropertyChangedEventHandler> notifier, params Expression<Func<object>>[] dependents)
    {
        // The name of the property which is dependent on the value of other properties
        var name = GetPropertyName(property);
        // The names of the other properties
        var dependentNames = dependents.Select<Expression, string>(GetPropertyName).ToArray();
       
        INotifyPropertyChanged sender = (INotifyPropertyChanged)notifier.Target;
        sender.PropertyChanged += (o, e) => {
            // If one of our dependents changes, emit a PropertyChanged notification for our property
            if (dependentNames.Contains(e.PropertyName)) {
                var h = notifier();
                if (h != null)
                    h(o, new PropertyChangedEventArgs (name));
            }
        };
    }

    public static ChangeNotifier<TValue> Create<TValue>(Expression<Func<TValue>> expression, Func<PropertyChangedEventHandler> notifier)
    {
        return new ChangeNotifier<TValue>(expression, notifier);
    }
}

The only change is that I need to use a slightly more complicated method of getting the property name as it's possible for certain types to get wrapped in a ConvertExpression.

Yet another INotifyPropertyChanged with Expression Trees

There are dozens of examples out there showing you how to avoid having to refer to method names as strings when implementing INotifyPropertyChanged. The most important reason why you don't want to have to do this is because method names can get refactored but the hardcoded strings might be forgotten. No-one wants to end up getting a Changed notification for a property which doesn't exist.

My issue with all these examples is that none of them thought far enough ahead. Fine, they all show you how refer to properties without using hardcoded strings but they still require you to write lots of boilerplate code to raise the PropertyChanged event - boilerplate you have to write for every property. What I want is to be able to declare all my properties like:

public string Title {
    get { return title; }
    set { title = value; }
}

and yet still get my property change notifications. I also want this method to be reasonably high performance. I don't want every property change to have extra memory or CPU overhead as every developer expects that changing the value of a property will not do any complex calculations. So how can I accomplish this?

To start off with, we can all tell that it's impossible to achieve the required behaviour using just the snippet above. We're going to have to add (at least) one additional level of indirection. That means I should be able to implement my requirements using code like:

public string Title {
    get { return title.Value; }
    set { title.Value = value; }
}

The object 'title' must then contain all the logic required to raise the property changed notification. So what might this magical object look like?

public class ChangeNotifier<TValue>
{
    Action<string> notifyHandler;
    string propertyName;
    TValue value;

    public TValue Value {
        get { return value; }
        set {
            if (!EqualityComparer<TValue>.Default.Equals(this.value, value)) {
                this.value = value;
                notifyHandler(propertyName);
            }
        }
    }


    public ChangeNotifier(Expression<Func<TValue>> expression, Action<string> notifyHandler)
    {
        if (expression.NodeType != ExpressionType.Lambda)
            throw new ArgumentException("Value must be a lamda expression", "expression");
        if (!(expression.Body is MemberExpression))
            throw new ArgumentException("The body of the expression must be a memberref", "expression");

        MemberExpression m = (MemberExpression)expression.Body;
        this.propertyName = m.Member.Name;
        this.notifyHandler = notifyHandler;
    }
}

You're probably looking at this thinking "What the hell is this Expression<Func<TValue>> ? How do I even use that monstrosity?". Well... simples!

public class Book : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    ChangeNotifier<string> author;
    ChangeNotifier<decimal> price;
    ChangeNotifier<int> quantity;
    ChangeNotifier<string> title;

    public string Author {
        get { return author.Value; }
        set { author.Value = value; }
    }
    public decimal Price {
        get { return price.Value; }
        set { price.Value = value; }
    }
    public int Quantity {
        get { return quantity.Value; }
        set { quantity.Value = value; }
    }
    public string Title {
        get { return title.Value; }
        set { title.Value = value; }
    }

    public Book()
    {
        Action<string> notify = (propertyName) => {
            var h = PropertyChanged;
            if (h != null)
                h(this, new PropertyChangedEventArgs(propertyName));
        };

        author = new ChangeNotifier<string> (() => Author, notify);
        price = new ChangeNotifier<decimal> (() => Price, notify);
        quantity = new ChangeNotifier<int> (() => Quantity, notify);
        title = new ChangeNotifier<string> (() => Title, notify);
    }
}

All that happens here is that when constructing the ChangeNotifier object, an Expression referencing the required Property is passed into the constructor, along with a delegate which will raise the PropertyChanged event. We parse that expression tree to retrieve the method name and store it. After that everything Just Works (tm) with little to no performance penalty. The days of writing boilerplate code for INotifyPropertyChanged are gone! You also have the benefit that you can't make a mistake writing the boilerplate code because you don't write it anymore!

Can't you feel the Moonlight? Part deux

As I was saying yesterday, the live version of the silverlight toolkit site didn't work right in moonlight. All the pretty charts rendered as you see them below, very empty.

I figured that since a slightly older version worked near-flawlessly, surely I could fix the live version with only a few minor tweaks. It's not like the would've completely rewritten the Chart controls within the space of 1 release.

I checked everything from DataBinding, to TemplateBinding, to Styles, to Measure/Arrange bugs and nothing was showing up as causing the issue. I finally narrowed it down to a bug in VisualStateGroup. For some reason the Name property was empty even though it was declared with a name in xaml.

One. Tiny. Patch. Later.


Success. I can't believe that the bug was that simple. In the end, those bugs are actually by far the worst. There's no exception thrown or any kind of visible indication that something has failed other than an empty screen. The only reason I found the bug was because the toolkit is opensource and I was running it locally with a few dozen Console.WriteLines, gradually reducing the area of code where I thought the bug was. Unfortunately this fix arrived too late for the 1.99.9 release, but it will definitely be in the release after it.

Can't you feel the moonlight?

It's time for the obligatory screenshots again. This is what the Data Visualisation demos from the Silverlight Toolkit (March edition) looked like yesterday:

Note the empty graphs. It doesn't look very pretty now, does it? However, one very minor fix later we now have the following:

Things are near-perfect in all the Data Visualization demos. One graph is missing a background colour and the elements in one graph aren't clickable when they should be. Neither should be particularly difficult to fix, the only problem is figuring out the cause.

Unfortunately the version of the Toolkit Demo on the live site still doesn't render perfectly, but as we already have one version near-perfect, getting a newer revision to work shouldn't be hard! Things are shaping up to give us a great 2.0 release.

What does this say?

This is the logo for the London olympics. Wolff Olins were paid a whopping GBP 400,000 to design this. You'd think that if you spent that kind of money, you'd end up with a logo that can be understood by mere mortals, however that's not what we got.

When I look at that logo I see the numbers '2' and '0' at the top. That's relatively clear. However, the bottom section is just random gibberish. If I concentrate a little harder, I could convince myself that there's a '1' in the bottom left, but what the hell is at the bottom right? A square with a squiggle beside it? I can't for the life of me figure out what that little square is for. Is it part of the '2' or is it there because they wanted to write "20-12"? I really can't tell.

If you're going to design a logo for such an important worldwide event, why can't it be legible? Logo design should be about creating an inventive and visually appealing way of getting some information across. It isn't supposed to be about who can create the most obfusticated advertisement - that defeats the entire purpose! Out of this list of a dozen alternative designs my favourite is this one:

It may not be the best logo I've ever seen, but it does one thing right: It's instantly recognisable and understandable. I know exactly what it's about without having to spend 10 mins rotating the thing trying to make sense out of it.

Dear Thierry Henry

If you truly were sorry that you twice hit the ball with your hand to prevent it from going wide and then proceeded to score from that opportunity, why didn't you admit it on the spot. You knew what you did, you did it deliberately, it is useless claiming you're sorry now when it's too late for that to mean anything. Your chance to show your true colours was on the field and show them you did.

Since the match will not be replayed, why don't you step down from this world cup season if you truly do regret that decision which cost Ireland our world cup chance?