A picture is worth 1000 words

Two to three weeks ago if you ran this battery of garbage collection related tests in moonlight, they'd mostly be red. How things have changed :)

Writing a profiler for mono

As some of you may know, my day job is working on Moonlight, the FOSS implementation of Silverlight. This is a fairly complex piece of code as we have C++ interoperating with C# and Javascript, so managing the lifecycle of objects can be difficult. C++ refcounts, C# has an automatic garbage collector and Javascript... well, let's just say that it complicates things a *lot* when combined with the other two languages.

One of the tricks we use to ensure that objects do not die early is to use a (normal) GCHandle on our C# objects to ensure the garbage collector doesn't mark them as junk before they're definitely finished with in C++. However, we were having issues where some of our objects appeared to never ever get garbage collected even though they should be eligible. We suspected that excess GCHandles were causing it but we had no easy way of tracking this. Without detailed information on when and where the GCHandles were allocated, it would be next to impossible to track down where we were going wrong. After several aborted attempts to track this with Console.WriteLine I figured there must be a better way.

How about a custom profiler for mono?

Documentation on this is fairly sparse currently, so hopefully this will help. Thanks go to Paolo Molaro for helping me when I got stuck! This code is mostly copied/pasted from moonlight so it's C++ ish and also may contain some moonlight specific code.

Step 1:
Open up the "mono/metadata/profiler.h" file and check out all the profiler hooks that are available. They are all documented in profiler.c, so open that if you want a description of how they all work. Take note of the ones you'll need. All I needed to do was track GCHandle allocations, so I just wanted this hook:

void mono_profiler_install_gc_roots    (MonoProfileGCHandleFunc handle_callback, MonoProfileGCRootFunc roots_callback);

Step 2:
I now know what function I need to provide and the data I want to gather, so it's time to create a struct to store my data. This is an opaque struct which will be passed to the mono runtime and passed back in every profiler hook, so you can stick whatever you want in it. I know I want to store stacktraces, allocated gchandles and a typename, so I'll start with just those. I'll also declare my prototype for my profiler hook here.

struct _MonoProfiler {
const char *type_name;
GPtrArray *gchandles;
GPtrArray *stacktraces;
Moonlight::Mutex locker; /* used to ensure only one thread accesses the arrays */

static void track_gchandle (_MonoProfiler *prof, int op, int type, uintptr_t handle, MonoObject *obj);
};

typedef _MonoProfiler MonoProfiler;

Step 3:
Implementation time! First the constructor for the MonoProfiler. I use an environment variable to limit which types we store stacktraces for as they're expensive to generate and consume a lot of memory to store when you have 10,000s of them. The important functions are the last three. The first registers our profiler with the runtime. The second installs the hook so we can get notifications for gchandle allocations. The final call enables the gc_handle events in the profiler which makes the runtime invoke the hook we just registered.

_MonoProfiler::_MonoProfiler ()
{
   type_name = g_getenv ("GCHANDLES_FOR_TYPE");

   gchandles = g_ptr_array_new ();
   stacktraces = g_ptr_array_new_with_free_func (g_free);
   /* Register the profiler with mono */
   mono_profiler_install (this, NULL);
   /* Supply a function for the gc_roots hook */
   mono_profiler_install_gc_roots (track_gchandle, NULL);
   /* Enable gc_roots tracking in the profiler so our hook is invoked */
   mono_profiler_set_events (MONO_PROFILE_GC_ROOTS);
}

Now all I need is the implementation for track_gchandle and I'm done!

void
MonoProfiler::track_gchandle (MonoProfiler *prof, int op, int type, uintptr_t handle, MonoObject *obj)
{
   // Ignore anything that isn't a strong GC handle (docs say type == 2 is a strong gchandle)
   if (type != 2)
       return;

   prof->locker.Lock ();

   GPtrArray *gchandles = prof->gchandles;
   GPtrArray *stacktraces = prof->stacktraces;

   if (op == MONO_PROFILER_GC_HANDLE_CREATED) {
       // Add the GCHandle to this array
       g_ptr_array_add (gchandles, (gpointer) handle);
       // If the target of the gchandle is of the correct type, store its stack trace
       // Otherwise store NULL so that we can keep the index of the gchandle and corresponding
       // stack trace in sync.
       if (prof->type_name && !strcmp (prof->type_name, mono_class_get_name (mono_object_get_class(obj))))
           g_ptr_array_add (stacktraces, get_stack_trace ());
       else
           g_ptr_array_add (stacktraces, NULL);
   } else if (op == MONO_PROFILER_GC_HANDLE_DESTROYED) {
       // Walk our list of gchandles and when we find the index of the destroyed handle
       // remove the handle and corresponding stacktrace
       for (int i = 0; i < (int)gchandles->len; i++) {
           if (g_ptr_array_index (gchandles, i) == (gpointer) handle) {
               g_ptr_array_remove_index_fast (gchandles, i);
               g_ptr_array_remove_index_fast (stacktraces, i);
               break;
           }
       }
   }

   prof->locker.Unlock ();
}

Step 4:
Profit! When you're ready, you can iterate over the gchandle array and pull out whatever statistics you want. For example here is the code I use to iterate the gchandles array and work out how many instances of each object type there are.

void
accumulate_g_ptr_array_by_type (gpointer data, gpointer user_data)
{
   // The hashtable I passed in as the user_data in g_ptr_array_foreach
   // which i am using to link a type name to a count
   GHashTable *by_type = (GHashTable*) user_data;

   // Get the MonoObject from the gchandle.
   MonoObject *ob = mono_gchandle_get_target (GPOINTER_TO_INT (data));

   // Get the type name from the mono_object
   const char *name = mono_class_get_name (mono_object_get_class(ob));

   // Find out how many instances we have already
   int count = GPOINTER_TO_INT (g_hash_table_lookup (by_type, name)) + 1;

   // Update the hashtable with an incremented count.
   g_hash_table_insert (by_type, name, GINT_TO_POINTER (count));
}

I run that hashtable through another function to order to sort the list by the number of each type of object allocated so I can have a nicely formatted list as follows:

1 instances GCHandled of type Surface
1 instances GCHandled of type Deployment
4 instances GCHandled of type NameScope
20 instances GCHandled of type ControlTemplate
218 instances GCHandled of type MonoType
3985 instances GCHandled of type Uri

Needless to say, it's now immediately obvious to me that Uris are being GChandled and never freed, so i just have to enable tracing of their stacktraces and I can see exactly where we allocated the handle and from there figure out why we haven't freed it.

MonoTorrent Lives! (on github)

The mono project, who've been hosting MonoTorrent since day 1, recently decided to host all their code at github. As such, MonoTorrent now lives in github! I took this opportunity to rename the module from 'bitsharp' to 'monotorrent' to avoid confusing people.

So what does this mean? Well, it means it's trivial for you to commit patches now! All you do is set up a github account, fork monotorrent, commit your changes to your fork and issue a pull request. I'll then automagically be informed of your changes and can click a button to either merge them directly into monotorrent or reject them (with an explanation). No more emailing patches or attaching them to forum posts. Pretty nifty, eh?

So get forking and lets see how great we can make MonoTorrent!

Hackwee Day IV - The final showdown

Yesterday was a bit of an anti-climax codewise. Pretty much everything I had planned on doing I had completed by midday. I then had a nice long chat with lamalex about what I had to do to get ipod support into his udev branch and hashed out a rough idea about how to upstream these changes. I believe he's going to do a bit of work over the next week or so and once that's complete I'll be able to upstream the remainder of my patches.

So at the end of the 4 days the state of things is:

Device support:
1) Hotplugging works
2) Syncing music to and from the device works
3) Removing music from the device works
4) Basic properties can be read about the device and displayed in banshee
5) Playlists aren't supported yet but should be fairly simple to do

Code written:
1) I've upstreamed my patches for libgpod-sharp
2) I spoke with lamalex about upstreaming the rest of my work there.
3) There are now packages available for openSUSE 11.2 courtesy of FunkyM which can be used to provide most of the required packages.
4) There are still some bleeding edge libraries required which aren't packaged anywhere. These will have to be packaged.
5) I think the best way to support the libgpod based iDevice would be to create a new addin so that you can run both the old addin and the new addin in parallel as some of the required libraries for libgpod based iDevice support are very new and not widely available as of yet. This will take a little bit of time.

I'm off on holidays next week so when I get back I'll start pushing my remaining work upstream. Once that's done it's just the boring process of streamlining the build to remove the dozen manual steps which are currently required :) All in all, it's been a successful hackweek. But as with all good hacks, the 5:1 rule applies. For every day spent adding functionality, I'll need 5 to make it stable and usable for everyone.

I have to thank teuf again for all the help over the last few days (and for libgpod itself!), Nathaniel McCallum for the awesome start to the .NET bindings for libgpod, Alex Launi for the banshee gio/udev work which saved me a lot of time and also for sparing the time yesterday to talk about how I can get my work upstreamed.

If I've forgotten anyone or anything, sorry! But it's been a hectic week and it was hard enough keeping track of all the packages and patches I had floating around, never mind everyone I was talking to :)

I'll post again when I get everything upstreamed and I'll have a nice set of instructions for anyone wishing to test out their iDevice with banshee.

Hackweek Day III - The coffee shortage

Today was a slow but interesting day (again). I took all my banshee work and rebased it on top of lamalexs work to bring udev/gio support to banshee. This saved me having to write my own backend, but it meant that once again I had to start upgrading core parts of my OS. Luckily nothing blew up this time!

So after a few hours hacking and pulling my hair out because seemingly simple commands were failing (they still are failing, i just worked around it) I now have proper detection implemented. There are no more hardcoded horrible hacks :) Anyone with the right pre-requisites and the udev branch of banshee with my patch can have basic iDevice support (tracks only).

This is awesome!

This will not be pushed to the main banshee repository as it depends on udev/gio to be useful. I will instead push my patches upstream to lamalex and his udev branch. All that has to be done then is to push the udev work to mainline banshee and everyone can enjoy full iDevice support.

Tomorrow I hope to get in touch with lamalex and iron out a few bugs I've experienced and then upstream my work to him. My bugfixes to the libgpod-sharp bindings have already been upstreamed and should be merged with the main codebase very soon.

Hackweek Day II - Attack of the code

Today has been reasonably productive. I found some bugs in the libgpod bindings which were fairly major and implemented some more features in banshee.

First the fun bug. The libgpod bindings did the normal thing of defining a managed struct which mirrored the native struct that libgpod uses. This means we can do a trivial byte copy of the unmanaged memory into one of our managed structs and trivially access the information in a safe way. This is all well and good right up until you realise that when you update a property on that struct, you are not actually changing the value of the unmanaged memory. Essentially you have two copies of the same data which are disconnected - one in managed memory and one in native memory. Updating one copy is not reflected in the other and there's no easy way to keep them synced without the risk of losing data.

After a bit of brainstorming with Jeremie, we came up with a rather neat and nifty solution. Take this struct as an example:

struct NativeDate {
    public int Day;
    public int Month;
    public int Year;
}

If native code allocates one of those and passes it to .NET as an IntPtr, there is a trivial way to map this to managed code without requiring a copy of the data.

unsafe class Date {
    IntPtr Native {
        get; set;
    }

    public int Day {
        get { return ((NativeDate *)Native)->Day; }
        set { ((NativeDate *) Native)->Day = value; }
    }
    public int Month {
        get { return ((NativeDate *) Native)->Month; }
        set { ((NativeDate *) Native)->Month = value; }
    }

    public int Year {
        get { return ((NativeDate *) Native)->Year; }
        set { ((NativeDate *) Native)->Year = value; }
    }

    public Date (IntPtr native)
    {
        Native = native;
    }
}

All we do is use a bit of unsafe code to take the pointer that we got from native code and cast it to a NativeStruct* and read or write the data as appropriate. Now both managed land and unmanaged land are working off the same hunk of memory so they can never get out of sync.

Now for the features! Banshee can now sync music to my iPhone and remove it aswell! This means all the basic stuff that's required for basic syncing is supported now. It took a lot longer than it should've to get this working because my iPhone wasn't actually set up right. There was a command (i can't remember the name of it now) which should've been run automatically to populate iTunes_Control/Device with some required information. This was never run for some bizarre reason (yay for bleeding edge) which meant nothing I did could ever actually update the database.

However once I got that setup, things progressed pretty rapidly. There's one major feature left which is automatic detection when the device is mounted. If i can get this done tomorrow i'll try to upstream everything and get people testing this by the weekend (or friday!).

Hackweek V - Day 1

So at the end of Day 1, things have progressed well enough. I managed to get all my requirements installed (just about!) though to do it I needed to install older versions of some of the ipod stack. Hopefully this won't bite me in the ass as the week progresses ;)

So a brief synopsis of the state of the world as I found it after one day:

1) Things are still very bleeding edge. On opensuse you still need just the right packages and svn checkouts to even begin to get things working. Nothing you need is in the standard repositories, I had to use the awesome opensuse build service to provide some packages (gvfs >= 1.5.1, libplist, umuxd) and had to compile just the right version of libimobiledevice manually. Not a pleasant experience.

2) Once all that horribleness is done, actually detecting the ipod in banshee is difficult. HAL is useless. You need either gio# or udev to be able to detect the device when it's plugged in. This puts further requirements on your distro to ship the latest goodies. I worked around this by hardcoding my iphones mount point for now ;)

3) Banshee is good to work with. Integrating DAPs with banshee is easier than it used to be. Hopefully things will simplify the more I delve into things and learn what's available. Some things in the existing code I hope are only there for historical reasons and can be removed with this rewrite ;)

4) So far I have two big checkpoints: Banshee can load the tracks on my iphone and it can also import them to its collection. I'm in the process of allowing you to copy tracks to the device, but I hit a few issues with libgpod which have yet to be investigated.

So things are looking good for the first day. Hopefully day 2 will be full of interesting developments. Maybe I'll get playlist syncronisation working, maybe I'll get track uploading working. Who knows!

Hackweek V

Next week is Hackweek V in Novell, the week in which we can work on whatever we want. This year, my hackweek proposal is to give banshee a new ipod addin based on libgpod. This should fix some of banshees bugs.

While working on this I'll try live tweet as I hack so if you're interested, do tune in from tuesday morning onwards (GMT). Mondays a bank holiday so I'm going to have to complete my hack in 4 days.

If you can, do give the proposal a +1. I've no idea what affect it'll have but do it anyway! ;)

EDIT: Link the correct hackweek proposal. Whoopsie :)

Performance benchmarks... or lack thereof

A friend of mine was over the other day and I was showing him this cool website I had recently heard of, grooveshark. So after admiring it for a little while he came out with "pity it's written in flash". So of course I asked the obvious question "Why?" and got the obvious answer "Because it's inefficient and sucking up all your CPU". He popped open 'top' and sure enough it was using 35% of my cpu, his point was proven.

However, that is completely and utterly irrelevant. This does not make flash a bad technology nor does it make grooveshark a badly written application. Maybe 35% cpu usage is just the cost of running in the browser. It's like the old complaint against firefox, "Firefox uses too much memory". By itself that means absolutely nothing. It's nonsense. For that to have any real meaning you'd have to compare firefox to another browser doing the same task. A legitimate complaint might be "Firefox uses 3 times more memory than opera when I view www.example.com, 300 mb versus 100mb". An arbitrary decision as to what amounts to be "too much" is useless without a reference. This is exactly what far too many people are doing to flash.

Sure, grooveshark uses more CPU and memory than banshee, but banshee isn't running inside firefox. It annoys me that people make such arbitrary decisions and try to pass off as fact that flash is inherently bad and HTML5 is inherently so much better and yet have no reference to compare against. I haven't seen anything even close to the complexity of grooveshark written using html5. So before you get on your high horse decrying all plugins, show me a comparable application which uses $COMPETING_TECHNOLOGY_OF_CHOICE and performs measurably better.

Zencomic - increasing happiness since nineteen digity two

Today I had some spare time and decided to hack together a quick addin for Zencomic. I choose garfield minus garfield as I feel it reflects my life pretty closely in some aspects (not the manic depressive aspects though ;) ).

One of the downsides of working with a distributed team is that everyone starts work at a different time. On some of the quieter days when people are away on holidays, I'm sometimes the only hacker actively working on Moonlight in the wee small hours of 9am UTC. By the time the Americans wake up, I've sometimes left a nice little monologue in the IRC channel detailing my frustrations and successes during the morning.

Anyone, now for the obligatory screenshot


The addin was hacked together over the course of 30 minutes. It should be able to open any garfield-garfield comic, but if you find a particular one it fails on do let me know. Also, please don't suggest I switch my beautiful string splitting code into a Regex unless you plan on writing it yourself ;)

UPDATE: I just fixed the last known issue. The addin now detects exactly how many pages of g-g goodness there are rather than being hardcoded to assume 101 pages (the current number).

hapy birfday 2 me

My confusion this morning was evident

FOSDEM 2010

So Ruben has been working on getting a bunch of talks organised for the Mono room in FOSDEM and I've been lucky enough to get a slot. I have a 30 minute slot, so I figured that would be long enough to talk about 2 things properly and then take questions.

I've decided that I want to talk about two things.

1. The piece picker algorithm
2. How i handle threading

The piece picking code handles which segment of the file will be downloaded next. It has gone through several iterations, each time adding new features. Eventaully the whole class became so hard to maintain and test that I just had to rewrite things from the ground up using a new approach. This might give some people ideas on how to tackle problems in their own projects but even if it doesn't, it should still be interesting ;)

MonoTorrent handles a lot of concurrent connections. Using one thread per connection just wouldn't scale yet using plain asynchronous sockets and locking has huge potential for deadlocks and race conditions. Getting this right took well over a year - probably due to my lack of experience at this - but right now I think I've hit the winning threading solution. This part of the talk should be useful to anyone who's worked on an app which does a lot of socket operations. Maybe I'll wake up in a few months time and see a dozen apps using that technique.

Anyway, feel free to drop in for my talk (and all the others!). If you have any questions about any other aspects of MonoTorrent, or programming in general, feel free to find me and ask.