Moonlight 4.0 SDK... it's nearly there!

Three delicious screenshots.

1) Writing some code.


2) Launching from MonoDevelop:


3) Debugging!


4) Profit?

This should hopefully be available with the next MonoDevelop release :)

Mono.Nat 1.1.0

Quite a long time ago support for NAT-PMP devices was contributed to Mono.Nat. As I didn't own a NAT-PMP capable device, I had to trust that the code was good at it worked. From all reports, it did under most circumstances. However it had a couple of issues which prevented it from being enabled by default. Patches to fix these issues have finally been contributed and have now been merged into Mono.Nat and released!

If you want to forward ports from your application via your upnp or nat-pmp capable routers, look no further, Mono.Nat is here! There are also packages available from the opensuse build service project I have.

If you have any issues, submit a bug report and I'll get on them!

A picture is worth 1000 words

Two to three weeks ago if you ran this battery of garbage collection related tests in moonlight, they'd mostly be red. How things have changed :)

Writing a profiler for mono

As some of you may know, my day job is working on Moonlight, the FOSS implementation of Silverlight. This is a fairly complex piece of code as we have C++ interoperating with C# and Javascript, so managing the lifecycle of objects can be difficult. C++ refcounts, C# has an automatic garbage collector and Javascript... well, let's just say that it complicates things a *lot* when combined with the other two languages.

One of the tricks we use to ensure that objects do not die early is to use a (normal) GCHandle on our C# objects to ensure the garbage collector doesn't mark them as junk before they're definitely finished with in C++. However, we were having issues where some of our objects appeared to never ever get garbage collected even though they should be eligible. We suspected that excess GCHandles were causing it but we had no easy way of tracking this. Without detailed information on when and where the GCHandles were allocated, it would be next to impossible to track down where we were going wrong. After several aborted attempts to track this with Console.WriteLine I figured there must be a better way.

How about a custom profiler for mono?

Documentation on this is fairly sparse currently, so hopefully this will help. Thanks go to Paolo Molaro for helping me when I got stuck! This code is mostly copied/pasted from moonlight so it's C++ ish and also may contain some moonlight specific code.

Step 1:
Open up the "mono/metadata/profiler.h" file and check out all the profiler hooks that are available. They are all documented in profiler.c, so open that if you want a description of how they all work. Take note of the ones you'll need. All I needed to do was track GCHandle allocations, so I just wanted this hook:

void mono_profiler_install_gc_roots    (MonoProfileGCHandleFunc handle_callback, MonoProfileGCRootFunc roots_callback);

Step 2:
I now know what function I need to provide and the data I want to gather, so it's time to create a struct to store my data. This is an opaque struct which will be passed to the mono runtime and passed back in every profiler hook, so you can stick whatever you want in it. I know I want to store stacktraces, allocated gchandles and a typename, so I'll start with just those. I'll also declare my prototype for my profiler hook here.

struct _MonoProfiler {
const char *type_name;
GPtrArray *gchandles;
GPtrArray *stacktraces;
Moonlight::Mutex locker; /* used to ensure only one thread accesses the arrays */

static void track_gchandle (_MonoProfiler *prof, int op, int type, uintptr_t handle, MonoObject *obj);
};

typedef _MonoProfiler MonoProfiler;

Step 3:
Implementation time! First the constructor for the MonoProfiler. I use an environment variable to limit which types we store stacktraces for as they're expensive to generate and consume a lot of memory to store when you have 10,000s of them. The important functions are the last three. The first registers our profiler with the runtime. The second installs the hook so we can get notifications for gchandle allocations. The final call enables the gc_handle events in the profiler which makes the runtime invoke the hook we just registered.

_MonoProfiler::_MonoProfiler ()
{
   type_name = g_getenv ("GCHANDLES_FOR_TYPE");

   gchandles = g_ptr_array_new ();
   stacktraces = g_ptr_array_new_with_free_func (g_free);
   /* Register the profiler with mono */
   mono_profiler_install (this, NULL);
   /* Supply a function for the gc_roots hook */
   mono_profiler_install_gc_roots (track_gchandle, NULL);
   /* Enable gc_roots tracking in the profiler so our hook is invoked */
   mono_profiler_set_events (MONO_PROFILE_GC_ROOTS);
}

Now all I need is the implementation for track_gchandle and I'm done!

void
MonoProfiler::track_gchandle (MonoProfiler *prof, int op, int type, uintptr_t handle, MonoObject *obj)
{
   // Ignore anything that isn't a strong GC handle (docs say type == 2 is a strong gchandle)
   if (type != 2)
       return;

   prof->locker.Lock ();

   GPtrArray *gchandles = prof->gchandles;
   GPtrArray *stacktraces = prof->stacktraces;

   if (op == MONO_PROFILER_GC_HANDLE_CREATED) {
       // Add the GCHandle to this array
       g_ptr_array_add (gchandles, (gpointer) handle);
       // If the target of the gchandle is of the correct type, store its stack trace
       // Otherwise store NULL so that we can keep the index of the gchandle and corresponding
       // stack trace in sync.
       if (prof->type_name && !strcmp (prof->type_name, mono_class_get_name (mono_object_get_class(obj))))
           g_ptr_array_add (stacktraces, get_stack_trace ());
       else
           g_ptr_array_add (stacktraces, NULL);
   } else if (op == MONO_PROFILER_GC_HANDLE_DESTROYED) {
       // Walk our list of gchandles and when we find the index of the destroyed handle
       // remove the handle and corresponding stacktrace
       for (int i = 0; i < (int)gchandles->len; i++) {
           if (g_ptr_array_index (gchandles, i) == (gpointer) handle) {
               g_ptr_array_remove_index_fast (gchandles, i);
               g_ptr_array_remove_index_fast (stacktraces, i);
               break;
           }
       }
   }

   prof->locker.Unlock ();
}

Step 4:
Profit! When you're ready, you can iterate over the gchandle array and pull out whatever statistics you want. For example here is the code I use to iterate the gchandles array and work out how many instances of each object type there are.

void
accumulate_g_ptr_array_by_type (gpointer data, gpointer user_data)
{
   // The hashtable I passed in as the user_data in g_ptr_array_foreach
   // which i am using to link a type name to a count
   GHashTable *by_type = (GHashTable*) user_data;

   // Get the MonoObject from the gchandle.
   MonoObject *ob = mono_gchandle_get_target (GPOINTER_TO_INT (data));

   // Get the type name from the mono_object
   const char *name = mono_class_get_name (mono_object_get_class(ob));

   // Find out how many instances we have already
   int count = GPOINTER_TO_INT (g_hash_table_lookup (by_type, name)) + 1;

   // Update the hashtable with an incremented count.
   g_hash_table_insert (by_type, name, GINT_TO_POINTER (count));
}

I run that hashtable through another function to order to sort the list by the number of each type of object allocated so I can have a nicely formatted list as follows:

1 instances GCHandled of type Surface
1 instances GCHandled of type Deployment
4 instances GCHandled of type NameScope
20 instances GCHandled of type ControlTemplate
218 instances GCHandled of type MonoType
3985 instances GCHandled of type Uri

Needless to say, it's now immediately obvious to me that Uris are being GChandled and never freed, so i just have to enable tracing of their stacktraces and I can see exactly where we allocated the handle and from there figure out why we haven't freed it.

MonoTorrent Lives! (on github)

The mono project, who've been hosting MonoTorrent since day 1, recently decided to host all their code at github. As such, MonoTorrent now lives in github! I took this opportunity to rename the module from 'bitsharp' to 'monotorrent' to avoid confusing people.

So what does this mean? Well, it means it's trivial for you to commit patches now! All you do is set up a github account, fork monotorrent, commit your changes to your fork and issue a pull request. I'll then automagically be informed of your changes and can click a button to either merge them directly into monotorrent or reject them (with an explanation). No more emailing patches or attaching them to forum posts. Pretty nifty, eh?

So get forking and lets see how great we can make MonoTorrent!

Hackwee Day IV - The final showdown

Yesterday was a bit of an anti-climax codewise. Pretty much everything I had planned on doing I had completed by midday. I then had a nice long chat with lamalex about what I had to do to get ipod support into his udev branch and hashed out a rough idea about how to upstream these changes. I believe he's going to do a bit of work over the next week or so and once that's complete I'll be able to upstream the remainder of my patches.

So at the end of the 4 days the state of things is:

Device support:
1) Hotplugging works
2) Syncing music to and from the device works
3) Removing music from the device works
4) Basic properties can be read about the device and displayed in banshee
5) Playlists aren't supported yet but should be fairly simple to do

Code written:
1) I've upstreamed my patches for libgpod-sharp
2) I spoke with lamalex about upstreaming the rest of my work there.
3) There are now packages available for openSUSE 11.2 courtesy of FunkyM which can be used to provide most of the required packages.
4) There are still some bleeding edge libraries required which aren't packaged anywhere. These will have to be packaged.
5) I think the best way to support the libgpod based iDevice would be to create a new addin so that you can run both the old addin and the new addin in parallel as some of the required libraries for libgpod based iDevice support are very new and not widely available as of yet. This will take a little bit of time.

I'm off on holidays next week so when I get back I'll start pushing my remaining work upstream. Once that's done it's just the boring process of streamlining the build to remove the dozen manual steps which are currently required :) All in all, it's been a successful hackweek. But as with all good hacks, the 5:1 rule applies. For every day spent adding functionality, I'll need 5 to make it stable and usable for everyone.

I have to thank teuf again for all the help over the last few days (and for libgpod itself!), Nathaniel McCallum for the awesome start to the .NET bindings for libgpod, Alex Launi for the banshee gio/udev work which saved me a lot of time and also for sparing the time yesterday to talk about how I can get my work upstreamed.

If I've forgotten anyone or anything, sorry! But it's been a hectic week and it was hard enough keeping track of all the packages and patches I had floating around, never mind everyone I was talking to :)

I'll post again when I get everything upstreamed and I'll have a nice set of instructions for anyone wishing to test out their iDevice with banshee.

Hackweek Day III - The coffee shortage

Today was a slow but interesting day (again). I took all my banshee work and rebased it on top of lamalexs work to bring udev/gio support to banshee. This saved me having to write my own backend, but it meant that once again I had to start upgrading core parts of my OS. Luckily nothing blew up this time!

So after a few hours hacking and pulling my hair out because seemingly simple commands were failing (they still are failing, i just worked around it) I now have proper detection implemented. There are no more hardcoded horrible hacks :) Anyone with the right pre-requisites and the udev branch of banshee with my patch can have basic iDevice support (tracks only).

This is awesome!

This will not be pushed to the main banshee repository as it depends on udev/gio to be useful. I will instead push my patches upstream to lamalex and his udev branch. All that has to be done then is to push the udev work to mainline banshee and everyone can enjoy full iDevice support.

Tomorrow I hope to get in touch with lamalex and iron out a few bugs I've experienced and then upstream my work to him. My bugfixes to the libgpod-sharp bindings have already been upstreamed and should be merged with the main codebase very soon.