MonoTorrent

Moonlight 4.0 SDK... it's nearly there!

2011-04-19T16:44:00.003+01:00

Three delicious screenshots.

1) Writing some code.

2) Launching from MonoDevelop:

3) Debugging!

4) Profit?

This should hopefully be available with the next MonoDevelop release :)

Mono.Nat 1.1.0

2011-03-04T11:13:00.003Z

Quite a long time ago support for NAT-PMP devices was contributed to Mono.Nat. As I didn't own a NAT-PMP capable device, I had to trust that the code was good at it worked. From all reports, it did under most circumstances. However it had a couple of issues which prevented it from being enabled by default. Patches to fix these issues have finally been contributed and have now been merged into Mono.Nat and released!

If you want to forward ports from your application via your upnp or nat-pmp capable routers, look no further, Mono.Nat is here! There are also packages available from the opensuse build service project I have.

If you have any issues, submit a bug report and I'll get on them!

A picture is worth 1000 words

2010-12-17T14:23:00.005Z

Two to three weeks ago if you ran this battery of garbage collection related tests in moonlight, they'd mostly be red. How things have changed :)

Writing a profiler for mono

2010-12-06T11:49:00.006Z

As some of you may know, my day job is working on Moonlight, the FOSS implementation of Silverlight. This is a fairly complex piece of code as we have C++ interoperating with C# and Javascript, so managing the lifecycle of objects can be difficult. C++ refcounts, C# has an automatic garbage collector and Javascript... well, let's just say that it complicates things a *lot* when combined with the other two languages.

One of the tricks we use to ensure that objects do not die early is to use a (normal) GCHandle on our C# objects to ensure the garbage collector doesn't mark them as junk before they're definitely finished with in C++. However, we were having issues where some of our objects appeared to never ever get garbage collected even though they should be eligible. We suspected that excess GCHandles were causing it but we had no easy way of tracking this. Without detailed information on when and where the GCHandles were allocated, it would be next to impossible to track down where we were going wrong. After several aborted attempts to track this with Console.WriteLine I figured there must be a better way.

How about a custom profiler for mono?

Documentation on this is fairly sparse currently, so hopefully this will help. Thanks go to Paolo Molaro for helping me when I got stuck! This code is mostly copied/pasted from moonlight so it's C++ ish and also may contain some moonlight specific code.

Step 1:
Open up the "mono/metadata/profiler.h" file and check out all the profiler hooks that are available. They are all documented in profiler.c, so open that if you want a description of how they all work. Take note of the ones you'll need. All I needed to do was track GCHandle allocations, so I just wanted this hook:

void mono_profiler_install_gc_roots    (MonoProfileGCHandleFunc handle_callback, MonoProfileGCRootFunc roots_callback);

Step 2:
I now know what function I need to provide and the data I want to gather, so it's time to create a struct to store my data. This is an opaque struct which will be passed to the mono runtime and passed back in every profiler hook, so you can stick whatever you want in it. I know I want to store stacktraces, allocated gchandles and a typename, so I'll start with just those. I'll also declare my prototype for my profiler hook here.

struct _MonoProfiler {
const char *type_name;
GPtrArray *gchandles;
GPtrArray *stacktraces;
Moonlight::Mutex locker; /* used to ensure only one thread accesses the arrays */

static void track_gchandle (_MonoProfiler *prof, int op, int type, uintptr_t handle, MonoObject *obj);
};

typedef _MonoProfiler MonoProfiler;

Step 3:
Implementation time! First the constructor for the MonoProfiler. I use an environment variable to limit which types we store stacktraces for as they're expensive to generate and consume a lot of memory to store when you have 10,000s of them. The important functions are the last three. The first registers our profiler with the runtime. The second installs the hook so we can get notifications for gchandle allocations. The final call enables the gc_handle events in the profiler which makes the runtime invoke the hook we just registered.

_MonoProfiler::_MonoProfiler ()
{
   type_name = g_getenv ("GCHANDLES_FOR_TYPE");

   gchandles = g_ptr_array_new ();
   stacktraces = g_ptr_array_new_with_free_func (g_free);
   /* Register the profiler with mono */
   mono_profiler_install (this, NULL);
   /* Supply a function for the gc_roots hook */
   mono_profiler_install_gc_roots (track_gchandle, NULL);
   /* Enable gc_roots tracking in the profiler so our hook is invoked */
   mono_profiler_set_events (MONO_PROFILE_GC_ROOTS);
}

Now all I need is the implementation for track_gchandle and I'm done!

void
MonoProfiler::track_gchandle (MonoProfiler *prof, int op, int type, uintptr_t handle, MonoObject *obj)
{
   // Ignore anything that isn't a strong GC handle (docs say type == 2 is a strong gchandle)
   if (type != 2)
       return;

   prof->locker.Lock ();

   GPtrArray *gchandles = prof->gchandles;
   GPtrArray *stacktraces = prof->stacktraces;

   if (op == MONO_PROFILER_GC_HANDLE_CREATED) {
       // Add the GCHandle to this array
       g_ptr_array_add (gchandles, (gpointer) handle);
       // If the target of the gchandle is of the correct type, store its stack trace
       // Otherwise store NULL so that we can keep the index of the gchandle and corresponding
       // stack trace in sync.
       if (prof->type_name && !strcmp (prof->type_name, mono_class_get_name (mono_object_get_class(obj))))
           g_ptr_array_add (stacktraces, get_stack_trace ());
       else
           g_ptr_array_add (stacktraces, NULL);
   } else if (op == MONO_PROFILER_GC_HANDLE_DESTROYED) {
       // Walk our list of gchandles and when we find the index of the destroyed handle
       // remove the handle and corresponding stacktrace
       for (int i = 0; i < (int)gchandles->len; i++) {
           if (g_ptr_array_index (gchandles, i) == (gpointer) handle) {
               g_ptr_array_remove_index_fast (gchandles, i);
               g_ptr_array_remove_index_fast (stacktraces, i);
               break;
           }
       }
   }

   prof->locker.Unlock ();
}

Step 4:
Profit! When you're ready, you can iterate over the gchandle array and pull out whatever statistics you want. For example here is the code I use to iterate the gchandles array and work out how many instances of each object type there are.

void
accumulate_g_ptr_array_by_type (gpointer data, gpointer user_data)
{
   // The hashtable I passed in as the user_data in g_ptr_array_foreach
   // which i am using to link a type name to a count
   GHashTable *by_type = (GHashTable*) user_data;

   // Get the MonoObject from the gchandle.
   MonoObject *ob = mono_gchandle_get_target (GPOINTER_TO_INT (data));

   // Get the type name from the mono_object
   const char *name = mono_class_get_name (mono_object_get_class(ob));

   // Find out how many instances we have already
   int count = GPOINTER_TO_INT (g_hash_table_lookup (by_type, name)) + 1;

   // Update the hashtable with an incremented count.
   g_hash_table_insert (by_type, name, GINT_TO_POINTER (count));
}

I run that hashtable through another function to order to sort the list by the number of each type of object allocated so I can have a nicely formatted list as follows:

1 instances GCHandled of type Surface
1 instances GCHandled of type Deployment
4 instances GCHandled of type NameScope
20 instances GCHandled of type ControlTemplate
218 instances GCHandled of type MonoType
3985 instances GCHandled of type Uri

Needless to say, it's now immediately obvious to me that Uris are being GChandled and never freed, so i just have to enable tracing of their stacktraces and I can see exactly where we allocated the handle and from there figure out why we haven't freed it.

MonoTorrent Lives! (on github)

2010-10-02T19:38:00.004+01:00

The mono project, who've been hosting MonoTorrent since day 1, recently decided to host all their code at github. As such, MonoTorrent now lives in github! I took this opportunity to rename the module from 'bitsharp' to 'monotorrent' to avoid confusing people.

So what does this mean? Well, it means it's trivial for you to commit patches now! All you do is set up a github account, fork monotorrent, commit your changes to your fork and issue a pull request. I'll then automagically be informed of your changes and can click a button to either merge them directly into monotorrent or reject them (with an explanation). No more emailing patches or attaching them to forum posts. Pretty nifty, eh?

So get forking and lets see how great we can make MonoTorrent!

Hackwee Day IV - The final showdown

2010-06-12T14:44:00.002+01:00

Yesterday was a bit of an anti-climax codewise. Pretty much everything I had planned on doing I had completed by midday. I then had a nice long chat with lamalex about what I had to do to get ipod support into his udev branch and hashed out a rough idea about how to upstream these changes. I believe he's going to do a bit of work over the next week or so and once that's complete I'll be able to upstream the remainder of my patches.

So at the end of the 4 days the state of things is:

Device support:
1) Hotplugging works
2) Syncing music to and from the device works
3) Removing music from the device works
4) Basic properties can be read about the device and displayed in banshee
5) Playlists aren't supported yet but should be fairly simple to do

Code written:
1) I've upstreamed my patches for libgpod-sharp
2) I spoke with lamalex about upstreaming the rest of my work there.
3) There are now packages available for openSUSE 11.2 courtesy of FunkyM which can be used to provide most of the required packages.
4) There are still some bleeding edge libraries required which aren't packaged anywhere. These will have to be packaged.
5) I think the best way to support the libgpod based iDevice would be to create a new addin so that you can run both the old addin and the new addin in parallel as some of the required libraries for libgpod based iDevice support are very new and not widely available as of yet. This will take a little bit of time.

I'm off on holidays next week so when I get back I'll start pushing my remaining work upstream. Once that's done it's just the boring process of streamlining the build to remove the dozen manual steps which are currently required :) All in all, it's been a successful hackweek. But as with all good hacks, the 5:1 rule applies. For every day spent adding functionality, I'll need 5 to make it stable and usable for everyone.

I have to thank teuf again for all the help over the last few days (and for libgpod itself!), Nathaniel McCallum for the awesome start to the .NET bindings for libgpod, Alex Launi for the banshee gio/udev work which saved me a lot of time and also for sparing the time yesterday to talk about how I can get my work upstreamed.

If I've forgotten anyone or anything, sorry! But it's been a hectic week and it was hard enough keeping track of all the packages and patches I had floating around, never mind everyone I was talking to :)

I'll post again when I get everything upstreamed and I'll have a nice set of instructions for anyone wishing to test out their iDevice with banshee.

Hackweek Day III - The coffee shortage

2010-06-10T22:25:00.002+01:00

Today was a slow but interesting day (again). I took all my banshee work and rebased it on top of lamalexs work to bring udev/gio support to banshee. This saved me having to write my own backend, but it meant that once again I had to start upgrading core parts of my OS. Luckily nothing blew up this time!

So after a few hours hacking and pulling my hair out because seemingly simple commands were failing (they still are failing, i just worked around it) I now have proper detection implemented. There are no more hardcoded horrible hacks :) Anyone with the right pre-requisites and the udev branch of banshee with my patch can have basic iDevice support (tracks only).

This is awesome!

This will not be pushed to the main banshee repository as it depends on udev/gio to be useful. I will instead push my patches upstream to lamalex and his udev branch. All that has to be done then is to push the udev work to mainline banshee and everyone can enjoy full iDevice support.

Tomorrow I hope to get in touch with lamalex and iron out a few bugs I've experienced and then upstream my work to him. My bugfixes to the libgpod-sharp bindings have already been upstreamed and should be merged with the main codebase very soon.

Hackweek Day II - Attack of the code

2010-06-09T22:06:00.003+01:00

Today has been reasonably productive. I found some bugs in the libgpod bindings which were fairly major and implemented some more features in banshee.

First the fun bug. The libgpod bindings did the normal thing of defining a managed struct which mirrored the native struct that libgpod uses. This means we can do a trivial byte copy of the unmanaged memory into one of our managed structs and trivially access the information in a safe way. This is all well and good right up until you realise that when you update a property on that struct, you are not actually changing the value of the unmanaged memory. Essentially you have two copies of the same data which are disconnected - one in managed memory and one in native memory. Updating one copy is not reflected in the other and there's no easy way to keep them synced without the risk of losing data.

After a bit of brainstorming with Jeremie, we came up with a rather neat and nifty solution. Take this struct as an example:

struct NativeDate {
    public int Day;
    public int Month;
    public int Year;
}

If native code allocates one of those and passes it to .NET as an IntPtr, there is a trivial way to map this to managed code without requiring a copy of the data.

unsafe class Date {
    IntPtr Native {
        get; set;
    }

    public int Day {
        get { return ((NativeDate *)Native)->Day; }
        set { ((NativeDate *) Native)->Day = value; }
    }
    public int Month {
        get { return ((NativeDate *) Native)->Month; }
        set { ((NativeDate *) Native)->Month = value; }
    }

    public int Year {
        get { return ((NativeDate *) Native)->Year; }
        set { ((NativeDate *) Native)->Year = value; }
    }

    public Date (IntPtr native)
    {
        Native = native;
    }
}

All we do is use a bit of unsafe code to take the pointer that we got from native code and cast it to a NativeStruct* and read or write the data as appropriate. Now both managed land and unmanaged land are working off the same hunk of memory so they can never get out of sync.

Now for the features! Banshee can now sync music to my iPhone and remove it aswell! This means all the basic stuff that's required for basic syncing is supported now. It took a lot longer than it should've to get this working because my iPhone wasn't actually set up right. There was a command (i can't remember the name of it now) which should've been run automatically to populate iTunes_Control/Device with some required information. This was never run for some bizarre reason (yay for bleeding edge) which meant nothing I did could ever actually update the database.

However once I got that setup, things progressed pretty rapidly. There's one major feature left which is automatic detection when the device is mounted. If i can get this done tomorrow i'll try to upstream everything and get people testing this by the weekend (or friday!).

Hackweek V - Day 1

2010-06-08T23:03:00.002+01:00

So at the end of Day 1, things have progressed well enough. I managed to get all my requirements installed (just about!) though to do it I needed to install older versions of some of the ipod stack. Hopefully this won't bite me in the ass as the week progresses ;)

So a brief synopsis of the state of the world as I found it after one day:

1) Things are still very bleeding edge. On opensuse you still need just the right packages and svn checkouts to even begin to get things working. Nothing you need is in the standard repositories, I had to use the awesome opensuse build service to provide some packages (gvfs >= 1.5.1, libplist, umuxd) and had to compile just the right version of libimobiledevice manually. Not a pleasant experience.

2) Once all that horribleness is done, actually detecting the ipod in banshee is difficult. HAL is useless. You need either gio# or udev to be able to detect the device when it's plugged in. This puts further requirements on your distro to ship the latest goodies. I worked around this by hardcoding my iphones mount point for now ;)

3) Banshee is good to work with. Integrating DAPs with banshee is easier than it used to be. Hopefully things will simplify the more I delve into things and learn what's available. Some things in the existing code I hope are only there for historical reasons and can be removed with this rewrite ;)

4) So far I have two big checkpoints: Banshee can load the tracks on my iphone and it can also import them to its collection. I'm in the process of allowing you to copy tracks to the device, but I hit a few issues with libgpod which have yet to be investigated.

So things are looking good for the first day. Hopefully day 2 will be full of interesting developments. Maybe I'll get playlist syncronisation working, maybe I'll get track uploading working. Who knows!

Hackweek V

2010-06-04T17:32:00.004+01:00

Next week is Hackweek V in Novell, the week in which we can work on whatever we want. This year, my hackweek proposal is to give banshee a new ipod addin based on libgpod. This should fix some of banshees bugs.

While working on this I'll try live tweet as I hack so if you're interested, do tune in from tuesday morning onwards (GMT). Mondays a bank holiday so I'm going to have to complete my hack in 4 days.

If you can, do give the proposal a +1. I've no idea what affect it'll have but do it anyway! ;)

EDIT: Link the correct hackweek proposal. Whoopsie :)

Performance benchmarks... or lack thereof

2010-05-17T23:02:00.002+01:00

A friend of mine was over the other day and I was showing him this cool website I had recently heard of, grooveshark. So after admiring it for a little while he came out with "pity it's written in flash". So of course I asked the obvious question "Why?" and got the obvious answer "Because it's inefficient and sucking up all your CPU". He popped open 'top' and sure enough it was using 35% of my cpu, his point was proven.

However, that is completely and utterly irrelevant. This does not make flash a bad technology nor does it make grooveshark a badly written application. Maybe 35% cpu usage is just the cost of running in the browser. It's like the old complaint against firefox, "Firefox uses too much memory". By itself that means absolutely nothing. It's nonsense. For that to have any real meaning you'd have to compare firefox to another browser doing the same task. A legitimate complaint might be "Firefox uses 3 times more memory than opera when I view www.example.com, 300 mb versus 100mb". An arbitrary decision as to what amounts to be "too much" is useless without a reference. This is exactly what far too many people are doing to flash.

Sure, grooveshark uses more CPU and memory than banshee, but banshee isn't running inside firefox. It annoys me that people make such arbitrary decisions and try to pass off as fact that flash is inherently bad and HTML5 is inherently so much better and yet have no reference to compare against. I haven't seen anything even close to the complexity of grooveshark written using html5. So before you get on your high horse decrying all plugins, show me a comparable application which uses $COMPETING_TECHNOLOGY_OF_CHOICE and performs measurably better.

Zencomic - increasing happiness since nineteen digity two

2010-05-10T00:56:00.006+01:00

Today I had some spare time and decided to hack together a quick addin for Zencomic. I choose garfield minus garfield as I feel it reflects my life pretty closely in some aspects (not the manic depressive aspects though ;) ).

One of the downsides of working with a distributed team is that everyone starts work at a different time. On some of the quieter days when people are away on holidays, I'm sometimes the only hacker actively working on Moonlight in the wee small hours of 9am UTC. By the time the Americans wake up, I've sometimes left a nice little monologue in the IRC channel detailing my frustrations and successes during the morning.

Anyone, now for the obligatory screenshot

The addin was hacked together over the course of 30 minutes. It should be able to open any garfield-garfield comic, but if you find a particular one it fails on do let me know. Also, please don't suggest I switch my beautiful string splitting code into a Regex unless you plan on writing it yourself ;)

UPDATE: I just fixed the last known issue. The addin now detects exactly how many pages of g-g goodness there are rather than being hardcoded to assume 101 pages (the current number).

hapy birfday 2 me

2010-05-05T10:08:00.002+01:00

My confusion this morning was evident

FOSDEM 2010

2010-01-10T22:36:00.004Z

So Ruben has been working on getting a bunch of talks organised for the Mono room in FOSDEM and I've been lucky enough to get a slot. I have a 30 minute slot, so I figured that would be long enough to talk about 2 things properly and then take questions.

I've decided that I want to talk about two things.

The piece picker algorithm
How i handle threading

The piece picking code handles which segment of the file will be downloaded next. It has gone through several iterations, each time adding new features. Eventaully the whole class became so hard to maintain and test that I just had to rewrite things from the ground up using a new approach. This might give some people ideas on how to tackle problems in their own projects but even if it doesn't, it should still be interesting ;)

MonoTorrent handles a lot of concurrent connections. Using one thread per connection just wouldn't scale yet using plain asynchronous sockets and locking has huge potential for deadlocks and race conditions. Getting this right took well over a year - probably due to my lack of experience at this - but right now I think I've hit the winning threading solution. This part of the talk should be useful to anyone who's worked on an app which does a lot of socket operations. Maybe I'll wake up in a few months time and see a dozen apps using that technique.

Anyway, feel free to drop in for my talk (and all the others!). If you have any questions about any other aspects of MonoTorrent, or programming in general, feel free to find me and ask.

Expression Trees - serializing your data

2009-12-20T21:21:00.007Z

Update: Just to clarify - the code snippets below are under the MIT/X11 license.

I spent a few hours over the weekend writing a binary serializer using expression trees. I wanted to see how things would look using the new features available in .NET 4.0. My requirements were pretty simple:

1) Serialize all public properties in a type or a subset of them
2) Control the order in which they're serialized - sometimes you need to interop with an existing and you must write your data in a specific order
3) Control how a primitive is converted - Do you need to write value types in big endian, little endian, middle endian?
4) Easy to use API.

So lets start with the API. This is what I was hoping to use:

public class Secondary
{
public int First { get; set; }
public int Second { get; set; }
public int Third { get { return First + Second; } }
}

public class MyClass
{
public byte ByteProp { get; set; }
public short ShortProp { get; set; }
public int IntProp { get; set; }
public long LongProp { get; set; }
public string StringProp { get; set; }
}

static void Main(string[] args)
{
// Register a message so that all public fields will be serialized
Message.Register<MyClass>();

// Register a message so that only some fields are serialized and
// they are serialized in the specified order
Message.Register<Secondary>(
   d => d.Second,
   d => d.First
);

// Create a stream to serialize the data to
Stream s = new MemoryStream();
var message = new MyClass {
   IntProp = 1,
   LongProp= 2,
   ByteProp= 3,
   ShortProp = 4,
   StringProp = "Hello World"
};

// Encode the message to the stream
MessageEncoder.Encode(message, s);

// Rewind the stream and then decode the message
s.Position = 0;
var decoded = MessageDecoder.Decode<MyClass>(s);
}

It's pretty standard stuff. You can work with the standard serializer logic (serialize properties alphabetically) by registering an object without specifying any specific properties or you can customise which properties are serialized. This could also be done using attributes, but using attributes to control the order in which properties are serialized would be more error prone than the above.

Firstly, sometimes you need to write your data in big endian, others you need little endian. Sometimes you won't care. What you need is to be able to control this:

MessageEncoder.RegisterPrimitiveEncoder<int>((value, stream) => {
  stream.Write(BitConverter.GetBytes(value));
});

It's simple. Any type which can be directly converted to an array of bytes is classified as a 'primitive'. Each primitive can have an encoder/decoder pair registered as above.

public static class MessageEncoder
{
static Dictionary<Type, Delegate> encoders;
static Dictionary<Type, Delegate> primitives;

static MessageEncoder()
{
   encoders = new Dictionary<Type, Delegate>();
   primitives = new Dictionary<Type, Delegate>();
   RegisterPrimitiveEncoders();
}

static void RegisterPrimitiveEncoders()
{
   RegisterPrimitiveEncoder<byte>((value, stream) =>
       stream.WriteByte(value)
   );

   RegisterPrimitiveEncoder<short>((value, stream) =>
       stream.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(value)))
   );

   RegisterPrimitiveEncoder<int>((value, stream) =>
       stream.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(value)))
   );

   RegisterPrimitiveEncoder<long>((value, stream) =>
       stream.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(value)))
   );

   var intWriter = (Action<int, Stream>)primitives[typeof (int)];
   RegisterPrimitiveEncoder<string>((value, stream) => {
       var buffer = Encoding.UTF8.GetBytes(value);
       intWriter(buffer.Length, stream);
       stream.Write(buffer);
   });
}

public static void RegisterPrimitiveEncoder<T>(Action<T, Stream> encoder)
{
   primitives [typeof (T)] = encoder;
}

public static void RegisterMessage<T>(params Expression<Func<T, object>>[] properties)
{
   RegisterMessage<T>(properties.Select(p => p.AsPropertyInfo ()));
}

public static void RegisterMessage<T>(IEnumerable<PropertyInfo> properties)
{
   var propertyEncoders = new List<Expression>();

   // The encode function takes an instance of the class we're decoding and the Stream
   // which we should write the data to.
   ParameterExpression source = Expression.Parameter(typeof(T), "source_param");
   ParameterExpression stream = Expression.Parameter(typeof(Stream), "stream");

   // For each property, get the encoder which will convert the value of the property to a byte[]
   // which can be written to the stream.
   foreach (var property in properties) {
       // Get the encoder for this property type
       var action = primitives[property.PropertyType];
       // Create a var which holds the Action <T, Stream> which encodes the data to the stream
       Expression converter = Expression.Constant(action, action.GetType ());
       // Invoke the encoder passing the value of the property and the 'stream'
       Expression invoker = Expression.Invoke(converter, Expression.Property(source, property), stream);
       // Add the encoder for this property to the list.
       propertyEncoders.Add(invoker);
   }

   // Create an expression block which will execute each of the encoders one by one
   Expression block = Expression.Block(propertyEncoders);
   encoders.Add(typeof(T), Expression.Lambda<Action<T, Stream>>(
       block,
       source,
       stream
   ).Compile());
}

public static void Encode<T>(T message, Stream s)
{
   var encoder = (Action<T, Stream>)encoders[typeof (T)];
   encoder (message, s);
}
}

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Linq.Expressions;
using System.Reflection;
using System.Net;
using System.IO;

namespace Encoder
{
public static class MessageDecoder
{
   static Dictionary<Type, Delegate> decoders;
   static Dictionary<Type, Delegate> primitives;

   static MessageDecoder()
   {
       decoders = new Dictionary<Type, Delegate>();
       primitives = new Dictionary<Type, Delegate>();
       RegisterDefaultDecoders();
   }

   static void RegisterDefaultDecoders()
   {
       RegisterPrimitiveDecoder<byte>((s) => {
           var val = s.ReadByte();
           if (val == -1)
               throw new EndOfStreamException();
           return (byte)val;
       });

       RegisterPrimitiveDecoder<short>((s) => IPAddress.NetworkToHostOrder (s.ReadShort()));
       RegisterPrimitiveDecoder<int>(s => IPAddress.NetworkToHostOrder (s.ReadInt()));
       RegisterPrimitiveDecoder<long>(s => IPAddress.NetworkToHostOrder (s.ReadLong()));

       var intDecoder = (Func<Stream, int>)primitives[typeof(int)];
       RegisterPrimitiveDecoder<string>(s => {
           var length = intDecoder(s);
           var buffer = new byte[length];
           s.Read(buffer, 0, buffer.Length);
           return Encoding.UTF8.GetString(buffer);
       });
   }

   public static void RegisterPrimitiveDecoder<T>(Func<Stream, T> decoder)
   {
       primitives.Add(typeof(T), decoder);
   }

   public static void RegisterMessage<T>(params Expression<Func<T, object>>[] properties)
   {
       RegisterMessage<T>(properties.Select(d => d.AsPropertyInfo()));
   }

   public static void RegisterMessage<T>(IEnumerable<PropertyInfo> properties)
   {
       var propertyDecoders = new List<Expression>();

       // The decode function takes an instance of the class we're decoding and the Stream
       // containing the data to decode.
       ParameterExpression source = Expression.Parameter(typeof(T), "source_param");
       ParameterExpression stream = Expression.Parameter(typeof(Stream), "stream");

       // For each property, get the primitive decoder which will read data from the stream and
       // return a value of the correct type.
       foreach (var property in properties) {
           var action = primitives[property.PropertyType];
           // Create a var which holds the Func <Stream, T> which decodes the data from the stream
           Expression decoder = Expression.Constant(action, action.GetType());
           // Invoke the decoder passing 'stream' as the parameter
           Expression invoker = Expression.Invoke(decoder, stream);
           // Store the return value of the decoder in the property.
           Expression setter = Expression.Call(source, property.GetSetMethod(), invoker);
           // Add the decoder for this property to the list.
           propertyDecoders.Add(setter);
       }

       // Create a block which will execute the decoders for all the fields one after another.
       Expression block = Expression.Block(propertyDecoders);
       decoders.Add (typeof (T), Expression.Lambda<Action<T, Stream>>(
           block,
           source,
           stream
       ).Compile ());
   }

   public static T Decode<T>(Stream s) where T : class, new()
   {
       T t = new T();
       var decoder = (Action<T, Stream>)decoders[typeof(T)];
       decoder(t, s);
       return t;
   }
}
}

The idea is quite simple. For each class we can generate an ideal serializer using expression trees which doesn't require boxing or casting. This way we can avoid the use of reflection when serializing objects and so avoid the performance penalties incurred that. The code above only handles the simple case where a class consists of primitive types (int, long, string) , though it'd be easy enough to extend it to support more complex scenarios.

The serializer as you see it could not have been written with .NET 3.0. Some of the key components like BlockExpression were only introduced with .NET 4.0. If your object contains an array which needs to be serialized, you'll need the new IndexExpression too. Sure, it's possible to fake these using some anonymous delegates and Actions, but that's not pretty :)

The total implementation is less than 170 LOC. I'd be willing to bet that with another 100 LOC you could support most constructs. If you're currently a heavy user of reflection to provide object serialization, it's time to update ;)

New years resolutions

2009-12-15T00:13:00.002Z

It's tradition in quite a lot of countries to make a new years resolution on the 1st of January. Most people forget about them within a few days or weeks. This year, I'll be making one I'm going to keep!

I want to take part in a dancing [0] flash mob whether it's in this country or another.

What ideas do you have? Anything strange, interesting, unusual? Leave a comment and let me know, maybe you have a better idea than being part of a flash mob.

[0] Me and dancing don't get on particularly well, so it'll be an interesting challenge ;)

Yet another INotifyPropertyChanged with Expression Trees - Part 2

2009-12-06T13:33:00.010Z

In my last post, I described a method whereby you can implement INotifyPropertyChanged with zero performance overhead and near-zero boilerplate code. The only boilerplate left was the delegate you had to create to invoke the event:

public Book()
{
    // Boilerplate - eugh!
    Action<string> notify = (propertyName) => {
        var h = PropertyChanged;
        if (h != null)
            h(this, new PropertyChangedEventArgs(propertyName));
    };

    author = new ChangeNotifier<string> (() => Author, notify);
    price = new ChangeNotifier<decimal> (() => Price, notify);
    quantity = new ChangeNotifier<int> (() => Quantity, notify);
    title = new ChangeNotifier<string> (() => Title, notify);
}

The entire point of my implementation was to avoid writing boilerplate, so this was slightly irritating. Unfortunately, there's no trivial way around the problem as the .NET framework really limits what you can do with events. The first thing you'd think of is "pass the actual object into the ChangeNotifier constructor and just raise the event that way". For example my constructors would change to:

new ChangeNotifier<string>(() => Author, this);

That's well and good, right up until you realise that it's impossible for one object to raise an event that's declared on another object.

public class A
{
    public event EventHandler MyEvent;
}

public class B
{
    public void AccessEvent (A a)
    {
        // Invalid - you can't raise an event which is declared in another class
        a.MyEvent(this, EventArgs.Empty);

        // Invalid - you can't copy the event either
        EventHandler h = a.MyEvent;
        h(this, EventArgs.Empty);
    }
}

Another alternative would be to pass the event itself into the ChangeNotifier object:

new ChangeNotifier<string> (() => Author, PropertyChanged);

But this won't work because a copy of the delegate list is created. That means if anyone adds event handlers later on, they won't be invoked when the property changes. So with that stuck firmly in my mind, I never gave much thought to removing that last remaining bit of boilerplate. That's about to change!

What I really want is for my final implementation to look more like this:

public class Book : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    ChangeNotifier<string> author;

    public string Author
    {
        get { return author.Value; }
        set { author.Value = value; }
    }

    public Book()
    {
        author = ChangeNotifier.Create(() => Author, ????);
    }
}

That's short and sweet . The generic types should be automatically inferred, you shouldn't have to create the delegate to raise the event, it's beautiful! The only problem is to figure out what I should replace the question marks with. I need something that will allow me to get at the current list of event handlers from outside of the Book object, i.e. something along the lines of this:

Func<PropertyChangedEventHandler> getter = delegate { return PropertyChanged; };

Prettying it up a little, this is how my Book class looks:

public class Book : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    ChangeNotifier<string> author;

    public string Author {
        get { return author.Value; }
        set { author.Value = value; }
    }

    public Book()
    {
        author = ChangeNotifier.Create (() => Author, () => PropertyChanged);
    }
}

Beautiful! The more astute readers might notice a problem at this stage. Fine, the ChangeNotifier object can get the event list and raise the event, but it can't fill in the 'sender' - it has no reference to the 'book' object! Have no fear, it's already taken care of! The getter delegate has a reference to the book object (Delegate.Target), so we can fill everything in perfectly! The final implementation of the ChangeNotifier class is this:

public static class ChangeNotifier
{
    public static ChangeNotifier<TValue> Create<TValue>(Expression<Func<TValue>> expression, Func<PropertyChangedEventHandler> notifier)
    {
        return new ChangeNotifier<TValue>(expression, notifier);
    }
}

public class ChangeNotifier<TValue>
{
    Func<PropertyChangedEventHandler> notifier;
    string propertyName;
    TValue value;

    public TValue Value {
        get { return value; }
        set {
            if (!EqualityComparer<TValue>.Default.Equals(this.value, value)) {
                this.value = value;
                // Get the current list of registered event handlers
                // then invoke them with the correct 'sender' and event args
                PropertyChangedEventHandler h = notifier();
                if (h != null)
                    h(notifier.Target, new PropertyChangedEventArgs(propertyName));
            }
        }
    }

    public ChangeNotifier(Expression<Func<TValue>> expression, Func<PropertyChangedEventHandler> notifier)
    {
        if (expression.NodeType != ExpressionType.Lambda)
            throw new ArgumentException("Value must be a lamda expression", "expression");
        if (!(expression.Body is MemberExpression))
            throw new ArgumentException("The body of the expression must be a memberref", "expression");

        MemberExpression m = (MemberExpression)expression.Body;
        this.notifier = notifier;
        this.propertyName = m.Member.Name;
    }
}

I have one final trick up my sleeve. Suppose you have a field (Progress) whose value is calculated based on other values (CurrentStep, TotalSteps) and you want to get Notifications whenever any of those fields changes, well, that's easy!

public class Worker : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    ChangeNotifier<int> currentStep;
    ChangeNotifier<int> totalSteps;

    public int CurrentStep {
        get { return currentStep.Value; }
        set { currentStep.Value = value; }
    }
    public int TotalSteps {
        get { return totalSteps.Value; }
        set { totalSteps.Value = value; }
    }
    public double Progress
    {
        get { return (double)CurrentStep / TotalSteps; }
    }

    public Worker()
    {
        Func<PropertyChangedEventHandler> notifier = () => PropertyChanged;

        currentStep = ChangeNotifier.Create(() => CurrentStep, notifier);
        totalSteps = ChangeNotifier.Create(() => TotalSteps, notifier);

        // A PropertyChanged notification will be created for Progress every time
        // either the CurrentStep *or* TotalSteps changes.
        ChangeNotifier.CreateDependent(
            () => Progress,
            notifier,
            () => CurrentStep,
            () => TotalSteps
        );
    }
}

And the new helper methods are:

public static class ChangeNotifier
{
    static string GetPropertyName(Expression expression)
    {
        while (!(expression is MemberExpression)) {
            if (expression is LambdaExpression)
                expression = ((LambdaExpression)expression).Body;
            else if (expression is UnaryExpression)
                expression = ((UnaryExpression)expression).Operand;
        }
       
        return ((MemberExpression)expression).Member.Name;
    }

    public static void CreateDependent<TValue>(Expression<Func<TValue>> property, Func<PropertyChangedEventHandler> notifier, params Expression<Func<object>>[] dependents)
    {
        // The name of the property which is dependent on the value of other properties
        var name = GetPropertyName(property);
        // The names of the other properties
        var dependentNames = dependents.Select<Expression, string>(GetPropertyName).ToArray();
       
        INotifyPropertyChanged sender = (INotifyPropertyChanged)notifier.Target;
        sender.PropertyChanged += (o, e) => {
            // If one of our dependents changes, emit a PropertyChanged notification for our property
            if (dependentNames.Contains(e.PropertyName)) {
                var h = notifier();
                if (h != null)
                    h(o, new PropertyChangedEventArgs (name));
            }
        };
    }

    public static ChangeNotifier<TValue> Create<TValue>(Expression<Func<TValue>> expression, Func<PropertyChangedEventHandler> notifier)
    {
        return new ChangeNotifier<TValue>(expression, notifier);
    }
}

The only change is that I need to use a slightly more complicated method of getting the property name as it's possible for certain types to get wrapped in a ConvertExpression.

Yet another INotifyPropertyChanged with Expression Trees

2009-12-05T14:44:00.001Z

There are dozens of examples out there showing you how to avoid having to refer to method names as strings when implementing INotifyPropertyChanged. The most important reason why you don't want to have to do this is because method names can get refactored but the hardcoded strings might be forgotten. No-one wants to end up getting a Changed notification for a property which doesn't exist.

My issue with all these examples is that none of them thought far enough ahead. Fine, they all show you how refer to properties without using hardcoded strings but they still require you to write lots of boilerplate code to raise the PropertyChanged event - boilerplate you have to write for every property. What I want is to be able to declare all my properties like:

public string Title {
    get { return title; }
    set { title = value; }
}

and yet still get my property change notifications. I also want this method to be reasonably high performance. I don't want every property change to have extra memory or CPU overhead as every developer expects that changing the value of a property will not do any complex calculations. So how can I accomplish this?

To start off with, we can all tell that it's impossible to achieve the required behaviour using just the snippet above. We're going to have to add (at least) one additional level of indirection. That means I should be able to implement my requirements using code like:

public string Title {
    get { return title.Value; }
    set { title.Value = value; }
}

The object 'title' must then contain all the logic required to raise the property changed notification. So what might this magical object look like?

public class ChangeNotifier<TValue>
{
    Action<string> notifyHandler;
    string propertyName;
    TValue value;

    public TValue Value {
        get { return value; }
        set {
            if (!EqualityComparer<TValue>.Default.Equals(this.value, value)) {
                this.value = value;
                notifyHandler(propertyName);
            }
        }
    }


    public ChangeNotifier(Expression<Func<TValue>> expression, Action<string> notifyHandler)
    {
        if (expression.NodeType != ExpressionType.Lambda)
            throw new ArgumentException("Value must be a lamda expression", "expression");
        if (!(expression.Body is MemberExpression))
            throw new ArgumentException("The body of the expression must be a memberref", "expression");

        MemberExpression m = (MemberExpression)expression.Body;
        this.propertyName = m.Member.Name;
        this.notifyHandler = notifyHandler;
    }
}

You're probably looking at this thinking "What the hell is this Expression<Func<TValue>> ? How do I even use that monstrosity?". Well... simples!

public class Book : INotifyPropertyChanged
{
    public event PropertyChangedEventHandler PropertyChanged;

    ChangeNotifier<string> author;
    ChangeNotifier<decimal> price;
    ChangeNotifier<int> quantity;
    ChangeNotifier<string> title;

    public string Author {
        get { return author.Value; }
        set { author.Value = value; }
    }
    public decimal Price {
        get { return price.Value; }
        set { price.Value = value; }
    }
    public int Quantity {
        get { return quantity.Value; }
        set { quantity.Value = value; }
    }
    public string Title {
        get { return title.Value; }
        set { title.Value = value; }
    }

    public Book()
    {
        Action<string> notify = (propertyName) => {
            var h = PropertyChanged;
            if (h != null)
                h(this, new PropertyChangedEventArgs(propertyName));
        };

        author = new ChangeNotifier<string> (() => Author, notify);
        price = new ChangeNotifier<decimal> (() => Price, notify);
        quantity = new ChangeNotifier<int> (() => Quantity, notify);
        title = new ChangeNotifier<string> (() => Title, notify);
    }
}

All that happens here is that when constructing the ChangeNotifier object, an Expression referencing the required Property is passed into the constructor, along with a delegate which will raise the PropertyChanged event. We parse that expression tree to retrieve the method name and store it. After that everything Just Works (tm) with little to no performance penalty. The days of writing boilerplate code for INotifyPropertyChanged are gone! You also have the benefit that you can't make a mistake writing the boilerplate code because you don't write it anymore!

Can't you feel the Moonlight? Part deux

2009-12-04T16:44:00.005Z

As I was saying yesterday, the live version of the silverlight toolkit site didn't work right in moonlight. All the pretty charts rendered as you see them below, very empty.

I figured that since a slightly older version worked near-flawlessly, surely I could fix the live version with only a few minor tweaks. It's not like the would've completely rewritten the Chart controls within the space of 1 release.

I checked everything from DataBinding, to TemplateBinding, to Styles, to Measure/Arrange bugs and nothing was showing up as causing the issue. I finally narrowed it down to a bug in VisualStateGroup. For some reason the Name property was empty even though it was declared with a name in xaml.

One. Tiny. Patch. Later.

Success. I can't believe that the bug was that simple. In the end, those bugs are actually by far the worst. There's no exception thrown or any kind of visible indication that something has failed other than an empty screen. The only reason I found the bug was because the toolkit is opensource and I was running it locally with a few dozen Console.WriteLines, gradually reducing the area of code where I thought the bug was. Unfortunately this fix arrived too late for the 1.99.9 release, but it will definitely be in the release after it.

Can't you feel the moonlight?

2009-12-02T17:44:00.005Z

It's time for the obligatory screenshots again. This is what the Data Visualisation demos from the Silverlight Toolkit (March edition) looked like yesterday:

Note the empty graphs. It doesn't look very pretty now, does it? However, one very minor fix later we now have the following:

Things are near-perfect in all the Data Visualization demos. One graph is missing a background colour and the elements in one graph aren't clickable when they should be. Neither should be particularly difficult to fix, the only problem is figuring out the cause.

Unfortunately the version of the Toolkit Demo on the live site still doesn't render perfectly, but as we already have one version near-perfect, getting a newer revision to work shouldn't be hard! Things are shaping up to give us a great 2.0 release.

What does this say?

2009-11-30T00:29:00.004Z

This is the logo for the London olympics. Wolff Olins were paid a whopping GBP 400,000 to design this. You'd think that if you spent that kind of money, you'd end up with a logo that can be understood by mere mortals, however that's not what we got.

When I look at that logo I see the numbers '2' and '0' at the top. That's relatively clear. However, the bottom section is just random gibberish. If I concentrate a little harder, I could convince myself that there's a '1' in the bottom left, but what the hell is at the bottom right? A square with a squiggle beside it? I can't for the life of me figure out what that little square is for. Is it part of the '2' or is it there because they wanted to write "20-12"? I really can't tell.

If you're going to design a logo for such an important worldwide event, why can't it be legible? Logo design should be about creating an inventive and visually appealing way of getting some information across. It isn't supposed to be about who can create the most obfusticated advertisement - that defeats the entire purpose! Out of this list of a dozen alternative designs my favourite is this one:

It may not be the best logo I've ever seen, but it does one thing right: It's instantly recognisable and understandable. I know exactly what it's about without having to spend 10 mins rotating the thing trying to make sense out of it.

Dear Thierry Henry

2009-11-19T14:32:00.002Z

If you truly were sorry that you twice hit the ball with your hand to prevent it from going wide and then proceeded to score from that opportunity, why didn't you admit it on the spot. You knew what you did, you did it deliberately, it is useless claiming you're sorry now when it's too late for that to mean anything. Your chance to show your true colours was on the field and show them you did.

Since the match will not be replayed, why don't you step down from this world cup season if you truly do regret that decision which cost Ireland our world cup chance?

Don't BinaryReader.PeekChar () at me!

2009-11-02T10:14:00.010Z

Update: As pointed out in the comments, there are errors in the implementation of SeekableStream below. That's what happens when you write a class at 2am ;) The errors are that Writing to the stream will be off by 1 byte if you've peeked, and Position will be off by 1 byte if you've peeked. I'll re-read and update the class later.

Update 2: I updated the implementation of PeekableStream to make it a read-only stream wrapper. I'm not pushed about supporting writing on it as it's not something I require. I'll leave it as an exercise to the reader to support writing if they need it ;)

How many times has that urge hit you to peek inside a stream and see the secrets hidden inside? Have there been times when you wished that Stream exposed a 'public byte Peek ()" method so you could do this easily? Were you delighted when you discovered that BinaryReader exposed PeekChar () , which did nearly the same thing [0]? If that describes you, you're in for a horrible horrible surprise.

I received a bug report during the week saying that MonoTorrent reads all data twice from disk when it is hashing. I responded with "No it doesn't! That'd be crazy!", to which I was handed a screenshot of a ~310MB torrent which in the windows task manager reported ~750MB of I/O Read Bytes. I was told this happened after loading the torrent and calling HashCheck () on it. This was irrefutable evidence that something was up, but I was still unconvinced .

So over the weekend I fired up windows and double checked. I could replicate the bizarre statistic. But strangely enough, it wasn't hashing that was causing it! The I/O Read Bytes was up at around 350MB before hashing even started. But that was strange, because the only thing that happened before that was:

Torrent torrent = Torrent.Load (path);

There's no way a simple forward-only parser reading a 100kB file could possibly result in 350MB of IO reads, could it? Actually, it could! Changing the declaration to the following completely vanished the extra 350MB:

Torrent torrent = Torrent.Load (File.ReadAllBytes (path))

So what was going wrong? Internally the parser used BinaryReader.PeekChar () to figure out the type of the next element so that correct decoder could be called. I thought this would be a simple array access, or something similar. However what actually happens is that one byte is read from the underlying stream, then the stream seeks 1 byte backwards . In the case of FileStream, this meant that the entire read buffer was refilled from 'disk' [1] every time I peeked. A 100kB file really was really being turned into a 350MB monstrosity! And yes, the Mono implementation unfortunately has to do the same. So how could I fix this?

Simples! I could write a PeekableStream, one that's smart enough to not need to do horrible buffer killing seeks. What was the end result? Well, that particular .torrent file loaded nearly 5x faster, ~100ms for everything instead of ~500ms. An average file would experience a much smaller speedup. This one is a bit different in that it contains over 2000 files and the speed up is proportional to the number of BEncoded elements in the .torrent file.

public class PeekableStream : Stream
{
    bool hasPeek;
    Stream input;
    byte[] peeked;

    public PeekableStream (Stream input)
    {
        this.input = input;
        this.peeked = new byte[1];
    }

    public override bool CanRead
    {
        get { return input.CanRead; }
    }

    public override bool CanSeek
    {
        get { return input.CanSeek; }
    }

    public override bool CanWrite
    {
        get { return false; }
    }

    public override void Flush()
    {
        throw new NotSupportedException();
    }

    public override long Length
    {
        get { return input.Length; }
    }

    public int PeekByte()
    {
        if (!hasPeek)
            hasPeek = Read(peeked, 0, 1) == 1;
        return hasPeek ? peeked[0] : -1;
    }

    public override int ReadByte()
    {
        if (hasPeek)
        {
            hasPeek = false;
            return peeked[0];
        }
        return base.ReadByte();
    }

    public override long Position
    {
        get
        {
            if (hasPeek)
                return input.Position - 1;
            return input.Position;
        }
        set
        {
            if (value != Position)
            {
                hasPeek = false;
                input.Position = value;
            }
        }
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        int read = 0;
        if (hasPeek && count > 0)
        {
            hasPeek = false;
            buffer[offset] = peeked[0];
            offset++;
            count--;
            read++;
        }
        read += input.Read(buffer, offset, count);
        return read;
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        long val;
        if (hasPeek && origin == SeekOrigin.Current)
            val = input.Seek(offset - 1, origin);
        else
            val = input.Seek(offset, origin);
        hasPeek = false;
        return val;
    }

    public override void SetLength(long value)
    {
        throw new NotSupportedException();
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        throw new NotSupportedException();
    }
}

This code is under the MIT/X11 license, so everywhere you use PeekChar () and actually just want to Peek at a byte, use this class instead. Your harddrives will love you for it. If you actually want to peek at a char, extend this class to be able to read a (multi-byte) char from the underlying stream and cache it locally just like the current PeekByte method . A side benefit is that you can now peek at unseekable streams. Not too bad, eh?

[0] PeekChar does exactly what it says on the tin. It reads one (multi-byte) character from the stream. So if you're using PeerChar on a binary stream which does not contain valid data as defined by the current Encoding, you're going to corrupt some data or get exceptions. I mention this here in case anyone is using PeekChar () as a way of reading bytes from the stream.

[1] I say that the FileStream buffer was being filled from disk, but that's not quite accurate. It was actually being refilled from either the windows cache or the harddrives cache. It's physically impossible for a mere 7200 RPM harddrive to supply data that fast. However I still was physically copying 350MB of data around in memory so that was a huge penalty right there.

MonoTorrent 0.80 - Up up and away

2009-10-18T19:22:00.006+01:00

MonoTorrent 0.80 has been released. I'd like to say "It's the best release ever", but that always makes me think "If it wasn't the best release ever, why would I release it?"

The full release notes can be read on www.monotorrent.com. For the lazy, I'll put a quick blurb about the two new most exciting new features available:

Metadata Exchange
http://www.bittorrent.org/beps/bep_0009.html

Put simply, this means you can click on a link like this: magnet:?xt=urn:btih:12345678901234567890 and then the torrent will magically [0] be able to download. Behind the scenes what happens is that peers are found via DHT and then they are queried for the .torrent metadata. Once the metadata has been obtained, the actual downloading can commence and away you go. Finally, I can start a download via text message!

Local Peer Discovery

This allows MonoTorrent to find other peers who are downloading the same torrent on the local network. A simple UDP broadcast message is used for discovery. This is an implementation of style LDP and so is fully compatible with uTorrent and other clients which have implemented this style. The main benefit of this is that in corporate or educational environments, it's possible that many people will be trying to access the same torrent at the same time. This approach allows all these peers to connect to each other and thus transfer the bulk of their data over the internal LAN rather than all of them fighting for bandwidth on the (usually) limited WAN connection.

As per usual, there are a bunch of bug fixes and enhancements. This is one more milestone on the way to the final 1.0 release. One which I'm really looking forward to. I might even do some nostalgia posts about the big disasters I created while learning C# and implementing this library ;)

[0] Actual product does not contain magic.

EDIT: Just clarified that the LPD implementation is the uTorrent style.

Mono.Nat 1.0.2

2009-06-29T23:05:00.002+01:00

I just tagged and released Mono.Nat 1.0.2 . It's a fairly minor bugfix release which addresses a number of minor issues:

Added workaround for certain versions of miniupnpd which incorrectly advertise their available services (bug has been reported upstream)
Fixed some other minor issues with routers reporting incorrect services.
Added extra API to make it easy to log the full handshake/request process to help diagnose issues
Stopping and Starting discovery will rediscover all available devices correctly
Full support for computers with multiple network cards on multiple subnets
Rewrote the internals to ensure that the asynchronous API is 100% asychronous - prevents calls to BeginXXX blocking on some slower routers.

Precompiled binaries and sourcecode can be downloaded here and packages will soon be winding their way to a repository near you.

If you want to forward ports automagically on a upnp empowered router near you, this is the library for you!

Monsoon - blowing down barriers

2009-05-28T02:18:00.003+01:00

Yes, Monsoon is now using Mono.Addins for some delicious plugability. Support for this has only just been added, so there is a severe lack of extension points defined in monsoon, but those can be added as time goes on. Right now there is one extension point. I'm sure you've already guessed what it is.

Yup, that's right. That little nifty thing at the bottom is DHT bootstrapping itself. As you can see, it's currently displaying a rather disappointing value of '1' for 'Nodes'. This is either because the bootstrap node is currently unavailable, or my router is playing silly buggers again and not forwarding my ports right. Luckily you can also bootstrap into DHT by just downloading a normal torrent. Other peers advertise when they support DHT and provide the required info to allow you to use them as a bootstrap node.

What this means is that opensuse users will finally have easy access to a DHT enabled torrent client without having to enable additional repositories. Things will work right out of box... well, it'll work as soon as you click to enable the addin which fetches it from the monsoon website ;)

Once I solidify everything there'll be a preview release of Monsoon with these features and another slightly big one I've been working on. More on that later. I've reached my word quota for this post ;)

Book memes - Reloaded

2009-05-18T19:19:00.003+01:00

* Grab the nearest book.
* Open it to page 56.
* Find the fifth sentence.
* Post the text of the sentence in your journal along with these instructions.
* Don't dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

"Ponder and Ridcully waited for a few moments, but the city stayed full of normal noise, like the collapse of masonry and distant screams"

Not a bad quote, eh? And yes, I am still a kid at heart :p

Polymorphism, why do you fail me?

2009-05-17T22:09:00.003+01:00

Polymorphism, it's the cornerstone of object oriented programming. We couldn't live without it. So then, tell me why this rather trivial case fails to compile.

public class B : A { }

public void Foo (ref A bar) {  }

public void Baz ()
{
    B b = new B ();
    Foo (ref b);
}

The issue is hat you can't pass a 'B' parameter type by ref where a 'ref A' is expected. Why is this? What case could possibly fail if this was allowed?

So what've I been working on?

2009-05-15T14:09:00.005+01:00

I haven't actually blogged much about my job. That was getting kind of weird, so I thought it was high time I wrote something. Then I thought to myself "A picture is worth a thousand words, so why bother wasting my time with words". So here are two pictures which show what I've been doing over the last 5 days. Though technically i've been working on parts of this over the last 2-3 weeks ;)

Moonlight - Monday 11th May 2009:

Moonlight - Friday 15th May 2009 (with local patches):

Pretty sweet, eh?

UPDATE: If you want to look at the actual site, it's available here: http://silverlight.net/samples/sl2/toolkitcontrolsamples/run/default.html

Hopefully our next alpha preview will contain the necessary fixes to load the site up so you can see it in all its glory.

Monsoon 0.21

2009-04-20T22:55:00.002+01:00

Monsoon 0.21 has been released. It now uses MonoTorrent 0.72. Other than bumping to a newer version of monotorrent, there were only a few minor fixes for Monsoon 0.21. The most notable of which is a fix for a change in firefoxs behaviour when opening files directly from the browser.

The update should come to a repository near you soon!

nat-pmp support in Mono.Nat

2009-04-16T22:30:00.003+01:00

Is there anyone out there with a week or so to complete the implementation of nat-pmp support in Mono.Nat that'd be awesome. The implementation is 80-90% complete. The only requirement is that you have a router supporting nat-pmp with which to test against. Unfortunately I don't have one and I can't get the daemon working on my router.

If you want to work on it, send me an email and I'll fill you in on what needs to be done.

MonoTorrent 0.72 released

2009-04-06T11:56:00.002+01:00

This is a bugfix release to address a few reported issues and also a few issues that were discovered via my own testing.

* Add a helper method which ensures all data is flushed to disk
* Added additional error handling to prevent malformed DHT messages crashing the library
* Fixed issue when zeroing unused bits for torrents with an exact multiple of 32 pieces
* Fixed issue where data could be written to the wrong file if a file with the same name existed in multiple torrents
* Fixed the handling of torrents where the last file(s) are of zero length
* Fixed regression with global download rate limiting
* Fixed a performance regression with the new piece picking pipeline which resulted in lots of CPU cycles being used up on peers which have not sent an unchoke message

In other news, monotorrent.com is changing its hosting provider. It still brings you to the old website, but it'll be moved to http://projects.qnetp.net/projects/show/monotorrent soon enough. This is were future releases will be made.

Google Summer of Code - 2009

2009-03-31T10:10:00.002+01:00

The final deadline for applications for the gsoc is fast approaching. You now have 4 days left, right up until 19:00 UTC on April 3rd. There are so many cool projects available this year, so make sure you apply soon before they're all taken!

Some of my personal favourites would be:
Writing SIMD code in C#
VDPAU VC-1/H.264 Support
Codec Demuxers
A Dirac decoder

And many more! Funnily enough, video encoding is what got me interested in programming in the beginning, so if you students don't snap up those projects, I will ;) Hack on something cool for the summer and earn yourself some cash while you're at it. It'll be great!

We got the grand slam!

2009-03-22T23:12:00.004Z

We've done it! We've won the Grand Slam for the first time in 61 years. It was the most nail-biting match I've watched in a while. It was so close that the result was actually down to the very last kick of the match, which was a penalty awarded* against Ireland. Thankfully Wales missed it and sealed Irelands victory.

Go on the green!

* The ref made a bad call here and awarded a penalty against Ireland because the ball bounced forward after it was dropped. The rule are pretty clear that this should be a scrum and *not* a penalty. It wasn't the first time he made that mistake either. Still, it didn't matter in the end, woo!

Update: A friend just told me that the ref originally signalled a scrum but then changed it to a penalty. Sounds like an irish guy mouthed off or did something off the ball. Did anyone notice what actually made him change his mind?

MonoTorrent 0.70

2009-03-20T01:39:00.003Z

With all the excitement going on, I forgot to blog about MonoTorrent 0.70, so here's a belated summary of what went on:

* Fixed an issue for torrents with no trackers
* Optimised the Bitfield class to allow for higher performance piece picking
* Rewrote the piece picking API which resulted in a very extensible, easily testable, less buggy and faster implementation
* Fixed an issue where the announce wouldn't happen immediately after a torrent completes
* Fixed several issues with webseeding and rate limiting
* Fixed an issue which resulted in an incorrect encryption level been chosen in a small proportion of cases
* Ensure that announces to trackers always time out correctly
* Fixed a race condition when stopping a torrent while an incoming connection is being processed
* Increased the performance of disk IO so that hashing a torrent is about 10% faster
* Vastly improved performance of the BanList parser
* Fixed issue where fast pieces could be requested multiple times
* Fixed issue with selective downloading where the start/end indices of files could be offset by 1

A precompiled binary can be found here and the source tarball can be found here. Enjoy!

What does it take to download a torrent?

2009-03-15T18:25:00.003Z

The simple answer to this question is "All you need is the .torrent file. Stupid n00bs." This is true. The .torrent file contains all the metadata required to A) download the data and B) verify the data. Without this, you can't really do anything. So, do you actually need the .torrent file to begin a download?

No! All you need is the infohash for the .torrent file! The infohash is the SHA1 hash of the metadata in a .torrent file. As such, it can be used as a unique identifier for a particular .torrent file. With this infohash, you can query the BitTorrent DHT for a list of peers downloading that torrent. Then, with the help of the Metadata Exchange extension, you can connect to these peers and request that they send you the metadata from the .torrent file and you're away and downloading. Great!

"But what if some malicious peer sends you corrupt metadata, then you'd never be able to download the torrent properly!", I hear you asking. Well, in a rather beautiful twist, this is next to impossible. As I said earlier, the infohash is generated by putting the metadata in the .torrent file through a SHA1 hash. So all you have to do is hash the metadata once you have received it and then compare the result of that to the SHA1 hash you used to start the download. If they match, then you can be fairly confident that the metadata has not been corrupted/altered in any way.

As of 17:00GMT, March 15th MonoTorrent has completed its first download using only a 20 byte hash to begin the download. This is possible because of some tireless work by Olivier Dufour, who also implemented Peer Exchange, a good few parts of DHT, WebSeeding and SuperSeeding. The code for this still hasn't quite hit SVN, a bit of refactoring remains to be done. It should be in SVN within a week. I'm looking forward to his next patch of awesomenesss now.

The monsoon-devel list

2009-03-02T10:05:00.002Z

I've started a mailing list for monsoon so that packagers can be kept updated and various interested people can post questions and all that jazz. Basically it's the standard -devel mailing list. That's mostly why it's called "monsoon-devel" ;)

If you're interested in keeping up to date on Monsoon, please join the group at: http://groups.google.com/group/monsoon-devel

Banshee - For all your medical record needs?

2009-02-27T10:00:00.004Z

I'm sure everyone's familiar with Banshee, the most awesome media player of all time ever ;) So, how does it feel to know it has expanded into the medical record field? Good eh? Next time you go to your doctor do you know what they'll be running? That's right, Banshee - Medical Record Edition. Check out its website here:

http://gnumed.org/index.html

The regular banshee website can be found here:

http://banshee-project.org/

It's fun to open both in tabs and switch between them.

Monsoon 0.20

2009-02-19T22:51:00.003Z

Monsoon 0.20 has been released and should be in a repository near you soon. Release notes can be found here. Fun stuff!

Excitement abounds!

2009-01-24T13:49:00.002Z

Next weekend is going to be an interesting one. Several releases are coinciding:

1) MonoTorrent 0.70 is prepped and ready for release. Contains numerous bugfixes and performance enhancements.
2) Monsoon 0.20 will in a repository near you. This will be using MonoTorrent 0.70, so a nice upgrade from the previous 0.40 release.
3) The DBus daemon for monotorrent is ready for its first release. Banshee will be using this to add support for torrent based podcasts.
4) Mono.Nat will be getting it's first official package release too. It's a reasonably mature library which supports UPnP port forwarding/mapping. There's also support for nat-pmp in the library, but it's disabled due to lack of testing. If someone has a nat-pmp capable router, feel free to test the code, fix the few remaining issues and submit a patch.

But for now, it's holiday time for a week ;)

A workaround!

2008-12-22T01:47:00.003Z

So there exists *one* way and one way only to safely set the Range header when using 64bit indices. This method was found with the help of NotZhila on #mono:

MethodInfo method = typeof(WebHeaderCollection).GetMethod
                        ("AddWithoutValidate", BindingFlags.Instance | BindingFlags.NonPublic);

HttpWebRequest request = (HttpWebRequest) WebRequest.Create ("http://www.example.com/file.exe");

long start = int32.MaxValue;
long end = int32.MaxValue +  100000;

string key = "Range";
string val = string.Format ("bytes={0}-{1}", start, end);

method.Invoke (request.Headers, new object[] { key, val });

You need to call the protected method of the WebHeaderCollection class so you can add the "Range" header without being forced to go through the broken HttpWebRequestAddRange method. I tried about a half dozen more legitimate methods, but the API is so locked down that it was impossible.

For example, HttpWebRequest.Headers is a get/set property, but if you set a new collection, it checks to make sure "Range" isn't there, and then just copies the keys from your new collection into it's internal one and ignores the collection you just gave it. What this means is that:

CustomCollection c = new CustomCollection ();
request.Headers = c;
Console.WriteLine (request.Headers == c);

prints false.

Ouch.

Anyway, the hack works. If you need 64bit indices when downloading files via HttpWebRequest, here's your guaranteed working hack, on both Mono and MS.NET.

System.Net.WebRequest -> System.Net.WebSuck

2008-12-20T23:22:00.002Z

I can see that throughout the lifetime of the .NET framework, no-one realised that files greater than 2GB exist on the web.

System.Net.WebRequest.AddRange (int startOffset, int endOffset)

Thankfully I don't have to rewrite my own class from scratch, I can cannibalise one from Mono and add in a workaround, though that's likely to be a pain in the ass too :(

PiecePicking and runtime method overriding

2008-12-17T11:53:00.004Z

Deciding which chunk of data to download next is a pretty complex task in the bittorrent world. You have to take into account a number of factors, such as:
1) You may want to prefer to download rare pieces first, maybe not.
2) You may want to pick a piece randomly, or maybe you want to download from the start of the file and get pieces in order
3) The user may set a higher priority on a certain file, so you should prefer its pieces.
4) The user may want to ignore certain files and never select their pieces.

Then imagine trying to NUnit test all the combinations and you'll find that it becomes very difficult to ensure the implementation is correct and impossible to extend with new behavior because the interactions become too complex.

The basic premise I had when redesigning this area of code was this: When I want to change the picking behaviour, I want to override one single method and then just slot the picker into the pipeline. So here's a reduced version of the PiecePicker class (3 methods instead of 10) and an example of how it's implemented.

public abstract class PiecePicker
{
  protected PiecePicker(PiecePicker picker)
  {
      this.picker = picker;
  }

  void CheckOverriden()
  {
      if (picker == null)
          throw new InvalidOperationException("This method must be overridden");
  }

  public virtual void Initialise(BitField bitfield, TorrentFile[] files, IEnumerable&lt;Piece&gt; requests)
  {
      CheckOverriden();
      picker.Initialise(bitfield, files, requests);
  }

  public virtual bool IsInteresting(BitField bitfield)
  {
      CheckOverriden();
      return picker.IsInteresting(bitfield);
  }

  public virtual MessageBundle PickPiece(PeerId id, BitField peerBitfield, List&lt;PeerId&gt; otherPeers, int startIndex, int endIndex, int count)
  {
      CheckOverriden();
      return picker.PickPiece(id, peerBitfield, otherPeers, startIndex, endIndex, count);
  }
}

    public class RandomisedPicker : PiecePicker
    {
        Random random = new Random();

        public RandomisedPicker(PiecePicker picker)
            :base(picker)
        {

        }

        public override MessageBundle PickPiece(PeerId id, BitField peerBitfield, List<PeerId> otherPeers, int startIndex, int endIndex, int count)
        {
            MessageBundle message;
            int midpoint = random.Next(startIndex, endIndex);
            if ((message = base.PickPiece(id, peerBitfield, otherPeers, midpoint, endIndex, count)) != null)
                return message;
            else
                return base.PickPiece(id, peerBitfield, otherPeers, startIndex, midPoint, count);
        }
    }

Here's a usage example:

// StandardPicker provides an implementation for every method in PiecePicker
PiecePicker p = new RandomisedPicker(new StandardPicker);

So what happens is this:
1) p.PickPiece is called which results in RandomisedPicker.PickPiece being called
2) This splits the range between startIndex/endIndex into two and then makes two calls to base.PickPiece.
3) base.PickPiece will call standardPicker.PickPiece (because the 'picker' is non-null)
4) standardPicker.PickPiece will see if there are any pieces to request, and choose the first available one

Suppose I wanted to be able to ignore all the pieces which I already own, well, that's easy, I just use an IgnoringPicker (not a great name I'll admit):

    public class IgnoringPicker : PiecePicker
    {
        BitField bitfield;
        BitField temp;

        public IgnoringPicker(BitField bitfield, PiecePicker picker)
            : base(picker)
        {
            this.bitfield = bitfield;
            this.temp = new BitField(bitfield.Length);
        }

        public override MessageBundle PickPiece(PeerId id, BitField peerBitfield, List<PeerId> otherPeers, int startIndex, int endIndex, int count)
        {
            // In binary operations: temp = peerBitfield & (~bitfield);
            temp.SetAll(false).Or(peerBitfield).NAnd(bitfield);
            return base.PickPiece(id, temp, otherPeers, startIndex, endIndex, count);
        }

        public override bool IsInteresting(BitField bitfield)
        {
            temp.SetAll(false).Or(bitfield).NAnd(this.bitfield);
            return base.IsInteresting(temp);
        }
    }

Now i just change my usage to:

Picker picker = new StandardPicker ();
picker = new RandomisedPicker (picker);
picker = new IgnoringPicker (myBitfield, picker);

The biggest benefit is that each individual picking logic unit can very easily tested. They can then be combined in any order you want to change the picking behaviour. Random->Rarest->Standard would be different to Rarest->Random->Standard. The former will give you the rarest piece in a random set, the latter will give you a random piece from the rarest set.

The implementation of a logic unit is also trivial - you only have to override the behaviour you want to change rather than implementing a massive interface where for 90% of the methods you end up calling basePicker.Method anyway. So I'm quite happy with this change. This code already passes all the NUnit tests from the old picker as well as all the new tests I've written, so it should be going live in the near future. There's still some more work to be done before it's complete.

MonoTorrent 0.62

2008-11-08T18:54:00.000Z

MonoTorrent 0.62 has now been released which addresses a few major and minor issues with the 0.60 release. Here's the details:

* Fixed regression in message handling which resulted in 0.60 not transferring properly. Caused by not running the right NUnit tests before tagging.
* Performance optimisation for sending/receiving messages
* Fixed bug creating torrents with 2gb+ files with TorrentCreator
* Fixed issue with udp sockets in the Dht code which could cause dht to stop sending/receiving messages

I'd highly recommend upgrading from 0.60 to 0.62 as soon as possible.

Binary - http://monotorrent.com/Files/0.62/monotorrent-0.62-bin.zip
Source - http://monotorrent.com/Files/0.62/monotorrent-0.62.tar.gz

Are you mapping those dlls?

2008-11-06T13:50:00.002Z

One of the issues with writing cross platform applications is that if you do P/Invoke into a native library, the name of that library changes depending on the OS. Mono has built in support for selecting the right file at runtime. The problem is that it's hard to ensure that you've correctly mapped all the methods you P/Invoke.

So I wrote a little tool to do that for you. It was inspired by an attempt by Aaron Bockover which parsed the raw cs files. I figured it'd be much better to just parse the compiled assembly ;)

using System;
using System.Collections.Generic;
using System.Xml;
using Mono.Cecil;
using System.IO;

namespace DllImportVerifier
{
    class MainClass
    {
        static void Main (string [] args)
        {
            args = new string [] { Environment.CurrentDirectory };
            if (args.Length != 1) {
                Console.WriteLine ("You must pass the path to the assemblies to be processed as the argument");
                return;
            }

            List<string> assemblies = new List<string> ();

            if (Directory.Exists (args[0])) {
                assemblies.AddRange (Directory.GetFiles (args [0], "*.dll"));
                assemblies.AddRange (Directory.GetFiles (args [0], "*.exe"));
            }
            else if (File.Exists (args [0])) {
                assemblies.Add (args [0]);
            }
            else {
                Console.WriteLine ("{0} is not a valid file or directory", args [0]);
                return;
            }

            foreach (string assembly in assemblies)
                ProcessConfig (assembly);
        }

        static void ProcessConfig (string assemblyName)
        {
            AssemblyDefinition assembly = null;
            try {
                assembly = AssemblyFactory.GetAssembly (assemblyName);
            } catch {
                Console.WriteLine ("{0} is not a valid .NET assembly", Path.GetFileName (assemblyName));
                return;
            }

            List<string> dlls = new List<string> ();
            try {
                XmlTextReader doc = new XmlTextReader (assemblyName + ".config");
                while (doc.Read ()) {
                    if (doc.Name != "dllmap")
                        continue;

                    string dll = doc.GetAttribute ("dll");
                    if (!dlls.Contains (dll))
                        dlls.Add (dll);
                }
            } catch {
                // Ignore malformed or invalid config files. If the config is malformed,
                // drop any successfully parsed dll maps
                dlls.Clear();
            }

            List<string> unreferenced = VerifyReferences (assembly, dlls);
            foreach (string dll in unreferenced)
                Console.WriteLine("Assembly: '{0}' references '{1}' without mapping it", Path.GetFileName (assemblyName), dll);
        }

        static List<string> VerifyReferences (AssemblyDefinition assembly, List<string> dlls)
        {
            List<string> unreferenced = new List<string> ();
            foreach (TypeDefinition type in assembly.MainModule.Types) {
                foreach (MethodDefinition method in type.Methods)  {
                    if (!method.IsPInvokeImpl)
                        continue;
                    
                    string dll = method.PInvokeInfo.Module.Name;
                    if (!dlls.Contains (dll) && !unreferenced.Contains (dll))
                        unreferenced.Add (dll);
                }
            }

            return unreferenced;
        }
    }
}

BarrierHandle

2008-11-05T14:17:00.003Z

Did you ever have a case where you have a big dataset which can be processed in parallel, so long as all threads finish Step 1 before any thread starts Step 2? Thing is, there's no built in class to handle this case.

AutoResetEvent won't do it because it because it only signals *one* of the waiting threads, not *all* of them. What you need is a manual reset event and some complex handling.

I give you BarrierHandle:


using System;
using System.Threading;
 
class SpectralNorm
{
    public class BarrierHandle : System.Threading.WaitHandle
    {
        int current;
        int threads;
        ManualResetEvent handle = new ManualResetEvent (false);
         
        public BarrierHandle (int threads)
        {
            this.current = threads;
            this.threads = threads;
        }
 
        public override bool WaitOne()
        {
            ManualResetEvent h = handle;
            if (Interlocked.Decrement (ref current) > 0) {
                h.WaitOne ();
            }
            else {
                handle = new ManualResetEvent (false);
                Interlocked.Exchange (ref current, threads);
                h.Set ();
                h.Close ();
            }
 
            return true;
        }
    }
     
    public static void Main(String[] args)
    {
        int threadCount = 5;//Environment.ProcessorCount;
 
        // Lets assume that there's 20 units of data for each thread
        int[] dataset = new int [threadCount * 20];
 
        // This is the handle we use to ensure all threads complete the current step
        // before continuing to the next step
        BarrierHandle barrier = new BarrierHandle(threadCount);
 
        // Fire up the threads
        for (int i = 0; i < threadCount; i++) {
            ThreadPool.QueueUserWorkItem (delegate {
                int jjj = i;
                try {
                    Step1 (dataset, jjj * 20, 20);
                    barrier.WaitOne ();
                    Step2 (dataset, jjj * 20, 20);
                    barrier.WaitOne ();
                    Step3 (dataset, jjj * 20, 20);
                } catch (Exception ex) {
                    Console.WriteLine (ex);
                }
            });
        }
 
        System.Threading.Thread.Sleep (3000);
    }
 
    private static void Step1 (int[] array, int offset, int count)
    {
        Console.WriteLine ("Step1: {0} - {1}", offset, offset + count);
        Thread.Sleep (500);
    }
    private static void Step2 (int[] array, int offset, int count)
    {
        Console.WriteLine ("Step2: {0} - {1}", offset, offset + count);
        Thread.Sleep (500);
    }
    private static void Step3 (int[] array, int offset, int count)
    {
        Console.WriteLine ("Step3: {0} - {1}", offset, offset + count);
        Thread.Sleep (500);
    }
}

MonoTorrent 0.60

2008-11-03T11:34:00.001Z

So here are the bugfixes and features for 0.60 as compared to 0.50:

Bug fixes:
* Fixed critical regression in 0.50 whereby transfers would be incredibly slow.
* Fixed issue where announce/scrape requests to offline trackers may never time out
* Added a few memory optimisations when reading piece data from disk.
* Add the ability to report an alternate IP/Port combo to the tracker
* Optimised the encrypted handshake - its now a bit faster
* Fixed bug where wrong message ID was sent for extension messages
* Fixed bug where pieces which do not exist would be requested from webseeds.
* Fixed several bugs in TorrentCreator class where relative paths were used instead of absolute paths.
* Disabled UdpTracker support as the current implementation locked up until a response was received.
* Fixed a possible issue with multi-tier torrents.
* Added the ability to tell whether peers come from PeerExchange, DHT or the tracker.
* Some big performance boosts for webseeds by using a better method of picking pieces.
* Fixed a corner case whereby certain torrents may not get their last piece requested.
* When hosting a tracker,"/announce" is appended when you use the construtor overload which takes an IPEndPoint.

New Features:
* Implemented DHT support.
* Implemented sparse file support on the NTFS file system.

All in all, an awesome release. I have to give a shout out to Matthew Raymer who has been tirelessly testing MonoTorrent for the last few weeks. Without his constant bug reports, some of those bugs would never have been found, such as the one with UdpTrackers. Great stuff.

Packages:
Binary: http://www.monotorrent.com/Files/0.60/monotorrent-0.60-bin.zip
Source: http://www.monotorrent.com/Files/0.60/monotorrent-0.60.tar.gz

There are also packages for openSUSE in the openSUSE Build Service for those of you interested in those.

A hack too far

2008-10-31T12:03:00.003Z

I was just looking at the PackUriHelper class, with a view to implementing it in Mono. One of it's many methods converts a regular Uri into a 'pack' Uri. For example:

http://www.test.com/main/page.html?query=val#middle

converts to

pack://http:,www.test.com,main,page.html?query=val#middle/

So I says to myself, this is easy, it's just combining two Uris. Simple!

Wrong

Uri original = new Uri ("http://www.test.com/main/page.html?query=val#middle");

Uri complete = new Uri ("pack://" + original.OriginalString); // FAILS
Uri complete = new Uri (new Uri ("pack://", original)); //FAILS
Uri complete = new Uri (new Uri ("pack://", original.OriginalString)); // FAILS

How about escaping the second string...
string escaped = Uri.HexEscape (original.OriginalString);
Uri complete = new Uri ("pack://" + escaped); // FAILS

So at this stage I've lost all faith in humanity, so i try a basic test just to make sure i'm not insane. I'll try create a pack uri object myself, just to prove that it really is possible to parse them.

Uri complete = new Uri ("pack://http:,www.test.com,main,page.html?query=val#middle/");

EPIC FAIL

You can't do that. While they can *generate* that Uri for you, you can't generate one for yourself. Funnily enough, they do register a custom parser for the pack scheme, it's just incapable of parsing pack URI's. Don't ask me why.

After several hours and a lot of wasted time I finally came up with this:

Uri complete = new Uri (new Uri ("pack://"), escaped);

This is the *only* way to create a Pack URI. You have to hex escape the original uri and then call the constructor overload which takes a Uri and a string.

That is one of the stupidest things I've ever seen.

MonoTorrent 0.60

2008-10-25T16:55:00.003+01:00

MonoTorrent 0.60 is being prepared for release. 0.50 was released a mere 3 weeks ago, so why 0.60 already?

Well:

There was a big regression in 0.50 with regards to download speeds. Transfers were a *lot* slower than 0.40. To backport this fix would be very complex as there have been a lot of changes since 0.50 was branched.
DHT support is now good enough for me to activate by default. This was my milestone for releasing 0.60.
There have been a number of other important bugs fixed aswell (including a critical one with UdpTrackers) which need to be released.
There have been a number memory/performance optimisations since 0.50 which would be nice to release ;)

So, if you're using MonoTorrent, I urge you to check out the branch (when it is created) from http://anonsvn.mono-project.com/source/branches/bitsharp-0.60.

Until I create the branch, check out revision 117059 of monotorrent from svn. If you have any bugs/issues, let me know and I'll try to get a fix in for 0.60.

Walking can be tricky...

2008-10-20T22:40:00.002+01:00

Awesome!
(No, it's not just a knee-high stocking with sticky tape, it's a cast)

The big wahooni

2008-10-14T20:01:00.002+01:00

Sparsely populated, just the way I like it

2008-10-11T17:40:00.002+01:00

The NTFS filesystem has support for sparse files, but this has to be specially enabled when creating a file. I was originally linked to a blog post on the idea, but unfortunately the license pretty much forbids me from using that code.

So I spent most of yesterday evening and this morning googling API documentation and eventually came up with a fairly basic implementation of sparse file support. The only two operations supported are:

Creating a sparse file
Setting the size of the sparse data.

The benefits of this are seen only on the NTFS filesystem, but what it gives you is the ability to write pieces at arbitrary places in the file without having to preallocate up to that point. There's a bit of overhead, but other than that you only use up the space which you've actually written to, i.e. only the pieces which have been downloaded.

Finally, if you don't have an NTFS filesystem supporting sparse files, you will not be affected by this. Those of you on HFS+ can never get this support, and those of you on ext3, I'm unsure how to enable sparse files. I think it's enabled by default, but if not, any recommendations on setting this up would be appreciated.

MonoTorrent 0.50 - The Good, The Bad, and the seriously awesome

2008-10-06T21:59:00.004+01:00

It's release time! Yes, MonoTorrent 0.50 has hit. There have been a lot of changes since the last release, and this time, it's more than just under the hood fixes. There are several reasons why the new release is so much better than previous releases, I've listed the more important ones below, but first, the packages!

You can grab a precompiled binary suitable for Windows, Mac OS and Linux.
You can grab the sourcecode here as a .tar.gz archive.
There are also packages available on the OpenSuse build service, and of course the tag for the release can be gotten from svn.

Now, to the features and updates.

WebSeeding Support
There is provisional support for Http Web Seeding. This means when you're hosting a torrent, you can add standard Http servers as 'seeds'. No extra configuration is needed. This is still an experimental feature, and still has some corner cases where it doesn't work, all bug reports on this are welcome!

IP Address Banning
You can now ban individual IP addresses or IP Address ranges. Block lists from Emule, PeerGuardian and SafePeer are supported out of box by the built in parser, and any custom list can easily by loaded so long as you can parse the list into IPAddress objects. Internally the banlist is stored using the extremely efficient RangeCollection written by Aaron Bockover.

Efficent Torrent Streaming
Thanks to the efforts of Karthik Kailash and David Sanghera, we now have a special downloading mode in MonoTorrent which allows you to efficiently stream audio/video. Psuedo random piece picking is used to ensure you download pieces from a 'high priority' range before anything else. User code can set this 'High priority' range to be the next X bytes of data. When everything in the high priority range is downloaded, standard rarest-first picking is used.

Peer Exchange
uTorrent style Peer Exchange support is supported thanks to the tireless efforts of Olivier Dufour. This extension allows peer information to be passed across a bittorrent connection. In practice this means that if the tracker only gives you 1 peer, you can discover (potentially) hundreds more via peer exchange.

Enhanced compatibilty with broken clients
There are still clients out there which transmit corrupted BEncodedDictionary objects. These guys need to read the spec and ensure that their dictionaries keys are sorted using a binary comparison. In the cases where the order appears to not matter, I've implemented support for ignoring the error. This should reduce the number of clients which are disconnected due to sending corrupt messages - this means higher performance.

Simplified Threading API
The core of MonoTorrent has undergone a complete rewrite. Previously, all the worker threads interacted with the core by taking out locks, then doing their work. This meant that implementing something as trivial as cancelling a pending asynchronous request was actually pretty hard. That method was actually horrendously prone to deadlocking the engine.

Nowadays all the worker threads add a task to the main thread, and the main thread does all the work. "What about the performance" i hear you ask, well, it performs the exact same, but it's so much easier to maintain and add new features to.

It also means the engine should be deadlock free, because there are no locks anymore. Nice.

NUnit Tests
As with all big software projects, regressions are bad. A year ago I had virtually no NUnit tests. Nowadays there are over 130 NUnit tests for the engine. While this doesn't even test 1/2 the code in MonoTorrent, each test adds that little bit more certainty that I don't regress.

There are also a bunch bugfixes here and there, and more big features in the pipeline. As a taster, DHT support is already active and enabled in SVN should you wish to test it out.

Who loves lamdas?

2008-10-03T02:21:00.007+01:00

I finally started a project where I can use all the new fanciness. How would you process a list of names so that each name is printed out 'i' times?

int i = 5;
IEnumerable<string>() names = new List<string>() { "Alan" };
names.ForEach (n => i.Times().Apply(j => Console.WriteLine(n)));

All your lambdas are belong to us!

Webserver for NUnit tests

2008-10-02T15:53:00.002+01:00

What's the best way of setting up a simple HTTP file server to be used purely for NUnit tests. What I'm looking for is a way to set up a server so that I can host some files in. My NUnit tests will then initiate HTTP requests to 'download' those files.

Ideally whatever method I choose won't require me to write temporary files to disk. Currently what I do is host a HttpListener and manually fulfill the requested data by creating an in-memory byte[] and writing that into the response. I'd like to do something like:

webServer.Add("path/where/file/is", CreateFakeByteArray());

Then the server will fulfill requests from this faked byte[].

The reason for this setup is that I need a better way to test my implementation of WebSeeding in MonoTorrent without having to code up my own buggy file server ;)

tee-sharp a.k.a. CopyStream

2008-09-29T02:16:00.004+01:00

How many times do you have one stream, but you actually want to write the data to multiple places simultaneously[0]? Well, now you can[1]!

I took an hour and spun up this awesome asynchronous beast of a stream splitter. There is an optimisation that could be applied to it: Reads can be performed at the same time as writes. I figured that for a 1.0 implementation, this was good enough. If anyone wants to try their hand at making the read perform in parallel with the writes, feel free. Patches are welcome ;)

[0] http://tirania.org/blog/archive/2008/Sep-24.html
[1] http://www.monsoon-project.org/jaws/data/files/TeeStream.cs

EDIT: There's also a 'deliberate' bug there. 10 points to the first person to spot it and bonus 10 points if you can fix it with less than 5 lines of extra code.

Captchas, a bridge too far?

2008-09-24T02:03:00.003+01:00

I was just trying to open a new gmail account when I encountered the unsolvable captcha. I couldn't for the life of me figure out what was written in that horrible mangled imagine of what i can only assume were standard roman-alphabet letters. I can only assume that I am in fact illiterate and that would explain why I couldn't read it.

Thankfully, google have a second option. They have a little link which reads out the captcha for you. It turns out that I can't understand english either. I couldn't make out a word of what was said. So, if anyone out there can speak english, could you please decypher this for me:

http://monotorrent.com/GoogleSucks.wav

It lives!

2008-09-21T13:08:00.002+01:00

Revision 113625 - Enable DHT by default in MonoTorrent.

Awesome!

How verbose is too verbose?

2008-09-20T00:18:00.006+01:00

A common thing to do in code is to perform an action on each element in an array of objects. In C# there are two main ways to write this:

// Lets just assume this list of strings has been populated with lots of strings
List<string> allStrings = GetLotsOfStrings ();

// Method 1: The for loop
for (int i = 0; i < allStrings.Count; i++)
DoStuff (allStrings[i]);

// Method 2: The foreach loop
foreach (string s in allStrings)
DoStuff (s);

However, both of those methods are far too verbose. There is another way, which is much much nicer!

allStrings.ForEach(DoStuff);

How awesome is that, eh?

MonoTorrent is teh awesome

2008-09-03T18:21:00.002+01:00

How many lines to host a torrent server?

Any ideas?

Guess!

Check it out yourself

So what happened in SoC 2008?

2008-09-03T06:02:00.003+01:00

So, after ~3 months of hacking while travelling up the east coast of Austrlia, what exactly have I managed to accomplish in this years SoC? Well, quite a lot :) Here's a list of new stuff and upcoming stuff in no real order:

DHT

DHT support is available in SVN. It's pretty much complete, but lacks some real world testing. There are currently about 35 NUnit tests covering all important modules in the code. I need to give this a week or two of solid testing and then I'll be enabling it by default. A few updates will need to be applied to MonoTorrent so that the 'private' flag will be obeyed now that we have DHT support.

IP Address Banning

Awesome support for this has been added to SVN using a combination of my own code and code written by The Great Bocky. It uses an extremely efficient way of storing IP Address ranges so that they can be represented by two integers plus a little overhead. There is a parser which supports all the main ban lists and users can parse other formats manually and add the addresses in that way.

Extended Messaging Protocol

Support for the LibTorrent extension protocol is complete. This allows custom messages to be sendable to remote peers over the standard bittorrent protocol. So if you require the ability to send arbitrary data to a remote peer and have them react in a special way, you can!

Http Seeding (Web Seeding)

Support for the getright style Http Seeding is complete. This (better) specification allows a standard HTTP server act as a seed with no special software required. If MonoTorrent decides that there aren't enough peers available in the swarm to allow the torrent to complete, it will automatically start downloading the necessary chunks from the server.

Peer Exchange

Support for this has been completed by Olivier Dufour. This pretty cool idea allows peers you are connected to to send you details about other peers which are active in the swarm. This way you gain information about more peers even if the tracker goes offline.

Misc

Faster SHA1

Due to both an algorithm change and architecture changes, hashing performance has nearly doubled. This means that hashing a file takes less than 1/2 the time it used to *and* that CPU usage while downloading is reduced.

Better Encryption

There was a bug in header-only encryption that prevented it from working correctly before. As MonoTorrent always defaulted to Full Encryption, this wasn't such a huge issue before. Along with this bugfix, encryption is now more performant than before - using less CPU and less memory. The code also shrunk considerably in size and is much more maintainable than before.

Deadlock Free

It's now impossible to deadlock the library. This isn't so important for the end-user, but for anyone programming with MonoTorrent it's great news. If you are extending the library to add extra functionality internally, it's now easy to ensure that you do everything in a thread-safe and non-deadlocking manner.

Abort long connection attempts

Sometimes an operating system might wait an incredibly long time before aborting a connection attempt. This meant that if MonoTorrent tried to connect to a peer that was no longer available, sometimes the OS would take up to 150 seconds to abort the attempt. Worst case scenario is that the first 5 peers you connect to all take 150 seconds to abort and it looks like MonoTorrent is doing nothing. Now MonoTorrent hard-aborts a connection attempt if it takes more than 10 seconds.

Streaming torrents ahoy!

Two guys Karthik Kailish and David Sanghera have created a new way to download a torrent with MonoTorrent. Generally speaking, the rarest piece of a torrent is downloaded first, then the second rarest and so on. This new code allows you to specify a range of bytes which is High, Medium or Low priority. Then, from within these ranges the rarest first algorithm is active. For example if you are a video player and you want to start playback from byte 1000, you can tell MonoTorrent that the range from 1000 -> 5000 is important so those bytes are downloaded first, which allows you to start playback as soon as enough of that high-priority data has arrived.

Unit Tests

The number of tests covering MonoTorrent has doubled over the summer, from 55 up to 111. Every test makes the liklehood of accidently introducing a bug less and less. I like tests ;)

Banshee Plugin

Finally, while not quite related to MonoTorrent itself - A plugin for banshee has been created which allows the downloading of torrent based podcasts. It's still a work in progress, but hopefully that can be cleaned up and completed pretty soon.

So with all these changes and features, i'm hoping to push out the next release of MonoTorrent by the end of september. This release is unlikely to include DHT, but I hope to have a second release shortly afterwards which will include DHT.

Anyway, I'm off to pack my bags now so I'm ready to head to Ha Long City at 7am in the morning. I'm enjoying my last week in Viet Nam at the moment. It's been a blast, though sometimes i wonder if they deliberately decide to not understand what I say just because I'm mispronouncing it slightly. 'Ho Chi Minh' isn't *that* hard to understand, is it?

Socks proxies in .NET

2008-07-17T08:09:00.004+01:00

I'm just wondering if anyone out there knows a library which has a nice MIT/X11 (or similar) license which will allow me to use a socks based proxy. Alternatively, 10 points to the first person who writes the wrapper to do it ;)

I had found one before, but I seem to have misplaced the link and google is failing me now.

EDIT: Ideally it'd be available in source form so i don't have to bundle a binary. I *really* don't want to bundle a library if I can avoid it.

MonoTorrent 0.4 and Monsoon 0.15

2008-06-20T22:42:00.004+01:00

MonoTorrent 0.40 has been released. There weren't many changes feature-wise, but there's been quite a lot of under the hood changes. Details can be found on www.monotorrent.com

Also, Monsoon 0.15 has been released. The release notes are available and your packages can be gotten from here.

Fun times, eh?

MonoTorrent 0.50 is slated for a few weeks time (don't hold me to this). There are a bucket load of features in the works which will definitely kick some ass. I've been getting some great patches recently from Olivier Dufour, which he has detailed in his post. These should all make 0.50. I've also been getting some awesome work from Karthik Kailash, and his friend David (whose second name i can't find now), implementing a fancy debugging GUI which exposes all the internals in a nice GUI to make it easy for me to detect bugs/issues. He's also implementing Ono support, which helps get faster transfers; Bit-tyrant like unchoking which prioritises peers who reciprocate data resulting in faster transfers along with a new Piece Picking algorithm which allows you to stream a media file via torrent efficiently.

I'm unsure how many of those features will hit 0.50, it depends on when they hit SVN and how much testing i can get in. But hopefully a few of them will get there.

MonoTorrent - Expanding your universe

2008-06-11T16:14:00.002+01:00

As i'm sure everyone has heard at this stage, Banshee 1.0 has been released. It's a huge step up from the old 0.13.x releases, and well worth checking out!

So, now that banshee has some kickass podcast support, along with video support, wouldn't it be nice if you could download video podcasts which have a .torrent payload?

Wouldn't it be awesome if there as a .NET based torrent library, that maybe was exposed via a DBus service that could be integrated with banshee with just a few lines of code. Of course, once you've integrated the actual torrent downloading, how do you make banshee realise that .torrent files need to be handled specially? Well, write another few lines of code.

So all in all, because of banshees awesome extension framework, i wrote less than 200 lines of code of banshee code to enable banshee to download torrents. I was surprised by how easy everything was. I was up with creating the new extension within about 10 minutes. So, if you're interested in this extension, attach yourself to the bug report and you'll be able to keep up-to-date with the latest happenings.

After all this, what exactly does it look like when you download a torrent podcast? Well, it looks exactly like it does for a regular podcast download. You don't have to do anything special, it's all just MAGIC! Check out the screencast.

Titles!

2008-05-25T21:42:00.001+01:00

Why the hell did my blogspot account have titles disabled by default? Did i stupidly manage to turn them off the day i opened my blogspot account?

Anyway, i have titles... finally!

2008-05-15T00:43:00.003+01:00

The Summer of Code is starting soon, and I'm a student once again. So what are the big plans this year? Well, hopefully a lot! Here's a brief outlook on what to expect during this summer. Note, these are in no particular order

MonoTorrent
1) DHT support in MonoTorrent. This is probably the most time-consuming feature that is planned.

2) DBus based daemon for monotorrent. The idea behind this is to allow applications to consume torrent files without worrying about interop-ing with .NET. This would provide a system-wide torrent service which any application, or many simultaenous applications, could take advantage of.

This daemon will expose a simplified API as compared to the monotorrent library itself.

3) HTTP/Socks proxy support.

4) A proper NUnit testing framework. All the essentials are now implemented in MonoTorrent to allow me to test stuff deep inside monotorrent relatively easily. Now i just need to implement my test harness and then some tests (work on that has started as of yesterday actually :) ).

5) Implement support for both the Azureus and Libtorrent messaging protocols. Support is mostly there for the Libtorrent protocol. This will also allow user-defined messages to be sent via the torrent connection.

6) As per usual, i'll be spending time with a profiler seeing where performance can be improved. Don't expect too much from this though, things are already pretty good ;)

Monsoon
Now that we have a sweet GUI for monotorrent, which is going to be available in Suse 11.0 (other distros also have packages these days), we need to keep improving it. Once again there are in no particular order:

1) Get Monsoon portable to both MacOS and Windows. 95% of this work has been completed already. So it's nearly done. Once this has been completed, i need to look into packaging installers for these platforms (or someone can offer to do that for me ;) ).

2) Next we get cracking on the buglist and try and resolve those. Quite a number of cosmetic things need to be looked at and there are also a few bigger issues, such as that memory issue (which is still proving to be quite elusive!).

3) General nicing up of everything. There isn't really a firm plan for monsoon yet. A more detailed timeline and suchlike will be created as soon i can get together with buchan after my exams and see where we want to take this.

All in all, a busy summer.

2008-05-09T14:16:00.003+01:00

It's funny. I always thought that DRM should be used as a way to make life difficult for people pirating software/music. I never though it would be used as a way to convince people that pirating software is the better way:

http://www.theinquirer.net/gb/inquirer/news/2008/05/07/mass-effect-wins-award-worst-pc

Seriously, what are these people thinking?

2008-04-30T11:39:00.009+01:00

I haven't done a mono-specific post in a while, so i'm glad i can change that with some good news! A few days ago, Scott Peterson, (who took part in the Summer of Code last year to port banshee to windows) was complaining that monsoon took an hour to hash a 60GB torrent. I was a bit surprised at this, because it shouldn't be that slow! I did know that SHA1 hashing in mono was a tad slow, but no way should it be *that* slow.

Anyway, he decided to fire up a profiler and get cracking on optimising the hell out of the SHA1 class in mono, and i decided that i'd skip some study and do the same. We had two main aims in this:

To make the fastest possible implementation using no unsafe code
To make the fastest possible managed SHA1 implementation

Aim 1:
In order to do this, we couldn't use unsafe code, everything had to be done without pointers. We also had a constraint that we couldn't make the size of the class significantly longer, in fact, *reducing* the lines of code was also an aim because the more lines of code, the slower it is for the JIT to process.

We did a number of iterations on the code, testing different things out. We fully unrolled the three main loops, we partially unrolled some of em and benchmarked everything in between. The things we learned in the end were:

1) In this particular algorithm, there was a great benefit to using local variables rather than fields. That gave a biggish boost. EDIT: This is because mono doesn't perform array out of bounds check removal (abcrem) on fields. It only performs it on local vars. As array access is *very* frequent, removing these checks is a huge benefit.

2)

buff[i] = ((buff[i-3] ^ buff[i-8] ^ buff[i-14] ^ buff[i-16]) << 1)
              | ((buff[i-3] ^ buff[i-8] ^ buff[i-14] ^ buff[i-16]) << 31));

The above code actually performed significantly slower than:

uint temp = buff[i-3] ^ buff[i-8] ^ buff[i-14] ^ buff[i-16];
buff[i] = (temp << 1) | (temp >> 31);

)

3)Re-rolling some of the loops did not affect performance significantly. The performance delta between when all three of the loops were fully unrolled and when two of them were only partially unrolled was less than 6% on my system, but the IL was reduced by a fairly massive proportion.

4) Massive methods perform slower. We found that by splitting the 'ProcessBlock' method up, performance increase noticeably. Bear in mind that when we originally tested this, we had all three big loops fully unrolled and they were all in the same method. Still, it's worth bearing in mind.

So, what was the final performance delta with all these changes?

Core 2 Duo T7400 @ 2.16GHz
2.54x faster

Athlon 64 3200+
3.10x faster

Pentium 4 @ 2.80GHz
1.36x faster

Xeon X5355 @ 2.66GHz
1.38x faster

64bit PPC:
1.60x faster

EDIT: It seems I've mixed up my numbers when i was recording them yesterday. Some of the numbers compare against SVN head and some compare against the version in Mono 1.9. I think the first 2 results are against the version in 1.9 and the last three are against SVN head.

Not too shabby.

Aim 2:
More to come on this later, we're still in the optimisation process, but i think it's fair to say that the unsafe version is quite a fair bit faster thant the safe version so far.

2008-04-21T22:40:00.003+01:00

So opensuse 11.0 beta 1 has been released. I decided i'd be adventurous and give it a whirl, so i fired up my mac and downloaded the ISO while i was in college. When i got home i realised that i had never actually burned an ISO on my mac before, and there didn't seem to be any built in software to do it.

So, i fired up google and expected to have to download trial software to burn the image and suffer all sorts of hassle and annoyance. I was pleasantly surprised to find that it was frikin simple to burn an iso:

$ hdutil burn image.iso

It's now 5 miutes later, and i'm just about to reboot into the Live CD environment, very nice!

2008-04-19T00:16:00.002+01:00

MonoTorrent 0.30 has been tagged and released. All i have to do now is update the website. Here's the short changelog.

Highlights include:
[Client]
* Amazing extensibility - Active connections can be injected from any source you want. So if your application already has an active connection to a peer, that connection can be passed to MonoTorrent and it will be used.
* Udp Tracker support implemented - Can now use the low bandwidth udp protocol if the tracker supports it.
* Initial support for the libtorrent messaging protocol
* Implemented semi-intelligent memory buffer to reduce disk reads/writes.
* Fixed several race conditions when Stopping/Unregistering torrent managers.
* Fixed issue whereby monotorrent would stop connecting to new peers
* Can now handle torrents with 1,000's of files gracefully.
* Implemented IPV6 support.
* 15% faster hash checking
* Enhanced the accuracy of the ratelimiting code when a global limit is applied
* Per-file progress correctly updated when FastResume data is loaded
* If a file is set to 'Do Not Download', it now will definitely not be downloaded.
* Abort a connection attempt if it takes more than 10 seconds to complete. Some operating systems default to 3 minute timeouts which kills performance.
* Some speed and memory enhancements, as always.

[Tracker]
* Correctly removes zombie peers (peers who crash before telling the tracker they're stopping)
* Added ability to compare peers based on an arbitrary key rather than only based on IP.
* Minor speed and memory enhancements

A precompiled binary can be found here.
A tarball can be found here

Coinciding nicely with this is the release of Monsoon 0.11.3. The changelog looks something as follows:
* Adding/Removing/Renaming of labels is completely context-menu driven now
* Can drag and drop torrents to add and remove them from a label.
* Fixed several issues persisting state across application restarts
* Global rate limits can be set by clicking on the labels which display the global download/upload rates
* Now takes advantage of the new FastResume API.
* Correctly invoking libgtk and libX11.
* Now supports nat-pmp through the use of a newer mono.nat
* Multi-select enabled in the file view
* When creating a torrent, hashchecking is skipped if you choose to seed it immediately
* You will always be prompted if you choose to remove/delete a torrent
* Made Monsoon fully translatable
* Added tooltips to the main items

Monsoon can be gotten via 1-Click install or the GNOME:Community repository:

2008-04-08T18:30:00.004+01:00

So, some quick news on the MonoTorrent front. As i said a week or two ago, Monsoon is going to be part of the Suse 11 distribution (woo!). Since then a few things have happened.

1) Monsoon has hit feature freeze and has been branched. This is the version that will be included with suse. If you want to check out the code and put it through it's paces use this url:

http://anonsvn.mono-project.com/source/branches/monsoon-0.11

If you find any bugs, please use the novell bugzilla to file a report. Some time early next week, this will be officially tagged and released. So make any bug reports earlier to increase the chance that they'll be fixed.

2) MonoTorrent itself has also been branched for it's 0.3 release. If you're a developer using MonoTorrent, check the code out from:

http://anonsvn.mono-project.com/source/branches/bitsharp-0.30

Bug reports are welcome (as always), same place as above (Except use the MonoTorrent module, not the Monsoon module). I'll have full release notes available when the release is made. Some pretty cool stuff has been done along with the usual bug quashing ;)

3) Finally, mono-curses has been updated again to run against MonoTorrent 0.30. So if you want a slick cool ncurses GUI for MonoTorrent, check the code out from:

http://anonsvn.mono-project.com/source/trunk/mono-curses

Finally, I just want to add: All Your Torrent Are Belong To Us - Use Monsoon! ;)

2008-04-04T10:24:00.002+01:00

There was a meeting yesterday for openSUSE Gnome, one of the important decisions for the day (in my eyes) was the decision about which BitTorrent client was going to be bundled with suse.

Torrent default app decision:
Monsoon seems the more dynamic app, good response from its maintainer
Transmission seems to have a better UI, actively developed, although a bit mac centric
BitTorrent-gtk very basic but should work for basic needs
AI: add both monsoon and transmission, monsoon as default
AI: suseROCKS to run tests with both apps
AI: FunkyPenguin to package Transmission today
AI: vuntz to move Transmission and monsoon to autobuild and drop gnome-btdownload

We're in :)

Monsoon had an open bug report on making it translatable, after the above decision was made, the priority on translations became pretty critical ;) Meebey volunteered to go and get translations all set up and spent a few hours yesterday getting that all sorted and creating the first translation (German). Olivier Dufour, who created a Winforms based GUI for MonoTorrent has also volunteered to do a French translation.

So, if anyone out there wants to translate Monsoon into their native language (or at least one they're good at ;) ), please join us on our new irc channel, #monsoon on irc.gimp.net/irc.gnome.org. We'll get you sorted out.

2008-03-29T15:30:00.005Z

I got some amazing news just a few minutes ago. I was just chatting away in #mono about how i was pushing a new release out soon, when anf6 (who is also called Alan) piped up with this:

|anf6| alan: I've been working on a WebUI for MonoTorrent, most of the functionality works ^_^
|alan| anf6: are you serious?
|anf6| yes
|alan| :o

It turns out that anf6 had been chatting to the developer of the uTorrent WebUI (Directrix) about the possibility of creating a MonoTorrent backend for the UI. Directrix was all for that! So, without further ado, here's the obligatory screenshot:

All i can say is: WOW! I can't wait for this to hit release!

UPDATE: I'd just found out that less than 300 lines of code were needed for the monotorrent -> uTorrent WebUI integration. How slick is that? That includes hosting MonoTorrent and a mini asp.net host to serve the pages.

2008-03-28T00:27:00.004Z

A few days ago, i heard the suse team are looking for a new default torrent client for Suse 11.0. So of course I jumped in saying "Well, why not Monsoon?". So i fired off a response with a ton of links suggesting that Monsoon was the most amazing torrent client ever and there was no need to review any other clients. Of course that didn't work, but Monsoon has been taken into consideration as a viable candidate.

Andrew Wafaa took it upon himself to do a little review of his own, and Monsoon fared pretty well. I was happy enough with that. Some of the points mentioned on the thread include:

1) Applications which need wizards are bad. For a default client, you want something rough and ready.
2) Something minimalistic but packed with features is good. No massive complicated apps like Azureus.

Of course, 2 is a bit of a contradiction. It's quite hard to get both. In GUIs to expose a feature or some statistic, you need a new GUI element. So there's a fine balancing act between keeping the GUI clean and small while still exposing all the nice features in an easy to use way.

Here's a few screenshots of Monsoon anyway, which i hope expose the full diversity of what it can do.

The first time you start Monsoon, you aren't greeted by a wizard, everything is set up nicely for you. Everything is shown to you by default.

When you load torrents, they automatically start. If the files already exist (like the kubuntu one did), monsoon check that file to see how much of it is valid data and then resume the download using it. Standard stuff.

The ramp-up time is fairly fast. After 30 seconds Monsoon can connect to 50 people (the default limit for a torrent). Note, it is smart in how it connects to people. It doesn't have more than 5 pending connection attempts at any one time. This prevents overloading of cheaper routers.

For the people out there who want a really minimalistic client, you can hide everything. The GUI can be reduced down to something even slimmer than Transmission should you so wish. There is also plans to implement a mini-mode, which will consist of just a single progress bar per torrent.

So what if you want to set the global upload/download rate? Why, it's simple! Just click on the global upload/download speed indicators and away you go.

If you want to organise your torrents, it's just a simple context menu. You can add/remove/rename the labels with a right click, or double click. Slick, eh? There is also a patch in the works which will allow you to drag'n'drop torrents from the main view into a label and drag'n'drop them between labels. This will replace the existing way of doing it which involves ticking checkboxes in the 'Options' menu.

2008-03-26T21:37:00.002Z

There's now a proper issue tracker for Monsoon and MonoTorrent. It's being hosted on the novell bugzilla (thanks guys!) so file your bugs there if at all possible. It makes it easier for me to monitor and record what exactly goes on.

The issue tracker can be found at in the 'Monsoon' section of Mono:Tools. If you have a bugzilla account already, this link should bring you right to the submission form:

https://bugzilla.novell.com/enter_bug.cgi?classification=&product=Mono%3A+Tools&submit=Use+This+Product&component=Monsoon

2008-03-09T00:57:00.002Z

This is just a shout out to any budding Artistes, (especially that guy who did all those cool looking MonoDevelop icons): I'm looking for a nice logo for Monsoon. I have no ideas in mind, so any suggestions are welcome. Drop a comment here or send me an email if you have anything to show me :)

2008-03-09T00:06:00.003Z

This is just a quick follow up from my last post. I decided to do a few benchmarks to see what transmission was like as compared the Monsoon (the GTK# gui for the MonoTorrent library). I've heard about tranmission before, but i've never used it. I was pretty shocked by the results. This is a quick summary of what I found:

1) Memory usage.
Transmission used less memory than MonoTorrent. That came as no surprise to me. No matter how efficently I code, i cannot ever get as efficent as an app written in C/C++. Mostly because in a .NET based app, the .NET runtime/jit must be loaded which consumes a few MB of memory. Percentage wise, yes, the difference is huge, but if you take a look at overall memory, it's not massive.

MonoTorrent: 35 Res / 20 Shared
Transmission: 20 Res / 13 Shared

This figure was gotten after extensively using the GUI by opening menu's and flicking around pages.

2) Hashing performance
I suppose another important metric - how long does it take to hash a complete file.
MonoTorrent: 95 seconds
Transmission: 85 seconds

This difference is fairly negligible. A full hash will rarely be performed. However, i suppose it was worth measuring. One optimisation i could make in monotorrent which would reduce that gap slightly would be if i read the next chunk of data off disk asynchronously while hashing the current chunk. At the moment it's all sequential, but i suppose it could easily enough be made parallel. It shouldn't make a huge difference though.

3) Download speeds
This would be the most important metric. This is where the biggest surprise came.
MonoTorrent: 15 seconds - 400kB/sec, 30 seconds - stabilised at 550-600kB/sec (maxed my connection) and connected to 50 people, the maximum allowed by my settings.
Transmission: 15 minutes: still at less than 50kB/sec and still only connected to 6 peers.

What am i doing wrong with transmission to make it so slow? It's not NAT (even though transmissions uPnP support cannot detect my uPnP enabled router) because i manually forwarded it in the end. I was using both svn head (r97353) of MonoTorrent and svn head (r5227) of Transmission when i ran this quick test.

EDIT: Just as i finished this, Transmission managed to connect to 3 additional people and one of them had a massive upload capacity which let Transmission reach ~480kB/sec. Still, why did that take so long? These results were consistent every time i started/stopped both transmission and Monsoon. Monsoon consistently maxed out my connection quickly whereas transmission consistently took forever to even break 40 kB/sec.

UPDATE: I just want to add that i tested using the ubuntu-7.10-desktop-i386.iso torrent on Suse 10.2.

2008-03-04T01:28:00.004Z

So, a thought struck me. Ubuntu currently has a horrible default bittorrent client. It's about to be replaced by Transmission which, while definately a step up in the world, is still feature lacking (in my opinion) as compared to other available clients, most especially the great work done on the GTK Gui for MonoTorrent.

I propose that that the MonoTorrent GUI should be bundled with Ubuntu. MonoTorrent supports everything transmission does (except for Peer Exchange[1] and scheduling[2]) along with these extra cool features:

* Fast Peer Extensions - allow you to start a torrent faster
* Udp Tracker Protocol - allows you to use the bandwidth saving UDP protocol for tracker communications
* LibTorrent Extension Protocol - Ok, this is only partially supported. While it's implemented in SVN, it's not fully tested or enabled by default yet.
* RSS Feeds - Automagic downloading from RSS feeds. (see video)
* Organise downloads into tags (see video)
* Multi-Tracker Protocol support - If the main tracker is down, no worries! You can use the backup ones!
* Retarget Files before/while they are downloading i.e. you can rename a file as you download!
* Can monitor a folder and automatically load and download new torrents
* Minimizes to the gnome notification area.

The GUI including all supporting libs is a 384 kB package, and i think in the region of 600kB when installed[3]. What we're looking at is 600kB for a fully fledged, feature rich, uber snazzy bittorrent client. The best thing is [b]all[/b] the dependencies are already on the live CD.

So, if there are any Ubuntu devs out there, would you like to consider MonoTorrent as the default bittorrent app? If not, why not? I'll gladly do my best to fix any issues you have. Leave your comments below and lets get some discussion going!

[1] It'd take a few hours to complete support for this with tests.

[2] I did actually receive a patch to implement this a long while ago, but in the end, it fell by the wayside. I am a horrible person :(

[3] The package can be reduced a fair bit by splitting out the Tracker component of MonoTorrent using the Linker and removing bundled libraries which are available as packages nowadays. I'd guesstimate at least 100kB can be saved from the installed size from these optimisations. 70kB by removing the bundled DBus stuff and at the very least, 30kB by shrinking MonoTorrent.dll.

UPDATE: I'd just like to point out that an official name has been chosen for the GUI: [b]Monsoon[/b]. The next release will have everything rebadged to this name.

2008-02-27T02:54:00.004Z

Its that time of the year when all us Europeans gather together for a big competition to decide who has the most friends. It's called the Eurovision. Last year, we did spectacularly. We were within inches of achieving our goal of finishing with zero points until those albanian b******s decided to give us 5 points. I swear, that was our plan all along!

Anyway, we learned our lesson last year. This time we're going to put in an entry that simply cannot fail! Ladies and gents, i present to you, Dustin The Turkey, singing our Eurovision entry. Save us all.

2008-02-22T00:27:00.002Z

After my rather lengthy post about all the cool stuff I was doing with MonoTorrent, i was asked this question:

I'm very curious about what are the real possibilities of MonoTorrent library. I mean, I always thought Bittorrent was just a protocol for P2P file exchange, but when I see so much modularity on MonoTorrent... The question is, how can MonoTorrent help me as a programmer?

Well, that's a pretty good question.

I'm going to talk briefly about the three important new features that are (or will soon) be available:

1) Custom Peer Connections
Aim: To allow the user to route all peer traffic over whatever medium they want.

Use case 1: You are writing an application which requires that all connections are encrypted end-to-end. This means you have complicated routines to set up the connections. You want to add the ability to transfer files.

Use case 2: You want to route bittorrent traffic over a network such as Tor.

Use case 3: Restrictive firewall. Suppose only certain kinds of traffic are allowed through a particular firewall you have. You can push the bittorrent traffic inside a different protocol to allow it to pass through the firewall.

In case 1 and 2, we assume that normally it is impossible for monotorrent to create the connections itself. So what you do is create the necessary connections manually, then wrap them in the IConnection interface and pass them directly into monotorrent. Everything else is automatic.

For case 3, it might be possible to implement a HttpPeerConnection, whereby you push the bittorrent messages inside of a HTTPWebRequest and then send that to the remote peer and he decodes it back into it's orginal form for processing. Note: I don't recommend you do this to avoid getting around a firewall in work. Also, if you do implement something like that, only other clients who implemented that feature would be able to communicate with you, but it is an interesting application.

2) Custom Trackers
Aim: To allow someone to add peer data into the engine manually.

Use case 1: You want to type in your friends ip:port so you can directly connect to him

Use case 2: In your application, you keep a list of people you can/should connect to, and you know that MonoTorrent can create connect to them directly.

Use case 3: You want to implement an alternative peer source like the bittorrent DHT protocol

In all these cases, you just have to implement two simple methods to allow MonoTorrent to get data from your source. For case 1, whenever you enter the ip:port combination, you just raise an event with that IP:Port stored in the event args.

Case 1 is a manual event, and so MonoTorrent would never query that 'Tracker' for peer details, but for case 2 and 3, it's more fun. All you do is wait for MonoTorrent to call the 'Announce' method on your tracker, then you go off and find the peers and raise the AnnounceComplete event when you're ready. It's as easy as pie.

3) Custom Messages

Use case 1: You're lazy and don't want to define your own messaging protocol or message format and want things to 'Just Work (tm)', so just piggy back all your messages over the MonoTorrent system.

Use case 2: You want/need to send a few pieces of information between torrent clients but don't want to have to set up a separate communication channel to do so in.

Use case 3: You want to make Bittorrent 2.0.

You need to implement a slightly more complex interface than the previous two examples, but it's nothing amazingly difficult. The helper methods for reading/writing the basic types from a byte[] are all there to make creating a message that much easier. So you're left with only having to implement a mere 4 abstract methods. Whilest this feature isn't finished as yet, it will be as simple as defining your custom message, then making a call like:

byte messageId = 9;
Message.Register(messageId, delegate { return new MyCustomMessage(); });

or

Message.register(messageId, delegate (TorrentManager m) {
return new MyCustomMessage(m.SomeData, m.OtherData);
});

Once that's done, you can queue up your message to be sent and react to it's arrival with ease. What you might want to send, i don't know. But the ability is there for you to send it.

So in brief, MonoTorrent can be used as a drop in replacement for either '1 to 1', '1 to many', 'many to 1' or even 'many to many' transfers. I'm not saying it *should* be used for all those cases, it may be the heavy weight alternative to a simple Socket.Send, but the possibility is there. You can send customised messages to the other clients so that they can perform custom logic and you can insert peer details easily from any source.

2008-02-18T00:59:00.002Z

It was on 28th January that i released MonoTorrent 0.20. Here we are, less than a month later and already i have some cool new stuff to blog about, this is a lot better than the time gap between the last two MonoTorrent releases.

So, what's new? LOTS! Let me give a brief overview of what's possible nowadays.

1) Custom Peer Connections
It's now possible to plug monotorrent directly into another application and have monotorrent uses connections created by that application. The advantage of this is that you can tunnel MonoTorrent through anything now. You can create an encrypted connection whatever way you want and then just pass it straight into monotorrent and it'll be used. This also makes it easy to push traffic through networks like Tor (or whatever).

2) Custom Trackers
First, i'm not talking about the server aspect of MonoTorrent. I'm talking about the clientside class used to deal with announcing/scraping to a server. It's now possible to (very easily) implement your own tracker class. What use is this you say? Well, quite a bit actually! If you want to be able to add peer data into monotorrent (but not active connections as i described above) you can subclass the MonoTorrent.Client.Tracker class and do it from there.

Suppose you want to implement a new protocol for client-tracker communications. If you want MonoTorrent to support it, just inherit from MonoTorrent.Client.Tracker, register that implementation with the TrackerFactory and you're done! You don't have to modify one letter of MonoTorrent source code to add in support for your new protocol.

3) Custom Messages
Ok, so technically speaking, this still isn't possible yet, however all the architecture to enable it is there. I just need to expose it publicly in a nice way. Anyway, when i implemented support for the libtorrent extension protocol, I realised that the way i handled peer messages just wouldn't work. Basically i used a giant switch statement to select the correct message to decode. This meant that all messages had to be defined within MonoTorrent and had to exist at compile. There was no way for the user to define their own message type.

So, instead i updated the API to use some .NET 2.0 loveliness to allow for custom messages to be defined outside of MonoTorrent and still allow them to be decoded and handled correctly.

4) Custom Writers
If i was to pick a favourite change out of the list, it would be the custom messages, but this comes a close second ;) You can now define you're own 'PieceWriter' class. What this allows you to do is to redirect disk read/write requests wherever you want them to go. Of course, the logical first implementation is a MemoryBuffer.

Yes, MonoTorrent will now cache reads/writes in memory where prudent. For example, each 'piece' you download consists of multiple 'blocks'. Previously each block was written straight to disk as it arrived. When all the blocks had arrived, they were all read straight back off the disk so that a SHA1 hash of the piece could be generated. This was very inefficient.

5) NUnit test-able!
Yeah, so this isn't really a new feature. In fact, it's something i should've been doing all along, but due to the implementation of MonoTorrent, certain things were far too awkward to test. Previously, in order to test internal logic i'd have to:

Create a fake .torrent file and write it to disk.
Create fake data and write it to disk so that it can be 'downloaded' by one client and 'seeded' by the other
Instantiate a MonoTorrent.Tracker and load the torrent into it
Instantiate two MonoTorrent.Clients and load the torrent into them.
Hit 'start' and measure what little i could

This method means that it's quite possible temp files will be left on the disk - not good. Also, if something does go wrong, there is no way for me to find out exactly what went wrong. The Client would close the connection and all i'd get would be a ConnectionClosed event. Finally, it was just plain awkward! Creating a fake torrent and the necessary fake files was a pain in the ass.

Once the above changes had been complete, it became so much easier. Now I create a fake torrent in memory (no need to write it to disk!). I create one MonoTorrent.Client using the in-memory torrent and redirect its reads/writes using a custom PieceWriter. This writer just peforms in-memory operations to fill in the fake data as required. I then create a custom PeerConnection and pass one end of the connection into the engine and hold the other end myself.

This means when i send a message into the engine, i can receive the reply and verify that it is as expected using NUnit tests. Pretty sweet.

In other news, the following other nifty things have been implemented:

A) Udp Tracker support
I finally got around to implementing UDP tracker support. It has been requested by a few people, so here it is. Enjoy!

B) Better FastResume support
There were a few issues with the old fast resume support. In certain advanced use cases, fast resume support just plain didn't work. This has now been fixed. It's now up to the user to handle where/how fast resume data is stored. This is nearly finished, but not quite. It's on my TODO list for the coming week.

C) Message Bundles
An entirely invisible change, but one i want to talk about. Whenever a piece is downloaded succesfully, a 'have' message should be sent to all peers you're connected to. This is used as a way to keep track of who has what. However, have messages are tiny and typically you'd complete a few pieces a second. This means that several times a second, for each connected peer, a single 12 byte message has to be sent. This is hardly worth the effort of calling socket.BeginSend!

So, why not encode several messages at a time into the send buffer and send em all at once? Well, that just doesn't work. The architecture just does not allow for that to happen. One message and one message alone can be encoded and sent at a time.

The solution? The MessageBundle. The MessageBundle is just another implementation of the PeerMessage base class, except the message bundle can store multiple messages inside it. When you call Encode() on the bundle, it encodes all it's stored messages into the buffer. This allows me to send multiple messages at a time, but for all intents and purposes, MonoTorrent thinks i'm sending one.

Now HaveMessages are delayed and bundled together.

I'll release all this fanciness sometime in the near future.

2008-02-07T00:14:00.000Z

What kind of people do credit card companies like to give credit cards to? People who always pay their bills? People who occasionally forget and pay a few days late? How about people who forget quite often, or can't afford to pay the full bill in one month?

Well, it looks like the last group is exactly who Egg want! They have issued notice to 7% of their customers informing them that due to their bad credit rating, their credit card is being cancelled with no right to appeal. According to a fair number of customers, they have excellent credit ratings and always paid their bill in full and on time. It looks like they're being dumped because they're just not profitable.

It's kind of ironic that the 'best' customers are the ones being dumped in favour of customers who may not actually be able to pay their bills.

2008-02-03T16:41:00.000Z

So, after asking around about my new internet situation, i was informed that there was no way for me to set it up as i wanted to. I had to have my WRT54G as the slave and the new neatgear WGR614 as my master device. With that in mind, i reattached the netgear to my new NTL broadband and put the WRT54G back to it's old position beside the two wired computers.

Just in case anyone else needs to do something like this, this is how you create a wireless bridge between two routers:

Equipment needed:
1) A router which supports the 'Client' mode. In my case, a linksys WRT54G with the openwrt firmware, the 'slave' router.
2) Any other wireless router, the 'master' router.

Setup:

Master Router
1) Connect it to your internet box (be it an ADSL modem or whatever).
2) Set up the wireless connection as per normal.
3) Enable uPnP and DHCP as required.

This router will accept all wireless connections, and can also accept wired connections via it's LAN ports as per usual. All port forwarding should be done via this router.

Slave router
1) Connect any wired computers to LAN ports on the router. Make sure nothing is connected to the WAN port. You won't be using this.
2) Set the wireless to use the same SSID (network name) and frequency as the master router.
3) Set the wireless connection to use 'Client' mode. In the case of openwrt, the correct mode is called 'Client (bridged)'. In this mode, the router will look for the wireless network with the SSID you specified and connect to it like a regular laptop.
4) Disable uPnP and DHCP on this router.

Note: The slave router cannot accept wireless connections anymore. It can only forward wired computers to your master router.

With that done, my two desktops remain in the same old room as always and connect wirelessly to my master router, thus giving them access to the net and having them accessible to the wireless clients.

The only problem is that the new router is a bit limited in it's configuration and so will end up being replaced in the near future. For the moment, it does the job fine.

2008-02-01T22:01:00.000Z

I recently changed internet providers, and i have this situation:

The new internet arrives in via a cable in my front room. There are two desktops in the house, which are in the back room. The old internet came through a hole in the wall into this room, so everything was peachy.

I'm now in possession of a linksys WRT54G (with openwrt firmware) and a neatgear WGR614 V7. What i want to do is to bridge the two wireless routers so that i have a single network containing my wireless+wired computers.

Does anyone know if there's a modified/alternate firmware for the netgear router which supports 'client' mode in wireless networks so i can just make it connect to my main router (the linksys). I'd much prefer to use this as my main router as opposed to the netgear. Any advice on getting this set up would be great. Ideally i wouldn't have to buy new hardware, but if it comes to it, thats what i'll do ;)

2008-01-28T13:28:00.000Z

MonoTorrent 0.20 has just been shoved unwillingly out of it's cosy spot in SVN into full public view. The last time i tagged and released was way back on July 4th. Far too long ago! Full details of the release can be read here. Download links here aswell.

Here's the changelog:
Client
Features
Linear piece picking possible
- This mode of operation chooses pieces towards the start of the file where possible. This should not be a publicly settable option in a GUI. It could be automatically set until the first 20 pieces in a torrent are received to allow for previewing. For general usage, this is a very inefficient way to download.
Full choke/unchoke algorithm implemented (Andy Henderson)
- Full tit-for-tat algorithm implemented. This improves download rates and upload rates and is pretty slick.
Sub-optimal implementation of code used to connect to peers made optimal
- Fixed corner case where a connection wouldn't be cleaned up correctly
- Significantly faster startup performance in cases where remote peers don't support encryption and encryption is enabled
Rate limiting for Disk IO implemented
Configurable amount of open filestreams
Reduced disk thrashing when multiple torrents hashing simultaneously
Files are not preallocated anymore
TorrentManager will never block when .Start() is called
The Tracker base class made more extensible (Eric Butler)
- An example usage would be if a person had peer details stored in a database. They could implement a 'DataBaseTracker' class, register it with the engine, then monotorrent could announce to that tracker and retrieve peers from the database. Thanks to Eric, this is hugely simplified and is now trivial to implement.

Bug Fixes
Piece Requesting
Fixed bug where a piece could be requested twice
When a corrupt piece is received, ensure all contributing peers are marked
Fixed a possible null ref when removing pending piece requests when a connection is closed

Torrent Creator
When creating an (optional) MD5 hash in the torrent creator, open the files in read-only mode (Roger Zander)
Creating torrents no longer spins up an extra thread when using the synchronous method (Eric Butler)
Fixed possible race conditions when creating a torrent using the asynchronous methods
Private key is retained when creating torrents

Misc
Per-file progress is updated correctly.
Fixed detection of when a tracker supports scrape - It should have been case insensitive
Added extra check to make sure a scrape request is only performed if the tracker supports scraping
Added extra logic checks to help prevent the loading of incorrect fastresume data
When a peer sends a HaveAll message - it is correctly marked as being a seeder
PieceHashed event fired under all circumstances

Tracker
The MonoTorrent.Tracker got a major rewrite. Highlights include:
- more than 1000x memory reduction for trackers hosting a large number of torrents. 1000 torrents can be hosted in ~30kB. Previously would have required ~30MB.
- Faster more efficient announce handling. For large trackers, lookups are significantly faster
- Vastly simplified announce handling. It is now trivial to write code to handle incoming connections from differing sources. A typical example would be to create an AspNetHandler to handle connections in an ASP.NEt project. It requires less than 20 lines of code to be written.

It should be fairly solid, with a lot of cool feature enhancements under the hood. There are a few cool things which i managed to implement this time round like rate limiting disk IO, not pre-allocating 100% of the file on startup, being able to properly handle a torrent with 100,000 files in it. Good stuff.

Plans for the future are pretty cool aswell. I'll blog more about that later though.

2008-01-26T15:42:00.000Z

Today i'm going to talk about optimisation a little, this is mostly in the MonoTorrent Tracker context, but some of the ideas still apply to other situations.

Sometimes you hear people talking about how they want to optimise their code to make it faster, and they ask questions like 'what's the fastest way to multiply by 2, bitshift or regular multiplication?', or 'Should I null out objects the very second I'm finished using them or just wait for them to fall out of scope?'. These kind of questions are probably the epitome of premature optimisation. These kind of optimisations won't make your application faster or better.

I had done a lot of work on the Tracker code (the 'server' portion of the bittorrent specification) recently. I had precomputed values which are used regularly, i had optimised the hashcodes used in my dictionary lookups, i had reduced the amount of data that needs to be kept in-memory significantly. I wanted to see what effect all this had on the actual running of the tracker. This was the first time i had run a benchmark on the tracker.

I went on the net, found a big tracker and checked it's stats. Using these, i decided that this was benchmark was representative of a heavy real-world load:

1) Load 2000 torrents into the engine, each of which contains 1000 peers.
2) Hammer the server with 1000 requests a second choosing a random torrent and random peer from the list and make a fake request from them to the server.

So, once that was written, i fired up the tracker and ran the benchmark. My system locked up and i was forced to hard-reboot. What had gone wrong! I started the benchmark again, but monitored memory and CPU usage carefully. I was surprised to find that memory usage was rocketing, which is what caused the massive slowdown of my system! I couldn't understand why. I did a few quick calculations to figure out how much memory i'd expect the tracker to use, they put final memory usage at far less than 300MB. I quickly whipped out my allocation profiler and began optimising. Here are the 'issues' i fixed:

1) The objects i was using as the 'key' in a dictionary were being recreated every time i used them. Typically this means that for every request to the tracker, at least a dozen complex objects were created/garbage collected needlessly. In this kind of scenario, the objects should be declared as 'static readonly' and reused. I implemented this change.

The benchmark still couldn't run.

2) I decided that the next problem was that i was pre-generating two byte[] for each peer when they were added to the server. This was so that a request could be fulfilled by simply copying pregenerated byte[]. I changed this to generate the byte[] at request time rather than storing it in memory. I expected this to fix the issue.

The benchmark still couldn't run after this change.

3) Finally, i noticed there were a huge number of hashtable related objects being retained in memory. This was a bit weird. There shouldn't have been that many around. A few minutes of checking the code made me realise that the probably cause was keeping a NameValueCollection object in-memory for each peer. I rewrote the peer class to extract the necessary information from the collection and then dump it, rather than holding a reference to it.

The benchmark could run!

Now, the memory improvements were gigantic. Previously i had stats like this:
Active Torrents: 500
Active Peers per Torrent: 500
Memory: 350MB

Now i had:
Active Torrents: 500
Active Peers per Torrent: 500
Memory: 40MB

Active Torrents: 2000
Active Peers per Torrent: 1000
Memory: 140MB

There is no way in hell I'd ever have found the cause of the issue unless i had run a profiler. So, for anyone who is trying to make their code run fast and efficiently, you need a profiler. You can't get away without it.

2008-01-10T16:18:00.000Z

As you've probably heard by now, Aaron has just released a new version of Banshee has just. Good stuff! As part of the blogpost detailing the changes and features in the new version he says:

Alan McGovern has nearly completely rewritten our MTP support and we are close to enabling it by default. The next release should have solid MTP support, and we hope to roll 0.13.3 within the next two weeks. The new MTP support uses libmtp instead of libgphoto2. This decision was made for a number of reasons, though I am really not informed enough to convey them properly.

Well, i never detailed the reasons for the switch in my previous blogpost, so i may as well do it now. I'll also mention the current issues with libmtp for those of you interested in what those issues are.

Reasons for the switch:
The primary reason is that to use libgphoto as the mtp backend, you need svn head of libgphoto and svn head of libgphoto-sharp. This issue is actually a pretty major issue. If banshee wanted to ship MTP support, it would have to ship it's own version of libgphoto and libgphoto-sharp. The other alternative was to enable MTP support by default, but the end-user wouldn't be able to use it until they compiled and installed libgphoto themselves or their distro packaged the required version of libgphoto. This version is likely to be released in the next month or two (i believe), but could take several months to propagate through the various distros.

With libmtp, banshee can run with version 0.2.0 or higher, which means anything newer than Aug 4th 2007. This is a much more workable solution.

Secondly, API: The libmtp API is aimed at mp3 players. Banshees API is also aimed at mp3 players, this makes it significantly easier to map the banshee API to the libmtp API. Several tasks such as getting track listings, uploading tracks, and retrieving tracks are much simpler in libmtp.

Thirdly, with libgphoto, downloading or uploading a track requires copying the entire file to memory in C#, then pushing that into libgphoto (which duplicates the memory usage) then the data is pushed to the device. There is an alternative method whereby i create a unix file descriptor which is then passed into libgphoto and the file is read/written to that. However, this is not ideal as there's a good chance that half written temp files can be left over in the event of a crash.

With libmtp i can pass in a file path and libmtp will write the file directly from that path or read it directly from that path. This means the memory copying is completely removed resulting in much better performance.

Fourthly, Whilest libgphoto can (technically speaking) run on windows, the necessary code hasn't been written to allow this to happen. libmtp can be compiled and run on windows (or so the documentation says). As there is work to make banshee run on windows, this is an advantage.

Finally, playlists. I'm still unsure how to create playlists using libgphoto. It'd require a fair bit of messing around and experimenting for me to figure out exactly how it should work. With libmtp, this is trivial to do.

However, libmtp has some pretty big issues at the moment, which only really came to light a few hours before the release.

If you connect an MTP device and banshee loads it up, connecting a second DAP (of any kind) will make libmtp choke. Once there's an active device, any call into libmtp to list available devices throws a (non-fatal) error. This means if you plug in a second device, you get an annoying error message popping up. If you do own two devices and were trying to copy from one to the other, this is impossible.

There was also an issue linking a device reported by libmtp to a device reported by HAL. There is no sure-fire way of doing this using the libmtp API. Under libgphoto this was possible.There are a few possible workarounds for this, but none of them are ideal.

2008-01-06T20:35:00.000Z

Banshee now has support for MTP devices in trunk. It's a long and torturous story, but here's the short of it:

Way back when, the work on implementing MTP support was started by a guy called Patrick. At the time, there were two options:
1) libgphoto
2) libmtp

libmtp was still in it's early stages of development and wasn't mature enough to be a viable option, so libgphoto was chosen althought it wasn't that much better. At this time, libgphoto didn't have metadata reading/writing implemented, didn't show any stats on the different filesystems in the device. Note: libmtp was no better at the time. So, a C# binding to libgphoto was started and basic support for syncing was eventually pushed into banshee.

Fast forward about 18 months. libgphoto has matured significantly in its support for mp3 players. Track metadata has full read/write support, you can get stats on the filesystems, patches and optimisations have been pushed in so that all the basic operations are fast. However, there's still issues with long filename support - there are hardcoded limits in libghoto itself which means if it encounters a path or filename longer than the limit, it just bails out. For mp3 players, this was a serious issue as these limits were quite frequently reached. For instance, if you were a fan of Sufjan Stephens, you were out of look. His incredibly long track titles like 'Out of Egypt, Into the Great Laugh of Mankind, and I Shake the Dirt From My Sandals as I Run' really throw a spanner in the works

Around this time, i stepped in to complete the binding and complete banshee MTP support. I pestered poor Marcus for quite a while until i got various fixes/updates into libgphoto such as the change to allow unlimited filename length and path length, and also to remove 'long' types from the public API (as a 'long' type can be 32bit or 64bit depending on architecture and leads to nasty complications binding it in C#). I also completed the initial revision of the libgphoto-sharp API restructuring. The entire endevour probably took about 2 months from start to finish.

However, the problem now was that in order to have MTP support in banshee, you needed SVN head of libgphoto, svn head of libgphoto2-sharp and a pot of luck that you could get it installed without screwing up your packaged version. A new release of libgphoto was several months away and probably still a further few months until it managed to trickle down into the various distros.

It was then that i was reminded by Aaron that libmtp still existed and was quite mature. I took a quick look at the API showed that it was tailored specifically for the kind of tasks that banshee was trying to do. Better yet, it had a faster release cycle, it didn't require svn head of anything in order to get all the features that were required and it was already available in nearly every distro.

I spent about 4 days writing a brand new C# wrapper for libmtp and then replaced the libgphoto-sharp code in banshee with the new libmtp-sharp code. While there are things in libmtp that are missing as compared to libgphoto, the simplicity of the new API makes it trivial to implement features which previously would have been quite complicated. For example, Playlist support under libmtp is trivial to implement, yet under libgphoto i'm not 100% sure how i'd go about it!

What i regret most of all out of this is that i put Marcus through so much hassle for minimal benefits as Banshee is no longer using libgphoto. While i do feel bad about putting so much effort into the libgphoto-sharp binding to never use it myself, it was still the best decision from banshee's point of view.

So, if anyone out there has an MTP device and wants to make sure that their device works with banshee before the release (so i can fix any bugs should there be any), check out the guide here. Post any bug reports on http://bugzilla.gnome.org under the DAAP component in Banshee.

2007-12-09T22:22:00.000Z

Looky here, who wants a giant network accessible harddrive which won't let you share the 26 most common filetypes. Honestly, why would anyone buy a piece of hardware which is unbelievably crippled and completely unsuitable for the purpose it's being advertised for. And more importantly, why would someone design such a horrible piece of hardware?

On a similar, but more sinister note, if america thinks you might be guilty of a crime, then they feel they can just kidnap you. The mentality of guilty until proven innocent really is showing through here. I wonder what would happen if i kidnapped an american citizen and brought them to ireland because i suspected them of a crime. I'm sure America wouldn't view that too kindly.

2007-12-06T01:44:00.000Z

My christmas exams (worth 35% of my degree) are getting closer and closer. Pretty much every day starting from last saturday has turned into a 10:30am->7:30pm study-a-thon. I'm trying to learn in the space of a 10 days what i should've been learning over the last 12 weeks. It's mostly working.

As a result, i've become a minor insomniac. Tis only 2pm, i've only been trying to sleep for 3 hours now. This describes it pretty well:

2007-11-26T09:26:00.000Z

In two short days i'll be flying out to madrid for the mono summit. Woohoo! Turns out i'm missing my last 3 days of term, in which i should be getting two tutorials and our first problem sheet for a particular subject. So, not the best time to be fecking off to a foreign country.... ah well!

Also, if that F-Spot guy is about the place who was asking me about upgrading F-Spot to use the latest libgphoto2-sharp, give me a shout via email and let's organise something before i head over. Otherwise, just try and find me between wednesday evening and friday evening.

p.s. if i fail any exams because of this, i'm holding Aaron responsible, he tempted me over with a Chicken Pag! So no new MTP code for banshee if i fail ;)

2007-11-07T12:31:00.000Z

I heard some great news earlier in the week and just got around to checking it out. There's another C# implementation of the BitTorrent protocol being developed! Good stuff! They've been working on it for about a year, and have "the core of the beast" pretty much complete. Thats great stuff! Considering i thought i was the only person with a C# based bittorrent implementation, i thought it'd be great to have a chat with them, see how they solved different issues etc.

So, about 15 mins ago, i loaded up that link and decided to download their alpha client, just to see how it works etc. Thats when i became seriously pissed off.

Without even having run the 'alpha client' i realised that the seedpeer codebase was actually MonoTorrent*. Yes, these guys had the cheek to take my code and pretend it was theirs. They tried to pass off about 18 months of my development time as their development time, without so much as giving me credit or mentioning that they were, in fact, lying when they said "we have almost finished coding the core of the beast". The worst thing is, they weren't even smart enough to do it properly.

So, for anyone out there who has actually used SeedPeer, or has a blog who've blogged about it, please try to make it known that seedpeer == MonoTorrent, except that they don't even use all the features i have available. For example, here's there 'TODO' list:

MultiTracker
Torrent Creation
Custom DHT (faster)
Rss Feeds
Integrated Search Function (with commenting)
Custom Peer Exchange
Fast Resume
Global/Per torrent Settings
Download Priority
more...

Here's a list of the features that are already in monotorrent that are in their TODO list:

MultiTracker
Torrent Creatio
Rss Feeds
Fast Resume
Global/Per torrent Settings
Download Priority
more...

I suppose this is what you expect by making the code public. People will use it (which is what i want), but then they'll try to pass it off as their own (which is just plain wrong). So, if anyone wants to email the seedpeer people, their contact form is here, so feel free to let them know you know ;) Also, if you want a nice GUI for monotorrent, check out www.monotorrent.com :)

* Yes, i am 100% sure of this. I haven't used reflector to verify, but i'm 100% sure that at least 95% of the code is a direct copy/paste from monotorrent. It might even be 100%. If someone is bored, they can use reflector (or whatever) so they can see i'm not lying. No better proof than looking yourself.

Also, if the seedpeer guys do read this, using an obfusticator won't prevent me from recognising my own code. I can think of a dozen ways offhand to check if a client is my client without having to see the code. It's ridiculously easy.

EDIT: For those of you who know these things, The middle paragraph of the MIT/X11 license states:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

Does that mean if they distribute a binary, they have to include that license header in some form or other?

2007-11-02T12:07:00.000Z

I decided to install the English dictionary in firefox today so I could get proper spelling corrections when writing emails and whatnot. So I fired up Firefox and browsed my way over to here to grab the English dictionary.

So it turns out that 'American English' is actually 'English' whereas 'British English' is what i wanted. Now, considering 'English' originated in Britain, why the hell do americans insist on referring to 'American English' as 'English' and referrring to English proper as 'British English'. British English is 'English'. You're the one's who created your own seperate spelling and dialect.

It's the equivalent of referring to 'Canadian French' as 'French' and referring to French proper as 'France French'. C'mon guys, you didn't invent it, stop trying to make it sound like ye did!

On other news, my macbook is in for repair for an unknown amount of time after it just stopped booting up. So i'll be MIA until it gets back to me. I'm still contactable by email should anyone need to.

2007-10-14T19:46:00.000+01:00

So, i'm looking for a bit of help. It's not a particularly hard task, it's just it'll take me a while and i need to go do some study. Basically, i need a method which parses XML-like data and adds the element name and the element contents to a Dictionary. It is fairly trivial, but i don't like the idea of having to think through all the string mangling required ;) First person to get me a method which works gets 10 brownie points.

You need to be able to handle something like this:
http://monoport.com/5078 (sorry for pasting there, but blogger borks on the XML tags)

Thing is, this is, technically speaking, invalid XML. The tag should be escaped, and the '&' in ObjectFileName should also be escaped. Unfortunately, i can't rely on the source of this XML being fixed any time soon, so i need to be able to hand parse the XML. I've outlined a procedure at the end which should be able to handle most of the mess that will be thrown at it. The only thing i'll add is that the manual parser doesn't have to cope with every eventuality. If something goes wrong (for example an artist called themselves which breaks the parsing) then don't worry, just abort. This is a last ditch effort to parse. It should succeed 99% of the time if you follow the procedure below.

Here's some psuedo code:
while(currentIndex < nexttag =" string.IndexOf('<'," elementname =" string.SubString(nextTag,">', currentPosition); // read the element name

currentPosition += the number of characters i've just 'parsed'.
string data = string.SubString(currentPosition, string.IndexOf("'); // The contents are between currentPosition and the end tag.
currentPosition += data.Length;
currentPosition += length of closing tag;

if(nextCharacter != '<') AbortParse(); // If the next character is not '<', then something has gone wrong, so give up. myDictionary.Add(elementName, contents);

2007-10-06T15:59:00.000+01:00

When i read the news about gcc supporting CIL a few days ago, the very first thought to cross my mind wasn't "Wow, that's really cool", it was "Wow, i wonder how many people are going to start shouting to not use GCC anymore because it's 'contaminated' with microsoft code and you'll get sued if you try use it for anything".

So should i take it as a good sign that there's no uproar? Have people realised that implementing an open specification cannot get you sued? You never know, it might have happened. Still, it's an amazing achievement. I can't say it's something i'm ever likely to use, but it does have some nice possibilities.

2007-10-03T00:32:00.000+01:00

With the mono summit coming up soon, I'm hoping to be able to spare the time from college to head over for at least 2-3 days of it. It'll be interesting to get together with a bunch of people for some hacking or some talking or whatever's going on.

So, if anyone's out there who's interested in talking about/hacking on (or redesigning ;) ) any of the projects i've been working on, or projects similar to them:

MonoTorrent: My pet bittorrent library project
Lunar Eclipse: The xaml designer for silverlight
libgphoto-sharp: The csharp binding for libgphoto - for MTP based media devices
Banshee MTP support: using the above binding

do give me a shout and i'll see what can be arranged. I'm already looking forward to getting some hacking in on F-Spot over the week to port it to the new libgphoto-sharp bindings. Should be fun!

For anyone else who's kinda undecided about going, ah sure gwon! Tis a bit of craic ;)

2007-09-28T18:17:00.001+01:00

So over the last few weeks i've been working on improving the C# binding for libgphoto2.. Thankfully, the backend C wrapper was in fairly good shape with thanks to the guy who originally developed it, and trickv, who became the maintainer of it and is the guy responsible for implementing MTP support in banshee.

For me looking at the code as a newcomer to the libgphoto-sharp 'team', the first thing i realised was that the c# api was a direct copy/paste of the C api. There was no proper frontend which simplified the use of the libghoto2 library. For example, to get a list of connected devices and then connect to a specific one required detailed knowledge of the libgphoto2 API and over 100 lines of code and also required you to be very careful about disposing of objects correctly.

So, my first task in getting full MTP support in banshee was to write up a new API for libgphoto2-sharp which hid all that nastiness from the end-user. The new API is, i suppose, 80% complete. Quite a few of the methods in the API are blocking, and so asynchronous equivalents will have to be added. One of the more immediate benefits is that detecting and connecting to a camera takes 3 lines of code now ;)

So, if anyone out there wants to use the new simpler API (fspot and banshee devs, i'm talking to you) you can check the code out with this command:

svn co http://gphoto.svn.sourceforge.net/svnroot/gphoto/trunk/bindings/libgphoto2-sharp gphoto

The binding should be considered API unstable until (probably) the release of libgphoto3.x.

2007-09-21T18:58:00.000+01:00

As both a cyclist and a humanitarian, i think i'm fully qualified to say: Women with babies should not be allowed out in public.

"What are you on about" i hear you ask. Well, the reason is simple. Whenever a woman sees a baby, they instantly become hypnotised by it's hideous features and are reduced to mumbling cryptic phrases such as "Goochie goochie goo" and "Who's a big boy then?" to said baby. Unfortunately, this distracts them from real-world issues such as how to cross the road safely.

I was cycling to college yesterday, as i've done for the last 3 years (i cycled to secondary school for the 6 years before that and primary school for 3 years before that). All was going well, i was a mere 2 minutes from my house and was just preparing to take a left turn (in ireland we drive on the left-hand side of the road) when all of a sudden, a woman with a baby decided to cross the road in front of me whilest talking to her baby. So there i was, travelling at approximately 25 miles an hour with a baby in a pram being pushed by an idiot a mere 15-20 metres in front of me.

I jammed on the brakes, the back tire locked, the wheel skidded on the wet ground and i went flying. Whilest i got away with some scrapes and bruises, my MP3 player wasn't quite so lucky. I only got music from the left earpiece. I was pissed! So, as an electronic engineer, i tried to fix it. A quick google got me instructions on cracking open the case, so i did.

Initial inspection made it look like very minor damage:

However as i gently poked it, more and more bits started coming off:

Finally, by the end of it, the entire lefthand side was toasted and the top bit was also completely broken off. I was none to happy:

So, my task now is to find something to wedge along the side of the earphone jack which will hold the metally bits in place. Everything works fine at the moment, but unless i support those metally bits, they will bend back out of position through use. Worst case scenario, it's a 30 gig portable harddrive which can play films on any tv via TV-Out. Still pretty useful.

2007-09-10T00:48:00.000+01:00

I don't care how i get one, but i need one of these!

Clicky Linky.

It looks so, so, so nice! I just wish public free wireless was as prevalent in ireland as it is in the US.

2007-09-06T09:14:00.000+01:00

Have no fears my American friends, your President has your best interests at heart. Check out this interview where he reveals where that 50 billion in defense money is going: