MonoTorrent 0.12.1 has been tagged in svn. I'd recommend anyone running MonoTorrent under MS.NET to update to this revision as soon as possible. There is an issue in the Mono.Security lib whereby you'd get strange exceptions in MS.NET if more than one calculation of BigInteger.ModPow() was being run simultaneously.

I've just tagged monotorrent 0.12 in SVN, so anyone using the library in their application is *highly* advised to use this new release. It contains several very important changes which affect library performance in a major way. Check the forum for a changelog of the important stuff ;)

I'm now officially into the third week of my internship at Novell. Tis interesting stuff. I've managed to put faces to a lot of names and learning a lot of interesting things! Since i've gotten here, i've been working a bit on moonlight. I wasn't quite part of the hack-a-thon as i have zero knowledge of C++, but i did get the fun job of excerising the API and pointing bugs over and over until people went insane ;)

As you may or may not know, it's Hack Week at Novell. What this means is that every novell employee gets to work on whatever project they want for this week.

While my appointed task for the summer is a Silverlight Designer, i'm also doing that for hackweek as it's great fun! The idea is to end up with a visual designer for silverlight which can run from a WinForms GUI, a GTK# GUI or just from your browser! At the moment i'm implementing it with GTK# as it makes my life easy, but sticking a new frontend (i.e. web based frontend) should be relatively simple.

So, you might be wondering what the designer can do at the moment. The short answer: nothing useful. The long answer: lots of stuff!

At the moment you can draw any of the basic shapes: Squares, Rectangles, Ellipses, Circles, Lines and Pencil. There's support for undo/redo. There's support for converting your amazing drawing into XAML (there will soon be support for allowing you to hand edit the xaml and load those changes back into the drawing surface).

So all in all, nothing useful, but it still does a lot ;)

Miguel was nice enough to upload my screencast to youtube which demonstrates the amazing awesomeness of the designer! By the end of the summer i'd hope to be able to record animations in the designer so you can easily create that stunning animation to incorporate into your website/desktop application/whatever.




In other news, MonoTorrent has gone through an extensive bug-fixing phase which fixed a lot of those annoying bugs which i'd never managed to track down before. The incentive to keep me hunting for these elusive bugs was due to Andy Henderson. Thanks a lot for all those charts, graphs, explanations and reports andy! I couldn't have done it without you.

Here's a partial list of recent changes/fixes:
* Allocating less byte[], but making them larger. This results in less objects being pinned, less heap fragmentation and (hopefully) slightly better performance
* Fixed crash when trying to parse UDP trackers
* Fixed bug where threads wouldn't stop correctly when calling Dispose on the engine
* Incoming encrypted connections being received correctly - thank you azureus for hiding this from me
* Individual files now containing their % completion
* Fixed potential threading issue in encryption code
* Rewrote the public API
* Fix to make sure that both global and per-torrent connection limits are obeyed
* Fixed typo in ratelimiting which meant i was getting slightly less stable rate limiting than i should be
* Starting a torrent manager which has not been hashed and doesn't have fast resume data now works correctly.
* Added option to enable/disable have message suppression - Recommended value is to Disable have supression.
* Fixed several critical issues in piece picking code
---- Reduced CPU usage hugely
---- Reduced memory usage noticeably in certain scenarios
---- Rarest first algorithm now chooses rarest pieces as opposed to most common pieces (doh!)
---- Rarest first algorithm falls back to trying to choose less rare pieces if all the rarest pieces are already downloading
* Now handling older clients which request 32kB pieces correctly.
* Correctly dropping incoming connections when the TorrentManager is not running
* All download/upload speeds and amounts are given in kB/sec and kB respectively
* Fixed bug where large single-file torrents wouldn't load correctly - multi-file torrents were unaffected
* MessageTransferred event firing for incoming messages.
* Updated the PeerID code to make it *much* easier to use
* Fixed bug where an array out of bounds exception would be thrown when trying to request the last piece of a torrent whose number of pieces was an exact multiple of 32 (that was a fun one to debug ;) )

So these changes basically mean you have a *much* faster, more stable, better monotorrent! I have a few more ideas on reducing memory usage and CPU usage which shouldn't take too much effort to implement. I just have to find the time ;)

In less than 24 hours, I'll be flying off to Boston for the summer with only a small suitcase, a bicycle and a guitar to keep me company. I've been lucky enough to be accepted as an intern with Novell for the summer. I get to be chief coffee monkey and photocopier for a few months, isn't that great! ;)

This is an amazing opportunity, firstly i get to work on Mono (or related projects). That's gonna kick some serious ass. Secondly i get to visit the states for more than a week or two, which i've never been able to do before. This means i get to visit everywhere i want to visit as opposed to being short on time and having to compromise. Thirdly, it's just frikin awesome ;)

My only hope is that i have enough time to do everything i want to do over there. Visit NY, go to the Cheers bar, see the sites, visit my friend in Pennsylvania, go sky-diving (or not... who knows). So if anyone has any suggestions on what i should visit in Boston, let me know. I don't want to miss out on something good.

A few weeks ago, myself, Miguel, Aaron and Michael were roped into being in a podcast for the Google Summer of Code. So if anyone wants to tune in and take a listen to it, it's available here.

For those of you taking part in the SoC this year, it should be worth a listen!

In an interesting turn of events, CSS (the "copy protection" system used on regular DVDs) is no longer considered an "effective technological method" to prevent copy protection under the law. The direct result from this is that it does not qualify for the European DMCA's anti-circumnavigation laws.

The European DMCA only bans the circumnavigation of "effective technological methods" thanks to an amendment what was pushed through a while ago. It seems that once a method is well and truly broken then it's no longer illegal to circumnavigate the DRM on your favourite DVD's, Bluray discs or HD-DVD discs.

More info here: http://www.valimaki.com/docs/finnish_css.pdf

I gave mono.nat a onceover yesterday and chopped about 15% from the codebase through an ingenious set of refactors (basically i turned proof-of-concept code into maintainable code).

It has more complete support for uPnP than before, i think i now support every method exposed by the routers. So if anyone wants uPnP support in their application, grab the source from the MonoSVN (websvn link: http://anonsvn.mono-project.com) in the "Mono.NAT" folder.

For an example on how it works, look at the program in the Examples folder. And as always, send me all bug reports. Finally, if you have a uPnP router, and it is *not* detected by the example program when it's run, or some of the methods do *not* work as expected, let me know and i'll get some special debug builds to figure out the problem.

Finally, if anyone wants to supply me what a NAT-PMP capable router, or knows how i could easily host a NAT-PMP "server", i'll implement support for that aswell (over the coming weeks).

For those die-hard fans of Starcraft who've been waiting eagerly for Starcraft 2 to appear, you need wait no longer. Click. While i don't expect this game to hit the streets for quite a few months (if not a year or more), it's because of scenes like this that i can't wait to start playing. Thankfully Blizzard are the ones developing the game. I have full faith in them to not screw it up like so many other sequels.

Hell, if we're really lucky there'll be an open beta (or maybe a closed beta and someone gets me an invite ;) ) where we can test everything out for some multi-player zerg rushes.

Brilliant. Despite telling customers that there would be no change in the anti-copy protection on their latests DVD's, Sony has backed down. They're now issuing exchanges for every faulty disc.

The only thing i find odd is that they claim to have received complaints for less than one thousandth of one percent of affected discs shipped. Now, the pedant in me says instantly shouts "false statistics". Whats the point in giving stats based on units shipped. Surely the only stats that would matter would be on units sold. Obviously if you have 1000 discs in-shop, you're not going to receive complaints until someone buys them.

Despite AACS being broken for both BluRay and HD-DVD once already, and a huge list of keys for DVD's being created for both HD-DVD and BluRay with some parts of AACS permanently bypassed, these mega-corps still haven't learnt that DRM does not work.

Once again, in a desperate attempt to make their discs copy-proof, Sony has managed to screw up severely. A year ago, Sony decided that they'd install a secret rootkit which could not be detected by regular means on computers which attempted to play their Audio CD's. As bad as that was, at least the CD's were playable!

Now they've come up with a new ploy: Make their DVD's unplayable on their own DVD drives. If you can't play it, you can't copy it. To be honest, this must be a PR nightmare for them. They are knowingly selling bastardised imitation DVD's that do not play on dvd drives. Hell, they are still selling their own DVD drives without a warning that it will *not* play new sony DVDs.

I suppose the worst thing out of this is that other dvd drive manufacturers may be forced to spend time and effort updating their drive's firmware in order to allow their customers to use their legitimately bought entertainment discs.

I had a rather interesting experience on the bittorrent mailing list recently. There was a guy who had a server which was severely IO limited and he was trying to figure out how best to improve performance. The problem was that due to the randomness of how a piece is chosen to be downloaded in bittorrent, his server was having to do a lot of random disk seeking thus limiting his maximum upload rate to about 11-15MB/sec even though he had the bandwidth to handle much more than that.

There are hacks to avoid this kind of IO limiting, such as advertising that you have no pieces available When you connect to a peer and then telling them about selective pieces later. The idea is that the piece you advertise having, you'd buffer in memory and so when the peer requests the piece, you don't have to seek to disk to get it for them. With a sufficient memory cache you significantly reduce the amount of random disk access needed and increase throughput hugely

However, this has several disadvantages. The biggest disadvantage of this method is that it can (and does) reduce overall throughput. Take a situation where a torrent has 100 pieces. You put the first 10 pieces into memory and then send messages advertising those 10 pieces to everyone. If everyone already has those pieces, then no uploading will occur until you swap out those pieces and upload the next batch.

Secondly, you *must* disconnect every peer you've connected to every time you decide to swap out those 10 pieces. If you don't, the other peers will remember about the previous pieces you've advertised and may request those even though you no longer have them in memory. This means you once again have random seeking. However, the amount of random seeking would still be significantly reduced as each peer would only know about 20 pieces out of the actual 100 if they weren't disconnected.

Thirdly, this method of seeding requires special logic. Therefore if you want this kind of special handling you must either hack it into an existing open source client yourself, pay someone to do it, or find a client that supports this and use it (regardless of whatever other problems/deficits that client may have).

Now for the fun part: This issue was addressed in the Fast Extensions for the bittorrent protocol with the "SuggestPiece" message.

Suggest Piece is an advisory message meaning "you might like to download this piece." The intended usage is for 'super-seeding' without throughput reduction, to avoid redundant downloads, and so that a seed which is disk I/O bound can upload continguous or identical pieces to avoid excessive disk seeks.

As you can see, the Suggest Piece was designed to fix the exact problems i've described above, unfortunately due to either bad wording, or just bad interpretation, this message is going to fail miserably in that.

The SuggestPiece message is an "Advisory" message. My own opinion is that Advisory was added there to signify that you do *not* have to act on that message. The reason for this is that you could already be downloading piece 3 off peer X when peer Y sends a "SuggestMessage" suggesting you download Piece 3 off him instead. In this case, you would not follow the suggestion, you would just ignore it. That makes sense. It's logical. Downloading the same piece twice would just be stupid ;)

Unfortunately, several bittorrent developers who've implemented the Fast Extensions have mis-interpreted this as "You can completely ignore this message if you want to". One developer cited his reason for completely ignoring this message as "i have no use for this, so i ignore it". This is a stupid reason for not implementing proper handling for the SuggestPiece message. Even "I'm too lazy" would be a better excuse as you'd at least be acknowledging that you have an incomplete implementation.

A "proper" implementation of the SuggestPiece message would be to request that piece off the peer that sent you the suggestion at the earliest convenient time, i.e. the very next piece request off them should be for the suggested piece.

Lets assume that every torrent client implemented the suggest piece message handling as i described above. Now, i'll replay the situation above using SuggestPiece messages as opposed to selective piece advertising.

Firstly, you advertise having all the pieces as being available when you connect to each peer (this requires no special extra logic). Every time you receive a request from a peer to send them a piece, you load the entire piece into memory. As soon as that piece is loaded into memory, you send a SuggestPiece message (this requires no special extra logic) to every other peer. The other peers will then act on that message (this requires no special extra logic) and their next request from you will be that piece. That way every time you load a piece into memory once, you can then send it from memory to every other peer that wants it.

The benefits of this method are that every time you load a piece into memory, you can make other peers request it off you (if they need that piece). You will *never* have a situation where you will not be uploading. You will *always* have better performance than the initial scenario where you were IO limited. Assuming that you're still doing the same number of random disk reads as before, you will be able to increase your upload rate significantly because each piece that enters memory will be sent to more than 1 peer.

I'd guess that you could increase performance by, at the very least, 5 fold in a torrent with 100 other peers using the SuggestPiece method. The other major advantage is that you require no special logic in either the seeding client or downloading client! All you need is a proper implementation of the SuggestPiece message. It's not that hard!

EDIT: Despite promising myself i wouldn't, i think i will name and shame the clients that completely ignore the SuggestPiece message despite claiming support for the Fast Extensions:

KTorrent
Bitflu

The clients that i know of that fully support the extensions are:
MonoTorrent
MainLine

I was just talking to J.D. Conley (part of the Coversant team. He also has a blog on the net) about some if the issues i've been having in monotorrent with threading and async sockets. He had a few good ideas about what i could try, so hopefully over the next week or three i'll be able to test out a few of the ideas to help improve performance. Anyway, as part of the discussion i realised that i could probably replace a lot of my lock(object) statements with a much nicer ReaderWriterLock().

The way the code works in monotorrent is that i read from my lists of peers much more than i modify them, therefore it makes sense to have a locking structure that allows multiple readers and a single writer! This gave me the idea for this blog post.

You see, the problem with the ReaderWriter lock is that you have to be careful in how you use it. It's not as simple as the lock(object) statement. If you forget to release a reader lock, or worse, a writer lock, you will experience deadlocking and it will be very hard to track down the exact place where you haven't released the lock!

Also, suppose you want to a writer lock, but you forget to check if your thread already has a reader lock, you will end up throwing an exception as you will not be able to get the writer lock.

However, it's very easy to abstract away all those horrible problems with two simple structs:

public struct ReaderLock : IDisposable
{
    public ReaderWriterLock Locker;

    public ReaderLock(ReaderWriterLock locker)
    {
        Locker = locker;
        locker.AcquireReaderLock(1000);
    }

    public void Dispose()
    {
        Locker.ReleaseReaderLock();
    }
}

public struct WriterLock : IDisposable
{
    private bool upgraded;
    private LockCookie cookie;
    public ReaderWriterLock Locker;

    public WriterLock(ReaderWriterLock locker)
    {
        Locker = locker;
        upgraded = locker.IsReaderLockHeld;
        cookie = default(LockCookie);

        if (upgraded)
            cookie = locker.UpgradeToWriterLock(1000);
        else
            locker.AcquireWriterLock(1000);
    }

    public void Dispose()
    {
        if (upgraded)
            Locker.DowngradeFromWriterLock(ref cookie);
        else
            Locker.ReleaseWriterLock();
    }
}

Suppose you have the following code snippet:
ReaderWriterLock myLocker = new ReaderWriterLock();

Instead of having to manually use: myLocker.AcquireReaderLock() and remember to call myLocker.AcquireWriterLock() in a finally statement, and instead of messing around trying to decide if you should call myLocker.AcquireWriterLock or myLocker.UpgradeReaderLock() you can just use the corresponding ReaderLock or WriterLock as needed inside a using statement!

For example, take this code snippet:

try
{
   DoSomeStuffWhichMayOrMayNotThrowAnException();
   myLocker.AcquireReaderLock();
   DoLotsOfStuff();
   myLocker.ReleaseReaderLock();
}
catch(WhateverException)
{
   if(myLocker.IsReaderLockHeld)
        myLocker.ReleaseReaderLock();

   DoRestOfErrorHandling();
}

You can use the much nicer:

try
{
   DoSomeStuffWhichMayOrMayNotThrowAnException();
   using(new ReaderLock(myLocker)
       DoLotsOfStuff();
}
catch(WhateverException)
{
   DoRestOfErrorHandling();
}

Supposing you actually needed a writer lock, just change the using statement to:
using(new WriterLock(myLocker)

The constructor for writer lock will then check to see if it should upgrade an existing reader or if it should get a standard WriterLock. If it upgraded, it will remember to downgrade it correctly later.

Using this method not only makes you need less lines of code, makes your code more readable, but it also completely removes the possibility of creating a deadlock by accidentally forgetting to release a lock!

Just implemented an important bugfix for the last monotorrent release. Somehow i missed the fact that it didn't close connections properly ;) Fun stuff!

There's a new release up on monotorrent.com for anyone who downloaded the other release.

Enjoy!

I've opened a forum for monotorrent discussion which can be found here. If it proves unstable or crap, i'll hunt out better forum hosting, but it'll do for the moment.

MonoTorrent - Prebeta 3

So it's been a long time in the coming, but we now have MonoTorrent prebeta 3 released into the wild. Let fly with the bug reports!

Check out http://www.monotorrent.com/ for the juicy details.

Also, i'd like to give a shout out to jbevain, whose amazing linker, based on the even more amazing cecil, combined with the equally amazing merge tool managed to reduce the size of the libraries i have to distribute down from 520kB to a mere 190kB. Zip that up, and you have a mere 90KB file to distribute. Not too shabby!

A nice (depending on your point of view) side effect of this was also the fact that i know have a single MonoTorrent.exe file to distribute instead of over 1/2 a dozen different files.

Not too bad at all!

Once again, i'd like to say thanks to David Wang, for coding the Encryption support. Also to David Muir for his bugreports and patches (i hope to get more off ya ;) ) and everyone else who has bugged me about problems, or asked me questions.

If i don't reply to your emails, it's because i've forgotten about them. Email again!

I'm just putting a shout out to David Wang who has been working tirelessly over the last week or two to implement encrypted transfers for MonoTorrent. I have to say, he's done a great job.

The current SVN has around a 90% complete implementation. As far as i can tell, there's still a bug in accepting incoming connections. Other than that, all that needs to be done is to make sure it works with more than just uTorrent and then it's good to go!

I've been asked how i go about profiling, but i think the best explanation i could possibly give would be to look at this (very interesting) video.

Frederico talking at Fosdem

Other than that, all i can say is Reduce, Reuse and Recycle.

1) Reduce: Every object you allocate is an object the GC has to clean up. Know how many objects you're allocating and reduce the number if it's excessive. For example i once was calling DateTime.Now several thousand times a second because of the loop's i had written. By moving the call before the loops, i reduced the allocations from that by a factor of several thousand.

2) Reuse/Recycle: I used to allocate a new byte array every time i wanted to send a message to another computer via a socket or receive a message from them. This resulted in a *lot* of 16kB byte array's floating around. Instead i created a BufferManager class. From this class i requested a buffer every time i needed one and then returned the buffer back to the manager every time i was finished with it. This way i never allocated more buffers than i needed (about 40-60 buffers typically). Previously i could be allocating as much as 40 new buffers each and every second.

I might blog more about that later if people really want me to. But that all depends on how much people beg :P

It's been a while since i sat down and profiled monotorrent, turns out there's not much that has adversely affected my memory optimisations that i implemented a while ago. Things are all still fine and dandy.

However one thing that i never liked were the large amount of allocations due to using the async sockets. It always annoyed me that the worst part of my code i had no control over. Well, it turns out that it isn't *just* the sockets causing those allocations, there's another thing i haven't looked at yet.

My timer!

I use a single timer in the library which fires about 20 times a second. This was originally needed as a way to make sure that certain events happened regularly. However, I've never liked this timer and i have been working to get rid of it. For example i removed the dependence of downloading pieces and piece picking from that timer (as much as is possible). Download/Upload speed monitoring is still dependant on the timer, so once i remove that dependence and the only other thing left is to make connecting to other peers independent of the timer. Currently i try to connect to a new peer every timer tick so long as i haven't reached the limit.

So, before i used to have an allocation graph that looks like this:


That big yellow bar is the allocations due to both the timer and the async sockets. There's a lot of execution contexts there :p After changing the timer interval to 1 second, i get the following allocation graph:


That tiny cyan coloured bar is the severely reduced amount of execution contexts being allocated. So before you all get to feel the benefits of that change, i have to modify a fair bit of code. Once thats done, i'll be releasing beta 3 which is (hopefully) a lot more stable than any previous version, and faster, and uses less ram, and uses less CPU and god knows what else ;)

Just to respond publicly to a few questions about monotorrent that have been asked over and over:

1) MonoTorrent does have a GUI! In fact, it has two. One of them is a console GUI, the other is a gui created using GTK#. This is *not* the same as winforms in .NET. It is an alternative framework for creating GUI's. While i consider the GUI very limited (which it is), it is usable. I would much prefer a Winforms based GUI so the application would be 100% portable .net with no other dependancies but i don't have the time to finish my 1/2 created one.

2) MonoTorrent does use less ram than azureus. Considerably less. Very very much so. Monotorrent does use more ram than uTorrent. But not that much more. If you disable the disk cache in uTorrent completely, monotorrent only uses 4-5 megabytes of ram more than utorrent. If you leave the disk cache enabled in uTorrent, then monotorrent uses slightly less ram than uTorrent.

3) MonoTorrent does not support encryption. I looked at the specs a while ago, found them confusing, created some initial support but never got back to it. Once again, time is a factor. I just don't have enough.

4) MonoTorrent is not a "Linux App" or a "Microsoft application". It's a cross-platform bittorrent library.

5) Yes, monotorrent can be used on whatever tracker you want. Private/public/personal, i don't care. It works. (Just so long as it's not a udp based tracker ;) ).

6) MonoTorrent is not related to the disease known as mono (also known as glandular fever in non-american areas) ;)

7) Yes, i do like people to contribute code. But please talk to me first and keep in touch when writing your code. The last thing i want to do is to say "no" after you spend 3 weeks writing a bucketload of code for monotorrent because i've already written the code, or you've done it wrong or whatever.

Heh heh heh, i feel all warm and fuzzy inside:

MonoTorrent and prebuild get slightly more famous ;)