Monday, October 06, 2008

MonoTorrent 0.50 - The Good, The Bad, and the seriously awesome

It's release time! Yes, MonoTorrent 0.50 has hit. There have been a lot of changes since the last release, and this time, it's more than just under the hood fixes. There are several reasons why the new release is so much better than previous releases, I've listed the more important ones below, but first, the packages!

You can grab a precompiled binary suitable for Windows, Mac OS and Linux.
You can grab the sourcecode here as a .tar.gz archive.
There are also packages available on the OpenSuse build service, and of course the tag for the release can be gotten from svn.

Now, to the features and updates.

WebSeeding Support
There is provisional support for Http Web Seeding. This means when you're hosting a torrent, you can add standard Http servers as 'seeds'. No extra configuration is needed. This is still an experimental feature, and still has some corner cases where it doesn't work, all bug reports on this are welcome!

IP Address Banning
You can now ban individual IP addresses or IP Address ranges. Block lists from Emule, PeerGuardian and SafePeer are supported out of box by the built in parser, and any custom list can easily by loaded so long as you can parse the list into IPAddress objects. Internally the banlist is stored using the extremely efficient RangeCollection written by Aaron Bockover.

Efficent Torrent Streaming
Thanks to the efforts of Karthik Kailash and David Sanghera, we now have a special downloading mode in MonoTorrent which allows you to efficiently stream audio/video. Psuedo random piece picking is used to ensure you download pieces from a 'high priority' range before anything else. User code can set this 'High priority' range to be the next X bytes of data. When everything in the high priority range is downloaded, standard rarest-first picking is used.

Peer Exchange
uTorrent style Peer Exchange support is supported thanks to the tireless efforts of Olivier Dufour. This extension allows peer information to be passed across a bittorrent connection. In practice this means that if the tracker only gives you 1 peer, you can discover (potentially) hundreds more via peer exchange.

Enhanced compatibilty with broken clients

There are still clients out there which transmit corrupted BEncodedDictionary objects. These guys need to read the spec and ensure that their dictionaries keys are sorted
using a binary comparison. In the cases where the order appears to not matter, I've implemented support for ignoring the error. This should reduce the number of clients which are disconnected due to sending corrupt messages - this means higher performance.

Simplified Threading API
The core of MonoTorrent has undergone a complete rewrite. Previously, all the worker threads interacted with the core by taking out locks, then doing their work. This meant that implementing something as trivial as cancelling a pending asynchronous request was actually pretty hard. That method was actually horrendously prone to deadlocking the engine.

Nowadays all the worker threads add a task to the main thread, and the main thread does all the work. "What about the performance" i hear you ask, well, it performs the exact same, but it's so much easier to maintain and add new features to.

It also means the engine should be deadlock free, because there are no locks anymore. Nice.

NUnit Tests
As with all big software projects, regressions are bad. A year ago I had virtually no NUnit tests. Nowadays there are over 130 NUnit tests for the engine. While this doesn't even test 1/2 the code in MonoTorrent, each test adds that little bit more certainty that I don't regress.

There are also a bunch bugfixes here and there, and more big features in the pipeline. As a taster, DHT support is already active and enabled in SVN should you wish to test it out.

Friday, October 03, 2008

Who loves lamdas?

I finally started a project where I can use all the new fanciness. How would you process a list of names so that each name is printed out 'i' times?

int i = 5;
IEnumerable<string>() names = new List<string>() { "Alan" };
names.ForEach (n => i.Times().Apply(j => Console.WriteLine(n)));

All your lambdas are belong to us!

Thursday, October 02, 2008

Webserver for NUnit tests

What's the best way of setting up a simple HTTP file server to be used purely for NUnit tests. What I'm looking for is a way to set up a server so that I can host some files in. My NUnit tests will then initiate HTTP requests to 'download' those files.

Ideally whatever method I choose won't require me to write temporary files to disk. Currently what I do is host a HttpListener and manually fulfill the requested data by creating an in-memory byte[] and writing that into the response. I'd like to do something like:

webServer.Add("path/where/file/is", CreateFakeByteArray());

Then the server will fulfill requests from this faked byte[].

The reason for this setup is that I need a better way to test my implementation of WebSeeding in MonoTorrent without having to code up my own buggy file server ;)

Monday, September 29, 2008

tee-sharp a.k.a. CopyStream

How many times do you have one stream, but you actually want to write the data to multiple places simultaneously[0]? Well, now you can[1]!

I took an hour and spun up this awesome asynchronous beast of a stream splitter. There is an optimisation that could be applied to it: Reads can be performed at the same time as writes. I figured that for a 1.0 implementation, this was good enough. If anyone wants to try their hand at making the read perform in parallel with the writes, feel free. Patches are welcome ;)

[0] http://tirania.org/blog/archive/2008/Sep-24.html
[1] http://www.monsoon-project.org/jaws/data/files/TeeStream.cs

EDIT: There's also a 'deliberate' bug there. 10 points to the first person to spot it and bonus 10 points if you can fix it with less than 5 lines of extra code.

Wednesday, September 24, 2008

Captchas, a bridge too far?

I was just trying to open a new gmail account when I encountered the unsolvable captcha. I couldn't for the life of me figure out what was written in that horrible mangled imagine of what i can only assume were standard roman-alphabet letters. I can only assume that I am in fact illiterate and that would explain why I couldn't read it.

Thankfully, google have a second option. They have a little link which reads out the captcha for you. It turns out that I can't understand english either. I couldn't make out a word of what was said. So, if anyone out there can speak english, could you please decypher this for me:

http://monotorrent.com/GoogleSucks.wav

Sunday, September 21, 2008

It lives!

Revision 113625 - Enable DHT by default in MonoTorrent.

Awesome!

Saturday, September 20, 2008

How verbose is too verbose?

A common thing to do in code is to perform an action on each element in an array of objects. In C# there are two main ways to write this:
// Lets just assume this list of strings has been populated with lots of strings
List<string> allStrings = GetLotsOfStrings ();

// Method 1: The for loop
for (int i = 0; i < allStrings.Count; i++)
DoStuff (allStrings[i]);

// Method 2: The foreach loop
foreach (string s in allStrings)
DoStuff (s);
However, both of those methods are far too verbose. There is another way, which is much much nicer!
allStrings.ForEach(DoStuff);
How awesome is that, eh?

Wednesday, September 03, 2008

MonoTorrent is teh awesome

How many lines to host a torrent server?

Any ideas?

Guess!

Check it out yourself

So what happened in SoC 2008?

So, after ~3 months of hacking while travelling up the east coast of Austrlia, what exactly have I managed to accomplish in this years SoC? Well, quite a lot :) Here's a list of new stuff and upcoming stuff in no real order:


DHT
DHT support is available in SVN. It's pretty much complete, but lacks some real world testing. There are currently about 35 NUnit tests covering all important modules in the code. I need to give this a week or two of solid testing and then I'll be enabling it by default. A few updates will need to be applied to MonoTorrent so that the 'private' flag will be obeyed now that we have DHT support.
IP Address Banning
Awesome support for this has been added to SVN using a combination of my own code and code written by The Great Bocky. It uses an extremely efficient way of storing IP Address ranges so that they can be represented by two integers plus a little overhead. There is a parser which supports all the main ban lists and users can parse other formats manually and add the addresses in that way.
Extended Messaging Protocol
Support for the LibTorrent extension protocol is complete. This allows custom messages to be sendable to remote peers over the standard bittorrent protocol. So if you require the ability to send arbitrary data to a remote peer and have them react in a special way, you can!
Http Seeding (Web Seeding)
Support for the getright style Http Seeding is complete. This (better) specification allows a standard HTTP server act as a seed with no special software required. If MonoTorrent decides that there aren't enough peers available in the swarm to allow the torrent to complete, it will automatically start downloading the necessary chunks from the server.
Peer Exchange
Support for this has been completed by Olivier Dufour. This pretty cool idea allows peers you are connected to to send you details about other peers which are active in the swarm. This way you gain information about more peers even if the tracker goes offline.
Misc
Faster SHA1
Due to both an algorithm change and architecture changes, hashing performance has nearly doubled. This means that hashing a file takes less than 1/2 the time it used to *and* that CPU usage while downloading is reduced.
Better Encryption
There was a bug in header-only encryption that prevented it from working correctly before. As MonoTorrent always defaulted to Full Encryption, this wasn't such a huge issue before. Along with this bugfix, encryption is now more performant than before - using less CPU and less memory. The code also shrunk considerably in size and is much more maintainable than before.
Deadlock Free
It's now impossible to deadlock the library. This isn't so important for the end-user, but for anyone programming with MonoTorrent it's great news. If you are extending the library to add extra functionality internally, it's now easy to ensure that you do everything in a thread-safe and non-deadlocking manner.
Abort long connection attempts
Sometimes an operating system might wait an incredibly long time before aborting a connection attempt. This meant that if MonoTorrent tried to connect to a peer that was no longer available, sometimes the OS would take up to 150 seconds to abort the attempt. Worst case scenario is that the first 5 peers you connect to all take 150 seconds to abort and it looks like MonoTorrent is doing nothing. Now MonoTorrent hard-aborts a connection attempt if it takes more than 10 seconds.
Streaming torrents ahoy!
Two guys Karthik Kailish and David Sanghera have created a new way to download a torrent with MonoTorrent. Generally speaking, the rarest piece of a torrent is downloaded first, then the second rarest and so on. This new code allows you to specify a range of bytes which is High, Medium or Low priority. Then, from within these ranges the rarest first algorithm is active. For example if you are a video player and you want to start playback from byte 1000, you can tell MonoTorrent that the range from 1000 -> 5000 is important so those bytes are downloaded first, which allows you to start playback as soon as enough of that high-priority data has arrived.
Unit Tests
The number of tests covering MonoTorrent has doubled over the summer, from 55 up to 111. Every test makes the liklehood of accidently introducing a bug less and less. I like tests ;)
Banshee Plugin
Finally, while not quite related to MonoTorrent itself - A plugin for banshee has been created which allows the downloading of torrent based podcasts. It's still a work in progress, but hopefully that can be cleaned up and completed pretty soon.
So with all these changes and features, i'm hoping to push out the next release of MonoTorrent by the end of september. This release is unlikely to include DHT, but I hope to have a second release shortly afterwards which will include DHT.
Anyway, I'm off to pack my bags now so I'm ready to head to Ha Long City at 7am in the morning. I'm enjoying my last week in Viet Nam at the moment. It's been a blast, though sometimes i wonder if they deliberately decide to not understand what I say just because I'm mispronouncing it slightly. 'Ho Chi Minh' isn't *that* hard to understand, is it?

Thursday, July 17, 2008

Socks proxies in .NET

I'm just wondering if anyone out there knows a library which has a nice MIT/X11 (or similar) license which will allow me to use a socks based proxy. Alternatively, 10 points to the first person who writes the wrapper to do it ;)

I had found one before, but I seem to have misplaced the link and google is failing me now.

EDIT: Ideally it'd be available in source form so i don't have to bundle a binary. I *really* don't want to bundle a library if I can avoid it.

Friday, June 20, 2008

MonoTorrent 0.4 and Monsoon 0.15

MonoTorrent 0.40 has been released. There weren't many changes feature-wise, but there's been quite a lot of under the hood changes. Details can be found on www.monotorrent.com

Also, Monsoon 0.15 has been released. The release notes are available and your packages can be gotten from here.

Fun times, eh?

MonoTorrent 0.50 is slated for a few weeks time (don't hold me to this). There are a bucket load of features in the works which will definitely kick some ass. I've been getting some great patches recently from Olivier Dufour, which he has detailed in his post. These should all make 0.50. I've also been getting some awesome work from Karthik Kailash, and his friend David (whose second name i can't find now), implementing a fancy debugging GUI which exposes all the internals in a nice GUI to make it easy for me to detect bugs/issues. He's also implementing Ono support, which helps get faster transfers; Bit-tyrant like unchoking which prioritises peers who reciprocate data resulting in faster transfers along with a new Piece Picking algorithm which allows you to stream a media file via torrent efficiently.

I'm unsure how many of those features will hit 0.50, it depends on when they hit SVN and how much testing i can get in. But hopefully a few of them will get there.

Wednesday, June 11, 2008

MonoTorrent - Expanding your universe

As i'm sure everyone has heard at this stage, Banshee 1.0 has been released. It's a huge step up from the old 0.13.x releases, and well worth checking out!

So, now that banshee has some kickass podcast support, along with video support, wouldn't it be nice if you could download video podcasts which have a .torrent payload?

Wouldn't it be awesome if there as a .NET based torrent library, that maybe was exposed via a DBus service that could be integrated with banshee with just a few lines of code. Of course, once you've integrated the actual torrent downloading, how do you make banshee realise that .torrent files need to be handled specially? Well, write another few lines of code.

So all in all, because of banshees awesome extension framework, i wrote less than 200 lines of code of banshee code to enable banshee to download torrents. I was surprised by how easy everything was. I was up with creating the new extension within about 10 minutes. So, if you're interested in this extension, attach yourself to the bug report and you'll be able to keep up-to-date with the latest happenings.

After all this, what exactly does it look like when you download a torrent podcast? Well, it looks exactly like it does for a regular podcast download. You don't have to do anything special, it's all just MAGIC! Check out the screencast.

Sunday, May 25, 2008

Titles!

Why the hell did my blogspot account have titles disabled by default? Did i stupidly manage to turn them off the day i opened my blogspot account?

Anyway, i have titles... finally!

Thursday, May 15, 2008

The Summer of Code is starting soon, and I'm a student once again. So what are the big plans this year? Well, hopefully a lot! Here's a brief outlook on what to expect during this summer. Note, these are in no particular order

MonoTorrent
1) DHT support in MonoTorrent. This is probably the most time-consuming feature that is planned.

2) DBus based daemon for monotorrent. The idea behind this is to allow applications to consume torrent files without worrying about interop-ing with .NET. This would provide a system-wide torrent service which any application, or many simultaenous applications, could take advantage of.

This daemon will expose a simplified API as compared to the monotorrent library itself.

3) HTTP/Socks proxy support.

4) A proper NUnit testing framework. All the essentials are now implemented in MonoTorrent to allow me to test stuff deep inside monotorrent relatively easily. Now i just need to implement my test harness and then some tests (work on that has started as of yesterday actually :) ).

5) Implement support for both the Azureus and Libtorrent messaging protocols. Support is mostly there for the Libtorrent protocol. This will also allow user-defined messages to be sent via the torrent connection.

6) As per usual, i'll be spending time with a profiler seeing where performance can be improved. Don't expect too much from this though, things are already pretty good ;)

Monsoon
Now that we have a sweet GUI for monotorrent, which is going to be available in Suse 11.0 (other distros also have packages these days), we need to keep improving it. Once again there are in no particular order:

1) Get Monsoon portable to both MacOS and Windows. 95% of this work has been completed already. So it's nearly done. Once this has been completed, i need to look into packaging installers for these platforms (or someone can offer to do that for me ;) ).

2) Next we get cracking on the buglist and try and resolve those. Quite a number of cosmetic things need to be looked at and there are also a few bigger issues, such as that memory issue (which is still proving to be quite elusive!).

3) General nicing up of everything. There isn't really a firm plan for monsoon yet. A more detailed timeline and suchlike will be created as soon i can get together with buchan after my exams and see where we want to take this.

All in all, a busy summer.

Friday, May 09, 2008

It's funny. I always thought that DRM should be used as a way to make life difficult for people pirating software/music. I never though it would be used as a way to convince people that pirating software is the better way:

http://www.theinquirer.net/gb/inquirer/news/2008/05/07/mass-effect-wins-award-worst-pc

Seriously, what are these people thinking?

Wednesday, April 30, 2008

I haven't done a mono-specific post in a while, so i'm glad i can change that with some good news! A few days ago, Scott Peterson, (who took part in the Summer of Code last year to port banshee to windows) was complaining that monsoon took an hour to hash a 60GB torrent. I was a bit surprised at this, because it shouldn't be that slow! I did know that SHA1 hashing in mono was a tad slow, but no way should it be *that* slow.

Anyway, he decided to fire up a profiler and get cracking on optimising the hell out of the SHA1 class in mono, and i decided that i'd skip some study and do the same. We had two main aims in this:
  1. To make the fastest possible implementation using no unsafe code
  2. To make the fastest possible managed SHA1 implementation
Aim 1:
In order to do this, we couldn't use unsafe code, everything had to be done without pointers. We also had a constraint that we couldn't make the size of the class significantly longer, in fact, *reducing* the lines of code was also an aim because the more lines of code, the slower it is for the JIT to process.

We did a number of iterations on the code, testing different things out. We fully unrolled the three main loops, we partially unrolled some of em and benchmarked everything in between. The things we learned in the end were:

1) In this particular algorithm, there was a great benefit to using local variables rather than fields. That gave a biggish boost. EDIT: This is because mono doesn't perform array out of bounds check removal (abcrem) on fields. It only performs it on local vars. As array access is *very* frequent, removing these checks is a huge benefit.

2)
buff[i] = ((buff[i-3] ^ buff[i-8] ^ buff[i-14] ^ buff[i-16]) << 1)
| ((buff[i-3] ^ buff[i-8] ^ buff[i-14] ^ buff[i-16]) << 31));


The above code actually performed significantly slower than:
uint temp = buff[i-3] ^ buff[i-8] ^ buff[i-14] ^ buff[i-16];
buff[i] = (temp << 1) | (temp >> 31);
)

3)Re-rolling some of the loops did not affect performance significantly. The performance delta between when all three of the loops were fully unrolled and when two of them were only partially unrolled was less than 6% on my system, but the IL was reduced by a fairly massive proportion.

4) Massive methods perform slower. We found that by splitting the 'ProcessBlock' method up, performance increase noticeably. Bear in mind that when we originally tested this, we had all three big loops fully unrolled and they were all in the same method. Still, it's worth bearing in mind.

So, what was the final performance delta with all these changes?

Core 2 Duo T7400 @ 2.16GHz
2.54x faster

Athlon 64 3200+
3.10x faster

Pentium 4 @ 2.80GHz
1.36x faster

Xeon X5355 @ 2.66GHz
1.38x faster

64bit PPC:
1.60x faster

EDIT: It seems I've mixed up my numbers when i was recording them yesterday. Some of the numbers compare against SVN head and some compare against the version in Mono 1.9. I think the first 2 results are against the version in 1.9 and the last three are against SVN head.

Not too shabby.

Aim 2:
More to come on this later, we're still in the optimisation process, but i think it's fair to say that the unsafe version is quite a fair bit faster thant the safe version so far.

Monday, April 21, 2008

So opensuse 11.0 beta 1 has been released. I decided i'd be adventurous and give it a whirl, so i fired up my mac and downloaded the ISO while i was in college. When i got home i realised that i had never actually burned an ISO on my mac before, and there didn't seem to be any built in software to do it.

So, i fired up google and expected to have to download trial software to burn the image and suffer all sorts of hassle and annoyance. I was pleasantly surprised to find that it was frikin simple to burn an iso:

$ hdutil burn image.iso

It's now 5 miutes later, and i'm just about to reboot into the Live CD environment, very nice!

Saturday, April 19, 2008

MonoTorrent 0.30 has been tagged and released. All i have to do now is update the website. Here's the short changelog.

Highlights include:
[Client]
* Amazing extensibility - Active connections can be injected from any source you want. So if your application already has an active connection to a peer, that connection can be passed to MonoTorrent and it will be used.
* Udp Tracker support implemented - Can now use the low bandwidth udp protocol if the tracker supports it.
* Initial support for the libtorrent messaging protocol
* Implemented semi-intelligent memory buffer to reduce disk reads/writes.
* Fixed several race conditions when Stopping/Unregistering torrent managers.
* Fixed issue whereby monotorrent would stop connecting to new peers
* Can now handle torrents with 1,000's of files gracefully.
* Implemented IPV6 support.
* 15% faster hash checking
* Enhanced the accuracy of the ratelimiting code when a global limit is applied
* Per-file progress correctly updated when FastResume data is loaded
* If a file is set to 'Do Not Download', it now will definitely not be downloaded.
* Abort a connection attempt if it takes more than 10 seconds to complete. Some operating systems default to 3 minute timeouts which kills performance.
* Some speed and memory enhancements, as always.

[Tracker]
* Correctly removes zombie peers (peers who crash before telling the tracker they're stopping)
* Added ability to compare peers based on an arbitrary key rather than only based on IP.
* Minor speed and memory enhancements

A precompiled binary can be found here.
A tarball can be found here

Coinciding nicely with this is the release of Monsoon 0.11.3. The changelog looks something as follows:
* Adding/Removing/Renaming of labels is completely context-menu driven now
* Can drag and drop torrents to add and remove them from a label.
* Fixed several issues persisting state across application restarts
* Global rate limits can be set by clicking on the labels which display the global download/upload rates
* Now takes advantage of the new FastResume API.
* Correctly invoking libgtk and libX11.
* Now supports nat-pmp through the use of a newer mono.nat
* Multi-select enabled in the file view
* When creating a torrent, hashchecking is skipped if you choose to seed it immediately
* You will always be prompted if you choose to remove/delete a torrent
* Made Monsoon fully translatable
* Added tooltips to the main items

Monsoon can be gotten via 1-Click install or the GNOME:Community repository:

Tuesday, April 08, 2008

So, some quick news on the MonoTorrent front. As i said a week or two ago, Monsoon is going to be part of the Suse 11 distribution (woo!). Since then a few things have happened.

1) Monsoon has hit feature freeze and has been branched. This is the version that will be included with suse. If you want to check out the code and put it through it's paces use this url:
http://anonsvn.mono-project.com/source/branches/monsoon-0.11
If you find any bugs, please use the novell bugzilla to file a report. Some time early next week, this will be officially tagged and released. So make any bug reports earlier to increase the chance that they'll be fixed.

2) MonoTorrent itself has also been branched for it's 0.3 release. If you're a developer using MonoTorrent, check the code out from:

http://anonsvn.mono-project.com/source/branches/bitsharp-0.30
Bug reports are welcome (as always), same place as above (Except use the MonoTorrent module, not the Monsoon module). I'll have full release notes available when the release is made. Some pretty cool stuff has been done along with the usual bug quashing ;)

3) Finally, mono-curses has been updated again to run against MonoTorrent 0.30. So if you want a slick cool ncurses GUI for MonoTorrent, check the code out from:

http://anonsvn.mono-project.com/source/trunk/mono-curses
Finally, I just want to add: All Your Torrent Are Belong To Us - Use Monsoon! ;)

Friday, April 04, 2008

There was a meeting yesterday for openSUSE Gnome, one of the important decisions for the day (in my eyes) was the decision about which BitTorrent client was going to be bundled with suse.

Torrent default app decision:
Monsoon seems the more dynamic app, good response from its maintainer
Transmission seems to have a better UI, actively developed, although a bit mac centric
BitTorrent-gtk very basic but should work for basic needs
AI: add both monsoon and transmission, monsoon as default
AI: suseROCKS to run tests with both apps
AI: FunkyPenguin to package Transmission today
AI: vuntz to move Transmission and monsoon to autobuild and drop gnome-btdownload
We're in :)

Monsoon had an open bug report on making it translatable, after the above decision was made, the priority on translations became pretty critical ;) Meebey volunteered to go and get translations all set up and spent a few hours yesterday getting that all sorted and creating the first translation (German). Olivier Dufour, who created a Winforms based GUI for MonoTorrent has also volunteered to do a French translation.

So, if anyone out there wants to translate Monsoon into their native language (or at least one they're good at ;) ), please join us on our new irc channel, #monsoon on irc.gimp.net/irc.gnome.org. We'll get you sorted out.

Hit Counter