I got some amazing news just a few minutes ago. I was just chatting away in #mono about how i was pushing a new release out soon, when anf6 (who is also called Alan) piped up with this:

|anf6| alan: I've been working on a WebUI for MonoTorrent, most of the functionality works ^_^
|alan| anf6: are you serious?
|anf6| yes
|alan| :o

It turns out that anf6 had been chatting to the developer of the uTorrent WebUI (Directrix) about the possibility of creating a MonoTorrent backend for the UI. Directrix was all for that! So, without further ado, here's the obligatory screenshot:

All i can say is: WOW! I can't wait for this to hit release!









UPDATE: I'd just found out that less than 300 lines of code were needed for the monotorrent -> uTorrent WebUI integration. How slick is that? That includes hosting MonoTorrent and a mini asp.net host to serve the pages.

A few days ago, i heard the suse team are looking for a new default torrent client for Suse 11.0. So of course I jumped in saying "Well, why not Monsoon?". So i fired off a response with a ton of links suggesting that Monsoon was the most amazing torrent client ever and there was no need to review any other clients. Of course that didn't work, but Monsoon has been taken into consideration as a viable candidate.

Andrew Wafaa took it upon himself to do a little review of his own, and Monsoon fared pretty well. I was happy enough with that. Some of the points mentioned on the thread include:

1) Applications which need wizards are bad. For a default client, you want something rough and ready.
2) Something minimalistic but packed with features is good. No massive complicated apps like Azureus.

Of course, 2 is a bit of a contradiction. It's quite hard to get both. In GUIs to expose a feature or some statistic, you need a new GUI element. So there's a fine balancing act between keeping the GUI clean and small while still exposing all the nice features in an easy to use way.

Here's a few screenshots of Monsoon anyway, which i hope expose the full diversity of what it can do.



The first time you start Monsoon, you aren't greeted by a wizard, everything is set up nicely for you. Everything is shown to you by default.





When you load torrents, they automatically start. If the files already exist (like the kubuntu one did), monsoon check that file to see how much of it is valid data and then resume the download using it. Standard stuff.





The ramp-up time is fairly fast. After 30 seconds Monsoon can connect to 50 people (the default limit for a torrent). Note, it is smart in how it connects to people. It doesn't have more than 5 pending connection attempts at any one time. This prevents overloading of cheaper routers.



For the people out there who want a really minimalistic client, you can hide everything. The GUI can be reduced down to something even slimmer than Transmission should you so wish. There is also plans to implement a mini-mode, which will consist of just a single progress bar per torrent.





So what if you want to set the global upload/download rate? Why, it's simple! Just click on the global upload/download speed indicators and away you go.





If you want to organise your torrents, it's just a simple context menu. You can add/remove/rename the labels with a right click, or double click. Slick, eh? There is also a patch in the works which will allow you to drag'n'drop torrents from the main view into a label and drag'n'drop them between labels. This will replace the existing way of doing it which involves ticking checkboxes in the 'Options' menu.

There's now a proper issue tracker for Monsoon and MonoTorrent. It's being hosted on the novell bugzilla (thanks guys!) so file your bugs there if at all possible. It makes it easier for me to monitor and record what exactly goes on.

The issue tracker can be found at in the 'Monsoon' section of Mono:Tools. If you have a bugzilla account already, this link should bring you right to the submission form:

https://bugzilla.novell.com/enter_bug.cgi?classification=&product=Mono%3A+Tools&submit=Use+This+Product&component=Monsoon

This is just a shout out to any budding Artistes, (especially that guy who did all those cool looking MonoDevelop icons): I'm looking for a nice logo for Monsoon. I have no ideas in mind, so any suggestions are welcome. Drop a comment here or send me an email if you have anything to show me :)

This is just a quick follow up from my last post. I decided to do a few benchmarks to see what transmission was like as compared the Monsoon (the GTK# gui for the MonoTorrent library). I've heard about tranmission before, but i've never used it. I was pretty shocked by the results. This is a quick summary of what I found:

1) Memory usage.
Transmission used less memory than MonoTorrent. That came as no surprise to me. No matter how efficently I code, i cannot ever get as efficent as an app written in C/C++. Mostly because in a .NET based app, the .NET runtime/jit must be loaded which consumes a few MB of memory. Percentage wise, yes, the difference is huge, but if you take a look at overall memory, it's not massive.

MonoTorrent: 35 Res / 20 Shared
Transmission: 20 Res / 13 Shared

This figure was gotten after extensively using the GUI by opening menu's and flicking around pages.

2) Hashing performance
I suppose another important metric - how long does it take to hash a complete file.
MonoTorrent: 95 seconds
Transmission: 85 seconds

This difference is fairly negligible. A full hash will rarely be performed. However, i suppose it was worth measuring. One optimisation i could make in monotorrent which would reduce that gap slightly would be if i read the next chunk of data off disk asynchronously while hashing the current chunk. At the moment it's all sequential, but i suppose it could easily enough be made parallel. It shouldn't make a huge difference though.

3) Download speeds
This would be the most important metric. This is where the biggest surprise came.
MonoTorrent: 15 seconds - 400kB/sec, 30 seconds - stabilised at 550-600kB/sec (maxed my connection) and connected to 50 people, the maximum allowed by my settings.
Transmission: 15 minutes: still at less than 50kB/sec and still only connected to 6 peers.

What am i doing wrong with transmission to make it so slow? It's not NAT (even though transmissions uPnP support cannot detect my uPnP enabled router) because i manually forwarded it in the end. I was using both svn head (r97353) of MonoTorrent and svn head (r5227) of Transmission when i ran this quick test.


EDIT: Just as i finished this, Transmission managed to connect to 3 additional people and one of them had a massive upload capacity which let Transmission reach ~480kB/sec. Still, why did that take so long? These results were consistent every time i started/stopped both transmission and Monsoon. Monsoon consistently maxed out my connection quickly whereas transmission consistently took forever to even break 40 kB/sec.

UPDATE: I just want to add that i tested using the ubuntu-7.10-desktop-i386.iso torrent on Suse 10.2.

So, a thought struck me. Ubuntu currently has a horrible default bittorrent client. It's about to be replaced by Transmission which, while definately a step up in the world, is still feature lacking (in my opinion) as compared to other available clients, most especially the great work done on the GTK Gui for MonoTorrent.

I propose that that the MonoTorrent GUI should be bundled with Ubuntu. MonoTorrent supports everything transmission does (except for Peer Exchange[1] and scheduling[2]) along with these extra cool features:

* Fast Peer Extensions - allow you to start a torrent faster
* Udp Tracker Protocol - allows you to use the bandwidth saving UDP protocol for tracker communications
* LibTorrent Extension Protocol - Ok, this is only partially supported. While it's implemented in SVN, it's not fully tested or enabled by default yet.
* RSS Feeds - Automagic downloading from RSS feeds. (see video)
* Organise downloads into tags (see video)
* Multi-Tracker Protocol support - If the main tracker is down, no worries! You can use the backup ones!
* Retarget Files before/while they are downloading i.e. you can rename a file as you download!
* Can monitor a folder and automatically load and download new torrents
* Minimizes to the gnome notification area.

The GUI including all supporting libs is a 384 kB package, and i think in the region of 600kB when installed[3]. What we're looking at is 600kB for a fully fledged, feature rich, uber snazzy bittorrent client. The best thing is [b]all[/b] the dependencies are already on the live CD.

So, if there are any Ubuntu devs out there, would you like to consider MonoTorrent as the default bittorrent app? If not, why not? I'll gladly do my best to fix any issues you have. Leave your comments below and lets get some discussion going!

[1] It'd take a few hours to complete support for this with tests.

[2] I did actually receive a patch to implement this a long while ago, but in the end, it fell by the wayside. I am a horrible person :(

[3] The package can be reduced a fair bit by splitting out the Tracker component of MonoTorrent using the Linker and removing bundled libraries which are available as packages nowadays. I'd guesstimate at least 100kB can be saved from the installed size from these optimisations. 70kB by removing the bundled DBus stuff and at the very least, 30kB by shrinking MonoTorrent.dll.

UPDATE: I'd just like to point out that an official name has been chosen for the GUI: [b]Monsoon[/b]. The next release will have everything rebadged to this name.

Its that time of the year when all us Europeans gather together for a big competition to decide who has the most friends. It's called the Eurovision. Last year, we did spectacularly. We were within inches of achieving our goal of finishing with zero points until those albanian b******s decided to give us 5 points. I swear, that was our plan all along!

Anyway, we learned our lesson last year. This time we're going to put in an entry that simply cannot fail! Ladies and gents, i present to you, Dustin The Turkey, singing our Eurovision entry. Save us all.

After my rather lengthy post about all the cool stuff I was doing with MonoTorrent, i was asked this question:

I'm very curious about what are the real possibilities of MonoTorrent library. I mean, I always thought Bittorrent was just a protocol for P2P file exchange, but when I see so much modularity on MonoTorrent... The question is, how can MonoTorrent help me as a programmer?

Well, that's a pretty good question.

I'm going to talk briefly about the three important new features that are (or will soon) be available:

1) Custom Peer Connections
Aim: To allow the user to route all peer traffic over whatever medium they want.

Use case 1: You are writing an application which requires that all connections are encrypted end-to-end. This means you have complicated routines to set up the connections. You want to add the ability to transfer files.

Use case 2: You want to route bittorrent traffic over a network such as Tor.

Use case 3: Restrictive firewall. Suppose only certain kinds of traffic are allowed through a particular firewall you have. You can push the bittorrent traffic inside a different protocol to allow it to pass through the firewall.

In case 1 and 2, we assume that normally it is impossible for monotorrent to create the connections itself. So what you do is create the necessary connections manually, then wrap them in the IConnection interface and pass them directly into monotorrent. Everything else is automatic.

For case 3, it might be possible to implement a HttpPeerConnection, whereby you push the bittorrent messages inside of a HTTPWebRequest and then send that to the remote peer and he decodes it back into it's orginal form for processing. Note: I don't recommend you do this to avoid getting around a firewall in work. Also, if you do implement something like that, only other clients who implemented that feature would be able to communicate with you, but it is an interesting application.

2) Custom Trackers
Aim: To allow someone to add peer data into the engine manually.

Use case 1: You want to type in your friends ip:port so you can directly connect to him

Use case 2: In your application, you keep a list of people you can/should connect to, and you know that MonoTorrent can create connect to them directly.

Use case 3: You want to implement an alternative peer source like the bittorrent DHT protocol

In all these cases, you just have to implement two simple methods to allow MonoTorrent to get data from your source. For case 1, whenever you enter the ip:port combination, you just raise an event with that IP:Port stored in the event args.

Case 1 is a manual event, and so MonoTorrent would never query that 'Tracker' for peer details, but for case 2 and 3, it's more fun. All you do is wait for MonoTorrent to call the 'Announce' method on your tracker, then you go off and find the peers and raise the AnnounceComplete event when you're ready. It's as easy as pie.


3) Custom Messages

Use case 1: You're lazy and don't want to define your own messaging protocol or message format and want things to 'Just Work (tm)', so just piggy back all your messages over the MonoTorrent system.

Use case 2: You want/need to send a few pieces of information between torrent clients but don't want to have to set up a separate communication channel to do so in.

Use case 3: You want to make Bittorrent 2.0.

You need to implement a slightly more complex interface than the previous two examples, but it's nothing amazingly difficult. The helper methods for reading/writing the basic types from a byte[] are all there to make creating a message that much easier. So you're left with only having to implement a mere 4 abstract methods. Whilest this feature isn't finished as yet, it will be as simple as defining your custom message, then making a call like:

byte messageId = 9;
Message.Register(messageId, delegate { return new MyCustomMessage(); });

or

Message.register(messageId, delegate (TorrentManager m) {
return new MyCustomMessage(m.SomeData, m.OtherData);
});

Once that's done, you can queue up your message to be sent and react to it's arrival with ease. What you might want to send, i don't know. But the ability is there for you to send it.

So in brief, MonoTorrent can be used as a drop in replacement for either '1 to 1', '1 to many', 'many to 1' or even 'many to many' transfers. I'm not saying it *should* be used for all those cases, it may be the heavy weight alternative to a simple Socket.Send, but the possibility is there. You can send customised messages to the other clients so that they can perform custom logic and you can insert peer details easily from any source.

It was on 28th January that i released MonoTorrent 0.20. Here we are, less than a month later and already i have some cool new stuff to blog about, this is a lot better than the time gap between the last two MonoTorrent releases.

So, what's new? LOTS! Let me give a brief overview of what's possible nowadays.

1) Custom Peer Connections
It's now possible to plug monotorrent directly into another application and have monotorrent uses connections created by that application. The advantage of this is that you can tunnel MonoTorrent through anything now. You can create an encrypted connection whatever way you want and then just pass it straight into monotorrent and it'll be used. This also makes it easy to push traffic through networks like Tor (or whatever).

2) Custom Trackers
First, i'm not talking about the server aspect of MonoTorrent. I'm talking about the clientside class used to deal with announcing/scraping to a server. It's now possible to (very easily) implement your own tracker class. What use is this you say? Well, quite a bit actually! If you want to be able to add peer data into monotorrent (but not active connections as i described above) you can subclass the MonoTorrent.Client.Tracker class and do it from there.

Suppose you want to implement a new protocol for client-tracker communications. If you want MonoTorrent to support it, just inherit from MonoTorrent.Client.Tracker, register that implementation with the TrackerFactory and you're done! You don't have to modify one letter of MonoTorrent source code to add in support for your new protocol.

3) Custom Messages
Ok, so technically speaking, this still isn't possible yet, however all the architecture to enable it is there. I just need to expose it publicly in a nice way. Anyway, when i implemented support for the libtorrent extension protocol, I realised that the way i handled peer messages just wouldn't work. Basically i used a giant switch statement to select the correct message to decode. This meant that all messages had to be defined within MonoTorrent and had to exist at compile. There was no way for the user to define their own message type.

So, instead i updated the API to use some .NET 2.0 loveliness to allow for custom messages to be defined outside of MonoTorrent and still allow them to be decoded and handled correctly.

4) Custom Writers
If i was to pick a favourite change out of the list, it would be the custom messages, but this comes a close second ;) You can now define you're own 'PieceWriter' class. What this allows you to do is to redirect disk read/write requests wherever you want them to go. Of course, the logical first implementation is a MemoryBuffer.

Yes, MonoTorrent will now cache reads/writes in memory where prudent. For example, each 'piece' you download consists of multiple 'blocks'. Previously each block was written straight to disk as it arrived. When all the blocks had arrived, they were all read straight back off the disk so that a SHA1 hash of the piece could be generated. This was very inefficient.

5) NUnit test-able!
Yeah, so this isn't really a new feature. In fact, it's something i should've been doing all along, but due to the implementation of MonoTorrent, certain things were far too awkward to test. Previously, in order to test internal logic i'd have to:

1. Create a fake .torrent file and write it to disk.
2. Create fake data and write it to disk so that it can be 'downloaded' by one client and 'seeded' by the other
   
3. Instantiate a MonoTorrent.Tracker and load the torrent into it
4. Instantiate two MonoTorrent.Clients and load the torrent into them.
5. Hit 'start' and measure what little i could

This method means that it's quite possible temp files will be left on the disk - not good. Also, if something does go wrong, there is no way for me to find out exactly what went wrong. The Client would close the connection and all i'd get would be a ConnectionClosed event. Finally, it was just plain awkward! Creating a fake torrent and the necessary fake files was a pain in the ass.

Once the above changes had been complete, it became so much easier. Now I create a fake torrent in memory (no need to write it to disk!). I create one MonoTorrent.Client using the in-memory torrent and redirect its reads/writes using a custom PieceWriter. This writer just peforms in-memory operations to fill in the fake data as required. I then create a custom PeerConnection and pass one end of the connection into the engine and hold the other end myself.

This means when i send a message into the engine, i can receive the reply and verify that it is as expected using NUnit tests. Pretty sweet.

In other news, the following other nifty things have been implemented:

A) Udp Tracker support
I finally got around to implementing UDP tracker support. It has been requested by a few people, so here it is. Enjoy!

B) Better FastResume support
There were a few issues with the old fast resume support. In certain advanced use cases, fast resume support just plain didn't work. This has now been fixed. It's now up to the user to handle where/how fast resume data is stored. This is nearly finished, but not quite. It's on my TODO list for the coming week.

C) Message Bundles
An entirely invisible change, but one i want to talk about. Whenever a piece is downloaded succesfully, a 'have' message should be sent to all peers you're connected to. This is used as a way to keep track of who has what. However, have messages are tiny and typically you'd complete a few pieces a second. This means that several times a second, for each connected peer, a single 12 byte message has to be sent. This is hardly worth the effort of calling socket.BeginSend!

So, why not encode several messages at a time into the send buffer and send em all at once? Well, that just doesn't work. The architecture just does not allow for that to happen. One message and one message alone can be encoded and sent at a time.

The solution? The MessageBundle. The MessageBundle is just another implementation of the PeerMessage base class, except the message bundle can store multiple messages inside it. When you call Encode() on the bundle, it encodes all it's stored messages into the buffer. This allows me to send multiple messages at a time, but for all intents and purposes, MonoTorrent thinks i'm sending one.

Now HaveMessages are delayed and bundled together.

I'll release all this fanciness sometime in the near future.

What kind of people do credit card companies like to give credit cards to? People who always pay their bills? People who occasionally forget and pay a few days late? How about people who forget quite often, or can't afford to pay the full bill in one month?

Well, it looks like the last group is exactly who Egg want! They have issued notice to 7% of their customers informing them that due to their bad credit rating, their credit card is being cancelled with no right to appeal. According to a fair number of customers, they have excellent credit ratings and always paid their bill in full and on time. It looks like they're being dumped because they're just not profitable.

It's kind of ironic that the 'best' customers are the ones being dumped in favour of customers who may not actually be able to pay their bills.

So, after asking around about my new internet situation, i was informed that there was no way for me to set it up as i wanted to. I had to have my WRT54G as the slave and the new neatgear WGR614 as my master device. With that in mind, i reattached the netgear to my new NTL broadband and put the WRT54G back to it's old position beside the two wired computers.

Just in case anyone else needs to do something like this, this is how you create a wireless bridge between two routers:


Equipment needed:
1) A router which supports the 'Client' mode. In my case, a linksys WRT54G with the openwrt firmware, the 'slave' router.
2) Any other wireless router, the 'master' router.


Setup:

Master Router
1) Connect it to your internet box (be it an ADSL modem or whatever).
2) Set up the wireless connection as per normal.
3) Enable uPnP and DHCP as required.

This router will accept all wireless connections, and can also accept wired connections via it's LAN ports as per usual. All port forwarding should be done via this router.

Slave router
1) Connect any wired computers to LAN ports on the router. Make sure nothing is connected to the WAN port. You won't be using this.
2) Set the wireless to use the same SSID (network name) and frequency as the master router.
3) Set the wireless connection to use 'Client' mode. In the case of openwrt, the correct mode is called 'Client (bridged)'. In this mode, the router will look for the wireless network with the SSID you specified and connect to it like a regular laptop.
4) Disable uPnP and DHCP on this router.

Note: The slave router cannot accept wireless connections anymore. It can only forward wired computers to your master router.

With that done, my two desktops remain in the same old room as always and connect wirelessly to my master router, thus giving them access to the net and having them accessible to the wireless clients.

The only problem is that the new router is a bit limited in it's configuration and so will end up being replaced in the near future. For the moment, it does the job fine.

I recently changed internet providers, and i have this situation:

The new internet arrives in via a cable in my front room. There are two desktops in the house, which are in the back room. The old internet came through a hole in the wall into this room, so everything was peachy.

I'm now in possession of a linksys WRT54G (with openwrt firmware) and a neatgear WGR614 V7. What i want to do is to bridge the two wireless routers so that i have a single network containing my wireless+wired computers.

Does anyone know if there's a modified/alternate firmware for the netgear router which supports 'client' mode in wireless networks so i can just make it connect to my main router (the linksys). I'd much prefer to use this as my main router as opposed to the netgear. Any advice on getting this set up would be great. Ideally i wouldn't have to buy new hardware, but if it comes to it, thats what i'll do ;)

MonoTorrent 0.20 has just been shoved unwillingly out of it's cosy spot in SVN into full public view. The last time i tagged and released was way back on July 4th. Far too long ago! Full details of the release can be read here. Download links here aswell.

Here's the changelog:
Client
Features
Linear piece picking possible
- This mode of operation chooses pieces towards the start of the file where possible. This should not be a publicly settable option in a GUI. It could be automatically set until the first 20 pieces in a torrent are received to allow for previewing. For general usage, this is a very inefficient way to download.
Full choke/unchoke algorithm implemented (Andy Henderson)
- Full tit-for-tat algorithm implemented. This improves download rates and upload rates and is pretty slick.
Sub-optimal implementation of code used to connect to peers made optimal
- Fixed corner case where a connection wouldn't be cleaned up correctly
- Significantly faster startup performance in cases where remote peers don't support encryption and encryption is enabled
Rate limiting for Disk IO implemented
Configurable amount of open filestreams
Reduced disk thrashing when multiple torrents hashing simultaneously
Files are not preallocated anymore
TorrentManager will never block when .Start() is called
The Tracker base class made more extensible (Eric Butler)
- An example usage would be if a person had peer details stored in a database. They could implement a 'DataBaseTracker' class, register it with the engine, then monotorrent could announce to that tracker and retrieve peers from the database. Thanks to Eric, this is hugely simplified and is now trivial to implement.




Bug Fixes
Piece Requesting
Fixed bug where a piece could be requested twice
When a corrupt piece is received, ensure all contributing peers are marked
Fixed a possible null ref when removing pending piece requests when a connection is closed

Torrent Creator
When creating an (optional) MD5 hash in the torrent creator, open the files in read-only mode (Roger Zander)
Creating torrents no longer spins up an extra thread when using the synchronous method (Eric Butler)
Fixed possible race conditions when creating a torrent using the asynchronous methods
Private key is retained when creating torrents

Misc
Per-file progress is updated correctly.
Fixed detection of when a tracker supports scrape - It should have been case insensitive
Added extra check to make sure a scrape request is only performed if the tracker supports scraping
Added extra logic checks to help prevent the loading of incorrect fastresume data
When a peer sends a HaveAll message - it is correctly marked as being a seeder
PieceHashed event fired under all circumstances



Tracker
The MonoTorrent.Tracker got a major rewrite. Highlights include:
- more than 1000x memory reduction for trackers hosting a large number of torrents. 1000 torrents can be hosted in ~30kB. Previously would have required ~30MB.
- Faster more efficient announce handling. For large trackers, lookups are significantly faster
- Vastly simplified announce handling. It is now trivial to write code to handle incoming connections from differing sources. A typical example would be to create an AspNetHandler to handle connections in an ASP.NEt project. It requires less than 20 lines of code to be written.



It should be fairly solid, with a lot of cool feature enhancements under the hood. There are a few cool things which i managed to implement this time round like rate limiting disk IO, not pre-allocating 100% of the file on startup, being able to properly handle a torrent with 100,000 files in it. Good stuff.

Plans for the future are pretty cool aswell. I'll blog more about that later though.

Today i'm going to talk about optimisation a little, this is mostly in the MonoTorrent Tracker context, but some of the ideas still apply to other situations.


Sometimes you hear people talking about how they want to optimise their code to make it faster, and they ask questions like 'what's the fastest way to multiply by 2, bitshift or regular multiplication?', or 'Should I null out objects the very second I'm finished using them or just wait for them to fall out of scope?'. These kind of questions are probably the epitome of premature optimisation. These kind of optimisations won't make your application faster or better.

I had done a lot of work on the Tracker code (the 'server' portion of the bittorrent specification) recently. I had precomputed values which are used regularly, i had optimised the hashcodes used in my dictionary lookups, i had reduced the amount of data that needs to be kept in-memory significantly. I wanted to see what effect all this had on the actual running of the tracker. This was the first time i had run a benchmark on the tracker.

I went on the net, found a big tracker and checked it's stats. Using these, i decided that this was benchmark was representative of a heavy real-world load:


1) Load 2000 torrents into the engine, each of which contains 1000 peers.
2) Hammer the server with 1000 requests a second choosing a random torrent and random peer from the list and make a fake request from them to the server.


So, once that was written, i fired up the tracker and ran the benchmark. My system locked up and i was forced to hard-reboot. What had gone wrong! I started the benchmark again, but monitored memory and CPU usage carefully. I was surprised to find that memory usage was rocketing, which is what caused the massive slowdown of my system! I couldn't understand why. I did a few quick calculations to figure out how much memory i'd expect the tracker to use, they put final memory usage at far less than 300MB. I quickly whipped out my allocation profiler and began optimising. Here are the 'issues' i fixed:

1) The objects i was using as the 'key' in a dictionary were being recreated every time i used them. Typically this means that for every request to the tracker, at least a dozen complex objects were created/garbage collected needlessly. In this kind of scenario, the objects should be declared as 'static readonly' and reused. I implemented this change.

The benchmark still couldn't run.

2) I decided that the next problem was that i was pre-generating two byte[] for each peer when they were added to the server. This was so that a request could be fulfilled by simply copying pregenerated byte[]. I changed this to generate the byte[] at request time rather than storing it in memory. I expected this to fix the issue.

The benchmark still couldn't run after this change.


3) Finally, i noticed there were a huge number of hashtable related objects being retained in memory. This was a bit weird. There shouldn't have been that many around. A few minutes of checking the code made me realise that the probably cause was keeping a NameValueCollection object in-memory for each peer. I rewrote the peer class to extract the necessary information from the collection and then dump it, rather than holding a reference to it.

The benchmark could run!

Now, the memory improvements were gigantic. Previously i had stats like this:
Active Torrents: 500
Active Peers per Torrent: 500
Memory: 350MB

Now i had:
Active Torrents: 500
Active Peers per Torrent: 500
Memory: 40MB

Active Torrents: 2000
Active Peers per Torrent: 1000
Memory: 140MB

There is no way in hell I'd ever have found the cause of the issue unless i had run a profiler. So, for anyone who is trying to make their code run fast and efficiently, you need a profiler. You can't get away without it.

As you've probably heard by now, Aaron has just released a new version of Banshee has just. Good stuff! As part of the blogpost detailing the changes and features in the new version he says:

Alan McGovern has nearly completely rewritten our MTP support and we are close to enabling it by default. The next release should have solid MTP support, and we hope to roll 0.13.3 within the next two weeks. The new MTP support uses libmtp instead of libgphoto2. This decision was made for a number of reasons, though I am really not informed enough to convey them properly.

Well, i never detailed the reasons for the switch in my previous blogpost, so i may as well do it now. I'll also mention the current issues with libmtp for those of you interested in what those issues are.

Reasons for the switch:
The primary reason is that to use libgphoto as the mtp backend, you need svn head of libgphoto and svn head of libgphoto-sharp. This issue is actually a pretty major issue. If banshee wanted to ship MTP support, it would have to ship it's own version of libgphoto and libgphoto-sharp. The other alternative was to enable MTP support by default, but the end-user wouldn't be able to use it until they compiled and installed libgphoto themselves or their distro packaged the required version of libgphoto. This version is likely to be released in the next month or two (i believe), but could take several months to propagate through the various distros.

With libmtp, banshee can run with version 0.2.0 or higher, which means anything newer than Aug 4th 2007. This is a much more workable solution.


Secondly, API: The libmtp API is aimed at mp3 players. Banshees API is also aimed at mp3 players, this makes it significantly easier to map the banshee API to the libmtp API. Several tasks such as getting track listings, uploading tracks, and retrieving tracks are much simpler in libmtp.

Thirdly, with libgphoto, downloading or uploading a track requires copying the entire file to memory in C#, then pushing that into libgphoto (which duplicates the memory usage) then the data is pushed to the device. There is an alternative method whereby i create a unix file descriptor which is then passed into libgphoto and the file is read/written to that. However, this is not ideal as there's a good chance that half written temp files can be left over in the event of a crash.

With libmtp i can pass in a file path and libmtp will write the file directly from that path or read it directly from that path. This means the memory copying is completely removed resulting in much better performance.

Fourthly, Whilest libgphoto can (technically speaking) run on windows, the necessary code hasn't been written to allow this to happen. libmtp can be compiled and run on windows (or so the documentation says). As there is work to make banshee run on windows, this is an advantage.

Finally, playlists. I'm still unsure how to create playlists using libgphoto. It'd require a fair bit of messing around and experimenting for me to figure out exactly how it should work. With libmtp, this is trivial to do.


However, libmtp has some pretty big issues at the moment, which only really came to light a few hours before the release.

If you connect an MTP device and banshee loads it up, connecting a second DAP (of any kind) will make libmtp choke. Once there's an active device, any call into libmtp to list available devices throws a (non-fatal) error. This means if you plug in a second device, you get an annoying error message popping up. If you do own two devices and were trying to copy from one to the other, this is impossible.

There was also an issue linking a device reported by libmtp to a device reported by HAL. There is no sure-fire way of doing this using the libmtp API. Under libgphoto this was possible.There are a few possible workarounds for this, but none of them are ideal.

Banshee now has support for MTP devices in trunk. It's a long and torturous story, but here's the short of it:

Way back when, the work on implementing MTP support was started by a guy called Patrick. At the time, there were two options:
1) libgphoto
2) libmtp

libmtp was still in it's early stages of development and wasn't mature enough to be a viable option, so libgphoto was chosen althought it wasn't that much better. At this time, libgphoto didn't have metadata reading/writing implemented, didn't show any stats on the different filesystems in the device. Note: libmtp was no better at the time. So, a C# binding to libgphoto was started and basic support for syncing was eventually pushed into banshee.

Fast forward about 18 months. libgphoto has matured significantly in its support for mp3 players. Track metadata has full read/write support, you can get stats on the filesystems, patches and optimisations have been pushed in so that all the basic operations are fast. However, there's still issues with long filename support - there are hardcoded limits in libghoto itself which means if it encounters a path or filename longer than the limit, it just bails out. For mp3 players, this was a serious issue as these limits were quite frequently reached. For instance, if you were a fan of Sufjan Stephens, you were out of look. His incredibly long track titles like 'Out of Egypt, Into the Great Laugh of Mankind, and I Shake the Dirt From My Sandals as I Run' really throw a spanner in the works

Around this time, i stepped in to complete the binding and complete banshee MTP support. I pestered poor Marcus for quite a while until i got various fixes/updates into libgphoto such as the change to allow unlimited filename length and path length, and also to remove 'long' types from the public API (as a 'long' type can be 32bit or 64bit depending on architecture and leads to nasty complications binding it in C#). I also completed the initial revision of the libgphoto-sharp API restructuring. The entire endevour probably took about 2 months from start to finish.

However, the problem now was that in order to have MTP support in banshee, you needed SVN head of libgphoto, svn head of libgphoto2-sharp and a pot of luck that you could get it installed without screwing up your packaged version. A new release of libgphoto was several months away and probably still a further few months until it managed to trickle down into the various distros.

It was then that i was reminded by Aaron that libmtp still existed and was quite mature. I took a quick look at the API showed that it was tailored specifically for the kind of tasks that banshee was trying to do. Better yet, it had a faster release cycle, it didn't require svn head of anything in order to get all the features that were required and it was already available in nearly every distro.

I spent about 4 days writing a brand new C# wrapper for libmtp and then replaced the libgphoto-sharp code in banshee with the new libmtp-sharp code. While there are things in libmtp that are missing as compared to libgphoto, the simplicity of the new API makes it trivial to implement features which previously would have been quite complicated. For example, Playlist support under libmtp is trivial to implement, yet under libgphoto i'm not 100% sure how i'd go about it!

What i regret most of all out of this is that i put Marcus through so much hassle for minimal benefits as Banshee is no longer using libgphoto. While i do feel bad about putting so much effort into the libgphoto-sharp binding to never use it myself, it was still the best decision from banshee's point of view.

So, if anyone out there has an MTP device and wants to make sure that their device works with banshee before the release (so i can fix any bugs should there be any), check out the guide here. Post any bug reports on http://bugzilla.gnome.org under the DAAP component in Banshee.

Looky here, who wants a giant network accessible harddrive which won't let you share the 26 most common filetypes. Honestly, why would anyone buy a piece of hardware which is unbelievably crippled and completely unsuitable for the purpose it's being advertised for. And more importantly, why would someone design such a horrible piece of hardware?

On a similar, but more sinister note, if america thinks you might be guilty of a crime, then they feel they can just kidnap you. The mentality of guilty until proven innocent really is showing through here. I wonder what would happen if i kidnapped an american citizen and brought them to ireland because i suspected them of a crime. I'm sure America wouldn't view that too kindly.

My christmas exams (worth 35% of my degree) are getting closer and closer. Pretty much every day starting from last saturday has turned into a 10:30am->7:30pm study-a-thon. I'm trying to learn in the space of a 10 days what i should've been learning over the last 12 weeks. It's mostly working.

As a result, i've become a minor insomniac. Tis only 2pm, i've only been trying to sleep for 3 hours now. This describes it pretty well:

In two short days i'll be flying out to madrid for the mono summit. Woohoo! Turns out i'm missing my last 3 days of term, in which i should be getting two tutorials and our first problem sheet for a particular subject. So, not the best time to be fecking off to a foreign country.... ah well!

Also, if that F-Spot guy is about the place who was asking me about upgrading F-Spot to use the latest libgphoto2-sharp, give me a shout via email and let's organise something before i head over. Otherwise, just try and find me between wednesday evening and friday evening.

p.s. if i fail any exams because of this, i'm holding Aaron responsible, he tempted me over with a Chicken Pag! So no new MTP code for banshee if i fail ;)

I heard some great news earlier in the week and just got around to checking it out. There's another C# implementation of the BitTorrent protocol being developed! Good stuff! They've been working on it for about a year, and have "the core of the beast" pretty much complete. Thats great stuff! Considering i thought i was the only person with a C# based bittorrent implementation, i thought it'd be great to have a chat with them, see how they solved different issues etc.

So, about 15 mins ago, i loaded up that link and decided to download their alpha client, just to see how it works etc. Thats when i became seriously pissed off.

Without even having run the 'alpha client' i realised that the seedpeer codebase was actually MonoTorrent*. Yes, these guys had the cheek to take my code and pretend it was theirs. They tried to pass off about 18 months of my development time as their development time, without so much as giving me credit or mentioning that they were, in fact, lying when they said "we have almost finished coding the core of the beast". The worst thing is, they weren't even smart enough to do it properly.

So, for anyone out there who has actually used SeedPeer, or has a blog who've blogged about it, please try to make it known that seedpeer == MonoTorrent, except that they don't even use all the features i have available. For example, here's there 'TODO' list:

• MultiTracker
• Torrent Creation
• Custom DHT (faster)
• Rss Feeds
• Integrated Search Function (with commenting)
• Custom Peer Exchange
• Fast Resume
• Global/Per torrent Settings
• Download Priority
• more...

Here's a list of the features that are already in monotorrent that are in their TODO list:

• MultiTracker
• Torrent Creatio
• Rss Feeds
• Fast Resume
• Global/Per torrent Settings
• Download Priority
• more...

I suppose this is what you expect by making the code public. People will use it (which is what i want), but then they'll try to pass it off as their own (which is just plain wrong). So, if anyone wants to email the seedpeer people, their contact form is here, so feel free to let them know you know ;) Also, if you want a nice GUI for monotorrent, check out www.monotorrent.com :)

* Yes, i am 100% sure of this. I haven't used reflector to verify, but i'm 100% sure that at least 95% of the code is a direct copy/paste from monotorrent. It might even be 100%. If someone is bored, they can use reflector (or whatever) so they can see i'm not lying. No better proof than looking yourself.

Also, if the seedpeer guys do read this, using an obfusticator won't prevent me from recognising my own code. I can think of a dozen ways offhand to check if a client is my client without having to see the code. It's ridiculously easy.

EDIT: For those of you who know these things, The middle paragraph of the MIT/X11 license states:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

Does that mean if they distribute a binary, they have to include that license header in some form or other?