Wednesday, February 27, 2008

Its that time of the year when all us Europeans gather together for a big competition to decide who has the most friends. It's called the Eurovision. Last year, we did spectacularly. We were within inches of achieving our goal of finishing with zero points until those albanian b******s decided to give us 5 points. I swear, that was our plan all along!

Anyway, we learned our lesson last year. This time we're going to put in an entry that simply cannot fail! Ladies and gents, i present to you, Dustin The Turkey, singing our Eurovision entry. Save us all.

Friday, February 22, 2008

After my rather lengthy post about all the cool stuff I was doing with MonoTorrent, i was asked this question:
I'm very curious about what are the real possibilities of MonoTorrent library. I mean, I always thought Bittorrent was just a protocol for P2P file exchange, but when I see so much modularity on MonoTorrent... The question is, how can MonoTorrent help me as a programmer?
Well, that's a pretty good question.

I'm going to talk briefly about the three important new features that are (or will soon) be available:

1) Custom Peer Connections
Aim: To allow the user to route all peer traffic over whatever medium they want.

Use case 1: You are writing an application which requires that all connections are encrypted end-to-end. This means you have complicated routines to set up the connections. You want to add the ability to transfer files.

Use case 2: You want to route bittorrent traffic over a network such as Tor.

Use case 3: Restrictive firewall. Suppose only certain kinds of traffic are allowed through a particular firewall you have. You can push the bittorrent traffic inside a different protocol to allow it to pass through the firewall.

In case 1 and 2, we assume that normally it is impossible for monotorrent to create the connections itself. So what you do is create the necessary connections manually, then wrap them in the IConnection interface and pass them directly into monotorrent. Everything else is automatic.

For case 3, it might be possible to implement a HttpPeerConnection, whereby you push the bittorrent messages inside of a HTTPWebRequest and then send that to the remote peer and he decodes it back into it's orginal form for processing. Note: I don't recommend you do this to avoid getting around a firewall in work. Also, if you do implement something like that, only other clients who implemented that feature would be able to communicate with you, but it is an interesting application.

2) Custom Trackers
Aim: To allow someone to add peer data into the engine manually.

Use case 1: You want to type in your friends ip:port so you can directly connect to him

Use case 2: In your application, you keep a list of people you can/should connect to, and you know that MonoTorrent can create connect to them directly.

Use case 3: You want to implement an alternative peer source like the bittorrent DHT protocol

In all these cases, you just have to implement two simple methods to allow MonoTorrent to get data from your source. For case 1, whenever you enter the ip:port combination, you just raise an event with that IP:Port stored in the event args.

Case 1 is a manual event, and so MonoTorrent would never query that 'Tracker' for peer details, but for case 2 and 3, it's more fun. All you do is wait for MonoTorrent to call the 'Announce' method on your tracker, then you go off and find the peers and raise the AnnounceComplete event when you're ready. It's as easy as pie.

3) Custom Messages

Use case 1: You're lazy and don't want to define your own messaging protocol or message format and want things to 'Just Work (tm)', so just piggy back all your messages over the MonoTorrent system.

Use case 2: You want/need to send a few pieces of information between torrent clients but don't want to have to set up a separate communication channel to do so in.

Use case 3: You want to make Bittorrent 2.0.

You need to implement a slightly more complex interface than the previous two examples, but it's nothing amazingly difficult. The helper methods for reading/writing the basic types from a byte[] are all there to make creating a message that much easier. So you're left with only having to implement a mere 4 abstract methods. Whilest this feature isn't finished as yet, it will be as simple as defining your custom message, then making a call like:

byte messageId = 9;
Message.Register(messageId, delegate { return new MyCustomMessage(); });


Message.register(messageId, delegate (TorrentManager m) {
return new MyCustomMessage(m.SomeData, m.OtherData);

Once that's done, you can queue up your message to be sent and react to it's arrival with ease. What you might want to send, i don't know. But the ability is there for you to send it.

So in brief, MonoTorrent can be used as a drop in replacement for either '1 to 1', '1 to many', 'many to 1' or even 'many to many' transfers. I'm not saying it *should* be used for all those cases, it may be the heavy weight alternative to a simple Socket.Send, but the possibility is there. You can send customised messages to the other clients so that they can perform custom logic and you can insert peer details easily from any source.

Monday, February 18, 2008

It was on 28th January that i released MonoTorrent 0.20. Here we are, less than a month later and already i have some cool new stuff to blog about, this is a lot better than the time gap between the last two MonoTorrent releases.

So, what's new? LOTS! Let me give a brief overview of what's possible nowadays.

1) Custom Peer Connections
It's now possible to plug monotorrent directly into another application and have monotorrent uses connections created by that application. The advantage of this is that you can tunnel MonoTorrent through anything now. You can create an encrypted connection whatever way you want and then just pass it straight into monotorrent and it'll be used. This also makes it easy to push traffic through networks like Tor (or whatever).

2) Custom Trackers
First, i'm not talking about the server aspect of MonoTorrent. I'm talking about the clientside class used to deal with announcing/scraping to a server. It's now possible to (very easily) implement your own tracker class. What use is this you say? Well, quite a bit actually! If you want to be able to add peer data into monotorrent (but not active connections as i described above) you can subclass the MonoTorrent.Client.Tracker class and do it from there.

Suppose you want to implement a new protocol for client-tracker communications. If you want MonoTorrent to support it, just inherit from MonoTorrent.Client.Tracker, register that implementation with the TrackerFactory and you're done! You don't have to modify one letter of MonoTorrent source code to add in support for your new protocol.

3) Custom Messages
Ok, so technically speaking, this still isn't possible yet, however all the architecture to enable it is there. I just need to expose it publicly in a nice way. Anyway, when i implemented support for the libtorrent extension protocol, I realised that the way i handled peer messages just wouldn't work. Basically i used a giant switch statement to select the correct message to decode. This meant that all messages had to be defined within MonoTorrent and had to exist at compile. There was no way for the user to define their own message type.

So, instead i updated the API to use some .NET 2.0 loveliness to allow for custom messages to be defined outside of MonoTorrent and still allow them to be decoded and handled correctly.

4) Custom Writers
If i was to pick a favourite change out of the list, it would be the custom messages, but this comes a close second ;) You can now define you're own 'PieceWriter' class. What this allows you to do is to redirect disk read/write requests wherever you want them to go. Of course, the logical first implementation is a MemoryBuffer.

Yes, MonoTorrent will now cache reads/writes in memory where prudent. For example, each 'piece' you download consists of multiple 'blocks'. Previously each block was written straight to disk as it arrived. When all the blocks had arrived, they were all read straight back off the disk so that a SHA1 hash of the piece could be generated. This was very inefficient.

5) NUnit test-able!
Yeah, so this isn't really a new feature. In fact, it's something i should've been doing all along, but due to the implementation of MonoTorrent, certain things were far too awkward to test. Previously, in order to test internal logic i'd have to:
  1. Create a fake .torrent file and write it to disk.
  2. Create fake data and write it to disk so that it can be 'downloaded' by one client and 'seeded' by the other
  3. Instantiate a MonoTorrent.Tracker and load the torrent into it
  4. Instantiate two MonoTorrent.Clients and load the torrent into them.
  5. Hit 'start' and measure what little i could
This method means that it's quite possible temp files will be left on the disk - not good. Also, if something does go wrong, there is no way for me to find out exactly what went wrong. The Client would close the connection and all i'd get would be a ConnectionClosed event. Finally, it was just plain awkward! Creating a fake torrent and the necessary fake files was a pain in the ass.

Once the above changes had been complete, it became so much easier. Now I create a fake torrent in memory (no need to write it to disk!). I create one MonoTorrent.Client using the in-memory torrent and redirect its reads/writes using a custom PieceWriter. This writer just peforms in-memory operations to fill in the fake data as required. I then create a custom PeerConnection and pass one end of the connection into the engine and hold the other end myself.

This means when i send a message into the engine, i can receive the reply and verify that it is as expected using NUnit tests. Pretty sweet.

In other news, the following other nifty things have been implemented:

A) Udp Tracker support
I finally got around to implementing UDP tracker support. It has been requested by a few people, so here it is. Enjoy!

B) Better FastResume support
There were a few issues with the old fast resume support. In certain advanced use cases, fast resume support just plain didn't work. This has now been fixed. It's now up to the user to handle where/how fast resume data is stored. This is nearly finished, but not quite. It's on my TODO list for the coming week.

C) Message Bundles
An entirely invisible change, but one i want to talk about. Whenever a piece is downloaded succesfully, a 'have' message should be sent to all peers you're connected to. This is used as a way to keep track of who has what. However, have messages are tiny and typically you'd complete a few pieces a second. This means that several times a second, for each connected peer, a single 12 byte message has to be sent. This is hardly worth the effort of calling socket.BeginSend!

So, why not encode several messages at a time into the send buffer and send em all at once? Well, that just doesn't work. The architecture just does not allow for that to happen. One message and one message alone can be encoded and sent at a time.

The solution? The MessageBundle. The MessageBundle is just another implementation of the PeerMessage base class, except the message bundle can store multiple messages inside it. When you call Encode() on the bundle, it encodes all it's stored messages into the buffer. This allows me to send multiple messages at a time, but for all intents and purposes, MonoTorrent thinks i'm sending one.

Now HaveMessages are delayed and bundled together.

I'll release all this fanciness sometime in the near future.

Thursday, February 07, 2008

What kind of people do credit card companies like to give credit cards to? People who always pay their bills? People who occasionally forget and pay a few days late? How about people who forget quite often, or can't afford to pay the full bill in one month?

Well, it looks like the last group is exactly who Egg want! They have issued notice to 7% of their customers informing them that due to their bad credit rating, their credit card is being cancelled with no right to appeal. According to a fair number of customers, they have excellent credit ratings and always paid their bill in full and on time. It looks like they're being dumped because they're just not profitable.

It's kind of ironic that the 'best' customers are the ones being dumped in favour of customers who may not actually be able to pay their bills.

Sunday, February 03, 2008

So, after asking around about my new internet situation, i was informed that there was no way for me to set it up as i wanted to. I had to have my WRT54G as the slave and the new neatgear WGR614 as my master device. With that in mind, i reattached the netgear to my new NTL broadband and put the WRT54G back to it's old position beside the two wired computers.

Just in case anyone else needs to do something like this, this is how you create a wireless bridge between two routers:

Equipment needed:
1) A router which supports the 'Client' mode. In my case, a linksys WRT54G with the openwrt firmware, the 'slave' router.
2) Any other wireless router, the 'master' router.


Master Router
1) Connect it to your internet box (be it an ADSL modem or whatever).
2) Set up the wireless connection as per normal.
3) Enable uPnP and DHCP as required.

This router will accept all wireless connections, and can also accept wired connections via it's LAN ports as per usual. All port forwarding should be done via this router.

Slave router
1) Connect any wired computers to LAN ports on the router. Make sure nothing is connected to the WAN port. You won't be using this.
2) Set the wireless to use the same SSID (network name) and frequency as the master router.
3) Set the wireless connection to use 'Client' mode. In the case of openwrt, the correct mode is called 'Client (bridged)'. In this mode, the router will look for the wireless network with the SSID you specified and connect to it like a regular laptop.
4) Disable uPnP and DHCP on this router.

Note: The slave router cannot accept wireless connections anymore. It can only forward wired computers to your master router.

With that done, my two desktops remain in the same old room as always and connect wirelessly to my master router, thus giving them access to the net and having them accessible to the wireless clients.

The only problem is that the new router is a bit limited in it's configuration and so will end up being replaced in the near future. For the moment, it does the job fine.

Friday, February 01, 2008

I recently changed internet providers, and i have this situation:

The new internet arrives in via a cable in my front room. There are two desktops in the house, which are in the back room. The old internet came through a hole in the wall into this room, so everything was peachy.

I'm now in possession of a linksys WRT54G (with openwrt firmware) and a neatgear WGR614 V7. What i want to do is to bridge the two wireless routers so that i have a single network containing my wireless+wired computers.

Does anyone know if there's a modified/alternate firmware for the netgear router which supports 'client' mode in wireless networks so i can just make it connect to my main router (the linksys). I'd much prefer to use this as my main router as opposed to the netgear. Any advice on getting this set up would be great. Ideally i wouldn't have to buy new hardware, but if it comes to it, thats what i'll do ;)

Hit Counter