Brilliant. Despite telling customers that there would be no change in the anti-copy protection on their latests DVD's, Sony has backed down. They're now issuing exchanges for every faulty disc.
The only thing i find odd is that they claim to have received complaints for less than one thousandth of one percent of affected discs shipped. Now, the pedant in me says instantly shouts "false statistics". Whats the point in giving stats based on units shipped. Surely the only stats that would matter would be on units sold. Obviously if you have 1000 discs in-shop, you're not going to receive complaints until someone buys them.
Wednesday, April 18, 2007
Despite AACS being broken for both BluRay and HD-DVD once already, and a huge list of keys for DVD's being created for both HD-DVD and BluRay with some parts of AACS permanently bypassed, these mega-corps still haven't learnt that DRM does not work.
Once again, in a desperate attempt to make their discs copy-proof, Sony has managed to screw up severely. A year ago, Sony decided that they'd install a secret rootkit which could not be detected by regular means on computers which attempted to play their Audio CD's. As bad as that was, at least the CD's were playable!
Now they've come up with a new ploy: Make their DVD's unplayable on their own DVD drives. If you can't play it, you can't copy it. To be honest, this must be a PR nightmare for them. They are knowingly selling bastardised imitation DVD's that do not play on dvd drives. Hell, they are still selling their own DVD drives without a warning that it will *not* play new sony DVDs.
I suppose the worst thing out of this is that other dvd drive manufacturers may be forced to spend time and effort updating their drive's firmware in order to allow their customers to use their legitimately bought entertainment discs.
Once again, in a desperate attempt to make their discs copy-proof, Sony has managed to screw up severely. A year ago, Sony decided that they'd install a secret rootkit which could not be detected by regular means on computers which attempted to play their Audio CD's. As bad as that was, at least the CD's were playable!
Now they've come up with a new ploy: Make their DVD's unplayable on their own DVD drives. If you can't play it, you can't copy it. To be honest, this must be a PR nightmare for them. They are knowingly selling bastardised imitation DVD's that do not play on dvd drives. Hell, they are still selling their own DVD drives without a warning that it will *not* play new sony DVDs.
I suppose the worst thing out of this is that other dvd drive manufacturers may be forced to spend time and effort updating their drive's firmware in order to allow their customers to use their legitimately bought entertainment discs.
Saturday, April 14, 2007
I had a rather interesting experience on the bittorrent mailing list recently. There was a guy who had a server which was severely IO limited and he was trying to figure out how best to improve performance. The problem was that due to the randomness of how a piece is chosen to be downloaded in bittorrent, his server was having to do a lot of random disk seeking thus limiting his maximum upload rate to about 11-15MB/sec even though he had the bandwidth to handle much more than that.
There are hacks to avoid this kind of IO limiting, such as advertising that you have no pieces available When you connect to a peer and then telling them about selective pieces later. The idea is that the piece you advertise having, you'd buffer in memory and so when the peer requests the piece, you don't have to seek to disk to get it for them. With a sufficient memory cache you significantly reduce the amount of random disk access needed and increase throughput hugely
However, this has several disadvantages. The biggest disadvantage of this method is that it can (and does) reduce overall throughput. Take a situation where a torrent has 100 pieces. You put the first 10 pieces into memory and then send messages advertising those 10 pieces to everyone. If everyone already has those pieces, then no uploading will occur until you swap out those pieces and upload the next batch.
Secondly, you *must* disconnect every peer you've connected to every time you decide to swap out those 10 pieces. If you don't, the other peers will remember about the previous pieces you've advertised and may request those even though you no longer have them in memory. This means you once again have random seeking. However, the amount of random seeking would still be significantly reduced as each peer would only know about 20 pieces out of the actual 100 if they weren't disconnected.
Thirdly, this method of seeding requires special logic. Therefore if you want this kind of special handling you must either hack it into an existing open source client yourself, pay someone to do it, or find a client that supports this and use it (regardless of whatever other problems/deficits that client may have).
Now for the fun part: This issue was addressed in the Fast Extensions for the bittorrent protocol with the "SuggestPiece" message.
As you can see, the Suggest Piece was designed to fix the exact problems i've described above, unfortunately due to either bad wording, or just bad interpretation, this message is going to fail miserably in that.
The SuggestPiece message is an "Advisory" message. My own opinion is that Advisory was added there to signify that you do *not* have to act on that message. The reason for this is that you could already be downloading piece 3 off peer X when peer Y sends a "SuggestMessage" suggesting you download Piece 3 off him instead. In this case, you would not follow the suggestion, you would just ignore it. That makes sense. It's logical. Downloading the same piece twice would just be stupid ;)
Unfortunately, several bittorrent developers who've implemented the Fast Extensions have mis-interpreted this as "You can completely ignore this message if you want to". One developer cited his reason for completely ignoring this message as "i have no use for this, so i ignore it". This is a stupid reason for not implementing proper handling for the SuggestPiece message. Even "I'm too lazy" would be a better excuse as you'd at least be acknowledging that you have an incomplete implementation.
A "proper" implementation of the SuggestPiece message would be to request that piece off the peer that sent you the suggestion at the earliest convenient time, i.e. the very next piece request off them should be for the suggested piece.
Lets assume that every torrent client implemented the suggest piece message handling as i described above. Now, i'll replay the situation above using SuggestPiece messages as opposed to selective piece advertising.
Firstly, you advertise having all the pieces as being available when you connect to each peer (this requires no special extra logic). Every time you receive a request from a peer to send them a piece, you load the entire piece into memory. As soon as that piece is loaded into memory, you send a SuggestPiece message (this requires no special extra logic) to every other peer. The other peers will then act on that message (this requires no special extra logic) and their next request from you will be that piece. That way every time you load a piece into memory once, you can then send it from memory to every other peer that wants it.
The benefits of this method are that every time you load a piece into memory, you can make other peers request it off you (if they need that piece). You will *never* have a situation where you will not be uploading. You will *always* have better performance than the initial scenario where you were IO limited. Assuming that you're still doing the same number of random disk reads as before, you will be able to increase your upload rate significantly because each piece that enters memory will be sent to more than 1 peer.
I'd guess that you could increase performance by, at the very least, 5 fold in a torrent with 100 other peers using the SuggestPiece method. The other major advantage is that you require no special logic in either the seeding client or downloading client! All you need is a proper implementation of the SuggestPiece message. It's not that hard!
EDIT: Despite promising myself i wouldn't, i think i will name and shame the clients that completely ignore the SuggestPiece message despite claiming support for the Fast Extensions:
KTorrent
Bitflu
The clients that i know of that fully support the extensions are:
MonoTorrent
MainLine
There are hacks to avoid this kind of IO limiting, such as advertising that you have no pieces available When you connect to a peer and then telling them about selective pieces later. The idea is that the piece you advertise having, you'd buffer in memory and so when the peer requests the piece, you don't have to seek to disk to get it for them. With a sufficient memory cache you significantly reduce the amount of random disk access needed and increase throughput hugely
However, this has several disadvantages. The biggest disadvantage of this method is that it can (and does) reduce overall throughput. Take a situation where a torrent has 100 pieces. You put the first 10 pieces into memory and then send messages advertising those 10 pieces to everyone. If everyone already has those pieces, then no uploading will occur until you swap out those pieces and upload the next batch.
Secondly, you *must* disconnect every peer you've connected to every time you decide to swap out those 10 pieces. If you don't, the other peers will remember about the previous pieces you've advertised and may request those even though you no longer have them in memory. This means you once again have random seeking. However, the amount of random seeking would still be significantly reduced as each peer would only know about 20 pieces out of the actual 100 if they weren't disconnected.
Thirdly, this method of seeding requires special logic. Therefore if you want this kind of special handling you must either hack it into an existing open source client yourself, pay someone to do it, or find a client that supports this and use it (regardless of whatever other problems/deficits that client may have).
Now for the fun part: This issue was addressed in the Fast Extensions for the bittorrent protocol with the "SuggestPiece" message.
Suggest Piece is an advisory message meaning "you might like to download this piece." The intended usage is for 'super-seeding' without throughput reduction, to avoid redundant downloads, and so that a seed which is disk I/O bound can upload continguous or identical pieces to avoid excessive disk seeks.
As you can see, the Suggest Piece was designed to fix the exact problems i've described above, unfortunately due to either bad wording, or just bad interpretation, this message is going to fail miserably in that.
The SuggestPiece message is an "Advisory" message. My own opinion is that Advisory was added there to signify that you do *not* have to act on that message. The reason for this is that you could already be downloading piece 3 off peer X when peer Y sends a "SuggestMessage" suggesting you download Piece 3 off him instead. In this case, you would not follow the suggestion, you would just ignore it. That makes sense. It's logical. Downloading the same piece twice would just be stupid ;)
Unfortunately, several bittorrent developers who've implemented the Fast Extensions have mis-interpreted this as "You can completely ignore this message if you want to". One developer cited his reason for completely ignoring this message as "i have no use for this, so i ignore it". This is a stupid reason for not implementing proper handling for the SuggestPiece message. Even "I'm too lazy" would be a better excuse as you'd at least be acknowledging that you have an incomplete implementation.
A "proper" implementation of the SuggestPiece message would be to request that piece off the peer that sent you the suggestion at the earliest convenient time, i.e. the very next piece request off them should be for the suggested piece.
Lets assume that every torrent client implemented the suggest piece message handling as i described above. Now, i'll replay the situation above using SuggestPiece messages as opposed to selective piece advertising.
Firstly, you advertise having all the pieces as being available when you connect to each peer (this requires no special extra logic). Every time you receive a request from a peer to send them a piece, you load the entire piece into memory. As soon as that piece is loaded into memory, you send a SuggestPiece message (this requires no special extra logic) to every other peer. The other peers will then act on that message (this requires no special extra logic) and their next request from you will be that piece. That way every time you load a piece into memory once, you can then send it from memory to every other peer that wants it.
The benefits of this method are that every time you load a piece into memory, you can make other peers request it off you (if they need that piece). You will *never* have a situation where you will not be uploading. You will *always* have better performance than the initial scenario where you were IO limited. Assuming that you're still doing the same number of random disk reads as before, you will be able to increase your upload rate significantly because each piece that enters memory will be sent to more than 1 peer.
I'd guess that you could increase performance by, at the very least, 5 fold in a torrent with 100 other peers using the SuggestPiece method. The other major advantage is that you require no special logic in either the seeding client or downloading client! All you need is a proper implementation of the SuggestPiece message. It's not that hard!
EDIT: Despite promising myself i wouldn't, i think i will name and shame the clients that completely ignore the SuggestPiece message despite claiming support for the Fast Extensions:
KTorrent
Bitflu
The clients that i know of that fully support the extensions are:
MonoTorrent
MainLine
Monday, April 02, 2007
I was just talking to J.D. Conley (part of the Coversant team. He also has a blog on the net) about some if the issues i've been having in monotorrent with threading and async sockets. He had a few good ideas about what i could try, so hopefully over the next week or three i'll be able to test out a few of the ideas to help improve performance. Anyway, as part of the discussion i realised that i could probably replace a lot of my lock(object) statements with a much nicer ReaderWriterLock().
The way the code works in monotorrent is that i read from my lists of peers much more than i modify them, therefore it makes sense to have a locking structure that allows multiple readers and a single writer! This gave me the idea for this blog post.
You see, the problem with the ReaderWriter lock is that you have to be careful in how you use it. It's not as simple as the lock(object) statement. If you forget to release a reader lock, or worse, a writer lock, you will experience deadlocking and it will be very hard to track down the exact place where you haven't released the lock!
Also, suppose you want to a writer lock, but you forget to check if your thread already has a reader lock, you will end up throwing an exception as you will not be able to get the writer lock.
However, it's very easy to abstract away all those horrible problems with two simple structs:
Suppose you have the following code snippet:
ReaderWriterLock myLocker = new ReaderWriterLock();
Instead of having to manually use: myLocker.AcquireReaderLock() and remember to call myLocker.AcquireWriterLock() in a finally statement, and instead of messing around trying to decide if you should call myLocker.AcquireWriterLock or myLocker.UpgradeReaderLock() you can just use the corresponding ReaderLock or WriterLock as needed inside a using statement!
For example, take this code snippet:
You can use the much nicer:
Supposing you actually needed a writer lock, just change the using statement to:
using(new WriterLock(myLocker)
The constructor for writer lock will then check to see if it should upgrade an existing reader or if it should get a standard WriterLock. If it upgraded, it will remember to downgrade it correctly later.
Using this method not only makes you need less lines of code, makes your code more readable, but it also completely removes the possibility of creating a deadlock by accidentally forgetting to release a lock!
The way the code works in monotorrent is that i read from my lists of peers much more than i modify them, therefore it makes sense to have a locking structure that allows multiple readers and a single writer! This gave me the idea for this blog post.
You see, the problem with the ReaderWriter lock is that you have to be careful in how you use it. It's not as simple as the lock(object) statement. If you forget to release a reader lock, or worse, a writer lock, you will experience deadlocking and it will be very hard to track down the exact place where you haven't released the lock!
Also, suppose you want to a writer lock, but you forget to check if your thread already has a reader lock, you will end up throwing an exception as you will not be able to get the writer lock.
However, it's very easy to abstract away all those horrible problems with two simple structs:
public struct ReaderLock : IDisposable
{
public ReaderWriterLock Locker;
public ReaderLock(ReaderWriterLock locker)
{
Locker = locker;
locker.AcquireReaderLock(1000);
}
public void Dispose()
{
Locker.ReleaseReaderLock();
}
}
public struct WriterLock : IDisposable
{
private bool upgraded;
private LockCookie cookie;
public ReaderWriterLock Locker;
public WriterLock(ReaderWriterLock locker)
{
Locker = locker;
upgraded = locker.IsReaderLockHeld;
cookie = default(LockCookie);
if (upgraded)
cookie = locker.UpgradeToWriterLock(1000);
else
locker.AcquireWriterLock(1000);
}
public void Dispose()
{
if (upgraded)
Locker.DowngradeFromWriterLock(ref cookie);
else
Locker.ReleaseWriterLock();
}
}
Suppose you have the following code snippet:
ReaderWriterLock myLocker = new ReaderWriterLock();
Instead of having to manually use: myLocker.AcquireReaderLock() and remember to call myLocker.AcquireWriterLock() in a finally statement, and instead of messing around trying to decide if you should call myLocker.AcquireWriterLock or myLocker.UpgradeReaderLock() you can just use the corresponding ReaderLock or WriterLock as needed inside a using statement!
For example, take this code snippet:
try
{
DoSomeStuffWhichMayOrMayNotThrowAnException();
myLocker.AcquireReaderLock();
DoLotsOfStuff();
myLocker.ReleaseReaderLock();
}
catch(WhateverException)
{
if(myLocker.IsReaderLockHeld)
myLocker.ReleaseReaderLock();
DoRestOfErrorHandling();
}
You can use the much nicer:
try
{
DoSomeStuffWhichMayOrMayNotThrowAnException();
using(new ReaderLock(myLocker)
DoLotsOfStuff();
}
catch(WhateverException)
{
DoRestOfErrorHandling();
}
Supposing you actually needed a writer lock, just change the using statement to:
using(new WriterLock(myLocker)
The constructor for writer lock will then check to see if it should upgrade an existing reader or if it should get a standard WriterLock. If it upgraded, it will remember to downgrade it correctly later.
Using this method not only makes you need less lines of code, makes your code more readable, but it also completely removes the possibility of creating a deadlock by accidentally forgetting to release a lock!
Thursday, March 22, 2007
Tuesday, March 20, 2007
I've opened a forum for monotorrent discussion which can be found here. If it proves unstable or crap, i'll hunt out better forum hosting, but it'll do for the moment.
MonoTorrent - Prebeta 3
So it's been a long time in the coming, but we now have MonoTorrent prebeta 3 released into the wild. Let fly with the bug reports!
Check out http://www.monotorrent.com/ for the juicy details.
Also, i'd like to give a shout out to jbevain, whose amazing linker, based on the even more amazing cecil, combined with the equally amazing merge tool managed to reduce the size of the libraries i have to distribute down from 520kB to a mere 190kB. Zip that up, and you have a mere 90KB file to distribute. Not too shabby!
A nice (depending on your point of view) side effect of this was also the fact that i know have a single MonoTorrent.exe file to distribute instead of over 1/2 a dozen different files.
Not too bad at all!
Once again, i'd like to say thanks to David Wang, for coding the Encryption support. Also to David Muir for his bugreports and patches (i hope to get more off ya ;) ) and everyone else who has bugged me about problems, or asked me questions.
If i don't reply to your emails, it's because i've forgotten about them. Email again!
So it's been a long time in the coming, but we now have MonoTorrent prebeta 3 released into the wild. Let fly with the bug reports!
Check out http://www.monotorrent.com/ for the juicy details.
Also, i'd like to give a shout out to jbevain, whose amazing linker, based on the even more amazing cecil, combined with the equally amazing merge tool managed to reduce the size of the libraries i have to distribute down from 520kB to a mere 190kB. Zip that up, and you have a mere 90KB file to distribute. Not too shabby!
A nice (depending on your point of view) side effect of this was also the fact that i know have a single MonoTorrent.exe file to distribute instead of over 1/2 a dozen different files.
Not too bad at all!
Once again, i'd like to say thanks to David Wang, for coding the Encryption support. Also to David Muir for his bugreports and patches (i hope to get more off ya ;) ) and everyone else who has bugged me about problems, or asked me questions.
If i don't reply to your emails, it's because i've forgotten about them. Email again!
Wednesday, March 14, 2007
I'm just putting a shout out to David Wang who has been working tirelessly over the last week or two to implement encrypted transfers for MonoTorrent. I have to say, he's done a great job.
The current SVN has around a 90% complete implementation. As far as i can tell, there's still a bug in accepting incoming connections. Other than that, all that needs to be done is to make sure it works with more than just uTorrent and then it's good to go!
The current SVN has around a 90% complete implementation. As far as i can tell, there's still a bug in accepting incoming connections. Other than that, all that needs to be done is to make sure it works with more than just uTorrent and then it's good to go!
Thursday, March 08, 2007
I've been asked how i go about profiling, but i think the best explanation i could possibly give would be to look at this (very interesting) video.
Frederico talking at Fosdem
Other than that, all i can say is Reduce, Reuse and Recycle.
1) Reduce: Every object you allocate is an object the GC has to clean up. Know how many objects you're allocating and reduce the number if it's excessive. For example i once was calling DateTime.Now several thousand times a second because of the loop's i had written. By moving the call before the loops, i reduced the allocations from that by a factor of several thousand.
2) Reuse/Recycle: I used to allocate a new byte array every time i wanted to send a message to another computer via a socket or receive a message from them. This resulted in a *lot* of 16kB byte array's floating around. Instead i created a BufferManager class. From this class i requested a buffer every time i needed one and then returned the buffer back to the manager every time i was finished with it. This way i never allocated more buffers than i needed (about 40-60 buffers typically). Previously i could be allocating as much as 40 new buffers each and every second.
I might blog more about that later if people really want me to. But that all depends on how much people beg :P
Frederico talking at Fosdem
Other than that, all i can say is Reduce, Reuse and Recycle.
1) Reduce: Every object you allocate is an object the GC has to clean up. Know how many objects you're allocating and reduce the number if it's excessive. For example i once was calling DateTime.Now several thousand times a second because of the loop's i had written. By moving the call before the loops, i reduced the allocations from that by a factor of several thousand.
2) Reuse/Recycle: I used to allocate a new byte array every time i wanted to send a message to another computer via a socket or receive a message from them. This resulted in a *lot* of 16kB byte array's floating around. Instead i created a BufferManager class. From this class i requested a buffer every time i needed one and then returned the buffer back to the manager every time i was finished with it. This way i never allocated more buffers than i needed (about 40-60 buffers typically). Previously i could be allocating as much as 40 new buffers each and every second.
I might blog more about that later if people really want me to. But that all depends on how much people beg :P
Sunday, March 04, 2007
It's been a while since i sat down and profiled monotorrent, turns out there's not much that has adversely affected my memory optimisations that i implemented a while ago. Things are all still fine and dandy.
However one thing that i never liked were the large amount of allocations due to using the async sockets. It always annoyed me that the worst part of my code i had no control over. Well, it turns out that it isn't *just* the sockets causing those allocations, there's another thing i haven't looked at yet.
My timer!
I use a single timer in the library which fires about 20 times a second. This was originally needed as a way to make sure that certain events happened regularly. However, I've never liked this timer and i have been working to get rid of it. For example i removed the dependence of downloading pieces and piece picking from that timer (as much as is possible). Download/Upload speed monitoring is still dependant on the timer, so once i remove that dependence and the only other thing left is to make connecting to other peers independent of the timer. Currently i try to connect to a new peer every timer tick so long as i haven't reached the limit.
So, before i used to have an allocation graph that looks like this:
That big yellow bar is the allocations due to both the timer and the async sockets. There's a lot of execution contexts there :p After changing the timer interval to 1 second, i get the following allocation graph:
That tiny cyan coloured bar is the severely reduced amount of execution contexts being allocated. So before you all get to feel the benefits of that change, i have to modify a fair bit of code. Once thats done, i'll be releasing beta 3 which is (hopefully) a lot more stable than any previous version, and faster, and uses less ram, and uses less CPU and god knows what else ;)
However one thing that i never liked were the large amount of allocations due to using the async sockets. It always annoyed me that the worst part of my code i had no control over. Well, it turns out that it isn't *just* the sockets causing those allocations, there's another thing i haven't looked at yet.
My timer!
I use a single timer in the library which fires about 20 times a second. This was originally needed as a way to make sure that certain events happened regularly. However, I've never liked this timer and i have been working to get rid of it. For example i removed the dependence of downloading pieces and piece picking from that timer (as much as is possible). Download/Upload speed monitoring is still dependant on the timer, so once i remove that dependence and the only other thing left is to make connecting to other peers independent of the timer. Currently i try to connect to a new peer every timer tick so long as i haven't reached the limit.
So, before i used to have an allocation graph that looks like this:
That big yellow bar is the allocations due to both the timer and the async sockets. There's a lot of execution contexts there :p After changing the timer interval to 1 second, i get the following allocation graph:
That tiny cyan coloured bar is the severely reduced amount of execution contexts being allocated. So before you all get to feel the benefits of that change, i have to modify a fair bit of code. Once thats done, i'll be releasing beta 3 which is (hopefully) a lot more stable than any previous version, and faster, and uses less ram, and uses less CPU and god knows what else ;)
Wednesday, February 28, 2007
Just to respond publicly to a few questions about monotorrent that have been asked over and over:
1) MonoTorrent does have a GUI! In fact, it has two. One of them is a console GUI, the other is a gui created using GTK#. This is *not* the same as winforms in .NET. It is an alternative framework for creating GUI's. While i consider the GUI very limited (which it is), it is usable. I would much prefer a Winforms based GUI so the application would be 100% portable .net with no other dependancies but i don't have the time to finish my 1/2 created one.
2) MonoTorrent does use less ram than azureus. Considerably less. Very very much so. Monotorrent does use more ram than uTorrent. But not that much more. If you disable the disk cache in uTorrent completely, monotorrent only uses 4-5 megabytes of ram more than utorrent. If you leave the disk cache enabled in uTorrent, then monotorrent uses slightly less ram than uTorrent.
3) MonoTorrent does not support encryption. I looked at the specs a while ago, found them confusing, created some initial support but never got back to it. Once again, time is a factor. I just don't have enough.
4) MonoTorrent is not a "Linux App" or a "Microsoft application". It's a cross-platform bittorrent library.
5) Yes, monotorrent can be used on whatever tracker you want. Private/public/personal, i don't care. It works. (Just so long as it's not a udp based tracker ;) ).
6) MonoTorrent is not related to the disease known as mono (also known as glandular fever in non-american areas) ;)
7) Yes, i do like people to contribute code. But please talk to me first and keep in touch when writing your code. The last thing i want to do is to say "no" after you spend 3 weeks writing a bucketload of code for monotorrent because i've already written the code, or you've done it wrong or whatever.
1) MonoTorrent does have a GUI! In fact, it has two. One of them is a console GUI, the other is a gui created using GTK#. This is *not* the same as winforms in .NET. It is an alternative framework for creating GUI's. While i consider the GUI very limited (which it is), it is usable. I would much prefer a Winforms based GUI so the application would be 100% portable .net with no other dependancies but i don't have the time to finish my 1/2 created one.
2) MonoTorrent does use less ram than azureus. Considerably less. Very very much so. Monotorrent does use more ram than uTorrent. But not that much more. If you disable the disk cache in uTorrent completely, monotorrent only uses 4-5 megabytes of ram more than utorrent. If you leave the disk cache enabled in uTorrent, then monotorrent uses slightly less ram than uTorrent.
3) MonoTorrent does not support encryption. I looked at the specs a while ago, found them confusing, created some initial support but never got back to it. Once again, time is a factor. I just don't have enough.
4) MonoTorrent is not a "Linux App" or a "Microsoft application". It's a cross-platform bittorrent library.
5) Yes, monotorrent can be used on whatever tracker you want. Private/public/personal, i don't care. It works. (Just so long as it's not a udp based tracker ;) ).
6) MonoTorrent is not related to the disease known as mono (also known as glandular fever in non-american areas) ;)
7) Yes, i do like people to contribute code. But please talk to me first and keep in touch when writing your code. The last thing i want to do is to say "no" after you spend 3 weeks writing a bucketload of code for monotorrent because i've already written the code, or you've done it wrong or whatever.
Monday, February 26, 2007
BitTorrent Distributed Hash Table (DHT):
I got bored over the weekend and decided to take a break from hacking on the main MonoTorrent libraries and/or Mono.Xna and instead work a little on some DHT support for MonoTorrent.
Unfortunately the only documentation is this rather undetailed document. There's quite a few places where important details are left undefined and other things just aren't mentioned or just defined in vague terms.
I'm keeping notes on these things to hopefully get a more concise and detailed specification together so that other developers attempting to implement DHT can have a better spec to work off. Also it might help with removing bugs in existing implementations.
For example one detail that wasn't mentioned in the spec was that you should never send out the details for a node (another bittorrent peer) unless you have checked that the node is still active. Some implementations don't do this and so it's possible that you will still be getting DHT requests for a torrent you turned off several days previously.
Finally, i've been bugging the developer of libtorrent for some tips on my own implementation, so thanks to him for asking my questions! It's appreciated.
Unfortunately the only documentation is this rather undetailed document. There's quite a few places where important details are left undefined and other things just aren't mentioned or just defined in vague terms.
I'm keeping notes on these things to hopefully get a more concise and detailed specification together so that other developers attempting to implement DHT can have a better spec to work off. Also it might help with removing bugs in existing implementations.
For example one detail that wasn't mentioned in the spec was that you should never send out the details for a node (another bittorrent peer) unless you have checked that the node is still active. Some implementations don't do this and so it's possible that you will still be getting DHT requests for a torrent you turned off several days previously.
Finally, i've been bugging the developer of libtorrent for some tips on my own implementation, so thanks to him for asking my questions! It's appreciated.
Saturday, February 17, 2007
MonoTorrent news:
MonoTorrent has gotten increasingly stable as time has gone on. I found the cause of a few long running bugs which have now been crushed into tiny tiny pieces. There shouldn't be any more ArgumentExceptions being thrown due to passing the wrong IAsyncResult into an End*** method. That was caused be me forgetting to clear out the Download/Upload queue when a peer is disconnected.
uPnP support has been enabled in MonoTorrent using Mono.Nat. So all you people with uPnP routers no longer have to worry about manually creating the port mapping in your router. It'll all be done automagically (all going well ;) ).
Disk writes are now fully asynchronous, but now will automatically throttle download speed if you are downloading faster than your harddisk can write. So you won't ever get 100s of megs of ram being used and 100% cpu usage when exceeding your write speed.
Upload and download speed calculations have been improved drastically (ish) for torrents. What i did before was calculate each individual peers upload and download speed, then sum up that for all connected peers to see a torrents overall download rate.
While this worked, it meant that any errors in calculation for each individual peer got magnified resulting in a fairly inaccurate overall download speed in certain situations. The quick solution was to give the Torrent it's own dedicated ConnectionManager.
There've been a few other minor changes here and there including enhancing download performance. All in all, some good stuff.
MonoTorrent has gotten increasingly stable as time has gone on. I found the cause of a few long running bugs which have now been crushed into tiny tiny pieces. There shouldn't be any more ArgumentExceptions being thrown due to passing the wrong IAsyncResult into an End*** method. That was caused be me forgetting to clear out the Download/Upload queue when a peer is disconnected.
uPnP support has been enabled in MonoTorrent using Mono.Nat. So all you people with uPnP routers no longer have to worry about manually creating the port mapping in your router. It'll all be done automagically (all going well ;) ).
Disk writes are now fully asynchronous, but now will automatically throttle download speed if you are downloading faster than your harddisk can write. So you won't ever get 100s of megs of ram being used and 100% cpu usage when exceeding your write speed.
Upload and download speed calculations have been improved drastically (ish) for torrents. What i did before was calculate each individual peers upload and download speed, then sum up that for all connected peers to see a torrents overall download rate.
While this worked, it meant that any errors in calculation for each individual peer got magnified resulting in a fairly inaccurate overall download speed in certain situations. The quick solution was to give the Torrent it's own dedicated ConnectionManager.
There've been a few other minor changes here and there including enhancing download performance. All in all, some good stuff.
Mono.XNA - the good, the bad and the ugly
There's been a good few changes for Mono.XNA recently. Firstly, we've moved to google project hosting. The idea of having an integrated issue tracker, integrated wiki, integrated email alerts was just to tempting to resist. Also we now have the ability to give write access to anyone we want as opposed to having to go through novell to grant users write access to the mono SVN which can take time.
So, the new URL for the SVN/Wiki/Issue Tracker etc is here: http://code.google.com/p/monoxna/
Feel free to drop in and start coding.
As part of the move, we've now toughened up on the contributing guidelines. Code must have tests, must follow the coding guidelines etc. Links to the relevant documentation about contributing can be seen on our google mailing list: http://groups.google.com/group/monoxna
Also, a little on my own work on XNA. I "reverse engineered" the .XNB format about 2-3 weeks ago. The only implementation so far is the Texture2DReader implementation i coded together. It's probably the most basic .XNB type in XNA, so it should serve as a good example as to how to code future files together. Unfortunately even the Texture2D format took quite a while to figure out. For anyone trying to hack some code together to decode other types, it isn't easy. You will spend hours with a hex editor.... well, it's quite possible you're better at it than me, but it took me hours :p
Still, that's another step forward. Things are looking bright for XNA, all we need now is a dozen coders to start hacking and planning and coding their way through the classes and then we can really start getting stuff done!
There's been a good few changes for Mono.XNA recently. Firstly, we've moved to google project hosting. The idea of having an integrated issue tracker, integrated wiki, integrated email alerts was just to tempting to resist. Also we now have the ability to give write access to anyone we want as opposed to having to go through novell to grant users write access to the mono SVN which can take time.
So, the new URL for the SVN/Wiki/Issue Tracker etc is here: http://code.google.com/p/monoxna/
Feel free to drop in and start coding.
As part of the move, we've now toughened up on the contributing guidelines. Code must have tests, must follow the coding guidelines etc. Links to the relevant documentation about contributing can be seen on our google mailing list: http://groups.google.com/group/monoxna
Also, a little on my own work on XNA. I "reverse engineered" the .XNB format about 2-3 weeks ago. The only implementation so far is the Texture2DReader implementation i coded together. It's probably the most basic .XNB type in XNA, so it should serve as a good example as to how to code future files together. Unfortunately even the Texture2D format took quite a while to figure out. For anyone trying to hack some code together to decode other types, it isn't easy. You will spend hours with a hex editor.... well, it's quite possible you're better at it than me, but it took me hours :p
Still, that's another step forward. Things are looking bright for XNA, all we need now is a dozen coders to start hacking and planning and coding their way through the classes and then we can really start getting stuff done!
Saturday, February 03, 2007
Asynchronous Process - Can't live with it, can't live without it
There are often times where you have two tasks running simultaenously at different speeds which depend on each other. For example, imagine transferring a file from computer X to computer Y. The simplistic approach is for computer X to transfer a part of the file to computer Y, wait for computer Y to write the data to the disk, then transfer the next part. This is a "pipelined" approach. It's good enough for most purposes, but if you have a large communication delay between the two computers, performance will suffer. However this method is very stable as you know exactly how much memory is needed and how much CPU time is needed to transfer and write a single block of data.
A second approach is for computer X to pass a part of the file to computer Y and (without waiting for Y to finish writing the data to disk) begin transferring the next part. This second approach can result in a *much* faster transfer rate when there is a large communication delay between the two computers simply because computer X no longer has to wait for the "I have written the data to disk" reply from computer Y.
However, the problem with the second approach is that if computer X sends 100 blocks of data a second and computer Y can only write 50 blocks of data a second to the disk, we either end up losing data by just dumping it or we end up storing it all in memory until we can write it to disk and memory usage climbs dramatically!
So, in order to gain the performance of the second method with the stability of the first method, you need some way to send feedback to computer X that it is sending data to fast. This is the problem i've just encountered in MonoTorrent when i converted the disk writes to be fully asynchronous to avoid blocking the main downloading thread. When i seed off my local machine, i can't write fast enough so i end up storing 1000's of chunks of data in memory until i tell my seeding client to stop. How easy this will be to fix, i'm not sure. But i am sure it's doable ;)
There are often times where you have two tasks running simultaenously at different speeds which depend on each other. For example, imagine transferring a file from computer X to computer Y. The simplistic approach is for computer X to transfer a part of the file to computer Y, wait for computer Y to write the data to the disk, then transfer the next part. This is a "pipelined" approach. It's good enough for most purposes, but if you have a large communication delay between the two computers, performance will suffer. However this method is very stable as you know exactly how much memory is needed and how much CPU time is needed to transfer and write a single block of data.
A second approach is for computer X to pass a part of the file to computer Y and (without waiting for Y to finish writing the data to disk) begin transferring the next part. This second approach can result in a *much* faster transfer rate when there is a large communication delay between the two computers simply because computer X no longer has to wait for the "I have written the data to disk" reply from computer Y.
However, the problem with the second approach is that if computer X sends 100 blocks of data a second and computer Y can only write 50 blocks of data a second to the disk, we either end up losing data by just dumping it or we end up storing it all in memory until we can write it to disk and memory usage climbs dramatically!
So, in order to gain the performance of the second method with the stability of the first method, you need some way to send feedback to computer X that it is sending data to fast. This is the problem i've just encountered in MonoTorrent when i converted the disk writes to be fully asynchronous to avoid blocking the main downloading thread. When i seed off my local machine, i can't write fast enough so i end up storing 1000's of chunks of data in memory until i tell my seeding client to stop. How easy this will be to fix, i'm not sure. But i am sure it's doable ;)
Sunday, January 28, 2007
MonoTorrent underwent a lot of under the hood changes recently to bring things more in line with the bittorrent specs. I also finished off the implementation of the Fast Peers Extensions, so downloads with monotorrent should be faster when used in swarms that have other clients supporting the Fast Peers Extensions.
A majorly important fix went in today which prevents incoming connections which have just been created from being closed. This should improve things substantially and hopefully remove a long running bug of invalid AsyncRequests being passed into EndXXX methods.
Also, after talking with Ty Norton (http://blog.aproductofsociety.org/) who is hosting www.monotorrent.com, he told me that there have been 211.37 megabytes of downloads of monotorrent so far this month. Considering the zip file is a mere 154kB in size, that's over 1300 downloads! Not too bad. Now, if only i got bug reports from one tenth of those people, i'd be happy.
Or maybe MonoTorrent is so good it doesn't throw any exceptions... who knows ;)
A majorly important fix went in today which prevents incoming connections which have just been created from being closed. This should improve things substantially and hopefully remove a long running bug of invalid AsyncRequests being passed into EndXXX methods.
Also, after talking with Ty Norton (http://blog.aproductofsociety.org/) who is hosting www.monotorrent.com, he told me that there have been 211.37 megabytes of downloads of monotorrent so far this month. Considering the zip file is a mere 154kB in size, that's over 1300 downloads! Not too bad. Now, if only i got bug reports from one tenth of those people, i'd be happy.
Or maybe MonoTorrent is so good it doesn't throw any exceptions... who knows ;)
Monday, January 22, 2007
MonoTorrent beta 2 is now out. It has a good few bug fixes, a good few extra features and hopefully is more stable than before.
Keep the bug reports coming in guys!
MonoTorrent home page
Keep the bug reports coming in guys!
MonoTorrent home page
Sunday, January 14, 2007
I leave my house in an hour for the sunny italian slopes where hopefully there'll be enough snow for a weeks snowboarding. It's my last break before i go back to college. I'm looking forward to it, mostly because i'm beginning to get bored at home and haven't seen some of my friends in a while. The results for our christmas tests are also coming out then, which isn't really something to look forward to ;) Sure we'll see how it goes :p
Hopefully by the time i get back, stubbing on Mono.XNA will be complete. We're oh so close to reaching that magical 100% stubbed mark. After that, with a bit of planning and discussion, we'll be able to start hacking our way through the implementation phases in a more controlled manner as opposed to dropping code in all over the place.
MonoTorrent will start getting some much needed love and attention over the coming weeks (or so i plan). I have a few things planned which i can't wait to get cracking on.
Mono.NAT (the uPnP library) is also in pretty good shape. I received a patch a few days ago which fixed a lot fo FxCop warnings and updated the library to CLS compliance. So feel free to give that a shot for all your uPnP port mapping needs.
Finally, do keep track of the Mono.XNA status at http://xna.taoframework.com/ClassStatus.html and http://xna.taoframework.com/ClassStatusGame.html. I won't be able to keep it updated while i'm away, but i did update it a few hours ago.
Hopefully by the time i get back, stubbing on Mono.XNA will be complete. We're oh so close to reaching that magical 100% stubbed mark. After that, with a bit of planning and discussion, we'll be able to start hacking our way through the implementation phases in a more controlled manner as opposed to dropping code in all over the place.
MonoTorrent will start getting some much needed love and attention over the coming weeks (or so i plan). I have a few things planned which i can't wait to get cracking on.
Mono.NAT (the uPnP library) is also in pretty good shape. I received a patch a few days ago which fixed a lot fo FxCop warnings and updated the library to CLS compliance. So feel free to give that a shot for all your uPnP port mapping needs.
Finally, do keep track of the Mono.XNA status at http://xna.taoframework.com/ClassStatus.html and http://xna.taoframework.com/ClassStatusGame.html. I won't be able to keep it updated while i'm away, but i did update it a few hours ago.
Friday, January 12, 2007
So Mono.XNA took a big (ish) step forward today with thanks to a big commit from RobLoach. Now, hopefully he'll write some NUnit tests to cover that code.
Take a look at the video linked at the end. It shows Mono.XNA successfully rendering an array of colours to the screen one after another in a loop. Render-licios. Next step, we want to be able to render multiple colours to the screen at the same time. I'm thinking a nice checkerboard pattern would be good. We'll see ;)
The video is encoded using x264. If you can't play it, VLC will play it. Just google "vlc" and download away.
http://xna.taoframework.com/SimpleExample.mp4
Take a look at the video linked at the end. It shows Mono.XNA successfully rendering an array of colours to the screen one after another in a loop. Render-licios. Next step, we want to be able to render multiple colours to the screen at the same time. I'm thinking a nice checkerboard pattern would be good. We'll see ;)
The video is encoded using x264. If you can't play it, VLC will play it. Just google "vlc" and download away.
http://xna.taoframework.
Subscribe to:
Posts (Atom)