MonoTorrent - Prebeta 3
So it's been a long time in the coming, but we now have MonoTorrent prebeta 3 released into the wild. Let fly with the bug reports!
Check out http://www.monotorrent.com/ for the juicy details.
Also, i'd like to give a shout out to jbevain, whose amazing linker, based on the even more amazing cecil, combined with the equally amazing merge tool managed to reduce the size of the libraries i have to distribute down from 520kB to a mere 190kB. Zip that up, and you have a mere 90KB file to distribute. Not too shabby!
A nice (depending on your point of view) side effect of this was also the fact that i know have a single MonoTorrent.exe file to distribute instead of over 1/2 a dozen different files.
Not too bad at all!
Once again, i'd like to say thanks to David Wang, for coding the Encryption support. Also to David Muir for his bugreports and patches (i hope to get more off ya ;) ) and everyone else who has bugged me about problems, or asked me questions.
If i don't reply to your emails, it's because i've forgotten about them. Email again!
11 comments:
I just looked shortly at the current version and one thing that bothered me for a very long time is still there:
Block is a class-type but really should be a structure.
Rationale with a somewhat extreme case:
You are downloading a few huge torrents. Each piece has 1023 blocks and you have 100 simultaneous pieces in transfer:
In the current implementation that would mean: 102,300 Block-objects.
Each Block-object (under 64-bit) has 20 byte overhead (12 object + 8 reference), so the memory wasted for overhead *only* is 2000kb. But that isn't even the worst effect. That would be the sheer amount of objects. Each of those objects has potentially a considerable lifespan, so it may actually survive GC collections. And 100,000 objects will put a LOT of pressure on the GC, even if they did not survive collections.
In fact a SINGLE huge torrent will produce more that 1,000,000 block objects in its livetime.
If Block gets changed into a structure you will get a single (ONE) object instead of 102,301 objects and the overhead will be 0 bytes instead of 2000kb.
I tried to make the change myself (in a previous version I already did that once and it worked, however I unfortunatelly lost the code), but there are some (small) changes required in the messages (blockIndex instead of block) that I don't have time to find out how to do right now.
And by the way:
It seems that the detailed events that have been removed a few versions before still aren't back.
Imho it would be usefull to be notified if:
* A block is requested
* A block has been received
* A piece has been completed
* A block/piece has been requested by a peer
Great Work, Alan and David!
MonoTorrent has been going through major advancements since i started following its development a few months ago.
As a side note, I do agree with andreas about a lack of detailed events for implementers. I've emailed you once or twice about it, But i'm not sure if the mails got to you.
In any case.. Good work, Keep it up! :)
Andreas, while your point may seem a good one, i've never actually come across that happening in real life. Allocations due to blocks have always been substantially less than 5% of the total.
Bear in mind that i create substantially more PeerMessages than any other kind of object yet even still their allocations are considerably less than other things.
The change from class to struct has a good chance of subtly breaking a lot of logic so i've been unwilling to make that change (even though i've thought of it before).
As for the comments about the events, there is a PeerMessaged event which fires every time a message is sent or received which could easily be used to check for PieceMessages or RequestMessages.
You can tell when a Piece is complete by hooking into the PieceHashed event.
@Ran: Email me again, sometimes i read an email but forget to reply. Sorry if that's happened to you.
alan, could you please explain where that PeerMessged event is and how you could use it?
I checked through the source and the most close thing I found was PeerMessageTransferred in ConnectionManager. However afaik it is impossible to get a reference to a currently used ConnectionManager, so you cant use that anyways.
And for the Blocks:
The 5% you mention are in bytes of data I guess. Obviously this is not dramatic, the point is the sheer number, which in "normal operation" is in the top 3 of the number of created objects. And please take into account that the Peermessages basically never survive a collection (And .Net is rather good at gen0 collections), while the blocks have a good chance to even reach gen 2.
Found a way through static calls:
ClientEngine.ConnectionManager
You are downloading a few huge torrents. Each piece has 1023 blocks
This would equate to each piece being 16 megabytes in size. I know of *no* torrent creator that allows pieces that size. So this can never happen. Call it 256 to be reasonable. Thats the most you should see out in the wild.
and you have 100 simultaneous pieces in transfer:
Perfectly reasonable.
In the current implementation that would mean: 102,300 Block-objects.
Actually it'd mean 256 * 100 = 256,000 block objects.
Each Block-object (under 64-bit) has 20 byte overhead (12 object + 8 reference)
I have no idea where this figure comes from. A "piece" contains one int and one List(Block). Each Block contains 3 ints and 3 bools. The overhead on being a class instead of a struct for each object is 4 bytes. So in your example that would mean an overhead of approx 100 * 256 * 4 bytes = 100 kB. Thats practically nothing.
Also, bear in mind that arrays are objects. Anything i put in an array ends up on the heap, not the stack. So even if i change to a struct, you'll still have a load of objects on the heap.
Lastly, supposing there are 256 blocks in a piece and 100 active pieces. This equates to a memory usage of approximately:
(blocks) 256 * 100 * 24 bytes = 614400
(pieces) 100 * 16 bytes = 1600
Total = 601 kB.
There is no way i'd want to store all that on the stack. I'd more than likely run out of stack space!
It's quite possible that changing Block from class to struct will result in less memory usage, but it'll be in the region of a few hundred kB, not MB.
Trust me - I do a lot of profiling as hobby and part of my job and the numbers I gave are correct (with a small mistake: I did miscalculate the padding, so actually the per-object overhead in this case is 16+8ref=24 bytes instead of the 20 I wrote). If you don't believe me just try for yourself:
using System;
using System.Collections.Generic;
using System.Text;
namespace MonoTorrent.Client
{
class Program
{
static void Main (string[] args)
{
Block[] blocks = new Block[100000];
for (int i = 0; i < blocks.Length; i++)
blocks[i] = new Block (0,0,0);
Console.ReadLine ();
}
}
}
Then profile the application (once using the existing Block and once just replacing "class" with "struct") and the results will be:
Struct:
1 MonoTorrent.Client.Block[] Object with a size of 1.600.024 bytes allocated on the large object heap (obviously not on the stack - I don't know what lead you to believe it would be on the stack ;)
Class:
100000 MonoTorrent.Client.Block Objects with a total size of 3.200.000 bytes
+
1 MonoTorrent.Client.Block[] with a size of 800.024 bytes
So in short:
100.001 Objects vs. 1 Object
4.000.024 bytes vs. 1.600.024 bytes
And all that with absolutely no negative consequences (well other than the work to change it ;)
You're right ;) I'll investigate how much effort it'll take to make the change. What i'm most worried about is the way objects are passed "by ref" whereas structs are copied, so previously where i was passing around a reference to the same object i'll now be making a new copy of the block. Still, it's a doable change.
I'll keep you posted on the progress.
As far as I've seen you could replace the object references with
int blockIndex
But perhaps I've overlooked something.
Well and unfortunatelly you'll have to replace the indexers when you need write access with the direct Array-call like
piece[1].Written = true -> piece.Blocks[1].Written = true
Done done and done ;) Also did a similar optimisation for storing the torrent hashes.
Post a Comment