Saturday, October 11, 2008

Sparsely populated, just the way I like it

The NTFS filesystem has support for sparse files, but this has to be specially enabled when creating a file. I was originally linked to a blog post on the idea, but unfortunately the license pretty much forbids me from using that code.

So I spent most of yesterday evening and this morning googling API documentation and eventually came up with a fairly basic implementation of sparse file support. The only two operations supported are:
  1. Creating a sparse file
  2. Setting the size of the sparse data.
The benefits of this are seen only on the NTFS filesystem, but what it gives you is the ability to write pieces at arbitrary places in the file without having to preallocate up to that point. There's a bit of overhead, but other than that you only use up the space which you've actually written to, i.e. only the pieces which have been downloaded.

Finally, if you don't have an NTFS filesystem supporting sparse files, you will not be affected by this. Those of you on HFS+ can never get this support, and those of you on ext3, I'm unsure how to enable sparse files. I think it's enabled by default, but if not, any recommendations on setting this up would be appreciated.


7 comments:

Anonymous said...

There's no magic needed on Linux. You just open the file, seek to the position and then write. The file will be sparse. (If you want to avoid a sparse file, you need to write up to the position instead of seeking beyond the current file size.)

"ls -lh file" will tell you the file size, while "du -sh file" will tell you how much space is actually allocated on disk.

For more information see:
http://www.answers.com/topic/sparse-file

Alan said...

Sweet, that's what I thought happened, but I wasn't 100% sure. When googling the topic, it seemed that a special commandline was used to create a sparse file.

There's also no documentation available on whether sparse files are created by default on EXT3 or not. Well, at least no documentation I could find ;)

Anonymous said...

I remember there being trouble with sparse files under Windows Vista. I'm not sure whether it was a bug or a functionality change in the implementation.


I also don't know if there was any truth to it, but there was some discussion of it over at the uTorrent forum: http://forum.utorrent.com/viewtopic.php?id=31998

Alan said...

It seems like there could be an issue using sparse files greater than 1.7 gb under vista.

Good catch. I'll have to work around this by leaving sparse file support disabled by default. Sub-optimal, but it's easier.

Anonymous said...

I've never cared much for sparse files in Windows since they're always considered "special". Although there certainly are cases that necessitate them.

I use lots of terrabyte sized sparse files on XFS, never had an issue. Rsync even handles them nicely which is great for disk to disk backups.

Anonymous said...

免費視訊,煙火,煙火工廠,視訊,正妹視訊,真愛密碼,真愛密碼,煙火秀,煙火圖片,衣蝶,衣蝶,AV女優,AV女優,
煙火批發,小泉彩,小泉彩,情趣用品,鞭炮,live119論壇,蜂炮,視訊

the sex toys said...

It's all erroneous the thing you are saying.

Hit Counter