Saturday, October 11, 2008

Sparsely populated, just the way I like it

The NTFS filesystem has support for sparse files, but this has to be specially enabled when creating a file. I was originally linked to a blog post on the idea, but unfortunately the license pretty much forbids me from using that code.

So I spent most of yesterday evening and this morning googling API documentation and eventually came up with a fairly basic implementation of sparse file support. The only two operations supported are:
  1. Creating a sparse file
  2. Setting the size of the sparse data.
The benefits of this are seen only on the NTFS filesystem, but what it gives you is the ability to write pieces at arbitrary places in the file without having to preallocate up to that point. There's a bit of overhead, but other than that you only use up the space which you've actually written to, i.e. only the pieces which have been downloaded.

Finally, if you don't have an NTFS filesystem supporting sparse files, you will not be affected by this. Those of you on HFS+ can never get this support, and those of you on ext3, I'm unsure how to enable sparse files. I think it's enabled by default, but if not, any recommendations on setting this up would be appreciated.


Anonymous said...

There's no magic needed on Linux. You just open the file, seek to the position and then write. The file will be sparse. (If you want to avoid a sparse file, you need to write up to the position instead of seeking beyond the current file size.)

"ls -lh file" will tell you the file size, while "du -sh file" will tell you how much space is actually allocated on disk.

For more information see:

Alan said...

Sweet, that's what I thought happened, but I wasn't 100% sure. When googling the topic, it seemed that a special commandline was used to create a sparse file.

There's also no documentation available on whether sparse files are created by default on EXT3 or not. Well, at least no documentation I could find ;)

Wyatt said...

I remember there being trouble with sparse files under Windows Vista. I'm not sure whether it was a bug or a functionality change in the implementation.

I also don't know if there was any truth to it, but there was some discussion of it over at the uTorrent forum:

Alan said...

It seems like there could be an issue using sparse files greater than 1.7 gb under vista.

Good catch. I'll have to work around this by leaving sparse file support disabled by default. Sub-optimal, but it's easier.

Anonymous said...

I've never cared much for sparse files in Windows since they're always considered "special". Although there certainly are cases that necessitate them.

I use lots of terrabyte sized sparse files on XFS, never had an issue. Rsync even handles them nicely which is great for disk to disk backups.

Anonymous said...


Anonymous said...

艾葳酒店經紀公司提供專業的酒店經紀, 酒店上班小姐,八大行業,酒店兼職,傳播妹,或者想要打工兼差打工,兼差,八大行業,酒店兼職,想去酒店上班, 日式酒店,便服店,制服酒店,ktv酒店,禮服店,整天穿得水水漂漂的,還是想去制服店日領上班小姐,水水們如果想要擁有打工工作、晚上兼差工作兼差打工假日兼職兼職工作酒店兼差兼差打工兼差日領工作晚上兼差工作酒店工作酒店上班酒店打工兼職兼差兼差工作酒店上班等,想了解酒店相關工作特種行業內容,想兼職工作日領假日兼職兼差打工、或晚班兼職想擁有鋼琴酒吧又有保障的工作嗎???又可以現領請找專業又有保障的艾葳酒店經紀公司!

一般的酒店經紀只會在水水們第一次上班和領薪水時出現而已,對水水們的上班安全一點保障都沒有!艾葳酒店經紀公司的水水們上班時全程媽咪作陪,不需擔心!只提供最優質的酒店上班,酒店上班,酒店打工環境、上班條件給水水們。心動嗎!? 趕快來填寫你的酒店上班履歷表

等相關服務 幫您快速的實現您的夢想~!!

the sex toys said...

It's all erroneous the thing you are saying.

Hit Counter