Saturday, October 11, 2008

Sparsely populated, just the way I like it

The NTFS filesystem has support for sparse files, but this has to be specially enabled when creating a file. I was originally linked to a blog post on the idea, but unfortunately the license pretty much forbids me from using that code.

So I spent most of yesterday evening and this morning googling API documentation and eventually came up with a fairly basic implementation of sparse file support. The only two operations supported are:
  1. Creating a sparse file
  2. Setting the size of the sparse data.
The benefits of this are seen only on the NTFS filesystem, but what it gives you is the ability to write pieces at arbitrary places in the file without having to preallocate up to that point. There's a bit of overhead, but other than that you only use up the space which you've actually written to, i.e. only the pieces which have been downloaded.

Finally, if you don't have an NTFS filesystem supporting sparse files, you will not be affected by this. Those of you on HFS+ can never get this support, and those of you on ext3, I'm unsure how to enable sparse files. I think it's enabled by default, but if not, any recommendations on setting this up would be appreciated.


8 comments:

Anonymous said...

There's no magic needed on Linux. You just open the file, seek to the position and then write. The file will be sparse. (If you want to avoid a sparse file, you need to write up to the position instead of seeking beyond the current file size.)

"ls -lh file" will tell you the file size, while "du -sh file" will tell you how much space is actually allocated on disk.

For more information see:
http://www.answers.com/topic/sparse-file

Alan said...

Sweet, that's what I thought happened, but I wasn't 100% sure. When googling the topic, it seemed that a special commandline was used to create a sparse file.

There's also no documentation available on whether sparse files are created by default on EXT3 or not. Well, at least no documentation I could find ;)

Anonymous said...

I remember there being trouble with sparse files under Windows Vista. I'm not sure whether it was a bug or a functionality change in the implementation.


I also don't know if there was any truth to it, but there was some discussion of it over at the uTorrent forum: http://forum.utorrent.com/viewtopic.php?id=31998

Alan said...

It seems like there could be an issue using sparse files greater than 1.7 gb under vista.

Good catch. I'll have to work around this by leaving sparse file support disabled by default. Sub-optimal, but it's easier.

Anonymous said...

I've never cared much for sparse files in Windows since they're always considered "special". Although there certainly are cases that necessitate them.

I use lots of terrabyte sized sparse files on XFS, never had an issue. Rsync even handles them nicely which is great for disk to disk backups.

Anonymous said...

免費視訊,煙火,煙火工廠,視訊,正妹視訊,真愛密碼,真愛密碼,煙火秀,煙火圖片,衣蝶,衣蝶,AV女優,AV女優,
煙火批發,小泉彩,小泉彩,情趣用品,鞭炮,live119論壇,蜂炮,視訊

the sex toys said...

It's all erroneous the thing you are saying.

梁爵 said...

2020.08.25不敢來酒店上班-酒店打工的原因一般人的觀念裡酒店是一個複雜、不好的環境,酒店小姐的基本介紹跟工作內容其實在酒店上班沒什麼不好,我在酒店上班的日子工作性質跟一般在聊天的美眉是同性質的。
例如:像跟一群人去錢櫃好樂迪參加聯誼會那種形式,認識交際不同的人,一樣是跟客人單純的閒聊、喝 酒炒熱氣份,但只是變成工作,還有錢可以賺,不好的是因為吸毒而去做酒店,甚至賺錢去養男人這是最笨、最愚蠢的作法。
酒店經紀〝俗稱經紀人〞我們常遇到的女孩子大部分都是單親媽媽要養小孩、要還父母的負債、要實現自己的目標、還有解決自己卡債問題等,這些並沒有什麼不好,人總是因為環境的因素不得已,你只要知道自己最終的目標是什麼就行。另外補充一點,小部分酒店是有做黑的沒錯,但不想做黑的也是大有人在,賣笑不賣身應該有聽過還是可以把大把銀子賺進口袋。
常有女孩們因為報紙裡的高薪而吸引,卻不知道這家酒店的型態而付出更高的犧牲本身在這一行業也有一段時間,真的有像電影裡的故事情節發生,其實女孩們真的因為經濟上的需要而這樣的工作,除了心理要有充足的準備之外,還要先了解這家店的性質,才不會後悔莫及,許多的女孩子對酒店都有很多的不了解就誤入了這個行業,更扯的是有些女孩子被報紙上的小廣告所騙,九成九不是被騙財就是被騙色。

Hit Counter