• Please review our updated Terms and Rules here

FAT12 minimum file size? saving space by merging small files

P

PgrAm

Experienced Member
Joined
Sep 28, 2011
Messages
276
Location
Toronto, Canada
Hi,

As I've been working on my DOS game I've been trying a few different things to save disk space as keeping it on a single floppy would save significant money on release. I've already used some compression on data files which has been a big win (thanks Trixter for help with that). I started thinking about a few files I had for the game that were about ~100 bytes each, but there were quite a few of them. These are things such as byte-code files for short scripts, walkbox data and some animation data. I thought that since the FAT12 sector size is 512 bytes that these files were wasting about ~400 bytes each, since I figured that each file gets allocated at least one sector. I think this is the reason why games like DOOM stored all the data in a single WAD file, I figured I should try doing something similar or at least make sure to merge files smaller than 512 bytes.

Does anyone have any info on this? Is my reasoning correct?
 
Right idea but you may want to make your files close to a multiple of 1024 bytes since 360k, 720k, and 2.88MB formats use clusters with 2 sectors.

Edit: That would be best for running the game from the floppy. You can get a lot more by using one of the special install formats (DMF, XDF, etc.) and compressing the complete contents. 10% to 20% more sectors and a reasonable compression could cut needed space in half.

Realized I suggested something already implemented.
 
Last edited:
Yes, that is correct reasoning. Ideally, any read-only non-optional resources should be merged in to one file, or a small set of files for certain data categories. Hard drives running FAT can waste even larger amounts of space for a single small file.

There is also the slight issue of "neatness". It is bad form to litter a user's hard drive with a bazillion small files that can slow down searches or other tools that might parse an entire disk hierarchy. (Although most "modern" software does exactly that). FAT also has a limit to the number of files that may be placed in a root directory. And from a DOS prompt, doing a "DIR" can be annoying when the name of the program executable is at the top and scrolls off the screen after several dozen data files.

Of course, this adds some complexity to your code. How complex depends on how flexible you want to be. Doom, for example, keeps track of data lumps by name, has sort of a faked hierarchy, lumps can be merged at runtime with external WADs, and so on. On the other hand, for a simple program you probably only need a binary index pointing to the offset of each compressed data lump. If you load everything at once, you may not even need much memory management.
 
Alright I think I have a good idea now what I'll do. Since the game is a sci-fi adventure game which takes place on several different planets, I'll merge all the files for each planet into one big resource file. Right now the game uses an index file to tell it where to find everything in the game, when it comes time to ship I'll make a utility that will strip all the filenames from the index and replace those with offsets into a bigger file.

Loading everything at once isn't really an option as the game would probably run out of memory.
 
PgrAm said:
Alright I think I have a good idea now what I'll do. Since the game is a sci-fi adventure game which takes place on several different planets, I'll merge all the files for each planet into one big resource file. Right now the game uses an index file to tell it where to find everything in the game, when it comes time to ship I'll make a utility that will strip all the filenames from the index and replace those with offsets into a bigger file.
Click to expand...

Yep. This is mostly a solved problem; I created a "WAD" library many moons ago, but the concepts are the same:

  1. Append all of your file data into a single big data file, keeping track of the file order and offsets
  2. Build an index to that data file that contains fixed-length records of the 8.3 filenames, the offset into the data file, and the length
  3. When you need a file's data, find the name in the index, seek to the offset, load the length
  4. For extra credit, append the data file to the index and adjust the offsets accordingly, then you'll only need to deliver one file
 
You could even use an existing file format, like LBR or (uncompressed) ARC or ZIP. That way you can use an existing utility to create the resource file.
 
uncompressed zip/lha etc.

or library system. I have a couple, one dedicated for realmode hooks dos in so opening/reading/writeing/exec becomes transparent, so it can be used by any realmode compiler/app etc. it might be on hornet or x2ftp from years ago. I know I have it at home somewhere. googling its name I find only the pmode/dos32/watcom version around.
 
I can't believe I didn't think of uncompressed ZIP before. I should have thought of that, because one of the Access games started doing that (I think Countdown, or maybe Martian Memorandum). That's a great idea.
 
BloodyCactus said:
or library system. I have a couple, one dedicated for realmode hooks dos in so opening/reading/writeing/exec becomes transparent, so it can be used by any realmode compiler/app etc. it might be on hornet or x2ftp from years ago. I know I have it at home somewhere. googling its name I find only the pmode/dos32/watcom version around.
Click to expand...

You're probably thinking of XLINK.

I would recommend against linkers -- they worked mostly ok back in the day, but they incurred some overhead, and they're less compatible than simply writing a simple filewad library. I haven't done extensive testing, but my gut feeling is that file linkers will have problems in freedos and dosbox.
 
Trixter said:
You're probably thinking of XLINK.

I would recommend against linkers -- they worked mostly ok back in the day, but they incurred some overhead, and they're less compatible than simply writing a simple filewad library. I haven't done extensive testing, but my gut feeling is that file linkers will have problems in freedos and dosbox.
Click to expand...

no, since I didnt write xlink, thats not what I am thinking off. I know my old stuff works fine in dosbox, freedos. the only overhead my library had was basically the list of files, an offset and a length * number of files. the realmode library hooked int 21h. never had issues. my pmode and later library stuff did not hook ints, and were much simpler and work fine across dos/pmode/linux/freebsd etc.

anyway, I found some of it, realised I wrote it in 93 in A86 :/ ugh. those were the days eh!
 
You must log in or register to reply here.
Back
Top