|
Programming a Virtual File System - Part III by (29 October 2001) |
Return to The Archives |
Introduction
 |
| Welcome back, guys! Today we'll have a look at the actual code =) I was quite busy for the past weeks, so I couldn't complete it too quickly, but here you go: article_vfs03.zip (306k) |
|
Implementation
|
|
For the implementation, I chose the platform independence approach (VFS1). This
has the advantage that the code is quite easy to port (see the next paragraph),
but it's not as fast as a native implementation could have been. Another point I want to mention is the "style" of the code. I tried to code understandably and avoided long, messy functions, but sometimes (*cough* VFS_Archives.cpp *cough*) I didn't succeed with this one. Also, I didn't optimize the VFS, just because this makes it easier to understand (the std::vector<>s are quite slow in VFS_Archive(s).cpp). Once again: This is a SAMPLE IMPLEMENTATION, not a full-fledged, fully-optimized one. Nonetheless, it has still ~ 4000 LOCs... =) Workspaces for both Microsoft Visual C++ 6 and 7 are provided. |
|
Mission Completed?
|
|
The following points are the "goals" we set in the first installment: a) Fast Access Times "The VFS should provide performance comparable with the native Win32 API Functions. By using few, huge files instead of lots of small ones, we could even improve the performance further" We didn't really succeed with this, mainly because we couldn't use File-Mapping and so; but also because the implementation is written for educational purpose, not for speed. b) Few Archives Instead Of Lots Of Small Files "The VFS should be able to handle so-called Archives, i.e. files that contain lots of other files. This comes in handy, because if we use only one file, we need only one File Handle, and so we won't exhaust too many System Resources. Another advantage is - especially for professional Game Projects - that it's not so easy to parse our file format." This works of course - a VFS without Archives wouldn't have been that cool...Archive handling is the biggest part of the VFS source, and it shouldn't contain too many bugs anymore, so I think this is pretty useful ;)  A small test program which demonstrates the usage of the VFS in a console application... actually I used it for testing VFS, but it might be useful for you... c) Debugging Features "Although Archives are a big pro for us, they are not the perfect solution for the development and debugging process since you can't quickly access files Imagine an image isn't display correctly on the screen and you want to check if the artist didn't save the file properly or if your code doesn't work: You would have to decompress the Archive File, look at the Image, perhaps fix the Image and compress everything again." Since directories and archives are treated the same way, you could use directories during the development process and ship with archives - this way you'd benefit from both the simplicity of the normal file system and the power of the VFS. d) Pluggable Encryption And Compression "Pluggable Encryption and Compression (PEC) is a really cool feature of our VFS. The VFS user can not only choose between several PEC-Plugins, he can even select more than one and determine the order of execution. For instance, you could create an Archive File for the BMP-Images with the PEC-Plugin "RLE-Compression" and another Archive File for TXT-Files with the PEC-Plugins "ZIP-Compression" and "RSA-Encoding"" This is completely covered by the Filter System. I've not only included a working zlib filter in the VPak application (compressed 30MB hard-to-compress-bitmaps to 7MB =), but also the long-expected CAddFilter (in the Test project) which does the awesome work of adding a specified number to each byte in the file *ggg* e) Security "We will store a MD5 key within each Archive file so that manipulation of the Archive Files will be detected instantly." I chose to not do this, mainly because it would be fairly slow to calculate an md5 of everything. If you still want this, go ahead and write your own filter - it's quite easy to do so... f) Multiple Root Paths "This is important especially for professional games, because you could for instance set the Root Paths to both the Installation Directory of the Game and the CD-ROM/DVD." This one works, too. g) Platform Independence (new) We almost reached this goal. You (I've not much experience with linux programming) just have to add an additional #elseif defined( LINUX ) section, that contains equivalent macros/functions/types like in the win32 section. Have a look at it in the VFS_Config.h file (former VFS_Types.h). Well, it seems that we reached most goals, so are we finished now? |
|
Not Really!
|
|
We've got a nice-working reference implementation with not too many bugs, but for a
real-world VFS you can't just write a VFS in ANSI C++ =) File Mapping etc. would
increase speed a lot, then of course Caching for the file name list etc. etc.
You got the point? ;) Another point I wanted to cover is that I removed the VFS functions to set Filters per Archive, not on a per-file base. Though the other approach would have been possible, I decided to skip it in the implementation (the parts of the implementation covering Archives is even without that feature not that easy to understand). In contrary to this decision, Filters are still applied per file, not per archive. Though file size increases for small files, it has the advantage that you can treat the files differently: For instance, you could only compress easily-compressable file types like .TXT-files and .BMP-files, and let the .MP3 files unchanged. Also, this makes the implementation of per-file Filter settings quite easy (Hint: add a bitfield or so to the ARCHIVE_FILE structure which indicates which filters are set in the specific file; got it?=). You see, there are some "exercises for the reader" *g* |
|
VPak
|
|
VPak is a small utility to create .VFSA Archives. It only supports creation of
Archives out of directories, and extraction of the whole Archive, but
nonetheless it's a cool, small tool. Some useful improvements AKA exercices for
the reader could be: - Creation of the archive out of a file list (VFS_Archive_CreateFromFileList()...). - Extraction of single files (VFS_Archive_ExtractFile()...). - Adding/Removing files (add: extract -> recreate; remove: extract -> remove -> recreate). This not only sounds slow, but it will also BE slow, so take care! It isn't that useful, either (remember, we don't want to copy WinZip=) - Make the user be able to select the order of the Filters (i.e. so that you can i.e. apply first encryption then compression or so).  VPak on duty... |
|
Bye, Bye, Bye, ...
|
|
Please tell me your opinion and experience with the VFS (if you decided to
give it a try). I'll be really happy if I get some feedback... By the way, if there'll be a lot of questions, I'll write a small FAQ covering these. I'm pretty sure Kurt will upload it (Editor's Note: I will.). Greetings, Michael PS: Credits to the zlib library that I used for the zlib Filter in VPak |
|
Article Series:
|