JCR6 pack system


(Posted 1 week ago) tricky

What I would like to show you is a bit of a nasty project to link to, as it has many kinds of stuff involved. This project is OOOOOOLD, btw ;)

JCR stands for "Jeroen's Collected Resource". Well "Jeroen" is my real first name, hence the name of the tool....

JCR has been set up from scratch 6 times by now, hence the number 6 in the name....

What is JCR6?

It's nothing more or less than a file packaging system, that I've practically used in nearly all my game productions and sometimes even in tools I write myself. The file format in which Kthura saves its map files are in JCR6. JCR6 can be seen as a kind of zip file, but where zip is most of all set up for archiving purposes, JCR6 is rather set up to for keeping files together, making it easier to distribute a game with tons of assets and scripts and stuff. As a result JCR6 cannot do "solid archiving", but it does support patching, internal aliasing, and some more cool things.

  • JCR6 is set up as a "modular" format. It was set up with expansion in mind. The original version of JCR6 was written in Python, and did support zlib compression. The version set up in BlitzMax has (unfortunately) only support for zlib, but if you can write modules for other compression methods, you can make the BlitzMax module of JCR6 support it easily. The current CLI tools are written in Go, and support zlib, lzw, flat, AND lzma. And I don't have to change the format in order to support new compression methods. (My C# version supports lzma and zlib, and I even have a C++ library, and supports zlib).
  • Do you remember how you could patch DOOM WAD files in order to play homemade levels? JCR6 does also support it, but JCR6 is by far more sophisticated than that?
  • I did however create drivers in multiple languages (including BlitzMax) which allows JCR6 to read the WAD files :)
  • JCR6 has an 'alias' system which allows multiple entries to point to the same data.
  • I could even make JCR6 able to use multiple files as it were one file and make its "dependencies" optional. My RPG games take a lot of space because of the music, but by simply removing the JCR6 file containing that music, the game is smallerr, and the game will run, but only without music.

Now BlitzMax as a few features that allows even pretty fun stuff....

global J:TJCRDir = JCR_Dir("MyJCR.JCR")
global I:TImage = LoadImage(JCR_B(J,"MyImage.png"))
global A:TAudio = LoadSound(JCR_B(J,"MyAudio.ogg"))
global B:TStream = ReadStream(JCR_B(J,"Text.txt"))
print ReadLine(B)
CloseStream(B)

JCR6 for BlitzMax is part of Tricky's Modules. Creating JCR6 files can best be done with JCR6 CLI Tools (If you use this in BlitzMax projects, please note that these tools use lzma by default which I could never get working in BlitzMax). And I got this GUI Interface, but all it can do is view and unpack JCR6 files.

The project is very very old. (JCR1 was written somewhere in the early 1990s I think)....

(Posted 1 week ago)

Awesome collection of mod you got there! and cool name btw. The WAD feature particularly is really nice. Does it follow the standard format with its limitations or you made some improvements?

(Posted 1 week ago)

When reading WAD files it JCR6 is bound to the limitations that WAD files brings. Please note, it's not able to do much with the data the WAD files contain, except for reading them and extracting them. Since WAD files only allow all-caps 8 letter file names JCR6 (which normally allows more than just that) will have to follow suit.

One thing I had to get around though, and that is the way DOOM levels are stored, since they contain multiple files all the same name each level, which is something JCR6 doesn't support, however, I solved this by converting these into folders, and then all is fine. I also got a driver for the PAK files Quake 1 uses (I don't know if Quake II is compatible with that. Since I never owned Quake II, I couldn't analyze that file).

Now I never used the WAD feature myself, but I placed it in so see if the JCR6 reader was extensible enough to handle multiple kinds of packed files, and WAD was one of simplest formats out there (and I've been "hacking" WAD files as a teen, so I still knew how that format worked).

JCR6 itself has seen many kinds of action. In the Star Story game I used it to store all scripts and assets (if you use my tools errors may occur due to a few bugs in early versions of my packers. My BlitzMax version of JCR6 didn't bother too much, but my Go and C# libraries do, and that is what my tools are written in), and not only, that... even the savegame files (in which those bugs are fixed, so they will work even in my current tools) are JCR6 files.... Oh, and what I forgot to mention, the BlitzMax drivers do have (limited) support for TAR files, too, although I'm not really game to use those as resource files for my games... :-/

(Posted 1 week ago)

wow, kudos for that! A lot of stuff to dig in. Yes, I thought about that WAD filename character limitation.

It would be nice to update it with some examples though, perhaps on each. I'm more interested to try out the WAD file support since I have modded a bit before.

(Posted 1 week ago)

Yeah, maybe I should come up with more examples when I got more time on my hands. When it comes to WAD, I must say that WAD is "read-only" for JCR6, however creating a tool which is able to convert "lump" files into WAD files is pretty easy, as long as they don't go beyond 8 letters. They will already have to be in the format that lump files normally have though Creating an entire BSP setup to generate nodes and such, will go beyond my scope of what I can do, I'm afraid. But here I can use my GJCR GUI interface tool displaying a WAD file ;)

(Posted 1 week ago)

Great! I see. maybe a very simple demo game using your WAD mod.

Btw, have you tried physfs? It's very much related to your mod here. I know an old blitzmax mod port named cower.physfs but can't find any archived copy.

(Posted 1 week ago)

I actually never heard of it, but thanks for the link so I can analyze it. Since the creator got their inspiration by the Quake III file system, I can say that JCR is older than that (since the first version of JCR dates back to the DOS era), and that JCR is actually a file format completely set up for the job where physfs is clearly relying on zip (JCR6 can outrank zip in compression ratios due to it using the lzma compression method, that is in my current C# based game engines, as I never got lzma to work in BlitzMax, and uses a case insensitive file table), however perhaps the project can still be valuable for me for JCR6 either in zip drivers for JCR6 relying on that module, or maybe some insight in the zip format itself, as documentation about zip files workouts I found before did appear to be inaccurate (based on the results from actual zipfiles tested with the content of those documents).

When it comes to reading writing outside JCR6 with a scripting language (as noted on the physfs site)... That could be possible in JCR6 too, but that is simply a matter of correctly configuring your scripting language. It's pretty easy in Lua to overwrite its entire "io" module with a new module which can only communicate with JCR6 in stead of the real file system... World of Warcraft did such kinds of protection in their addon system (since WoW addons are merely XML files and Lua scripts).

And now if you use my JCR6 tools on my games, you'll even see they cannot read certain entries, and no that was NOT a bug, but a security measure, which is possible due to JCR6's modular compression driver system, which is something zip cannot provide me, which is for me an important reason not to rely on zip based systems. Passwords in zip files are not that safe. :)

But analyzing the front page I can say that physfs does have some stuff that can work out... ;)

(Posted 6 days ago)

Yes, the physfs is a much newer format than WAD. I actually don't mind reading only files as this would be enough for most common game data usage which is read only and at least offers some sort of protection.

Since you mentioned that you also work mostly in C#, Here's a C sharp PhysFS implementation.. https://github.com/frabert/SharpPhysFS

as I never got lzma to work in BlitzMax, and uses a case insensitive file table),

I see, here's Otus LZMA mod that I just posted and entered into the Toolbox section if you have not checked it out yet.

(Posted 6 days ago)

Yeah, that was the module I used... When I tested the content of a plain text file with it, it got reduced to.... 0 bytes, which was pretty odd... Lzma is one of the best compression methods, when it comes to compression ratios,. but a 10K text file being reduced to 0 bytes, was a bit odd to me.

And thanks for the C# link. I will see if I have time soon to study the code.

Yeah C# is at this moment my 1st language for code that must compile to .exe -- But I just wanted to leave my Blitz work to whoever could still see a use for it, and if people have a use for it, I may maintain it as long as I can still get my Blitz compiler to work (which is a bit of a issue on any system that is not Mac on this moment) so I can test it... Or if people wanna take it over, I'm surely willing to discuss it. Some modules may surely be helpful I suppose... ;)

You may have noticed that I am trying to rely on myself as much as I can... A bit of a leftover of the 1980s when I had to do everything myself, so I'm used to that.... JCR is older than WAD, although the earliest versions were set up in an entirely different way due to the limitations back in the day.

(Posted 6 days ago)

Yeah, that was the module I used... When I tested the content of a plain text file with it, it got reduced to.... 0 bytes, which was pretty odd... Lzma is one of the best compression methods, when it comes to compression ratios,. but a 10K text file being reduced to 0 bytes, was a bit odd to me.

That's really odd. I did test the lzmastream.mod example earlier and it works fine. Converted a 60k text file reduced to 15k and it read back successfully.

You may have noticed that I am trying to rely on myself as much as I can... A bit of a leftover of the 1980s when I had to do everything myself, so I'm used to that.... JCR is older than WAD, although the earliest versions were set up in an entirely different way due to the limitations back in the day.

yea thats awesome. keep it up and sometimes you can also rely on solutions out there if it helps speed up the process. If you are used to that, then I guess it's understandable.

I just wanted to leave my Blitz work to whoever could still see a use for it, and if people have a use for it, I may maintain it as long as I can still get my Blitz compiler to work (which is a bit of a issue on any system that is not Mac on this moment) so I can test it... Or if people wanna take it over, I'm surely willing to discuss it. Some modules may surely be helpful I suppose... ;)

That's great and I'm eager to test out a WAD example when you have one ready.

(Posted 6 days ago)

I shall see if I have time to set one up, as my schedule is booked full for today. So I must ask for your patience. If I set it up, I wll just have to ask you to add DOOM.WAD yourself to that example, as I don't wanna get trouble with Id Software (or whoever owns the copyrights now)... :)

(Posted 5 days ago)

Ok great. You may want to add an example for each as well eventually.

Btw, would it be better if you add your nick as base/root of the mod so they are all in one place and organized?

ex.

Module Tricky.jcr6
(Posted 5 days ago)

As JCR6 was a bit of a too large project back at the time I did decide to give it its own base name. Also note that jcr6 is modular, so I made multiple modules, so I can make sure I only got support for what I needed in each project. For the same reason I gave JCR6 its own namespace in my current C# class setup.

Here I got a simple exampe for you: It will show the entries of DOOM.WAD, and it will show all the vertexes in E1M5 https://github.com/jcr6/BlitzMax-Examples/blob/master/Where's%20all%20the%20data/WADExample.bmx

(Of course, in order to avoid copyright issues, the DOOM.WAD file itself is not included, so you'll have to copy it in yourself).

(Posted 4 days ago)

That's great, will check it out.

(Of course, in order to avoid copyright issues, the DOOM.WAD file itself is not included, so you'll have to copy it in yourself).

I also found the FreeDoom WAD version, not sure if this is any better in terms of licensing and distribution.

https://freedoom.github.io/download.html

(Posted 4 days ago)

I will need to sort out the exact license for that... I'm always very careful with including other people's stuff into my own... I think the reasons for that are understandable ;)

(Quick note: As I realized WAD does not support compression I've removed the call to the zlib driver... Saves space in the created .exe file) ;)

(Posted 3 days ago)

I will need to sort out the exact license for that... I'm always very careful with including other people's stuff into my own... I think the reasons for that are understandable ;)

I agree and yea about that, it's a pretty common question to ask and hey! it's BSD

https://github.com/freedoom/freedoom/issues/335

just credit the author(s) and include the copy of the license and no problems

(Posted 1 day ago)

I see.... Thanks for the info ;)

Oh yeah, to get into the deep of the why not:

Module Tricky.jcr6

JCR6 is, like I said modular. The standard JCR6 module can only read uncompressed JCR6 files, and nothing else. All else is extra. Like zlib compression:

import jcr6.zlibdriver

This will allow JCR6 to perform zlib compression and decompression. If an lzma driver would ever be succesfully made it's just

import jcr6.lzmadriver

Or, if you wanna import them both

import jcr6.lzmadriver
import jcr6.zlibdriver

And now the big magic. If you would create your own compression algorithm, which you and you'd call it RonTekCompression, you could add a driver for JCR6 and then:

import jcr6.RonTekCompressionDriver

Would enable JCR6 to use it... at least in your program.....

And this is also how support for WAD files came to be. Although JCR6 is by far more sophisticated than WAD, some of the basics are the same. Files dumped together and a file table referencing to them, and that made it easy for JCR6 to support WAD files, as it's one of the simplest file formats I ever tried to "hack" as a teenager. And thus adding WAD support was as simple as:

import jcr6.WAD

And JCR6 even has a module that offers (limited) support to read out TAR files, although TAR does not have a file table in the way WAD and JCR6 do, it was still easy to do. Yeah, if you have the knowledge to do it, any archiving format can be made compatible with JCR6 in theory with 7z as the most notable exception as JCR6 does not have any possibility (yet) to support solid archiving, on which 7z heavily relies (rar supports solid archiving optionally, but it's not a default function). As JCR6 has in the first place been set up for quick read-outs in games, I didn't think solid archiving would be a good idea, so that's why I never implemented it (and have no serious plans in the nearby future either). This is of course in theory, as I never fully analyzed all archive systems out there, and as my schedule is full enough the way it is now, well I don't plan to ;)

This modular setup did allow me that when I translated JCR6 to Go (1st version of JCR6 was created in Python, and the 2nd version in BlitzMax, and the 3rd version in Go) I could easily add lzw support in Go (because Go has a standard module for that), while the Python and BlizMax version never supported it. And if better compression algorithms would ever be created in the future, then hey JCR6 is ready already even before they even came on the drawing table.

So that is why I didn't go for a simple: tricky.jcr6, but to make jcr6 a group on its own, and I did so for all languages that support JCR6 (with Turbo Pascal as a notable exception, due to it running on a platform where hardware is too limited for this approach, and expanding is therefore not a wonderful idea).

(Posted 12 hours ago)

That's good to know. Either way I just got used to author modules set into one mod folder.

Reply To Topic

Please log in to reply