dz utils: core tools for packages

[dizzy]

Here's a tool set for command line modders:

Today's addition: a simple hex editor...

ls: list contents of a package
extract: dump all or portions of a package to .dat files
create: create a new (blank) package file
append: add a .dat file's data to a package
xed: a simple hex editor

[J. M. Pescado]

Interesting. What would be nice if we had something that was able to read a package, load its entire contents into memory, presumbly for something in the future that would then be able to display, edit, and generally doctor its contents, and then write it back out. The business of reading packages seem to be largely the same as it was in the old disasims code which I adapted into hackdiffing (which right now is able to load a single item, and perform an agnostic md5sum on it for comparison), but we still don't have anything that is able to take a package, load it into memories, and then spit it back out intact (and hopefully recompressed). Even SimPE at present seems to have bugs in recompressing certain things.

[dizzy]

SimPE performs compression? Why in the world would you compress with only the ability to do the EA half-ass substitution method?

[jrd]

Filesize? I compress all text-based resources (including BCONs, BHAVs, Pie Menus) and it saves a lot of space. My "mascot clothes hider" is decreased in size almost ten-times by compression.

[J. M. Pescado]

Does this business handle compressed archives yet? I'm still trying to decipher what exactly should happen if you were to, say, edit a resources. Some sort of demonstration somewhere of how it would work to load a package into some kind of memory representation and then write it back out would be nice. Extracting and appending is rather limiting and awkward, and would undoubtedly produce something that ran like a dog if everytime you edited a resource, the entire package had to be extracted that way and then recomposited by appendation one at a time.

[dizzy]

The extract tool handles unpacking. I don't support packing.

Well, the editing portion of the toolset hasn't even been grokked yet. I proposed using a Lua/sed type tool, but you seem to have something else in mind.

Extracting and appending is no different from when you hit "Commit" and "Save" in SimPE, it just uses virtual memory instead of writing to a file (six vs half a dozen, IMHO).

If you want the editor tool to be highly optimized, then by all means write a highly optimized editor. 

[J. M. Pescado]

Well, the editing portion of the toolset hasn't even been grokked yet. I proposed using a Lua/sed type tool, but you seem to have something else in mind.

I was thinking "like SimPE, only different and nonsuck", such as with the ability to skip unnecessary file reading and not break when you try to open something while the game is running for read-only purposes. But for anything, we'd sort of need a core function set that can load, allow modification of a resource's binary guts-contents by some means, and then spit the thing out again, preferrably with the compression working. So far it is not entirely clear how to do that part.

[wes_h]

SimPE didn't have compression for a long time after it came out. I don't remember exactly, but it was quite a while after we had custom meshes working. Any file parts you changed were saved uncompressed, and those you used but hadn't changed were copied to the output in their original compressed form.
<* Wes *>

[dizzy]

This is all very basic and I honestly can't see how this could trip you up but:

Code:

ls [-g group] [-t type] [-n name] [-o g|t|u|i|x|s|n] [-r] package [output]
extract [-g group] [-t type] [-i instance] [-s subtype] [-q] package
append package data [data...]

and as an example:

Code:

ls -g 7FD46CD0 -t 42434F4E -o i objects.package dir.txt
extract -g 7FD46CD0 -t 42434F4E -i 106 objects.package
create new.package
append new.package 7FD46CD0-42434F4E-00000106-00000000.dat

If you wanted to mod this, simply edit the dat file between the extract and the append step.

[J. M. Pescado]

Code:

ls -g 7FD46CD0 -t 42434F4E -o i objects.package dir.txt
extract -g 7FD46CD0 -t 42434F4E -i 106 objects.package
create new.package
append new.package 7FD46CD0-42434F4E-00000106-00000000.dat

If you wanted to mod this, simply edit the dat file between the extract and the append step.

The flaws in that are firstly, the files are appended out of order, and secondly, there is no provision for removing the original file from the package, which makes it useful only for first-stage edits.

[dizzy]

Actually, it's more useful for full-blown rebuilds, but really. Who cares *how* it works, it works. 

[J. M. Pescado]

Actually, it's more useful for full-blown rebuilds, but really. Who cares *how* it works, it works. 

Yeah, but it'd be nice to see a sample of something that doesn't involve tearing apart the entire package into shredded bits all over the desktop.

[wes_h]

Well, I had thought about slapping the innards from those programs into a .DLL. While this doesn't directly help the non-MS people, it would allow something to be written in almost any language, including Visual Basic, by making function calls. You could then also make the command-line tools as not much more than a command-line parser and a function call.

But I am not doing that myself because I am doing other things that have sucked up all available non-RL time.