The Compressorizer! Mass DBPF Compressing Program

[MutantBunny]

I started working on my Dls with Delphy's tool. My dls folder waqs at 5.5 gigs and 12000+ files. Delphy's tool IDed 1100 or so as being orphans. That wasn't exactly correct, some files were fine while others were orphans. I got tired of going thru 1100+ files and just deleted them

Then I started in with the Squisher. I squished each folder separately. It Size went to 2.8 roughly. Then tried to stat my game. Only, I had forgotten to delete the groupcache files. Soooo I quit (hard shutdown) and went back to see what it had done to the files, curious if it had decompressed them. And yes, it had! But only some -probably because I shut down in the middle of the load.

So I resquished but I got lazy and did all files at once. Squished to 1.8 this time! Deleted thecache files, added 1 or 2 minutes to load time but other than that all is well in my game. ANd the files stayed squished this time.

[Magicmoon]

Perhaps my understanding of the actual string storage is not complete, but the code for the DDO works with basically every other file you can throw at it *except* the compressorised ones.

It's not choking on every compressed file. I'm throwing approx 50 errors now that the Downloads folder has been compressed, but there are 19,353 files in there, most of them package files, which the program is handling just fine (or so I assume since they are categorized correctly when the program finishes scanning).

[Indiasong]

Has anyone else noticed that duplicates are not spotted by the clean installer when you check items downloaded with a house to the compressed ones in your download folder?
About half of them seem to be spotted now. Yet when I throw them in their allocated folders, I am asked to replace or not.
I'm trying the simpe scan folder now, but that takes more than an hour. And I have to boot after re-installing the normal sized fonts, or I can't see the scanfolder controls.

[benrg]

I originally posted this reply in the "New DBPF library" thread, but I'm copying it here since the original message was in both threads.

Normal compressed SHPE ref, in Hex view in SimPE: 66 42 72 61 6e 64 69... etc.  This is 1 byte for the string length (0x66 - 102 bytes) and then immediately followed by the actual string (Brandi...).
For a compressorised SHPE ref, in Hex view in SimPE: 95 01 62 72 61 6e 64 69... etc.  This is 1 byte for the string length (0x95 - 149 bytes) and then a 01 and then followed by the actual string (brandi...)

SimPE uses System.IO.BinaryReader.ReadString() and its BinaryWriter counterpart for most string serialization. These do not use a single byte for the length -- they use a weird base-128 encoding documented here: http://msdn2.microsoft.com/en-us/library/system.io.binarywriter.write7bitencodedint.aspx. In this encoding a length of 0x95 is serialized as 0x95 0x01. I'm almost certain that this is your problem, not anything to do with compression.

I don't know whether this is a bug in SimPE or whether it actually is the encoding used by The Sims. If the former, you may have to use an ugly workaround anyway.

[Zazazu]

Slight issue, may have been mentioned before. The Collection Creator doesn't play nice with this. In fact, when opening folders in CC, I get nothing or one gobbly-gooked named file. Ick. I really need to clean my recolors and decorative objects, too. Any short-term suggestions?

[jsalemi]

Slight issue, may have been mentioned before. The Collection Creator doesn't play nice with this. In fact, when opening folders in CC, I get nothing or one gobbly-gooked named file. Ick. I really need to clean my recolors and decorative objects, too. Any short-term suggestions?

If CC = Collection Creator, I think jfade has an updated version of that on his site that plays nice with this.

[Zazazu]

Thanks. New version reads the files fine, the program just didn't do what I wanted it to. Still useful, though.

[jfade]

Thanks. New version reads the files fine, the program just didn't do what I wanted it to. Still useful, though.

What do you mean by this? I'm slightly confused now.

[Zazazu]

I had wanted something that would let me see what it was and delete the file if need be. This did half, and I couldn't see the real file name to cull them. I still use it for the intended purpose of collection creating, though.

[jsalemi]

I had wanted something that would let me see what it was and delete the file if need be.

Ah -- you want Paladin's Sims 2 Categorizer.  You can find it in the programs section over on http://www.simwardrobe.com/.

[americancindy]

My downloads folder got squished from 7.9 gigs to  3.1 gigs. This program is awesome! I just wish it wouldn't create problems with Delphy's Download Manager which was awfully useful. Is there anything I can do, so I can still run Delphys program, plus have my squished downloads folder?

[Rose Outlaw]

From what's been told so far: bothering Delphy to update his program would be, uhm, something like a waste of time, wouldn't it?

[Zazazu]

From what's been told so far: bothering Delphy to update his program would be, uhm, something like a waste of time, wouldn't it?

Delphy's posted a couple weeks ago sounding like he was going to look into it, unless I'm misreading and it was an "I give up".

And thanks, jsalemi. Too many "C" Sim programs and "-izers" on my desktop. Got confuzzled.

[Eva Aisling]

Until this program is able to work with Delphy's download organizer, it might be a good idea to take all compressed downloads out of the folder and just use the organizer for newly downloaded files before compressing.

[baratron]

Some more questions to add to the FAQ :

Can I run the Compressorizer on folders I've already compressed to make them even smaller?

I accidentally ran the Compressorizer on a folder that I'd already compressed! Will something bad happen?

I'm pretty sure the answers to both of these questions are "Once the file is compressed, it can't be compressed any smaller. So the utility will just ignore it.", but it would be nice to have that spelled out in idiot-friendly text in the manual.

Also, a question/bug report. I've noticed the program doesn't work with the standard Windows shortcuts of holding down CTRL to select multiple individual files, and SHIFT to select a block of files. How difficult would it be to add these in? I have all my Downloads sorted in folders depending on where they came from, and even with Recursive selected, it's still requiring something like 60 individual selections to compress everything if I don't feel brave enough (or am too impatient to wait) to select my entire Downloads folder in one chunk.

[notveryawesome]

I also have a question about using this along with the Pets CC Updater. I noticed that, when updating my CC for Pets, the PCCU decompresses the files (which I've already compressed using the Compressorizor). Does this mean that I will have to then re-compress them once I'm done updating them for Pets?

I know I'm doing things backward, but I'd decided not to bother updating my CC and then changed my mind after I'd already compressed all my files.

[jfade]

Some more questions to add to the FAQ :

Can I run the Compressorizer on folders I've already compressed to make them even smaller?

I accidentally ran the Compressorizer on a folder that I'd already compressed! Will something bad happen?

I'm pretty sure the answers to both of these questions are "Once the file is compressed, it can't be compressed any smaller. So the utility will just ignore it.", but it would be nice to have that spelled out in idiot-friendly text in the manual.

Also, a question/bug report. I've noticed the program doesn't work with the standard Windows shortcuts of holding down CTRL to select multiple individual files, and SHIFT to select a block of files. How difficult would it be to add these in? I have all my Downloads sorted in folders depending on where they came from, and even with Recursive selected, it's still requiring something like 60 individual selections to compress everything if I don't feel brave enough (or am too impatient to wait) to select my entire Downloads folder in one chunk.

Actually, the Compressorizer will just decompress and then recompress anything you throw at it, so if it's already compressed, it's just a waste of time, no harm done.

As for the multiple select, you mean with the open file dialog? Hmmm... I'm not sure you can toggle that to allow multiple selections since I've never tried. I'll have to experiment and see.

I also have a question about using this along with the Pets CC Updater. I noticed that, when updating my CC for Pets, the PCCU decompresses the files (which I've already compressed using the Compressorizor). Does this mean that I will have to then re-compress them once I'm done updating them for Pets?

I know I'm doing things backward, but I'd decided not to bother updating my CC and then changed my mind after I'd already compressed all my files.

Yeah, you will have to recompress them after updating them. I'm working on updating the updater (ha) to make it have more options for updating things for Seasons, make it faster, and add auto-recompression of things once it's saved them. This is, of course, low on my to do list at this point as I'm sort of focused on the Wardrobe Wrangler. (Yes, it's actually in alpha at this point, and I'm working on getting it to the beta stage. So far so good! )

[notveryawesome]

Thanks for the answer, jfade. I'd already discovered that I did need to re-compress the files but it's good to have confirmation. It definitely helps to do things in the correct order (update, THEN compressorize). I now have a system in place.

[witch]

This is, of course, low on my to do list at this point as I'm sort of focused on the Wardrobe Wrangler. (Yes, it's actually in alpha at this point, and I'm working on getting it to the beta stage. So far so good! )

Please, please don't lose focus. Don't waste your time on boring old things like eating, sleeping or having a life either.

If, umm, you know, you want some beta testers... I have a truckload of crap to test it on.

[jfade]

This is, of course, low on my to do list at this point as I'm sort of focused on the Wardrobe Wrangler. (Yes, it's actually in alpha at this point, and I'm working on getting it to the beta stage. So far so good! )

Please, please don't lose focus. Don't waste your time on boring old things like eating, sleeping or having a life either.

If, umm, you know, you want some beta testers... I have a truckload of crap to test it on.

Actually, I wouldn't mind a few more testers. You can hop over to http://forum.djssims.com/ and then register and apply to join the Beta Testers group there. To do that, just register, and then under the description of the beta testing section there's a link that you click that takes you to a page where you apply. Anyone is welcome!

[witch]

Cool, I'll check that out when I get home on Wednesday.

[crazyforwipeout]

I ran the Compressorizer on my downloads folder and went from 1.83GB to 1.15GB. I randomly checked my lots in game and everything sees to be running well.  Thanks.

[kuronue]

A thought occurs to me- any reason why compressed downloads wouldn't work in Stories? After all, if I'm putting in a set of default replacement skins, they ought to be compressed...

[jfade]

A thought occurs to me- any reason why compressed downloads wouldn't work in Stories? After all, if I'm putting in a set of default replacement skins, they ought to be compressed...

Should work fine, unless EA did something stupid and EA like, such as changing their own algorithm.

Try it and see.

[TashaYarrr]

Earlier in the thread, people talk about compressing their base game files or "EA GAMES" folders. I'm not sure if they're talking about the Program Files\ stuff (including going into each xpac's directory?), or the Documents and Settings stuff, or both. Would anyone be kind enough to tell me some good candidates for base game compression, or point me to a place I could find out?