Cutting Hair File Size

[CatOfEvilGenius]

I need to know which of these cases is more common:

CASE A: hair recolor with duplicate textures that REALLY ARE duplicates, byte for byte

CASE B: hair recolor with several textures that look like duplicates at first glance, but upon extremely close inspection, they're jittered by one or two pixels so an automated comparison considers them different, even though for all intents and purposes, they're the same damn texture and one wonders why the hell the recolor creator made them this way - edit: AmberDiceless points out there may occasionaly be legitimate reasons for this

I see Case A frequently in clothing and can handle it fine.  I've seen Case B in a few hair recolors.  Case B would require a UI that shows the textures and lets the user select which ones are duplicates.  This would be a pain in the butt to write so I need to know how often Case B occurs.  I mostly do clothing, not hair, so I don't know.  If Case B is infrequent, we can just keep fixing it manually in SimPE like we've been doing.  If its common, I'll write the GUI.

[J. M. Pescado]

CASE A: hair recolor with duplicate textures that REALLY ARE duplicates, byte for byte

This is the "trivial" case, so its commonality is not really the issue.

CASE B: hair recolor with several textures that look like duplicates at first glance, but upon extremely close inspection, they're jittered by one or two pixels so an automated comparison considers them different, even though for all intents and purposes, they're the same damn texture and one wonders why the hell the recolor creator made them this way

I don't BELIEVE it is that common, but one potential cause is that the images are saved in different compression formats, resulting in compressual artifacts that are different. Unfortunately, there is no way a COMPUTER can identify this with any degree of accuracy, so you're probably stuck doing the UI for the nontrivial cases. Also, you were added to Bowels, in case you didn't notice.

[CatOfEvilGenius]

Yes, A is the trivial case.  B I have seen in almost every hair package made by a certain creator, so perhaps its just that one person?  Been looking at more hair packages from different creators and different sites.  "Unfortunately", almost all the hair I have in Downloads is by creators who took the time to clean up their packages.  Anyone know a site with lots of hair packages in need of cleanup and dupe removal?

Or maybe someone can suggest some Maxis hair, that when recolored with BodyShop, gives you a recol package with duplicate TXTRs?  The "short cute" hair I tried recoloring with BodyShop had dupe images in Projects, but the final recol package in SavedSims did NOT have duplicate TXTRs.  When BodyShop recolors clothing with a single TXTR used by multiple TXMTs, the resulting recol has dupe TXTRs, but it doesn't make this mistake with hair, at least not for me with the hair I tried.

I did notice I got added to Bowels, thank you.

[Ambular]

I'm afraid I can't say with any certainty which case is more common, but in the case of B, just for clarification, there may sometimes be a legitimate reason why the creator did it that way.

When more than one hair mesh group references the same texture, BodyShop will clone multiples of that texture.  If the mesh happens to have, say, five parts, you wind up with umpteen duplicates: one for each group in each age category.  Now, in the case of a simple recolor, the extras may be unneeded (thus the stuff we've been discussing in this thread.)  However, in the case of one hairstyle I modified myself, I needed some of those extra textures in order to apply slightly different alpha effects and details on different parts of the mesh.

Also, with certain hairstyles (but not all of them), it seems as though some mesh groups pick up their display properties from the Material Definition (TXMT) of the Sim's skin, while others will use the TXMT of the hair itself.  When that happens, it may be necessary to color one texture a bit differently so it blends with those using a different TXMT.  Otherwise you'll get an unsightly two-tone effect.

These odd cases are the reason I said to take a close look at all the textures and make certain they really are identical before you delete them.  A visual way to confirm that would be highly advisable.

[CatOfEvilGenius]

An alpha version of the extraTXTRremove program is now available for testing.
It will automatically remove extra TXTRs in hair (and clothing) recolors.
If you want to test it and have any comments or bug reports, please put them in the MTS2 thread below.
I don't want to clutter this thread.

extraTXTRremove alpha version available for testing here
.

[J. M. Pescado]

The other useful form of reduction is to repack the Adult/YA propertysets together into a single propertyset and eliminate it from the BINX/3IDR/MATDs entirely, saving some more space that way. I also do this. I am ruthless like that.

[Ambular]

The other useful form of reduction is to repack the Adult/YA propertysets together into a single propertyset and eliminate it from the BINX/3IDR/MATDs entirely, saving some more space that way. I also do this. I am ruthless like that.

Combining Property Sets and TXMT's is easy, but it's a bitch trying to figure out which 3DIR goes with what.

[Theo]

You may have noticed that hairs in some stuff packs postedlinked in MATY were packaged using the 3 keys method. This makes 3IDR/BINX identification a lot easier, as well as making the XHTN obsolete. Old custom hairs may also be repackaged using this method.

[Ambular]

You may have noticed that hairs in some stuff packs postedlinked in MATY were packaged using the 3 keys method. This makes 3IDR/BINX identification a lot easier, as well as making the XHTN obsolete. Old custom hairs may also be repackaged using this method.

Actually, I hadn't, and I confess I'm not entirely sure what you just said.  XD  Can you link me to any discussions or explanations on the topic?

[Theo]

For example, take a look at any of the packages in "Fixed TS2 Store Hairs.zip", linked here. You'll find a number of differences:

• The XHTN resource has been replaced by a clothing-type COLL.
• The property sets no longer share their 3IDR's with the BINX. In fact, there are now twice as many 3IDR's (plus one for the COLL). But this is not a problem, since they're small in size.
• The property set references are no longer offset by 1. In the old packaging, PSET #1 was referenced by 3IDR #2, and so forth. (*)

(*) This explanation may have left you even more bamboozled, but in a nutshell, there are three types of resources involved, and each one has its own 3IDR resource (associated by Group and Instance).
It runs more or less like this:
— Begin with a BINX; find a 3IDR with the same id.

— On the BINX, get the 'binidx' value; this is the index of the line within the 3IDR that points to the COLL.
— The COLL (plus its own 3IDR) is used for cataloging purposes. You don't want to mess with, or delete this resource; leave it alone.

— On the BINX, get the 'objectidx' value; this is the index of the line within the 3IDR that points to the property set (GZPS).
— Get the GZPS; find a 3IDR with the same id. This is the object in hand.

Having understood that, we can now start with a property set we wish to delete, and work our way back to the resources that are referencing it, and delete them too. Easy, huh?

The old packaging (alas, the most common), combines a non-straightforward association of objects, with the requirement of 3IDR editing - in the event you'd wish to delete a resource.

[CatOfEvilGenius]

Combining adult / YA GZPSz and hunting references in 3IDRs sounds like something it would be useful to automate.  While I'm not yet very familiar with hair, I have chased instance / group references in clothing and makeup.  If you all would like me to take a crack at automating the processes Theo and Pescado discussed, I would be glad to.  Theo, if you already have plans to incorporate this into your color binner, let me know, so I don't duplicate your efforts.  Either way's fine with me.

[Theo]

Combining adult / YA GZPSz and hunting references in 3IDRs sounds like something it would be useful to automate.  While I'm not yet very familiar with hair, I have chased instance / group references in clothing and makeup.  If you all would like me to take a crack at automating the processes Theo and Pescado discussed, I would be glad to.  Theo, if you already have plans to incorporate this into your color binner, let me know, so I don't duplicate your efforts.  Either way's fine with me.

My idea for now would consist of a converter to this packaging method; it would accept a normal [hair|clothing|makeup] package, and then split and rearrange the 3 object types.

But I think you should finish the TXTR cleaner as a standalone tool first; It's a lot of work as it is now, and the more you add features, the more it becomes easy to miss the point, very much like what happened with the - now obsolete - binning tool. Also, because I need the TXTR cleaner NAO!

[CatOfEvilGenius]

I agree with you that adding the 3 object repackaging feature should wait until the texture cleaning is all done and thoroughly tested.  I've put it on my list'o'stuff'todo.  The repackager will probably end up being a separate program.

I need the TXTR cleaner NAO!

I also need it now, that's why I wrote it.   If you can live with no GUI, then the command line, single package version is available now:
http://www.modthesims2.com/showthread.php?t=318324

I will add a GUI and the ability to look for duplicates across multiple files, check back in a week or so.  I'll also add the optional ability to tell the program to remove user selected textures that are "almost identical", since this sometimes occurs in hair files.

[Ambular]

But I think you should finish the TXTR cleaner as a standalone tool first; It's a lot of work as it is now, and the more you add features, the more it becomes easy to miss the point, very much like what happened with the - now obsolete - binning tool. Also, because I need the TXTR cleaner NAO!

I don't consider the binning tool obsolete.  I still use it all the time.  Granted, jfade's Wardrobe Wrangler can also be used to bin hair, but the binning tool is frankly quicker and has a more straightforward interface that I much prefer (sorry jfade!)  Those extra features occasionally prove very useful.

(I'm aware of the issue with families and hair placed in the Custom bin, but since I seldom mess with putting anything in that bin anyway, it's not that big a deal to me.)

[WandersFar]

Can the technique described in part 2 of the original post be applied to skins as well? (Summary: Find duplicate textures and note their filenames. Go to Material Definitions and change the corresponding stdMatBaseTextureName references so they all point to a single texture. Delete the now unused duplicate textures.)

I have several skin packages where, for example, faces and scalps are shared across all genders and age groups. Could I delete all but one of the duplicate textures, and then point the material definitions to the remaining texture? Or would that just bork the package completely?

[CatOfEvilGenius]

Can the technique described in part 2 of the original post be applied to skins as well? (Summary: Find duplicate textures and note their filenames. Go to Material Definitions and change the corresponding stdMatBaseTextureName references so they all point to a single texture. Delete the now unused duplicate textures.)

Yes, you can (and should) do this with skins.  I have some CC skins created this way (not mine, I'm not that artistic).  If you want, try my automatic duplicate texture remover.  Back up your files first.  I think it works fine, but it's still in testing.

Hope I'm not being too off topicy discussing skins here, but the question seemed related.

[Ambular]

Can the technique described in part 2 of the original post be applied to skins as well? (Summary: Find duplicate textures and note their filenames. Go to Material Definitions and change the corresponding stdMatBaseTextureName references so they all point to a single texture. Delete the now unused duplicate textures.)

I have several skin packages where, for example, faces and scalps are shared across all genders and age groups. Could I delete all but one of the duplicate textures, and then point the material definitions to the remaining texture? Or would that just bork the package completely?

Yep, as Cat says, this is totally a good idea.  Just be very careful to verify that the files you think are duplicates, really are duplicates; sometimes they look very very similar but have subtle differences in shading that can leave seams on your Sim.  Cat's duplicate texture remover is probably the way to go, but make sure you test the cleaned up skin before you permanently delete the original.

[WandersFar]

Thank you, Cat and Amber, for your advice.

I am eagerly anticipating the duplicate texture remover, but I'm holding out for the GUI. Command lines make me a little nervous.

[CatOfEvilGenius]

The extra TXTR remover now has a GUI.
You can download it from MTS2, link below, it's in the FIRST post.

extra TXTR remover program

[CatOfEvilGenius]

Hair File Cleaning, and One Gray, Not Four... on a Mac

Mac users who make custom hair can bin with Wardrobe Wrangler, but WW does not remove extra grays.  Theo's SimPE plugin can't be used on a Mac.  So I plan to write a Mac app for cleaning hair files.  It'll do what Theo's plugin does.  Theo, or anyone else, if you plan to do this already, just let me know.  I like plans where I do no work.

Someone (Theo?) please tell me if the procedure for changing 4 grays to 1 gray is correct.  I did everything below manually with SimPE (not Theo's binner) on some hair with extra useless grays.  The updated hair worked fine in game, but I want to know if I missed anything, or did something that might break stuff without me noticing.

preliminary stuff...
1) Remove duplicate textures.
2) Combine A / YA
3) Make sure all 4 hairs are properly binned.
4) Make sure all 4 hairs are in the same family.
5) Give all hair correct genetics.  What are the correct genetic values for black, brown, red, blond?  I don't mean hairtone, I know about 1,2,3,4 and elder 5.  I mean, in XHTN, what does genetic=0 mean?  Should it be 0 for all four or not???

remove extra grays (unless you really, really want different shades of grey for different hair tones)
6) Nuke duplicate grey textures in the black hair and update TXMTs accordingly.
7) In the blond, red, and brown hairs, do the following
  a) nuke the GZPS for gray hair
  b) nuke all grey hair TXTRs
  c) find all TXMTs for gray hair, they have "emhair" or "efhair" in their name, note their Instance numbers
  d) find all 3IDRs that reference the Instance numbers of gray TXMTs from c), note the 3IDR Instance numbers
  e) nuke all BINXs with the same Instance numbers as the 3IDRs from d)
      nuke all 3IDRs from d)
      nuke all TXMTs from c)

Optionally, the user may want to townify the hair.  I assume this can be done?  Will changing the creator number to zeros do it?
Might make a separate "townifier" app that townifies hair / eyes / clothing, rather than bundling that function into the hair cleaner.

The user may also want to put the hairfile name in the tooltip, but that could also be its own separate app.
Might make that app if jfade doesn't include that function in the next version of Wardrobe Wrangler.

Is there anything else a hair cleaner should do?

[Theo]

5) Give all hair correct genetics.  What are the correct genetic values for black, brown, red, blond?  I don't mean hairtone, I know about 1,2,3,4 and elder 5.  I mean, in XHTN, what does genetic=0 mean?  Should it be 0 for all four or not???

The genetic binding is done by setting the hairtone guid in the property set to the value of the family guid in the XHTN.

The default XHTN's - which can be found in \TSData\Res\Catalog\Skins\Skins.package - have these 'genetic' values:
Black = 1
Brown = 1
Blond = 2
Red = 2
Grey = 0
Other = 0

Since hair binning binds your property sets to the default XHTN's, the custom XHTN seldom has any value of its own, except when custom hairtone guids are involved.

7)
  a) nuke the GZPS for gray hair
  c) find all TXMTs for gray hair, they have "emhair" or "efhair" in their name, note their Instance numbers

Yes, that's more or less how my tool worked, but since then I realized that the TXMT's were addressed according to this procedure:
7)
  a) Open GZPS; Fetch the values of all the 'overrideNresourcekeyidx';
  c) Open 3IDR with same instance as GZPS; Use the previously fetched indices against the resource list, to find the affected TXMT's.
Pseudocode:

Code:

int[] indices;
for (n=0; n<GZPS.property("numoverrides"); n++)
  indices[n] = GZPS.property("override{n}resourcekeyidx")

tgi[] txmts; // tgi = { dword type; dword group; qword instance; }
for (i=0; i<len(indices); i++)
  txmts[i] = 3IDR.items[indices[i]];

7)
  d) find all 3IDRs that reference the Instance numbers of gray TXMTs from c), note the 3IDR Instance numbers
  e) nuke all BINXs with the same Instance numbers as the 3IDRs from d)

The problem is that some hairs don't work like that.
For example, look at this hair package. The elder/grey GZPS has instance=5; so you open 3IDR #5, to find this:

Code:

00: CRES, 0x1C050000, 0x1311351AFF1E5AEF
01: SHPE, 0x1C050000, 0x84AEAA5AFF135801
02: TXMT, 0x5FBAFC0B, 0xDA51D2B0FFDC13AA
03: TXMT, 0x5FBAFC0B, 0xEC27220CFF93F4D1
04: TXMT, 0x5FBAFC0B, 0xDA7738FCFFFE22AB
05: <NULL>
06: STR#, 0x5FBAFC0B, 0x0000000000000001
07: COLL, 0x0FFEFEFE, 0x000000000FFE0080
08: GZPS, 0x5FBAFC0B, 0x0000000000000004 // 4!? WTF

The addressing scheme here is indices 00-04 used by GZPS #5, and 05-08 by BINX #5 that links to GZPS #4(!)

If you delete 3IDR #5, not only will you break the link to GZPS #4, but also this hair will stop working in game (or at least it did when I tried).

So my advice is to tread carefully around these resources. Besides that, the quick and dirty way to remove the extra grey hairs is to set the 'hide' flag on the property sets.

[CatOfEvilGenius]

Theo - Thanks for that.  You're right, can't match up GZPS, BINX, and 3IDR based just on the Instance # of the GZPS.  Have to double check reference numbers in the 3IDR.  Last night, I thought I was removing BINX and 3IDR for YA, and that broke my elder hair.  Good thing I back up.

BINX: These are for catalog entries in CAS, right?  Are they also used when sims change appearance in a mirror?

3IDR: These are a collection of references to other resources, used for grouping them together for each age, right?

From what I've read, the concensus seems to be that the above two resource types don't have to be removed for unused ages.  Ideally, I would like to nuke all unused stuff, but if there isn't a 100% accurate way to determine which BINX and 3IDR go with which age, I think I should leave them so I don't break stuff. 

edit: 3IDRs have references you can chase, but from your example, it looks like I have to check almost every single line in a 3IDR to figure out what all it goes with.  I could write code to do it, maybe write code to make new 3IDRs that only reference one GZPS.  Maybe.  Not sure it's worth the trouble.

[Ambular]

So my advice is to tread carefully around these resources. Besides that, the quick and dirty way to remove the extra grey hairs is to set the 'hide' flag on the property sets.

If you do that, and also change the Material Definition references and remove redundant textures, doesn't that clean up the vast majority of the resource-hogging and catalog-cluttering junk?  Everything else should be just 'invisible' data.  Seems like you'd have to have a very large number of custom hairstyles before it would become a serious problem.

[Theo]

BINX: These are for catalog entries in CAS, right?  Are they also used when sims change appearance in a mirror?

That's correct, for example if you look at the interim packages that are created by BodyShop (the ones in the \Projects folder ), you'll find that these are the same as the final packages, except that they lack the BINX resources, and thus cannot be displayed anywhere but the Create Content section of BodyShop.

3IDR: These are a collection of references to other resources

Period.

What really matters here are the main resources (BINX, GZPS, XTOL, etc...), that sometimes need to reference other resources.
They may do it directly, as seen in this example taken from BINX group 0x7F5AC619, instance 0x0CDF5EF4 (in \The Sims 2\TSData\Res\Catalog\Bins\globalcatbin.bundle.package):

Code:

bingroupid = 0x7FFBA28C
binid = 0x6CDF5E06
binrestypeid = 0x6C4F359D
creatorid = 00000000-0000-0000-0000-000000000000
icongroupid = 0x00000000
iconid = 0x00000000
iconrestypeid = 0x00000000
objectgroupid = 0x2C17B74A
objectid = 0xCC216E44
objectrestypeid = 0xEBCF3E27
(etc...)

Or, they can store those references on another resource (3IDR), and just use the list index:

Code:

binidx = 0
iconidx = 1
objectidx = 2
(etc...)

How can you find a 3IDR that belongs to a given resource? Just match its group and instance numbers.

If you do that, and also change the Material Definition references and remove redundant textures, doesn't that clean up the vast majority of the resource-hogging and catalog-cluttering junk?  Everything else should be just 'invisible' data.  Seems like you'd have to have a very large number of custom hairstyles before it would become a serious problem.

If total extirpation is not possible, then that may be considered a reasonable stump, yes. 

[CatOfEvilGenius]

When total extirpation is possible, you mentioned it's better to make the new file in the new format, with COLL and no XHTN.  Would such a file still be base game compatible or not?  If COLL is a collection resource, I'm guessing not BS compatible?