Testers wanted: Color Binning Tool 0.2.30

[Regina]

I am finally getting the opportunity to try out this tool, and let's just say this is no doubt going to set me off on a hair downloading spree!  For months I've been seeing hairstyles and just *harrumphing* because I knew if I downloaded them I'd just be causing myself a ton more work.  Woo-boy!  This changes everything!   

Motoki, I really and truly appreciate your tenacity in the non-properly binned hair arena.  Keep it going!  And don't let up on those site-owners, especially now that this tool is out!

And Theo, thank you so very much for taking the time to program this!  I've no doubt your time you've used in its development has already saved many times that amount of time on everyone else's end by not having to bin the old-fashioned way!

Okay--now before I forget!  I do have one question.  I noticed that when I binned a hairstyle, then saved it as one file instead of the original four, that the size of that file is a whopping 5.4 megs!  The hairfiles that went into it averaged 700 kb each.  This means that somehow or another that file grew almost 3 megs in size between binning and saving.  Is there any way around this because this is going to be absolute hell on people's games, especially on slower systems, not to mention what it could do to bandwidth and upload/download times? 

[Theo]

Hi Regina,

I've also seen the file size increase, and I think it may be related to the fact that the tool doesn't modify the existing packages - it creates new files from scratch, and afterwards processes and adds the resources to them.

There may be an option in SimPE's code that allows a more "conservative" file size setting, but until now I haven't found it

[twistingsims]

Just wanted to add my 2 cents worth if you don't mind.      It does save alot of time for binning hair and I know I could never create a program like this, but I did notice two things.

If a creator makes more than one version of blond or brown, etc.,  I have to put one in the custom bin but where would I put a second extra color?  And I would actually like it if I could put the all versions of blonds, browns, etc., in their respective bins instead of putting the extras back into custom, just my preference.

The second thing is when the binning process is finished, I noticed that textures don't change to the next or previous age group in Bodyshop.  Would making a unique GUID in options solve that?

Thank you for listening.   

[Motoki]

Hi Regina,

I've also seen the file size increase, and I think it may be related to the fact that the tool doesn't modify the existing packages - it creates new files from scratch, and afterwards processes and adds the resources to them.

There may be an option in SimPE's code that allows a more "conservative" file size setting, but until now I haven't found it

Theo, are the textures being saved as uncompressed somehow? SimPE can make use of the Nvidia DDS tools to provide DXT compression but you have to have the tools installed and point SimPE to the location where there are installed. Also even if you did this in the past and upgraded to a newer version of SimPE you may need to tell SimPE again the location of the DDS tools.

[Regina]

Motoki, you could be on to something there.  Now that you mention it I'm pretty sure that when I installed SimPE on this computer that I did NOT install the NVidia DDS tools.  I was only freshly acquainted with them and barely installed them on my old computer so it would only be right for my brain for me to completely forget that on my new machine.

Edit:  Nope--that didn't do the trick.  I installed the tools, fixed it up in SimPE and re-did that particular package.  The file size is still the same.

[Motoki]

So maybe the binning tool isn't correctly using the tools then to compress the textures? Hmm. I never noticed a file size increase when I exported the TXMT from SimPE and pasted it into a new package, but the tool must be decompressing the textures and then I guess either resaving them uncompressed or with less compression. Theo would have a better idea how it works though. Maybe Quaxi could give some insights too but his board was down last time I checked.

[pinhead]

Hi!

No! the problem with the size of the packages is just a matter of uncompressed resources.
SimPE, by default, when importing or creating resources never compress them (security reason).

You just need to open the package after completed and select some resources to
compress. But be very careful on this! some things can't be compressed.
And i never use the compression in 3IDR files because i got corrupted.

This information i'm a little worried. I'm not responsible of anything if something got corrupted
and people don't know what they are doing... So, if you are worried about using this, use just to TXTRs
and will decrease a lot of the package size. The other resources are all small ones and will not change
too much. But remember that i said, some can't be compressed. So, if it not compressed one or two,
forget it.

to do this, just select one or multiple resources (draggin a box around the resources with your mouse
or shift+click or shift+ctrl ... whatever) and select the Resource Dock (where
is showing the Group ID and Instances and where you can edit the IDs). There is a little box called compression.
Select "Yes" and go to File/Save (don't need to hit commit). Will make all selections compressed. Make sure to not hit commit!

note that simPE will freeze a little time, sometimes it takes more than 2 or 3 minutes if a lot of resources are selected.
i suggest select the resource by type first. This way is more fast.
And be patient! don't close the simPE task using task manager if hang up for too much time. lol
go take a bath, play dices or poker with your dog... PATIENCE!

make sure to make a little backup of the package. if something gots corrupt you can't use it.

[pioupiou]

Just a thank you post 
I only noticed today that you change the tool to allow choosing of the proxy for the custom color, thank you so so much !!!
I'm ashamed to notice only today
thanks again
pioupiou

[Theo]

I browsed SimPE's code documentation (the code itself! ), and indeed each resource has an option to make it compressed through code!

So following pinhead's tip, here's the code to compress the textures only:

Code:

IPackedFileDescriptor[] textureFiles = ret.FindFiles(SimPe.Data.MetaData.TXTR);
foreach (IPackedFileDescriptor pfd in textureFiles)
{
...
...
	if (this.settings.CompressTextures)
		pfd.MarkForReCompress = true;
}

Early tests show that compressing textures alone is enough to bring the file size down by a considerable amount.
The tradeoff is a noticeable lag when saving the packages.

I will provide an option to compress textures, available as a check box on the output options panel. 

By the way, I found another serious bug: the tool didn't take into account the "stdMatNormalMapTextureName" property of the material definitions, which caused some textures to be excluded from the output packages . The next release will have that correction as well.

[pinhead]

that will be cool, theo!
if the tool make the output textures compressed is less manually work that we have.
thanks a lot!

about the normal maps, i was thinking that bump map textures are only used by clothes..i never saw a hair using normal maps.
i'm curious. why you are worried about it?

[Theo]

i was thinking that bump map textures are only used by clothes..i never saw a hair using normal maps.

idtaminger PM'd me because she had a hair that was showing the skincolor instead of the hair's textures. The hair is "Nouk Dreads" that can be found here: http://noukiesims2.net/Genetics.html

After inspecting the package he sent me, it was clear that the TXMTs were referencing custom textures through the stdMatNormalMapTextureName property and that the tool had discarded those textures. The Filelist also had two entries: ufhaircornrowslong-all-bump and ufhaircornrowslong-brown (by that order).

It did take some reworking, but now the tool takes this special case into account. When reverting to the base textures, the texture names are taken with the correct order of the Filelist. Copying and pasting texture references will also consider base and bumpmap textures as one unit.

I'll let you know when I release these corrections so that you can test them.

Edited because idtaminger was in need of gender correction

[pinhead]

oh! i see.
that's new to me. I never saw a hair using bump textures. Now i know that the maxis dreads are using it.

good that you are working on this issue.
thanks for the info, theo. great job

EDITED:

Theo, i found something wrong with a package that i was working with your tool.
I downloaded some multi colored tinted hairs and i was categorizing those as "Custom".
If i try to use a base texture in a Material, your tool is changing the color name of Filelist to
"-custom" when pasting to stdMatBaseTextureName.
for example, the hair was cloned from a black hair. toddler, child, teen and elder texture was not changed
to be multi colored as adult texture. So, i was copying the base texture to the Materials. In file list, for example,
i have "puhairsimple-straight-black" and your tool is changing to "puhairsimple-straight-custom.
The texture will not be found, of course 

i hope that you can fix this later
i'm trying the version 0.2.7.1 btw.

[Syera]

to do this, just select one or multiple resources (draggin a box around the resources with your mouse
or shift+click or shift+ctrl ... whatever) and select the Resource Dock (where
is showing the Group ID and Instances and where you can edit the IDs). There is a little box called compression.
Select "Yes" and go to File/Save (don't need to hit commit). Will make all selections compressed. Make sure to not hit commit!

Theo, you might want to point out where the Compressed box is on your tutorial - I must've spent about five minutes looking for it!   

I compressed the textures - that decreased the filesize by about 1900k (nearly half the original size) - not bad! 

[pinhead]

Theo, i found something wrong with a package that i was working with your tool.
I downloaded some multi colored tinted hairs and i was categorizing those as "Custom".
If i try to use a base texture in a Material, your tool is changing the color name of Filelist to
"-custom" when pasting to stdMatBaseTextureName.
for example, the hair was cloned from a black hair. toddler, child, teen and elder texture was not changed
to be multi colored as adult texture. So, i was copying the base texture to the Materials. In file list, for example,
i have "puhairsimple-straight-black" and your tool is changing to "puhairsimple-straight-custom.
The texture will not be found, of course 

i hope that you can fix this later
i'm trying the version 0.2.7.1 btw.

just another small mistake in the "Custom" color tab is that the elder hair is the only one using another hairtone value at the PSET. It's
using the maxis 0000005 hairtone. You may need to exclude this change to only "Custom" color tab to keep the same hairtone value as
the other ages (to not broke the DNA if a sim has it). Looks like all the other ages aren't been changed in the hairtone field of PSETs.
 
I don't know if is possible isolate the changes that you are making to color tabs.

[Theo]

just another small mistake in the "Custom" color tab is that the elder hair is the only one using another hairtone value at the PSET. It's
using the maxis 0000005 hairtone. You may need to exclude this change to only "Custom" color tab to keep the same hairtone value as
the other ages (to not broke the DNA if a sim has it). Looks like all the other ages aren't been changed in the hairtone field of PSETs.

The hairtone field in PropertySets - elder age excluded - of custom hairs has the same value as the family guid, I guess you want this value applied also for the elder PropertySet?

Anyway, the tool has been updated with these changes.

Theo, you might want to point out where the Compressed box is on your tutorial - I must've spent about five minutes looking for it!

Thanks for pointing that out, I was waiting to have more changes in this release so that I could include them in that document. (In other words, I'd completely forgotten it! )
It has been updated as well.

If a creator makes more than one version of blond or brown, etc.,  I have to put one in the custom bin but where would I put a second extra color?  And I would actually like it if I could put the all versions of blonds, browns, etc., in their respective bins instead of putting the extras back into custom, just my preference.

Alas, the plugin doesn't directly support more than one custom color in the same family. What you can do is to process each custom package in separate sessions in the "Custom" tab, and making sure their family GUID (available in the options panel) is the same. The text box allows you to paste guid's.

The second thing is when the binning process is finished, I noticed that textures don't change to the next or previous age group in Bodyshop.  Would making a unique GUID in options solve that?

That would happen if you excluded those age groups, but if that's not the case, what hair package are you trying to bin?

[idtaminger]

idtaminger PM'd me because he had a hair that was showing the skincolor instead of the hair's textures. The hair is "Nouk Dreads" that can be found here: http://noukiesims2.net/Genetics.html

She! I'm a she.

Oh, and I'm a little confused about the texture referencing. Say I have a Maxis hair that was converted for all ages. Now, since each age is essentially using the same Maxis texture, how would I go about referencing the original Maxis adult hair?

[pinhead]

The hairtone field in PropertySets - elder age excluded - of custom hairs has the same value as the family guid, I guess you want this value applied also for the elder PropertySet?

Anyway, the tool has been updated with these changes.

yes. If other ages PSETs are using the same hairtone GUID (same as the family GUID), the elder must use it as well, since all custom hairs unbinned are using like this.
man, you are fast!

i tested redoing those packages and all is fine now. Plus, i tested other hairstyle with all colors properly binned and no problems were found.
this version is the best so far.
bravo!

But looks like the compress code for textures is not working. at least, not here.
and just a very small request, with low priority of course. When you have a time, if is doable.. could you please make the tool save the option checkboxes
as a preference or something? when i was testing, everytime that i create a new session the options checkboxes are changing to default.

now i will go back to work and stop bothering Theo with requests and small problems. lol

[Motoki]

Does the compress code use SimPE default compression or Nividia DDS tools? I remember Quaxi saying the default compression he put into SimPE wasn't so good, so that's why he added the option to get the Nvidia DDS tools and then have SimPE point to the location where they are installed and use them because he felt it was a better compression program.

At least that is what I seem to recall him saying.

[pinhead]

Motoki, i think that you are understanding wrong.
the compression is for resources (package compression). Not BMP or PNG textures to DDS.(image compression)
what the tool does isn't relating to recreating textures.
Is the same thing if you export a resource to *.simpe and reimporting. Will export/import as a binary file.
what that will be lost when importing again is the package compression, not the image compression.

[Motoki]

Oh, okay I see. Makes sense.

[Theo]

She! I'm a she.

Oh, that surely must have been a typo!

Oh, and I'm a little confused about the texture referencing. Say I have a Maxis hair that was converted for all ages. Now, since each age is essentially using the same Maxis texture, how would I go about referencing the original Maxis adult hair?

Those are functions to which the phrase "Motoki made me do it!" would apply .

What happens is that every hair cloned in BodyShop comes with an integral copy of the textures used. But what some hair creators want is to just change the underlying mesh and use the original base textures (that are buried somewhere in the TSData folder).

Pinhead told us that it was possible to make the hair packages point to the base textures, thus relieving the need to have those big texture files included in the package, and Motoki nagged me enough kindly insisted that I put that option available.

The "Copy/Use texture references" allows you to use the adult texture in the teenage TXMT's for example, and it's useful when you have situations in which each TXMT is using different textures, but you visually see that they are the same, so it will allow you to normalize the texture usage. (I hope this makes any sense)


Motoki & Pinhead:
I've tested the file compression without the DDS tools installed, and it works on my computer. Maybe it's time to query the SimPE forum on this one, because I don't know what it needs to work, I just set a property in code!

[Motoki]

Theo: Sorry for nagging.  But I REALLY appreciate your effort in doing this. If the creator doesn't change the Maxis texture then there's certainly no need to make a copy of it and take up extra space and loading time when we already have the same texture with the game files.

Oh and the SimPE forum is still down.

[Theo]

Lol Motoki, I said it in a jokingly manner, maybe missing the required smiley

Like Regina said before it was your tenacity and insistence for correctly binning the hair packages that made me do this tool in the first place!

cheers

[idtaminger]

The "Copy/Use texture references" allows you to use the adult texture in the teenage TXMT's for example, and it's useful when you have situations in which each TXMT is using different textures, but you visually see that they are the same, so it will allow you to normalize the texture usage. (I hope this makes any sense)

Well, what I mean is, I've got a hair that has added the toddler to teen stages. They use the same textures as the adult stage. The adult stage is a Maxis hair. So...what I want to do is to have no textures at all in the packages, and rather just have them reference the adult texture that the game has already provided. Possible? Feasible?

[Motoki]

Sure, it's possible to reference any Maxis texture for any mesh. Now whether the mesh's UV Map will work for that texture is a different story, but yes you can make new hair meshes that have no textures at all in them because they just call references to existing Maxis textures. My female afro is actually set up this way. And I didn't have this tool when I did it either so I had to do it the old fashioned tedious way.