Merging mods to increase Performance!

[Anach]

Support for the mod merge feature of s3pe is now located here. http://dino.drealm.info/den/denforum/index.php?topic=478.msg3049#msg3049


I've been experimenting with importing .packages to single .package/.dbc files using s3pe. Overall, upon initial tests, by decreasing 691 custom content .package files into 8 merged .package files, I've doubled my frame rate and halved my loading time for game start-up and entering CAS/Build/Buy. It is also possible to add new store content this way, by first unpacking the .sims3pack and then repacking it into a custom merged .package,

The only drawbacks with this system that I have come across are are that merging large amounts of .package into a single .package can take some time to complete, the difficulty of correctly merging different types of mods (including broken mods) and there is no way to uninstall one specific mod from the archive, and one is forced to re-merge the mods if they wish to make any updates or changes. The benefits are better performance, less files to manage, and the ability to learn the way mods work and discover problems or conflicts with mods you were otherwise unaware of.


As pointed out below in several posts, large single .package archives will have the same effect as .dbc archives, as they are essentially the same type of file. The performance increase is simply from merging, not from file type.

The process is pretty simple really.

Quick Guide

1. Get latest test version of s3pe


2. I sorted the .package files into categories that suited my method of organisation, and this is what I used for basis of creating separate .package archives (for me it was 5 total - body, clothing, objects, patterns, world.)

3. opened up s3pe, selected File>New.

4. From the "resource" menu of s3pe and chose "import as .dbc" and chose one of the categories of .package files. If you don't have an "Import as .dbc" option, then you have the wrong version of s3pe.

5. Hit CTRL+A to select all the .package files in the import dialogue box.

6. After the import was complete, I deleted all the type "0x73E93EEB" _xml files with a instance of "0x0000000000000000" , as these manifest files appear to be leftovers from the sims3pack versions of the file and don't seem to be required by the game, but will be a main source of conflicts.

7. After making any changes, i chose "Save" and saved the file as a .package. At this stage there doesn't seem to be any benefit to .dbc over a large .package, as the files seem to be one in the same. However, If you are using .dbc, you need to make sure your resource.cfg is configured to read .dbc files.

8. I run every new merged package through delphy's Sims 3 Dashboard tool to make sure the new package is readable and fix any possible mod issues at the same time.

Additional notes

• Before merging new untested mods into your package files, I would suggest first trying every new mod in-game via a probation folder within your mods folder. It's worthwhile testing every mod you install using process monitor and available modding tools.
• If you are wanting to extract  .sims3pack into .package (official or otherwise), then you will need this Delphy's sims3pack multi-extractor
• You can also safely delete ICON file types as well as THUM file types from the merged package in s3pe. ICON are only used by the official launcher and THUM will be re-generated by the game. Deleting of THUM files is handy in cases where you don't like the custom Thumbnails provided by the mod author.
  
  
• If you are extracting official store items from .sims3pack, run the extracted .package through s3rc.exe before attempting to merge with s3pe, or you will get errors.
  
  

aikea_guinea was kind enough to make up this visual guide with some handy screen-shots for those of you that find the text version difficult to follow.

Thanks to Inge and Peter Jones for the much appreciated updates to s3pe which now makes merging mods into a single file much easier..

[Baarogue]

Another question, is whether or not the files need to be named .dbc, or can simply remain as a large .package. Does the game handle .dbc archives differently than .package in therms of memory management/caching? Also, I'm interested in hearing from someone willing to try extracting store sims3packs and merging them via s3pe into one archive.

In the spirit of Learning By Doing, I'm willing to try this out. Unless you have a quick and handy tutorial somewhere, I now go to read that thread to figure out what I need to do. ^_^ Any other criteria for this experiment?

[J. M. Pescado]

There is no "benefit" to a dbc over a package, as they are identical in format and the game is unable to distinguish them. You could make them into .crap files and they would still work.

[Anach]

There is no "benefit" to a dbc over a package, as they are identical in format and the game is unable to distinguish them. You could make them into .crap files and they would still work.

Good to know. That means it's purely down to the sheer number of files which is causing the issue. My only worry was that possibly the game had some extra specific way of handling those files with a .dbc extension, as that is not the case, .crap files it will be from now on.

So with the above quickie guide, feel free to use .package in place of .dbc.

[Ozrae]

Just wanted to mention that I followed Anach's tutorial for creating .dbc archives and have also had extremely positive results.

My game, before doing what was suggested was slow and sluggish with horrendous load times. CAS/Build/Buy modes were almost unplayable. My game now loads quickly with no in-game lag and no negative side effects. All my cc is showing and working as it should.

I am a complete novice when it comes to using s3pe but by following the guide posted by Anach I found the process to be relatively painless. I now have a total of 9 categorized .dbc files which I merged 1,4009 .package files into . The only .package files I left untouched were hacks and careers and any default replacements as I noticed they didn't work when I merged them.

Thankyou to Anach for sharing his experiment as I now have my game back and am enjoying playing for the first time since I patched...actually I think it's even running better than it was pre-patch! :-)

[Inge]

The potential benefit of the .dbc is that is a way the .package content can be used from My Documents hierarchy instead of having to use the Framework in the game install folders.

It could also mean that CC gets included with shared lots etc.

Does anyone know if you can have more than one .dbc in the folder?

[Anach]

The potential benefit of the .dbc is that is a way the .package content can be used from My Documents hierarchy instead of having to use the Framework in the game install folders.

It could also mean that CC gets included with shared lots etc.

Does anyone know if you can have more than one .dbc in the folder?

Good suggestion in regards to the My Documents folder. As long as the .dbc are named differently, they should all work. I believe the game sets a maximum .dbc size (set in one of the .ini files) when creating them via launcher, and they are named in series dcdb0.dbc, dcdb1.dbc, etc. It would be good to know if items are included with exported lots or not, or whether the launcher is required to add more specific information to the archive for that feature to work.  

Just wanted to mention that I followed Anach's tutorial for creating .dbc archives and have also had extremely positive results.

My game, before doing what was suggested was slow and sluggish with horrendous load times. CAS/Build/Buy modes were almost unplayable. My game now loads quickly with no in-game lag and no negative side effects. All my cc is showing and working as it should.

I am a complete novice when it comes to using s3pe but by following the guide posted by Anach I found the process to be relatively painless. I now have a total of 9 categorized .dbc files which I merged 1,4009 .package files into . The only .package files I left untouched were hacks and careers and any default replacements as I noticed they didn't work when I merged them.

Thankyou to Anach for sharing his experiment as I now have my game back and am enjoying playing for the first time since I patched...actually I think it's even running better than it was pre-patch! :-)

I've merged a few default replacement textures without issue, such as roads, skins and other minor replacements. What is it you were trying to merge but couldnt? Dont forget to clear your caches. I also didnt bother merging hacks/tweaks/careers etc, as I'm always wanting to adjust them or update them.

Glad it worked out. Hopefully we can expand and improve upon this system and hopefully later some tools to make this type of mass merging easier in terms of adding and removing specific packages from a larger archive or better conflict detection.

[shiniki]

I too tried this, though I don't have nearly as much CC as Ozrae, perhaps less than 100 files altogether, mostly hair and some clothes, which tend to be larger files. I was unsure of what Anach meant by Step #6, so I cautiously  skipped it.

Have about 8 .dbc files after organizing the CC. I left the hacks alonel: I'm running AM, NRaas Supercomputer, 2xbill, NoMosaicWA (updated at the same time as trying out the .dbc thing), and MoreFreqOpp.

Cleared cache and prefetch just to be safe, ensmallened the savefile, and 3booted up. Didn't notice a quicker overall neighborhood load but usually I don't sit around and time it. House and furniture loading seemed faster but this was the beginning of a play session rather than the tail end of four or five hours. House switching seemed faster, even though only two pieces of CC were Buy Mode things (both are cars).

Had a CTD around the time YA Sim #1 was ready to come home from work at the Science Lab (he got singed) while YA Sim#2 was in the shower, maybe 2:30pm. Three kids at school. 3booted up again, played through the same day (YA #1 did not get singed this time), got to midnight, saved and quit. Not sure what caused the CTD but just now I checked in my Mods/Packages and realized I left the skins .package files in the folder with the .dbc. Wondering if that may have been a contributing factor, or if skipping Step 6 is to blame.

In any case, I am still willing to keep trying this, as anything that cuts down the house-switching load times can't be bad. That was the primary reason for me keeping to The One Fambly, and neglecting my other chosen.

[Anach]

I too tried this, though I don't have nearly as much CC as Ozrae, perhaps less than 100 files altogether, mostly hair and some clothes, which tend to be larger files. I was unsure of what Anach meant by Step #6, so I cautiously  skipped it.

Have about 8 .dbc files after organizing the CC. I left the hacks alonel: I'm running AM, NRaas Supercomputer, 2xbill, NoMosaicWA (updated at the same time as trying out the .dbc thing), and MoreFreqOpp.

Cleared cache and prefetch just to be safe, ensmallened the savefile, and 3booted up. Didn't notice a quicker overall neighborhood load but usually I don't sit around and time it. House and furniture loading seemed faster but this was the beginning of a play session rather than the tail end of four or five hours. House switching seemed faster, even though only two pieces of CC were Buy Mode things (both are cars).

Had a CTD around the time YA Sim #1 was ready to come home from work at the Science Lab (he got singed) while YA Sim#2 was in the shower, maybe 2:30pm. Three kids at school. 3booted up again, played through the same day (YA #1 did not get singed this time), got to midnight, saved and quit. Not sure what caused the CTD but just now I checked in my Mods/Packages and realized I left the skins .package files in the folder with the .dbc. Wondering if that may have been a contributing factor, or if skipping Step 6 is to blame.

In any case, I am still willing to keep trying this, as anything that cuts down the house-switching load times can't be bad. That was the primary reason for me keeping to The One Fambly, and neglecting my other chosen.

Step 6 is pretty much just about sorting conflicts (if there are any). If you import with the option to "Replace duplicates", it will mark any conflicts "deleted" in the s3pe window with a line through it. This is something that will only happen if there are conflicts, which in the perfect scenario there shouldnt be. Most the conflicts can be left as marked for deletion, but there were a few which werent really conflicts, such as the language strings and the 0x0000000000000000 files, which not all mods will have. So you were probably ok with skipping 6, if you didnt have any conflicts.

As for CTDs, every person's installed mods are going to be different so merging and conflicts are always going to be different, and that is really up to each person, but I do encourage every person to check their mods using process monitor before merging, as well as opening it in the appropriate editor to check for errors, as I found quite a lot of mods, even those from popular sites contain a multitude of errors, and this has been a major factor of errors while saving, CTDs, black textures and slow-downs. I'm now quite careful which mods I install.

[Ozrae]

I've merged a few default replacement textures without issue, such as roads, skins and other minor replacements. What is it you were trying to merge but couldnt? Dont forget to clear your caches. I also didnt bother merging hacks/tweaks/careers etc, as I'm always wanting to adjust them or update them.

Actually...taking another look, remerging my default replacements, clearing my caches and reloading my game they are now loading in .dbc files. I think I may have had some conflicts as I removed some files when I remerged.

[wendylady]

Will doing this make it show up with the CC icon also?

And can you add to the .dbc file after you make one?

[Anach]

Will doing this make it show up with the CC icon also?

And can you add to the .dbc file after you make one?

No they wont show with the CC icon, I believe they need to have a flag changed in the mod for that. There is a tool at MTS which may allow you to set that flag before importing mods.

Yes you can import as many new mods to an existing .dbc/.package as you wish.

[wendylady]

The program i found only changes it for EP1 files to give them a Icon , non of the EP0 will get one.

thanks for the fasts reply

[Inge]

I could ask Peter to have a look at making a tool for this...

[Zazazu]

To flag CC as CC? Sign me up...one of the major reasons I haven't been sharing stuff from TS3 is because I can't tell what of the things I have is CC, with the exception of HugeLunatic's windows since she edited her icons to have her insignia.

[Anach]

To flag CC as CC? Sign me up...one of the major reasons I haven't been sharing stuff from TS3 is because I can't tell what of the things I have is CC, with the exception of HugeLunatic's windows since she edited her icons to have her insignia.

I havent tried it, but looking at the packages, it seems Expansion packs are flagged 0x08 (EP1), Store content is flagged 0x01 (1), and normal content is flagged 0x00 (0). Maybe you can make a request of the author of the EP1 tool to do both?

As for tools, what we really need for the sake of game performance is a launcher capable of merging/removing .package files correctly from a single archive, and saving that either to my docs in .dbc or mods directory in .package, without the novice user having to worry about sorting conflicts.

[Inge]

[Flags]
public enum ContentCategory : uint
{
    kAllFlagsMask = 0xff000000,
    kCustomContent = 0x2000000,
    kEP1Content = 0x8000000,
    kEP2Content = 0x18000000,
    kLocalContent = 0x4000000,
    kNone = 0,
    kPaidContent = 0x1000000,
    kSP1Content = 0x10000000
}

[wendylady]

Well I tried to make the .dpc files and could not get them to show up, I did all the clean-up stuff and such. I tried to change my config file but then nothing showed up.

So I know I'Am missing something just not sure what

I got a few to work as .package files but not all worked that I made. Which makes me think a few files must be messed up and  that's why the hole thing does not show up.

The link to the process monitor just send me to a big thread, been reading thou it thou and found the links need, so going to spend a fun day with it seeing how well it can help.
Side note to others looking for it, the post with the link is # 240 and 283


Anach - Ill go ask him to look in to it when he's back from winter break.

[Inge]

Anyway, Peter has made good progress with this today.  He's adding it as an import option to s3pe, to batch import packages.  With any luck there will be something to test later today or tomorrow.

Edit:  This is all very well but wtf is a .dbc and where does it go?  I only have .ebc's

Edited again:  I found one!

[Anach]

Anyway, Peter has made good progress with this today.  He's adding it as an import option to s3pe, to batch import packages.  With any luck there will be something to test later today or tomorrow.

Edit:  This is all very well but wtf is a .dbc and where does it go?  I only have .ebc's

Edited again:  I found one!

.dbc is the file made by the launcher for content installed from your downloads folder.
.ebc is content installed via the in-game store.

You dont need to use .dbc, you can simply use .package and put it in your mods directory. I've not tested using a .dbc in my docs, but I am interested to see if the content shows in game when doing it that way.

[J. M. Pescado]

Edit:  This is all very well but wtf is a .dbc and where does it go?  I only have .ebc's

A dbc is a package. They appear to be functionally indistinguishable in every way.

[Inge]

http://sf.net/projects/sims3tools/files/s3pe/1001-03-1524/  start New file then use Resource/Import/as dbc.  Choose multiple packages.


Isn't the idea that you put the file into the DCCCache folder?

[Klinn]

I found quite a lot of mods, even those from popular sites contain a multitude of errors, and this has been a major factor of errors while saving, CTDs, black textures and slow-downs. I'm now quite careful which mods I install.

Anach, if you have the time and inclination, could you describe the types of errors you've found? Not in any great detail or necessarily naming the mods, just in point form, perhaps in a new thread.

This could serve as a handy "things to watch out for" checklist for new Sims modders such as myself. It seems every game develops some modding folklore concerning potential pitfalls to avoid.

Thanks!

[Anach]

I found quite a lot of mods, even those from popular sites contain a multitude of errors, and this has been a major factor of errors while saving, CTDs, black textures and slow-downs. I'm now quite careful which mods I install.

Anach, if you have the time and inclination, could you describe the types of errors you've found? Not in any great detail or necessarily naming the mods, just in point form, perhaps in a new thread.

This could serve as a handy "things to watch out for" checklist for new Sims modders such as myself. It seems every game develops some modding folklore concerning potential pitfalls to avoid.

Thanks!

I don't exactly have the patience for writing lengthy detailed and pretty looking tutorials, as you can probably tell from the above posts. The best advice I can give in that scenario, without having to document everything I install,  is this;

1. Install everything as a package first (even unpack custom sims3pack content), and put it in a probation folder within your mods/packages, so you can easily keep track of the latest content you install.
2. Run process monitor (check above posts for mts links of tutorials) and check the "count" of the file access. Often you will find a file that is well above everything else, and usually this is a troublesome file.
3. Open every package you download in s3pe or other relevent editor to see if it pops up for errors (s3pe, or delphy's CTU, etc). Many mods out there were made before tools were working correctly, while others simply havent been made correctly. If they error in the tools used to create them, then I tend not to trust them.

The better one is at knowing how mods are made (there are lots of tutorials available), the easier it is to work out what is dodgy. I certainly know a lot more than I did when I started playing Sims 3, but there is still a lot i'm unsure about when it comes to troubleshooting and merging mods (such as those 0x000000000000000 xml files), but there are a lot less mods installed now and I have far fewer issues. If I keep a clean game, Its often quite easy to tell when I've installed something dodgy, as suddenly I'll start seeing issues within game;

1. Sudden framerate drop.
2. Black (missing) textures on objects, sims or UI.
3. Crashes, freezing and very slow loading.

I do the same sort of troubleshooting for mods I use in Oblivion and Fallout3. Often you simply find stuff that is broken when opened in editors, or simply the modder doesn't know how to use the tools correctly, or is too lazy to bother. Similarly in those two games, mod merging is also very important for performance and reliability.

The good thing about this recent patch, with all it's issues, is it's forced me to have a good clean-out, and I must say, my game hasn't run this smooth and crash free since I first started playing. My Wife has also taken to reading tutorials to learn how to make mods, mainly so she can fix the broken stuff that she really likes.

http://sf.net/projects/sims3tools/files/s3pe/1001-03-1524/  start New file then use Resource/Import/as dbc.  Choose multiple packages.


Isn't the idea that you put the file into the DCCCache folder?

That is certainly an option, which I didnt actually try. Be interesting to see how it works. The limiting factor is whether the DCCache folder supports sub-folders for easier management. It's unlikely there will be any further performance increase by having a large .dbc in your DCCache folder rather than a large .package in your mods folder, as from what I can tell its purely about minimizing the sheer amount of files, and not about location or file type.

It also seems that when importing as .dbc within the new s3pe, it now merges the _key files correctly. As for the 0x00000000000000 xml files, it's possible those arent even needed? so I'm going to see how it goes with them removed.

*edit*

Seems custom .dbc within the my documents aren't supported, or at least I couldn't get them to show in-game. So there doesnt seem to be any benefit to the .dbc format at all. Saving them as merged .package files should be enough.

[Klinn]

Thanks for the quick response and info Anach.

Sorry, but I wasn't very clear in my post -- I wasn't looking for a description of how to tell when a mod is causing problems and isolating it, but the errors in the mods themselves that you happen to have come across. For example, if you've found a number of them with missing textures, or links to other components that don't exist, or whatever. Those examples are just guesses on my part, since I'm not too familar with Sims mods yet.
 
On the other hand, maybe it's too early to try to build up that Sims3 modding folklore, or things creators should keep in mind while building mods, since the tools may still be in a state of flux. As you mentioned:

Many mods out there were made before tools were working correctly, while others simply havent been made correctly.

I don't have any background in previous Sims games, but have dabbled in various areas of Sims3 modding, some retexturing, tweaking some meshes, some XML tuning, and some core modding, and I didn't want to accidentally repeat some typical noob errors you might have seen while examing other mods. If you happen to think of any common mod creation problems, great. If not, no worries, I'll continue to lurk & learn. ;-)

Back on topic, thanks for clearly describing how to combine packages into a .dbc file, I'm sure I'll find useful as I accumulate more mods.