Merging mods to increase Performance!

[Anach]

Thanks for the quick response and info Anach.

Sorry, but I wasn't very clear in my post -- I wasn't looking for a description of how to tell when a mod is causing problems and isolating it, but the errors in the mods themselves that you happen to have come across. For example, if you've found a number of them with missing textures, or links to other components that don't exist, or whatever. Those examples are just guesses on my part, since I'm not too familar with Sims mods yet.
 
On the other hand, maybe it's too early to try to build up that Sims3 modding folklore, or things creators should keep in mind while building mods, since the tools may still be in a state of flux. As you mentioned:

Many mods out there were made before tools were working correctly, while others simply havent been made correctly.

I don't have any background in previous Sims games, but have dabbled in various areas of Sims3 modding, some retexturing, tweaking some meshes, some XML tuning, and some core modding, and I didn't want to accidentally repeat some typical noob errors you might have seen while examing other mods. If you happen to think of any common mod creation problems, great. If not, no worries, I'll continue to lurk & learn. ;-)

Back on topic, thanks for clearly describing how to combine packages into a .dbc file, I'm sure I'll find useful as I accumulate more mods.

At the moment I certainly don't see myself as experienced enough in s3 modding to give any more advice than I already have, in that people take a look themselves. Even those mods which are often broken or missing stuff, i'd find it very difficult to make myself. Like you, a lot of it is simply guesses and my own testing for obvious stuff , So other than my own personal quality limits, I wouldn't feel comfortable in stating any thing else. I'd kinda feel akin to some noob coming in and telling Pescado how to make core mods.

I prefer to simply advise people to be very fussy when it comes to installing mods, as not every mod creator has the same level of expertise or quality control and quite a lot of mods will break your game in one way or another.

However, I will say that Hair mods seem to be one of the most frequently broken mods I come across, and one to watch out for. I now have very few CC hair in my game simply because it was causing game issues or simply had ugly visual issues. I've set my own limit of acceptable quality on that, and even while I might use one or two hairs from a one particular modder, it doesnt mean every hair from them will be fine, and more often than not, its the hair I actually want to use, unfortunately.

The trouble with threads that list troublesome mods, is that everyone has a different level of acceptable quality, often modders will update and fix mods, and quite often they end up so full of nonsense that they end up being ignored anyway. As much of a pain as it is, there really is no way to tell without testing every single mod prior to adding it permanently to your game.

Updated the main post to simplify the process with the latest information.

[Inge]

What about opening the original .dbc in s3pe, and adding the packages into that?  (make a safe backup of the original first, obviously   )

[Anach]

What about opening the original .dbc in s3pe, and adding the packages into that?  (make a safe backup of the original first, obviously  )

A scary thought, as i'm sure it will break the launcher horribly, but worth an experiment. I like to keep mine as clean as possible, that's why i'm excited about merging possibilities, as I can keep my CC separate to my SC, and not have to scroll through that horrible launcher to find stuff, since I generally never uninstall store stuff.

*edit* Doesn't work either. Tried importing 200Mb of various object mods to my existing official .dbc. Nothing showed in-game, similar to when I tried a freshly created .dbc in my docs.

[Inge]

Grrr!     Makes you wonder why it's there!  Maybe those xmls were more important than we thought.  And there was some "unknown" What happens if you remove them even for the bona fide stuff already in there?

[Moryrie]

I combined mine, and didn't seem to have any issues... when I was doing so however, I made sure none of the 'unknowns' or anything with a bunch of 0s (anywhere that I could see since I wasn't sure which column you were referring to) were deleted. It worked well. My game lags a lot less than it had even though I'd been running it with less CC than it has now. (I was only keeping what my sims were actively wearing in game.. anything else went to a folder on my desktop until I needed/wanted it).

I sorted mine into 4 files. 'Objects', 'CAS', 'Hair', and 'Patterns'.

Thanks for the guide though.

[Anach]

Grrr!     Makes you wonder why it's there!  Maybe those xmls were more important than we thought.  And there was some "unknown" What happens if you remove them even for the bona fide stuff already in there?

The 0x000-lots-of-zeros... xml files were only included with hair as far as I found. I also didn't find them in any official .dbc files, so I assumed they were probably added by DABOOBs or TSR Workshop and weren't actualy needed. The unknown files could very well be something to do with it. However, the merged content works perfectly fine in the mods directory, as either .dbc or .package, with no crashes or other issues without those zero xmls.

[Inge]

...but it's not proven that there is any advantage in making dbcs for the framework folders?

[Anach]

...but it's not proven that there is any advantage in making dbcs for the framework folders?

None as far as my own testing has shown. I initially did .dbc, then renamed (later remerged) as .package, and saw no difference performance wise. As JM stated, they appear to be identical files, except for whatever the official launcher does to allow the game to recognise the .dbc when in the docs folder.

However, it certainly seems that the fewer .package files in your mods folder, the better your game runs. I originally went from over 700 total, to 20 merged and 45 tweaks which I left as single files for ease of frequent modification and updates. I later cut that down to 5 merged and 39 tweaks and noticed that my larger household was running a few frames quicker average. I'm hoping to cut that down further to  6 main merged .package and a couple essential separate files (such as awesome and other tweaks that arent merge friendly) to see if it improves further. I would imagine that after a certain point there would be very little noticeable difference in performance.

*edit* as expected, it doesnt seem ill be gaining anymore performance by decreasing the amount of .packages further. I tried reducing my 45 total to 12 total, and didn't see a difference. Most of those were xml, ini, dll , ui type mods, so I didn't really expect much of a change, as they are fairly small and light compared to objects, clothing, or hair type mods. So as a recommendation, i'd suggest everyone leave those types of mods outside of the merged mods, due to it not making any difference performance wise, that they usually updated a lot more frequently than other types of mods and not all types will play nicely in a single file.

[Anach]

I combined mine, and didn't seem to have any issues... when I was doing so however, I made sure none of the 'unknowns' or anything with a bunch of 0s (anywhere that I could see since I wasn't sure which column you were referring to) were deleted. It worked well. My game lags a lot less than it had even though I'd been running it with less CC than it has now. (I was only keeping what my sims were actively wearing in game.. anything else went to a folder on my desktop until I needed/wanted it).

I sorted mine into 4 files. 'Objects', 'CAS', 'Hair', and 'Patterns'.

Thanks for the guide though.

The only files I manually removed from any of my merged mods were _xml (in the tag column) with resource keys of 0x00000000000000 (instance column). These only seemed to show in a few mods, and were my only conflicts that required manually merge (as alerted by s3pe). I haven't seen any negative effect of removing these, and I didn't find them in the official .dbcs, so I assumed they were created by the modding tools as some form of identification for those particular tools, but I really don't know at this stage.

[cefwyn]

Assuming these xml files you refer to are the ones tagged 0x73E93EEB, I have also been removing them as according to http://www.simswiki.info/wiki.php?title=Sims_3:PackedFileTypes they are just manifest files used by the launcher(at least as far as whoever posted that information can tell). I've been finding those resources in all manner of clothing packages, though I think they may all be ones that were created with TSR Workshop and later converted to a .package. In hair packages I've also been finding a lot of redundant resources which appear to reference bone structures for that hair which I have also been removing and havn't noticed any issues yet. Also, for those xml files it may be safer to, rather then deleting them completely, increment the instance value by one and keeping them as I did find some store packages which had multiple instances of those files with different data stored in them (Though again this may only be needed by the launcher).

I probably should have posted something on this ages ago since when the patch broke the launcher I extracted all my sims3packs with delphy's multi-installer and after finding the game suddenly unplayable with all those package files I had merged most of them together into a single storecontent.package file and had been using that for a while without problems. It's not really too surprising that Sims 3 isn't geared towards reading hundreds of small files efficiently since the best practice amongst game engines for some time now has been to create a virtual file system and read everything from a single file to reduce hard-drive access calls to the operating system (Which are very slow on windows).

Depending on your hardware Sims 3 should run without any noticable performance hit as long as you have no more then about 50-60 package files. If you've got a nice fast SCSI drive you may even be able to play without a performance hit without merging any packages and having a few hundred sitting on your drive, but it's always a good idea to reduce the amount of work the OS needs to do and have as few packages as possible. I can't really say what the limit is for Sims 3 as the performance hit may not even be related to the hardware limitations, but based on my own experience programming virtual file systems, trying to access more then 100 files (I definitely had more then 100 files after converting all the arr'd store content to .package) in a program is almost certainly going to produce a drop in performance, and while the hardware may be fast enough to access 45 or 50 files without a noticable hit you really want to try to get that number as close to 1 as possible as it also reduces fragmentation(depending on file size and hardware) and can make the entire game engine run smoother.

[graycurse]

you really want to try to get that number as close to 1 as possible as it also reduces fragmentation(depending on file size and hardware) and can make the entire game engine run smoother.

That COULD present a problem, though. I'll use programming practices as a metaphor. The more code you put in one class, the more likely you'll run into errors/the more difficult it will be to debug and maintain. If someone were to put EVERY piece of CC into one package, it's highly likely something will blow up. It'll take quite a bit of time to find the issue. I've tried both ways (with my CC packaged into hair/clothes/patterns/etc. packages or everything in one big package) and I haven't noticed a speed/performance difference with one package or ten.

[J. M. Pescado]

I don't think one package, or ten packages, is going to make a noticeable difference. The difference comes in those people who have OVER 9000 packages.

[cefwyn]

Yeah, on most hardware the difference between the game loading one package versus 10 packages is immeasurable, but you really want to keep the number down to as few as you can. Since you can't guarantee that packages made by other people are going to be 100% stable all the time (especially after EA releases a patch) it may not necessarily be a good idea to create a single package with everything in it as it would be difficult to debug, but less files will usually mean an increase in stability. I've still got about 60 package files and it runs just fine (I'm still not yet sure how best to merge some of them), but as I said in my last post you really want to keep it well below 100 as you'll definitely start noticing problems around that point on most hardware. Ideally 1 package should be more stable then even 10, but then if you have a lot of packages (>1GB worth) you start getting into problems with NTFS causing file fragmentation which will also reduce performance.

[dilpill]

When I was attempting to manually merge all of the packages extracted from the December Store files a few weeks ago, I gave up when I ran into about 50 instances of the 0x00000000000000 resource. They contained the description that is displayed when you mouse over the items in-game, so I assumed they were necessary. 


It's good to know that they are completely superfluous, though.

[cefwyn]

If those files were tagged as type 0x73E93EEB(_XML.manifest) then they appear to be safe to delete since they are probably only used by the launcher. To be on the safe side all I did (For most of them, I did delete some and havn't noticed anything missing) was undelete them, go into "Details..." for that resource and edit it's instance number to be something like 0x0000000001, or 2, etc.(Just as long as the number is unique in that package). I've been slowly experimenting with the internals of the package file so I may eventually get around to writing a package merging app which checks for hash collisions and then tries to merge them or at least provide a difference file so the user can interactively merge them like with windiff or similar apps.

[Tamsyn]

When I was attempting to manually merge all of the packages extracted from the December Store files a few weeks ago, I gave up when I ran into about 50 instances of the 0x00000000000000 resource. They contained the description that is displayed when you mouse over the items in-game, so I assumed they were necessary. 


It's good to know that they are completely superfluous, though.

Are you sure you were importing them correctly? I merged all of the store .package files by going to File > New in s3pe and then Resource > Import > From .package... and selecting them all, hitting Open and letting it run. I remodeled my nursery with some store objects and everything worked fine. Maybe you selected import from file? Something like that happened to me when I wasn't paying attention and did it that way.

[dilpill]

Oh, I did all of that, but I never got to the point of testing it in-game. When I saw all the "conflicts", I just assumed it was FUBAR.

[Anach]

You all have some very good points. I certainly suggest using a probation folder for testing individual packages before merging anything, and as I mentioned on the previous page of this thread, I like to test every single mod I install, simply to maintain a trouble free game. There is a lot of broken content out there, and it will do nasty things to the game if left in. My wife is currently going through her content 1 by 1, just as I did over a few days, as we both had some nasty game issues to track down. I removed a lot more than I would have expected to. Once each mod has been tested, It's certainly worth merging into fewer packages.

[Tamsyn]

Oh, I did all of that, but I never got to the point of testing it in-game. When I saw all the "conflicts", I just assumed it was FUBAR.

I just did the December store files, since I wasn't sure if I had included those, to make sure that it wasn't a December issue, and the screen closely resembles what I got when doing the whole thing. While I admit I haven't had time to thoroughly check each item, nothing exploded when I went in-game. I would be interested in seeing if the conflicts are indeed superfluous, or if there's a deeper issue.

[AloeOwl]

I've been reading through this post since the beginning and thought of giving this a try. I don't have THAT many CC files (around 200), but if it can boost performance why not?
I just tried this out but it didn't work out. One thing I'm confused about is that you said that the 0x00000000000000 xml files can be deleted, but more than three quarters of the files I imported are made of 0x00000000000000 xml files.
I tried this out with some build mode objects and my buy mode objects. And got two files (packages) and put them with my other mods in the mods/packages folder (not in sub-folders). Supposedly they should work but they aren't. I went back into the game and found nothing. Am I doing something wrong? Should I have deleted all those tons of 0x00000000000000 xml files?
Also when I click to import the packages I get a window listing all the files inside (The location of the packages) and it shows me things I can check like deleting duplicates or not, to compress them, and other things that are grayed out (I can't check those). Any suggestions on why this didn't work?

EDIT: When you say "0x00000000000000 xml files" are you referring to the group they belong to or the type? Examples of other groups: 0x00000000000001 or 0x00000000000002

[Anach]

I've been reading through this post since the beginning and thought of giving this a try. I don't have THAT many CC files (around 200), but if it can boost performance why not?
I just tried this out but it didn't work out. One thing I'm confused about is that you said that the 0x00000000000000 xml files can be deleted, but more than three quarters of the files I imported are made of 0x00000000000000 xml files.
I tried this out with some build mode objects and my buy mode objects. And got two files (packages) and put them with my other mods in the mods/packages folder (not in sub-folders). Supposedly they should work but they aren't. I went back into the game and found nothing. Am I doing something wrong? Should I have deleted all those tons of 0x00000000000000 xml files?
Also when I click to import the packages I get a window listing all the files inside (The location of the packages) and it shows me things I can check like deleting duplicates or not, to compress them, and other things that are grayed out (I can't check those). Any suggestions on why this didn't work?

EDIT: When you say "0x00000000000000 xml files" are you referring to the group they belong to or the type? Examples of other groups: 0x00000000000001 or 0x00000000000002

Get the latest s3pe, read the instructions on page 1, post 1. Keep in mind that this is all new, so even though I imported about 700 different types of mods, there are bound to be some troublesome mods or mod types that wont merge with other mod types. Also you can try import and choose "reject duplicates" as an alternative to import as .dbc. This process might require a bit of trial and error with certain files.

[Inge]

I need to point out the s3pe linked in this thread is *not* the latest public release you will find at MTS.  It is a special test release for the purposes of this experiment, so you need to get this one.

[Anach]

I need to point out the s3pe linked in this thread is *not* the latest public release you will find at MTS.  It is a special test release for the purposes of this experiment, so you need to get this one.

Thanks for the info. I updated the links on the first post to the one you provided.

I've also been doing some more testing by removing all custom content I had installed in sims3pack format via the launcher, extracting it and then merging it with the new import .dbc function. All worked very well. I didnt have any issues, except for the extraction using Delphy's tool. It seems that straight packages extracted from sims3pack dont like to work until they are merged into a fresh package, which I found very odd. The single packages on their own were not loading (20k+ in process moonitor) and the game was taking forever to load. As soon as I merged them to a fresh .package using the new import function they all worked without an issue, which makes me wonder if doing the same with other problematic .package files might fix the issue too.

Furthermore, I found that some packages will like to be merged in a specific order, which is entirely likely to be due to the way conflict resolution works. I was trying with a particular set of furniture (Cellar set) and found that I had to separate them into 3 files (paintings, furniture and clutter), then use the import feature and reject duplicates instead of the import .dbc function. So it will take a little toying with to get working right for each person.

As for the _xml manifest files, they seem to only be in certain files like clothing and hair, and I have deleted every single one without issue. Most likely it doesn't matter either way, and one could probably leave them all in a conflicted state and it wouldnt matter to the game. Though the _xml manifest files and icon files could also be excluded from the merge and save hassle of conflict resolution for those who might be inclined to panic

Compression is something I'm wondering about. I notice that while "Import from package" is all or nothing in terms of compression, the "import as .dbc" seems to be selective. Looking at the official .dbc, certain files seem to be almost always compressed, while others almost always not. Prior to the .dbc feature, I was compressing everything and didn't notice an issue?


PS. Wife forced me to correct spelling

[Inge]

I guess with import dbc he left the compression as he found it, whereas the user gets the choice (but *not* "leave as found" lol) with the normal import.

That's the trouble as things get bolted onto tools, they get out of step and inconsistent.   What would be the preferred action here?

[Anach]

I guess with import dbc he left the compression as he found it, whereas the user gets the choice (but *not* "leave as found" lol) with the normal import.

That's the trouble as things get bolted onto tools, they get out of step and inconsistent.   What would be the preferred action here?

I was actually thinking it would be nice to have a "leave is found" option when I was first playing around with the merging before the .dbc option, so that works well for me

As far as creating a trouble free merged file, with less worry for the end user, I would prefer to see only needed files merged. For single .package files it makes sense to leave in unofficial file types in case the user has need of them later when importing back into various tools, but if they are creating a merged .package for the purpose outlined in this thread, then there is really no need for files not used by the game, as the merged files can't be used by the tools that use those specific files. I'd imagine they could be filtered out of the merge or ignored when it comes to alerting about conflicts, so the user doesn't have to panic over 50 different  _xml manifest files with the same key.