Pathological loading times

[Soltis]

I've noticed as my saves approach the 40 MB mark (this is the combined size of everything in a given save folder), loading time skyrockets. It used to take around 20-30 seconds to load completely, and that's with a cleared cache and everything.

Now it's taking around five minutes to load a single saved game, and if I quit to the main menu and re-load another saved game, it can easily take 50% longer *every single time I do this*. Waiting 20 minutes on an i7 920 with 12 GB of RAM and an nV GTX 470 to load one single game is just a little ridiculous.

I've ensmallened and decrapified everything I can ensmallen and decrapify; I've used AM fixall and such to clear out any brokenness I can locate, etc etc; I just want to know, is this just inherent to TS3's broken programming/architecture, or is there another way to help the game load these save files more quickly?

[Madame Mim]

Obviously it's due to the endU virus (also known as PEBKAC) - you used the word pathological not me.

[wizard_merlin]

How much harddrive space do you have available?  You might be getting low.

My current saved game is at 137MB and still loads in maybe 30 seconds or so, never really timed it, so may be a bit quicker, or slightly longer, but nothing out of the ordinary.

[kiki]

Also check how big your mods folder is and how much CC you have.

[PolecatEZ]

I've noticed as my saves approach the 40 MB mark (this is the combined size of everything in a given save folder), loading time skyrockets. It used to take around 20-30 seconds to load completely, and that's with a cleared cache and everything.

Now it's taking around five minutes to load a single saved game, and if I quit to the main menu and re-load another saved game, it can easily take 50% longer *every single time I do this*. Waiting 20 minutes on an i7 920 with 12 GB of RAM and an nV GTX 470 to load one single game is just a little ridiculous.

I've ensmallened and decrapified everything I can ensmallen and decrapify; I've used AM fixall and such to clear out any brokenness I can locate, etc etc; I just want to know, is this just inherent to TS3's broken programming/architecture, or is there another way to help the game load these save files more quickly?

You got a sexy-like system, but load times are primarily dependant on HD speed.  Theres a program called HDTach you can get to test this.  You might want to think about setting up a soft RAID 0 if you have multiple HDs, or investing in 1 or 2 (for RAID) SSD's.  There's a few other solutions (in the $100 to $600 range), but I think if you check on that stuff first, you'll see the issue.  HDD read speed on a mid-range gaming rig like yours should hit in the 250 - 280 range, anything less and you're really bottlenecking your system.

Image below is from mine when I was testing out my 2 X WD1TB 6gb drives, but putting them on the Intel controller instead of the Marvel.  Tomorrow I put my second SSD in there to see if I can top it.  My saves are about 90mb and the load times stay under a minute on a 930 (OC'd to 3.6), 12gb, 2xGTX470 SLI (SLI does nothing for this game, btw), and RAID 0.

[lordrichter]

Also use Process Monitor to find out what CC you need to delete.

[BaronElectricPhase]

Defragment your Hard Drive

[PolecatEZ]

Defragment your Hard Drive

Back in the day, I used to bench my tech support guys that did this or advised this as a solution to anyone.  Defrag hasn't been a real solution for anything since Win95/98.  Basically, its irrelevant on an NTFS system, especially one that has good built-in background defrag.  Its still a good litmus test to tell the men from the boys though when it comes to sysadmins that have a clue what they're doing...if they use the word "defrag" seriously, then they're fit to be sacked, or, in my case, keep requisitioning techs from PERSCOM until you have a shop with 10 retards and 2 actual useful people.

Also use Process Monitor to find out what CC you need to delete.

This, though if he was troubleshooting well, he already has all his CC out.

[jezzer]

Also use Process Monitor to find out what CC you need to delete.

This, though if he was troubleshooting well, he already has all his CC out.

You've been around long enough to know that this is a pretty big "if".

[BaronElectricPhase]

Defragment your Hard Drive

...especially one that has good built-in background defrag.

Also use Process Monitor to find out what CC you need to delete.

This, though if he was troubleshooting well, he already has all his CC out.

Two interesting presumptions one after another... efficient, maybe... thorough, no.
Go ahead and sack *yourself*

[Zazazu]

Defragment your Hard Drive

Back in the day, I used to bench my tech support guys that did this or advised this as a solution to anyone.  Defrag hasn't been a real solution for anything since Win95/98.  Basically, its irrelevant on an NTFS system, especially one that has good built-in background defrag. 

You say this, but have you ever tested a rather low-end (but current) system that is 30% fragmented right after it has been defragged? Trust me, it makes a huge difference.

I do it monthly, but I'm very controlling with how clean and organized my system is.

One thing I noticed absolutely killed my loading times was CC, but more specifically the number of CC files. Merging cc into megafiles (and keeping the raw files somewhere on backup just in case) cut my load time, which had been around 5 or 6 minutes with just around 50 files, to less than 30 seconds. Now, I think I have 800 files merged into 12 and my load time is just around a minute.

[Soltis]

PolecatEZ, I doubt it's HD speed, because of two factors:

1. If IO was a big issue, the game ought to load slowly in general -- and my save files aren't big enough to (alone) be inducing enough IO load to make a difference.

2. The game loading slower every time I return to the main menu (not the launcher) suggests a memory leak, possibly caused by corrupt data  (or simple stupidity on the game's part)

3. My HDD is relatively new/fast; the 300 GB Raptor -- so not exactly in SSD territory, obviously, but no slouch. I've got the OS and apps on separate drives, and no swap file, so most extraneous issues of that nature are also taken care of.



As for CC, I just tried removing *all of it*, clearing every cache I could find, and loading the same saved games; it didn't make a difference.

The reason I didn't bother before is, wouldn't CC-related slowness be apparent from the start?

Keep in mind that the *game* loads just as fast as it ever has; it's only loading a saved game which takes longer and longer.

[Soltis]

Obviously it's due to the endU virus (also known as PEBKAC) - you used the word pathological not me.

If you want to bandy pretty semantics, the term also has a comp sci meaning referring to algorithmic behaviour which grows geometrically or exponentially; I could easily induce hour-long loading times with this little problem, if I just kept returning to the main menu and re-loading.

It bears mention that making foolish assumptions about situations one doesn't really understand is the actual basis of most ID 10 T and PEBKAC errors.

[PolecatEZ]

PolecatEZ, I doubt it's HD speed, because of two factors:

1. If IO was a big issue, the game ought to load slowly in general -- and my save files aren't big enough to (alone) be inducing enough IO load to make a difference.

2. The game loading slower every time I return to the main menu (not the launcher) suggests a memory leak, possibly caused by corrupt data  (or simple stupidity on the game's part)

3. My HDD is relatively new/fast; the 300 GB Raptor -- so not exactly in SSD territory, obviously, but no slouch. I've got the OS and apps on separate drives, and no swap file, so most extraneous issues of that nature are also taken care of.



As for CC, I just tried removing *all of it*, clearing every cache I could find, and loading the same saved games; it didn't make a difference.

The reason I didn't bother before is, wouldn't CC-related slowness be apparent from the start?

Keep in mind that the *game* loads just as fast as it ever has; it's only loading a saved game which takes longer and longer.

When you say no swap file, does that mean none on your gaming drive, or none whatsoever?  Just asking, because Sims 3 is horrible at memory management, happily ignoring 10 extra gigs of ram in favor of HD thrashing.  There's also some utilities that will actively periodically free up RAM that is no longer in use, such as the monitor with PowerPack 2.3.2+, which can help with some memory leaks depending on the timer setting. 

Another utility that I found helped quite well is RAMdisk, create a 4GB vHDD at 32bit and then have Windows 7 use the whole small thing as virtual memory.  Sims 3 will happily chew on that, and it also helps use that extra RAM you paid for

[Soltis]

No swap file anywhere; I never get above 70% RAM utilization, particularly since most apps I would actually push hard *coughs3pecough* are (at least currently) compiled as 32-bit and thus can't really consume all that much to begin with.

[GnatGoSplat]

I don't think slow game load times has anything to do with HDD access speed.  All slow loading save games I've had actually exhibit very little HDD access.  The LED just kind of flickers occasionally and remains dark for a great deal of the loading delay.  Pes once mentioned a save game can be bloated with an excess amount of compressed crap which keeps your PC busy uncompressing aforementioned crap.  You might try Twallan's Porter mod.  Export your entire hood using porter, then create a fresh hood, destroyallhumans with AM, place your Portered save, and use Porter to extract.  That should leave you with a new hood with your old sims that should load as quickly as a new, fresh hood.

[Madame Mim]

Obviously it's due to the endU virus (also known as PEBKAC) - you used the word pathological not me.

If you want to bandy pretty semantics, the term also has a comp sci meaning referring to algorithmic behaviour which grows geometrically or exponentially; I could easily induce hour-long loading times with this little problem, if I just kept returning to the main menu and re-loading.

It bears mention that making foolish assumptions about situations one doesn't really understand is the actual basis of most ID 10 T and PEBKAC errors.

paid - I crawl back in my box.

[Soltis]

I don't think slow game load times has anything to do with HDD access speed.  All slow loading save games I've had actually exhibit very little HDD access.  The LED just kind of flickers occasionally and remains dark for a great deal of the loading delay.  Pes once mentioned a save game can be bloated with an excess amount of compressed crap which keeps your PC busy uncompressing aforementioned crap.  You might try Twallan's Porter mod.  Export your entire hood using porter, then create a fresh hood, destroyallhumans with AM, place your Portered save, and use Porter to extract.  That should leave you with a new hood with your old sims that should load as quickly as a new, fresh hood.

Yeah, the same bears out in my experience -- but a lot of people still insist on doing unnecessary diligence to prove beyond a shadow of a doubt it's not the actual cause (especially strange since HDD speeds haven't meaningfully changed in ages, SSD not withstanding...).

At any rate, another interesting observation: the game loaded just fine (well, fine-ish) when my sims were in SimLa (or whatever) for a few days; the loading times spiked the moment I came back home.

I have the funny suspicion it's the house I built, or something; I tend to build big, and the house I have is 7 storeys tall (including basements) and 60-ish squares long, with a massive garden and considerable decoration. Still not enough objects that it *should* slow down the game, but I think anyone familiar with EA programming knows that "should" and "is" are frequently very different.

I will try the house exporting trick, though, just to see if there are any extraneous factors I can weed out; thanks for the suggestion.

[GnatGoSplat]

At any rate, another interesting observation: the game loaded just fine (well, fine-ish) when my sims were in SimLa (or whatever) for a few days; the loading times spiked the moment I came back home.

Yeah, that's because each WA town has its own .nhd (neighborhood) file.  The game won't load your bloated main hood until your sim comes home.

I have the funny suspicion it's the house I built, or something; I tend to build big, and the house I have is 7 storeys tall (including basements) and 60-ish squares long, with a massive garden and considerable decoration. Still not enough objects that it *should* slow down the game, but I think anyone familiar with EA programming knows that "should" and "is" are frequently very different.

I will try the house exporting trick, though, just to see if there are any extraneous factors I can weed out; thanks for the suggestion.

It's possible.  Come to think of it, my awful 460MB save folder that literally took 45-minutes to load was one where I had built several huge houses (main hood .nhd itself was 260MB).  However, I'd saved the houses to the library and put them back into a fresh hood, used Porter to restore all the old sims into the fresh hood, and the new save folder ended up only 30MB and taking under a minute to load.  Same hood, same houses, same sims, more buildings (added the Ambitions lots), but a lot smaller file size.  It seemed like the houses themselves don't bloat the save, but perhaps the save gets more bloated with every modification to the lot.  I haven't tried changing a lot and checking to see if the save file grows or not.  I do know that the more famblies you've made active does result in a bloatier save file.  About 5MB per fambly made active in my case.

[Soltis]

Very useful information, all in all; when I finally get pissed enough at the loading time to bother with the export/import I'll let you know how it works out.

[Soltis]

So I had some odd happenings when I tried using the Porter (the duplicate removal function appears to be completely non-functional), but I did migrate my extended family to a new game (I didn't really care about the rest of the town anyway; the family I actually liked had died already), and loading times are infinitely better.

I'm still very confused about exactly *what* was taking so long, especially since the new save file is almost as large as the old one, but whatever was going on has been fixed.

I'd still like to know what sort of accumulated crap is causing the problem, if anyone knows... I have the sneaking suspicion it has to do with the player behaviour tracking built into the game, but of course that's purely unfounded speculation.

If someone who actually knows anything about the structure of save files (what exactly they contain, what kind of indexes or such can get oversized easily, etc) I'd be interested to hear their thoughts.

[jezzer]

Hey, moron:  stop double-posting.  There's a "modify" button for a reason.

[Soltis]

I'm sure there is.

How about making sure it actually works before cunt-yapping about how it ought to be used; at least in Firefox 4b7, it does not.

[Soltis]

More accurately, the "Modify" button functions -- it brings up the post edit screen; hitting "save," however, does nothing.

[Madame Mim]

Try another browser?