Merging mods to increase Performance!

[Anach]

Instead of merging two large packages, try merging the individual smaller files and choosing the auto-save after each package option when asked.

[dramamine]

I decided to auto-save after importing the freshly downloaded hairs to delete the offending _KEY files, the attempt a remerging of just those files. When I tried, I got this error:

************** Exception Text **************
System.ArgumentNullException: Value cannot be null.
Parameter name: input
   at System.IO.BinaryReader..ctor(Stream input, Encoding encoding)
   at s3pi.Package.Package.packedChunk(ResourceIndexEntry ie)
   at s3pi.Package.Package.SaveAs(Stream s)
   at s3pi.Package.Package.SaveAs(String path)
   at s3pi.Package.Package.SavePackage()
   at S3PIDemoFE.MainForm.fileSave()
   at S3PIDemoFE.MainForm.fileSaveAs()
   at S3PIDemoFE.MainForm.menuBarWidget1_MBFile_Click(Object sender, MBClickEventArgs mn)
   at S3PIDemoFE.MenuBarWidget.OnMBFile_Click(Object sender, MB mn)
   at S3PIDemoFE.MenuBarWidget.tsMBF_Click(Object sender, EventArgs e)
   at System.Windows.Forms.ToolStripItem.RaiseEvent(Object key, EventArgs e)
   at System.Windows.Forms.ToolStripMenuItem.OnClick(EventArgs e)
   at System.Windows.Forms.ToolStripItem.HandleClick(EventArgs e)
   at System.Windows.Forms.ToolStripItem.HandleMouseUp(MouseEventArgs e)
   at System.Windows.Forms.ToolStripItem.FireEventInteractive(EventArgs e, ToolStripItemEventType met)
   at System.Windows.Forms.ToolStripItem.FireEvent(EventArgs e, ToolStripItemEventType met)
   at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEventArgs mea)
   at System.Windows.Forms.ToolStripDropDown.OnMouseUp(MouseEventArgs mea)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.ToolStrip.WndProc(Message& m)
   at System.Windows.Forms.ToolStripDropDown.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///C:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll
----------------------------------------
s3pe
    Assembly Version: 1012.2.14.25985
    Win32 Version: 1012.2.14.25985
    CodeBase: file:///C:/Program%20Files/s3pe/s3pe.exe
----------------------------------------
System.Windows.Forms
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Windows.Forms/2.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Drawing
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Drawing/2.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Configuration
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Configuration/2.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System.Xml/2.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
s3pi.CustomForms
    Assembly Version: 0.0.0.0
    Win32 Version: 0.0.0.0
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.CustomForms.DLL
----------------------------------------
s3pi.Interfaces
    Assembly Version: 1012.2.14.25971
    Win32 Version: 1012.2.14.25971
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.Interfaces.DLL
----------------------------------------
s3pi.WrapperDealer
    Assembly Version: 1012.2.14.25971
    Win32 Version: 1012.2.14.25971
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.WrapperDealer.DLL
----------------------------------------
s3pe.DDSPreviewWidget
    Assembly Version: 1012.2.14.25984
    Win32 Version: 1012.2.14.25984
    CodeBase: file:///C:/Program%20Files/s3pe/s3pe.DDSPreviewWidget.DLL
----------------------------------------
PresentationFramework
    Assembly Version: 3.0.0.0
    Win32 Version: 3.0.6920.1427 built by: SP
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/PresentationFramework/3.0.0.0__31bf3856ad364e35/PresentationFramework.dll
----------------------------------------
WindowsBase
    Assembly Version: 3.0.0.0
    Win32 Version: 3.0.6920.1427 built by: SP
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/WindowsBase/3.0.0.0__31bf3856ad364e35/WindowsBase.dll
----------------------------------------
PresentationCore
    Assembly Version: 3.0.0.0
    Win32 Version: 3.0.6920.1427 built by: SP
    CodeBase: file:///C:/WINDOWS/assembly/GAC_32/PresentationCore/3.0.0.0__31bf3856ad364e35/PresentationCore.dll
----------------------------------------
SlimDX.Wpf
    Assembly Version: 1.0.0.0
    Win32 Version: 1.0.0.0
    CodeBase: file:///C:/Program%20Files/s3pe/SlimDX.Wpf.DLL
----------------------------------------
s3pi.CASPartResource
    Assembly Version: 1012.2.14.25974
    Win32 Version: 1012.2.14.25974
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.CASPartResource.DLL
----------------------------------------
System.Custom
    Assembly Version: 1012.2.14.25970
    Win32 Version: 1012.2.14.25970
    CodeBase: file:///C:/Program%20Files/s3pe/System.Custom.DLL
----------------------------------------
s3pi.CatalogResource
    Assembly Version: 1012.2.14.25973
    Win32 Version: 1012.2.14.25973
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.CatalogResource.DLL
----------------------------------------
s3pi.ComplateResource
    Assembly Version: 1012.2.14.25973
    Win32 Version: 1012.2.14.25973
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.ComplateResource.DLL
----------------------------------------
s3pi.DefaultResource
    Assembly Version: 1012.2.14.25973
    Win32 Version: 1012.2.14.25973
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.DefaultResource.DLL
----------------------------------------
s3pi.Extensions
    Assembly Version: 1012.2.14.25971
    Win32 Version: 1012.2.14.25971
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.Extensions.DLL
----------------------------------------
s3pi.GenericRCOLResource
    Assembly Version: 1012.2.14.25972
    Win32 Version: 1012.2.14.25972
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.GenericRCOLResource.DLL
----------------------------------------
s3pi.Settings
    Assembly Version: 1012.2.14.25970
    Win32 Version: 1012.2.14.25970
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.Settings.DLL
----------------------------------------
s3pi.Helpers
    Assembly Version: 1012.2.14.25975
    Win32 Version: 1012.2.14.25975
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.Helpers.DLL
----------------------------------------
s3pi.ImageResource
    Assembly Version: 1012.2.14.25973
    Win32 Version: 1012.2.14.25973
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.ImageResource.DLL
----------------------------------------
s3pi.ModularResource
    Assembly Version: 1012.2.14.25974
    Win32 Version: 1012.2.14.25974
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.ModularResource.DLL
----------------------------------------
s3pi.NameMapResource
    Assembly Version: 1012.2.14.25974
    Win32 Version: 1012.2.14.25974
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.NameMapResource.DLL
----------------------------------------
s3pi.ObjKeyResource
    Assembly Version: 1012.2.14.25972
    Win32 Version: 1012.2.14.25972
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.ObjKeyResource.DLL
----------------------------------------
s3pi.Package
    Assembly Version: 1012.2.14.25974
    Win32 Version: 1012.2.14.25974
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.Package.DLL
----------------------------------------
s3pi.ScriptResource
    Assembly Version: 1012.2.14.25972
    Win32 Version: 1012.2.14.25972
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.ScriptResource.DLL
----------------------------------------
s3pi.StblResource
    Assembly Version: 1012.2.14.25973
    Win32 Version: 1012.2.14.25973
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.StblResource.DLL
----------------------------------------
s3pi.TextResource
    Assembly Version: 1012.2.14.25972
    Win32 Version: 1012.2.14.25972
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.TextResource.DLL
----------------------------------------
s3pi.TxtcResource
    Assembly Version: 1012.2.14.25974
    Win32 Version: 1012.2.14.25974
    CodeBase: file:///C:/Program%20Files/s3pe/s3pi.TxtcResource.DLL
----------------------------------------
SlimDX
    Assembly Version: 2.0.10.43
    Win32 Version:
    CodeBase: file:///C:/Program%20Files/s3pe/SlimDX.DLL
----------------------------------------
rsx-xqum
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/System/2.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
Accessibility
    Assembly Version: 2.0.0.0
    Win32 Version: 2.0.50727.3053 (netfxsp.050727-3000)
    CodeBase: file:///C:/WINDOWS/assembly/GAC_MSIL/Accessibility/2.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------

Againg, I'm not sure if this is useful, but I can consistently recreate this error.

[Anach]

Well seems those keys might be necessary, and just not being read correctly. When peter wakes up I'm sure I'll get a reply. We will know more then.

[Anach]

I decided to auto-save after importing the freshly downloaded hairs to delete the offending _KEY files, the attempt a remerging of just those files. When I tried, I got this error:

Peter had this to say.

Yes, it's reporting an error in the hair package.  Some of the hair packages have errors in their construction.

Which brings us back to why I would have originally removed all those hairs from my game. Bad packages, or bad hair which may have been made with old versions of creation tools. Whatever reason, It can't be good for the game.

[dramamine]

Thanks for the quick reply.

Well, that's a shame, but short of asking the offending creators to fix their hairs, I think I'll cope without.

Enjoy your holidays!

[Sigmund]

Using the autosave, rather than merging the smaller packages and re-merging the larger packages, fixed the 50 MB limit for me. I thought I'd tried that already, but apparently not since all is functioning normally again.

Dramamine, I've never been able to merge HP's packages. IIRC, she stated somewhere over at MTS that she's aware of the issue. Those hairs were made in the early days of TS3.

[Madame Mim]

I've never been able to merge HP hairs. The ones I chose to use don't seem to cause any other problems so I keep them in an unmerged folder.

[dramamine]

I've never been able to merge HP hairs. The ones I chose to use don't seem to cause any other problems so I keep them in an unmerged folder.

I was able to merge them back in Nov. with no problems, but I now have them in a similar setup. Thank you for the help, guys and gals.

Mim: Madam Mim was my FAVORITE character from the movie. I get a touch giddy when I see the name as her voice was so awesome and comical. Thank you for accidentally making my afternoon not suck.

[SpitSpitSpitty]

So I keep adding to my file little by little (this slow process of merging my mods) and I'm left wondering as I watch all of my HD space disappear on my C:/ drive... Where is all my space going? And how do I get it back?

I tried shutting down the program and reopening it after saving the file... and yet my space doesn't come back.
I tried restarting my computer... and yet my space doesn't come back.
I ran disc cleanup... and yet my space doesn't come back.
I uninstalled and reinstalled it on another drive with lots of space... but it CONTINUES to eat up my C:/ drive space and yet my space doesn't come back. 

I have been manually looking through my C:/ drive for this lost space and I can't find where it is! I'd estimate it's eaten up over 30 gigs so far and I have no idea how to get it back. Can anybody help me out here?

Thanks,

Spit

[spockblock]

I have been manually looking through my C:/ drive for this lost space and I can't find where it is! I'd estimate it's eaten up over 30 gigs so far and I have no idea how to get it back. Can anybody help me out here?

Perhaps you should try addressing this question to MR. COSMO SPACELEY, President of WINDOWS, instead of us, the undersigned, who collectively do not give a shit.

[SpitSpitSpitty]

Thanks spockblock! But if there's going to be any crap giving around here your going to have to take it up with my but, because it's the only thing around here that gives a crap. 

[spockblock]

You misspelled "butt."

[jezzer]

You misspelled "butt."

And also "you're".

[spockblock]

You misspelled "butt."

And also "you're".

See? This is why I got kicked out of Grammar Police Academy.

[jezzer]

See? This is why I got kicked out of Grammar Police Academy.

Really?  I thought it was because -- and I quote -- "Steve Guttenberg is a LITTLE BITCH".

[SpitSpitSpitty]

Surry i'm so darn unejamicated for ya toow speelin-B wizzamigers. Next time I be needen some heelp wit my game I'llz be sure'n not axe yall here on deez dare speel checkin boards disguised as help forums.

[Anach]

Could be a virus, could be a Windows issue, could be stupidity. Have never had the issue myself. I suggest doing a search of C: for *.package or using a program like tree-size and see where your space has gone.

[Simsample]

So I keep adding to my file little by little (this slow process of merging my mods) and I'm left wondering as I watch all of my HD space disappear on my C:/ drive... Where is all my space going? And how do I get it back?

Try deleting your temp files. Also untick the 'autosave' box when you do this- it seems to be due to temp files created during the autosave.

[ingeli]

The memory eating and how to solve it is discussed in this thread at S3PE-forum at Simlogical:
http://dino.drealm.info/den/denforum/index.php?topic=200.0

[HannahJoYo]

I'm sorry I have a headache from this already I'm getting this error but it still continues through what it's doing should I be worried?

Some resource names may not be displayed.

Error reading _KEY 0166038C:00000001:0000000000E5F018
Front-end Distribution: 10-1104-1243
Library Distribution: 10-1104-1243
Source: mscorlib
Assembly: mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
Exception has been thrown by the target of an invocation.
   at System.RuntimeMethodHandle._InvokeConstructor(Object[] args, SignatureStruct& signature, IntPtr declaringType)

   at System.RuntimeMethodHandle.InvokeConstructor(Object[] args, SignatureStruct signature, RuntimeTypeHandle declaringType)

   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

   at s3pi.WrapperDealer.WrapperDealer.WrapperForType(String type, Int32 APIversion, Stream s)

   at s3pi.WrapperDealer.WrapperDealer.GetResource(Int32 APIversion, IPackage pkg, IResourceIndexEntry rie, Boolean AlwaysDefault)

   at s3pi.WrapperDealer.WrapperDealer.GetResource(Int32 APIversion, IPackage pkg, IResourceIndexEntry rie)

   at S3PIDemoFE.BrowserWidget.CreateNameMap()
-----
Source: mscorlib
Assembly: mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)

   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)

   at NameMapResource.NameMapResource.Parse(Stream s)

   at NameMapResource.NameMapResource..ctor(Int32 APIversion, Stream s)

[HannahJoYo]

K I'm reading about the problem, which initially sucks, but thanks.

[tryguy]

OK, I was going to discuss this implementation issue over at http://dino.drealm.info/den/denforum/index.php?topic=220.0, but since I have to wait for log-in approval, I say I won't bother.

So, Anach, if you talk to those people in the near future, could you ask them to look into how they're generating the merged packages with the save-as-you-go approach?

What happens on my system is I've got a clothing directory that I merge together. I don't have to merge them together that often; only when I get something new. But right now, it merges to about 240 MB.

The "problem" I have is that to merge these together, I have to choose the save-as-you-go method (or else the program crashes when it runs out of memory), and that's OK because it does successfully make a merged package. But as a result, I get a horrendous amount of temp files created in my AppData/Local/Temp folder. With each file that's added, it puts a copy of the most recently merged file in there and adds to it. And then it moves on, copies that, adds. Copies that, adds. This goes on and on and on until you either run out of packages to merge, or you run out of diskspace.

I tried doing this with my other system, when I only had a 50 GB partition, which most of it spoken for and I was running into errors. That's when I noticed what this program was doing.

Now, since I know its behavior, I carefully watch those files and delete them as they build up. It's not a big issue at the beginning, but at the end, when you're adding packages in the order of kilobytes, and creating copies of 200+ MB on each addition, it soon consumes stupid amounts of diskspace for no reason.

And when the merge has completed, it doesn't even clean up after itself.

So... if you happen to talk to them, could you ask that they manage those files better? I mean, all they need is a copy of the most recent and a working file. But if they want a couple more than that, then fine. But they seriously don't need all of them, since they don't reference them at all, for anything.

OK, that's all I wanted to say. I really like merging all my packages together, and it's pretty easy to do. I'm just about to remerge the clothing package, and I'm going to be spending the next 30-60 minutes monitoring the process. That's why I thought I'd share my thoughts on that.

[Inge]

According to Peter, those files are not managed directly by s3pe at all.  It's something to do presumably with .Net and the OS as a result of some of the functions used.   Peter has looked through his code on many occasions trying to deal with this problem, and he says there is nowhere in his code itself that creates these files nor can find out what they are called in order to delete them.

I don't even recommend using the dbc squeezer at all - we added it as a favor to someone who thought it would be a good idea, as it seemed a harmless enough request.  To my mind the whole process is just a huge unnecessary hassle.  Why don't you use Granthes' launcher?  That deals with all the dbc stuff itself - apparently without the unwanted system overhead.

[tryguy]

Wow, such a quick response!

I feel a little sheepish. Umm, that's the first time I've heard of Granthe's Launcher. The reason I switched over to merging my packages in the first place was the massive performance increase at load-up. I'm sure this has been discussed elsewhere, but merging them brings load times down to "acceptable."

Anyway, I'll go look into that launcer. If it does the same job, I'll switch over. If I see longer loading times, I'll just persevere with the file-monitoring. Like I said, I don't merge that large file very often at all, mostly I'm just merging Hacks packages that get updated on a daily basis.

If you could thank Peter fo the implementation for me, Inge, that would be super. Thanks.

[Rushad]

I started using the Custom Launcher some months ago, after using this method for a long time. The launcher creates .package files up to 1 GB in size, and then makes new ones as needed. I'm currently at 4 files (yeah, I go a bit CC crazy) but there's barely any performance hit.

One thing to watch out for is that it actually keeps a backup of every file added, so twice the disk space is used. If you have disk space troubles, it might not be the best option. Otherwise, it works great.