SimSheet (a sim-tracking program) [Now with Aspiration and LTW tracking!]

[BattyCoda]

Hi, new tester here. I have been fooling around with this tonight and I have a question and a problem.

Am I supposed to have to rename the Rufio folder for each neighborhood? That was the only way I could get
it to do another neighborhood and save the data.

I loaded up my really old neighborhood, 1214 sims. The Rufio folder was created and the ExportedSims.txt file is there,
but when I tell it to Import Sims, it crashes with the SimSheet has encountered a problem and needs to close message.

Btw, the Rufio folder is being created in the SimSheet folder.

SimSheet:
C:\SimSheet-test

SimPE:
C:\ProgramFiles\SimPE\

[rufio]

Hi, new tester here. I have been fooling around with this tonight and I have a question and a problem.

Am I supposed to have to rename the Rufio folder for each neighborhood? That was the only way I could get
it to do another neighborhood and save the data.

No; the Rufio folder is only a temporary holding place for whatever neighborhood is being imported.  You can save via the LM dialog, which will create a <filename>.sls file in the SimSheet directory that can be loaded again.

I loaded up my really old neighborhood, 1214 sims. The Rufio folder was created and the ExportedSims.txt file is there,
but when I tell it to Import Sims, it crashes with the SimSheet has encountered a problem and needs to close message.

What happens before it crashes?  Any message boxes?

ETA: Kate, there appears to be a sim without a name in your neighborhood.  I can't see how that by itself would cause a crash, but after all it is crashing on the aspiration anyway.  I suspect there is something else wrong with the sim which is causing the crash; since "" come alphabetically before everything it gets loaded first.  I'm not real interested in looking through the entire output file to find out what's wrong though.  Inge, under what circumstances would the sim be output without a name?  Ok, I got the program to output - some of the lot descriptions have newlines in them, which causes the program to think that there are two different sims - one corresponding to the data before the newline, the other corresponding to the data after, and since the data is shorter than it should be for both sims it crashes trying to access things at the end of the list.  So what needs to be done to make the output digestible by SimSheet, Inge, is to have SimPE avoid outputting newlines in the actual data.

[Inge]

Thanks Rufio, I'll ask Peter to do a substitution with a normal unprintable character.

Having consulted, how about you get the strings enclosed in quotes?   We'll change the CSV to a fully conventional one with comma seperators.  I asked for the semicolons at the time as no one uses them in strings these days, but I forgot the newlines!! 

[J. M. Pescado]

See, this is why you need to escape all your strings. Even unprintables isn't safe, because you just know some weirdo is going to name a sim or a lot with a non-printable character, possibly one of the ones you use.

[BattyCoda]

I loaded up my really old neighborhood, 1214 sims. The Rufio folder was created and the ExportedSims.txt file is there,
but when I tell it to Import Sims, it crashes with the SimSheet has encountered a problem and needs to close message.

What happens before it crashes?  Any message boxes?

SimPE does its thing.

Generated list of sims.

Loaded data for 1214 sims.


SimSheet.exe has encountered a problem and needs to close.
We are sorry for the inconvenience.


ETA: This may be the newline problem. Here is a section of text that looks suspect:

N001;Pleasantview;314;Kiernan;19;Tricou;Tricou;21;House of Fallen Trees
;Y;N;;;316;16;0;Unknown;Unknown;51;Athletic;5;Aquarius;Romance;Male;Adult;14;864;80;37150;500;300;600;500;600;500;300;600;500;600;316;24;578;276;656;827;39;767;280;-27;953;409;418;175;478;921;400;400;295;755;606;455;171;294;285;0;706;135;0;N;;;;;;;Human;0;Romance;Nothing;Arts;12050
N001;Pleasantview;315;Fricorith;44;Tricou;Temptress;172;Sim State Dormitory (14 Rooms);Y;N;319;317;;Unknown;0;PrivateSchool;APlus;0;Unemployed;0;Capricorn;Reputation;Male;Adult;28;931;84;67450;300;700;600;300;600;300;700;600;300;600;400;998;400;1000;182;970;0;335;296;0;715;438;807;500;615;973;400;428;41;118;131;114;328;524;330;494;647;192;0;Y;0;1000;61;2;8000;Philosophy;Human;0;Reputation;Nothing;Film;8753
N001;Pleasantview;316;Nylissit;19;Tricou;Tricou;21;House of Fallen Trees
;Y;N;320;318;314;Unknown;0;Unknown;Unknown;0;Medical;6;Scorpio;Reputation;Female;Adult;14;732;71;20950;500;800;300;500;400;500;800;300;500;400;308;887;101;469;738;934;-28;442;231;39;573;447;837;400;500;823;400;963;135;295;0;0;702;436;301;500;803;524;0;N;;;;;;;Human;0;Reputation;Nothing;Science;11655
N001;Pleasantview;317;Kvornan;19;Tricou;Tricou;21;House of Fallen Trees
;Y;N;;;319;16;0;Unknown;Unknown;0;LawEnforcement;7;Cancer;Fun;Male;Adult;14;684;67;32750;500;500;500;500;500;500;500;500;500;500;373;705;890;420;673;413;33;1000;237;-23;326;767;985;14;493;615;400;896;140;102;492;0;606;204;324;0;987;76;0;N;;;;;;;Human;0;Fun;Nothing;Cuisine;10935
N001;Pleasantview;318;Jon Smith;19;Tricou;Tricou;21;House of Fallen Trees
;Y;N;;;320;16;0;Unknown;Unknown;36;Unemployed;0;Aquarius;Family;Male;Elder;2;900;70;7150;500;200;700;500;600;500;200;700;500;600;400;624;600;737;150;632;44;659;110;-29;347;873;71;400;400;168;400;973;510;399;506;0;81;177;654;0;678;417;0;N;;;;;;;Human;0;Family;Nothing;Arts;15434
N001;Pleasantview;319;Jennail;19;Tricou;Tricou;21;House of Fallen Trees
;Y;N;320;318;317;6;0;Unknown;Unknown;24;Medical;6;Cancer;Fortune;Female;Adult;14;964;86;52350;500;500;500;500;500;500;500;500;500;500;400;523;419;743;669;12;-23;678;270;27;669;350;148;400;418;804;400;330;762;357;0;0;702;403;0;500;600;709;0;N;;;;;;;Human;0;Fortune;Nothing;Sport;11990

Edit 2: Yep, that was it. I went into the .txt file and removed the newlines from the suspect lines and the neighborhood imported properly.

[Inge]

Fix is in progress.  Output will adhere more strictly to CSV format in future.  Strings quoted with quote-escaped embedded quotes, and commas instead of semicolons.

[rufio]

Man, I was hoping to avoid the text strings enclosed in quotes thing.  I've always had a heck of a time trying to get Windows to read newlines even if they were enclosed in quotes.  Sigh.  I don't even use the lot description in the program (nor do I have any plans to) and there should be no reason why any of the other fields would have newlines.  Can we just delete the newlines and forget about reformatting?

[Inge]

So you are unable to parse standard csv?

[rufio]

I'm sure I could find a way, but in Windows it seems to be too much of a pain.  In linux, my usual methods work fne.

[Inge]

Remind me what language you are using?

[Marhis]

Wait... maybe a stupid idea, but could it be because in unix and windows lines ends with different characters? IIRC in Windows are both "new line" and "carriage return" (\n and \r), while in unix there is only one? (or perhaps it's the vice versa, I really didn't remember well).

[Inge]

It's probably to do with the more powerful commandline file management utils Linux tends to have.

I see rufio's point.... but it seems a shame to have the output *only* of use to this one util.

[rufio]

Could be, Marhis.

Inge, I am using C++; in linux I would use cin.getline to get in everything up to the next comma, which would even get in newline characters, including \rs that I had added to the output in anticipation for use in Windows.  In Windows, if there is a newline before the next comma, getline will stop at the newline.  I don't know how to tell if getline stopped because of a comma, or because of a newline.  I have better I/O with Qt, so I'm sure I could hack together something, but it would be a pain and there is no reason to preserve newlines when I am not going to use them.  I don't think it would be a tragedy for excel either, if every single newline a creator put in the lot description was not preserved.

[Inge]

Newlines were the problem before, but there could also easily have been embedded commas in the strings (or semicolons as we were using).  It was really a mistake to ever have thought we could put out freely user-entered strings without protecting them in quotes.

As the output table has a fixed number of fields per record, could you not read in a count of fields?  The benefit of that is you don;t even have to parse the fields you don't want to deal with.

[rufio]

I am fine with parsing fields within quotes, as long as they do not contain newlines; I do that with the savefiles anyway.  What I do for those is use Qt I/O functions to break up the line into a list of strings as delineated by semicolons/commas, and combine any that appear to have quotes on the edges.  I get the info out of that list using indexes.  I don't think that counting semicolons until the right index is reached would work, because there could be semicolons in the output (encased in quotes) that would throw it off, and I would still have to go through and combine those to get the right index.  If that makes sense.

[Inge]

Yes it makes sense, but the same problem can arise if you *do* get a comma (or whatever the current seperator is) in a string field, mid-record.  It would still throw your fields off.

I am just not sure it should be SimPE's job to edit these user-entered fields though.  It means the output is sort of inaccurate.  It's not as if the commas or newlines got in there by error - they are actually valid content in those fields.  While empathising with your position, it feels to me as if it should be more of your tool's responsibility how it wants to display the data rather than the data's responsibility to pervert itself to fit - if you see what I mean?

[tunaisafish]

Yep, DOS uses CR+LF where linux would just use the NL.
Excel *does* handle the embedded line breaks fine, both importing and exporting them.
*The* spec to look for CSV's is rfc4180.
You should be able to find a CSV library to do the parsing for you for most languages.
I know that Perl Text::CSV took care of all the parsing and escaping headaches for me

I agree that it should be SimPE's job to just make the data available unaltered.
Then if somebody needs access to the information not used by SimSheet, they can easily use Excel, Perl etc...

If SimPE is going to destroy data for the transfer, then that should be a '-simsheetformat' option IMO.
I don't know what the ultimate goal of SimSheet is.  The next version may use more fields that it is using now?

[J. M. Pescado]

I still say the solution is just to escape any characters that are a not A-Za-z0-9

[Inge]

How would that help?  If she's splitting records on newline at first pass, and can't vary that action on the basis of the nl being within quotes, how is it going to be easier to detect an escape character preceding the newline?

[Marhis]

A couple of months ago I was trying to convert an Access output (csv indeed) in SQL, and it was a PITA, because of one text  field pretty much filled with carriage returns and various escape characters; in the end I found a program on Sourceforge which handled the conversion perfectly. Now, I don't remember what it was - I tried many other, first  - but having a look at the source of those kind of programs maybe could help.

[rufio]

How would that help?  If she's splitting records on newline at first pass, and can't vary that action on the basis of the nl being within quotes, how is it going to be easier to detect an escape character preceding the newline?

Maybe this?

Code:

string line = in.readLine ();
if (line[line.size () - 1] == '\\' && line[line.size () - 2] != '\\') line += "\r\n" + in.readLine ();

Or, you could simply use \n to represent the newline, and to hell with the actual newline character.

Escaping things would be simpler than messing around with quotes.  Even if you used quotes, you still have to escape the combination of "; as that could be mistaken for the end of quoted field if it occurred inside a quotation.

[Inge]

Now the \n substitution would come at a high cost in large hoods.

The quotes round strings comes at a small cost of the output file increasing in size by the number of " characters.   But because Peter already knows the data type of each field, there is no testing - certain fields always get "".

But if he had to test each character of the file to see if it was a newline you would be looking at a very significant slowdown in the output process.  People are already talking of "going out for the afternoon" while the sims are transferred from SimPE to your program - I am worried that would turn into "going for their vacation"

[coralleane]

I'm probably doing something very simple wrong here, but having downloaded the newest version, whenever I try to import sims, this is what I get after SimPE has run through its bit:



Clicking OK then brings up a message saying "Cannot find output file.  Check that you are using the AL version of SimPE" or words to that effect.  I see in the changelog that the output is now supposed to be going to the SimSheet folder, but it's not there - I went and checked the rufio folder in the main Sims folder that had been created previously, but it hadn't put it there either, it's still got the previous modified date on it.  I ran a search for "exportedsims.txt" on the whole computer just in case it had squirreled it away somewhere bizarre, but it only returned the afore-mentioned one in The Sims 2\Rufio folder.

ETA: Inge, the incredibly long loading time (which was with a hood of over 1,200 Sims) was only in a previous version - new versions have sped it up considerably, to the point where it only takes a few minutes to import a list that size for me now, compared to the hours it was taking before.  I don't know how much difference what you're discussing would make to that, though. 

[tunaisafish]

Rufio, any field with a \ before the NL in the original data would break the code you suggested.

Orig field
"/\/\ () /\/\
\/\/ () \/\/"

SimPE output would then have to be...
"/\\/\\ () /\\/\\\
\\/\\/ () \//\//"

I don't see what benefit there is to invent a new data standard, especially if in the process you corrupt the data and make it less readable.

There's C++ and .NET support listed here.
http://en.wikipedia.org/wiki/CSV_application_support

Inge, Peter will still need to scan all characters to escape existing quotes with a quote.
The slowdown of testing every character shouldn't be noticeable.
Also Rufio fixed the problem with the long transfer time.  Now if you blink you almost miss it

[rufio]

Inge - could he just automagically replace all newlines with \n right before writing the data to file?  That's what I do when I output non-newlines in place of newlines.  I'm using a Qt function, so I don't know exactly how it works, but I have seen no difference in the time it takes to output.

Cora, I'm not getting that problem, but I'll put up a debug version that will show the exact command it's executing.

tunafish - That's why it checks for \ as the last character and \ as the second-to-last.  Naturally, any \ in the data itself would be escaped with another \.

Also, I think I've probably gotten spoiled with Qt's I/O classes, but I would really prefer not to have to go back to using C I/O.  I've forgotten a lot of those functions and never really learned printf/scanf anyway.  And then I'd have to do a lot of messy conversion from character arrays to std::strings to QStrings and back again.  I understand the merits of using a "standard" file format, but CSV is not exactly a universal standard anymore, is it?  And it would be simple to add an option to SimPE to make it output in 100% standard CSV, I'm sure.