Another question:

Are there any Relationships or Family Ties which are invalid when an NPC is one of the sims? I know that Nervous Subject has the Grim Reaper as his father, so at least some of them are valid, but I'm seeing a surprising number of NPC Relationships and Family Ties in some of the downloaded neighborhoods.

For now, I'm assuming that they are all valid, unless the other sim doesn't exist.

Hum. Which neighbourhoods are these?

Half-aliens will have family ties to the PT, of course, and while Nervous didn't have a family tie to the Grim Reaper by default, I did add one in my version of Strangetown (assuming it was safe since the aliens have the NPC PT in their family trees). Other than that, I can't think of any good reason for sims to have relationships or family ties to NPCs, but again, if the known flag isn't checked, I don't think the relationships do any harm... On the other hand, I can't see how they do any good, either. And I'm pretty sure that having a "known" relationship to a NPC is a rather bad thing.

I just added my list of populated neighborhoods here:
http://www.simswiki.info/wiki.php?t...t:Neighborhoods

I have enough AnyGames set up on my machine to give each of these neighborhoods the configuration that they request, which gives a great testing platform. A lot of these neighborhoods have problems, so they make great test cases. My discussion with Quatchi about these neighborhoods was the inspiration for this tool.

I'm not sure that they are a problem. Certainly, you don't want to do anything which will cause changes to the objects.package, but relationships and family ties are stored in the neighborhood package. Guess I'll have to do some research.

The main areas of concern are memories which do not show up in SimPE. That includes memories which belong to the neighbohood (?), to the lot (including hiring of service sims), or to non-existent sims (which includes the full range of memories).

The main problem with memories is that there does not seem to be any way to tell whether the subject and target of a memory are sims or objects, other than by looking at each individual memory type. I'm reasonably sure that the game executable doesn't actually do that.

I believe that this is one of the reasons why invalid memories can cause such problems in neighborhoods. If the game has no way to know whether a memory parameter is a sim or an object, then invalid sims can be replaced by objects with the same instance.

I've added logic to produce a log entry for any sim who has a Sim Description (SDSC) but no User file, excluding known NPCs. The Sim Description contains a SimID which uniquely identifies a sim. The SimID is used to identify the sim by matching the Object Data (OBJD) GUID in either the User file or the objects.package (NPC).

The following sims are added to the game by EA, but they have no User file and there's no indication of their function in the objects.package, which means that they may just be garbage.

The questions are: Should I warn users about the presence of these sims? If so, then should I (eventually) "fix" these sims by removing their SDSC and all references? Note that I will not consider removing these sims until I can be sure of removing all references, including memories, wants, etc.

Since they are added by EA, I'm considering having the HoodChecker "recognize" them (ie, avoid giving an error if it encounters them). I have played a game with sims who have relationships with some of these "stealth NPCs" and nothing shows up in the game, so it's impossible to identify them. This is not surprising, since the majority of data about a sim is stored in the OBJD or other corresponding records in the User file or objects.package.

If anyone knows the purpose of these sims, I'd appreciate a comment.

The following SimID appears with no User files in all of the Pets, Bon Voyage, FreeTime, and Apartment Life stealth subhoods:
0x71B85E0D

The following SimIDs appear with no User files in the Seasons stealth subhood:
0x7040237A
0xF036D5C3
0xF03AE97B

The following SimIDs appear with no User files in the Bon Voyage stealth subhood:
0x73B38D46
0xF3B3896A

The following SimIDs appear with no User files in the Apartment Life stealth subhood:
0x01200000
0x01210000
0x01220000
0x01230000
0x158A6B8A
0x158A6BDD
0x158A6C30
0x558A6A92
0xF58A6C87

The following SimID appears with no User files in the Apartment Life neighborhood: Belladonna Cove. This is the only shipped neighborhood that I could find which has an stealth sim:
0x759390DF

Do these stealth NPCs show in SimPE (as Unknowns)?

Also, for the relationships to self, would it be possible to tag them depending on whether they have a 'known' flag or not?

Yes. That's why I decided to treat them as valid for now. However, since you can't see them in-game and since I haven't been able to find any references to them in the source code, I suspect that they're just garbage that EA didn't bother to clean up.

Probably. At this time, I'm not parsing that record at all, but it doesn't look too difficult:
http://www.simswiki.info/wiki.php?title=SREL

Do you think that's the only piece of information from that record which is important for self-relations: known -> bad; unknown -> OK? Or, should I ensure that all of the flags are unset and the relationship values are 0?

Last edited by Mootilda : 20th Apr 2011 at 02:25 AM.

Most likely. In the case of the stealth hoods, I don't think they are even copied over when merged into a main 'hood, just like other garbage. I'll have a look at the one in BDC. When I release the new version, I'll probably remove it first.



If you could do the latter, that would probably be the best.

Looks like that is pretty easy. I don't even need to parse the SREL record; I'm just checking the entire array for zeros. Any non-zero entry implies that the self-relation is invalid. I've made the changes and am testing before uploading.

Some interesting facts from my initial exploration of memories:
- I haven't found any memories which belong to invalid Lots, Families, or Sims, even in some of the neighborhoods which have a lot of problems.
- Sim memories make up about 2/3 of all memories. This means that only about 1/3 of all memories cannot be viewed in SimPE.
- If SimPE would display memories belonging to the Neighborhood, Lots, and Families, there would be no need for the HoodChecker to fix them.

I'm curious, do you know what purpose the neighbourhood/lot/family memories have? Are they "hidden" token type memories? All sorts of data that's not actually memories seems to be stored in a sim's memory, too, or at least that's how SimPE displays it. Pregnancy tokens, inventory items, gossip, etc.

Neighborhood memories are supposed to include subhoods, although I haven't gotten that far yet. Lot memories include the service sims assigned to a lot, which can be a major problem if the service sim is invalid. I think this is also where the food is stored (ie, fridge inventory). I believe that the family memories include games purchased, since they move with the family, rather than staying with the lot.

I'm not sure whether they're specifically hidden, but they're not displayed anywhere. SimPE only displays memories associated with Sims.

In this context, a "valid" sim is a sim with a valid Sim Description (SDSC) instance number, since that instance number is used in the Family Ties (FAMT), Sim Relations (SREL), and (most places within the) Neighborhood Memories (NGBH).

For an average sim, the only way to get the sim's name is to open every file in the Characters subfolder, and then match the SimID between the Sim Description (SDSC) in the neighborhood package and the Object Data (OBJD) in the User package. Once the correct User package has been found, the sim's name is in the Catalog Description (CTSS) within that file.

A sim who has a proper Sim Description but no User file can still be valid. The HoodChecker does not remove sims just because it cannot find their User file.

NPCs, for example, have their user data stored in the objects.package under Program Files. I have gone through the objects.package and the program recognizes all valid NPCs; at least, I hope that I've found them all (if not, please let me know). The HoodChecker does not open the objects.package each time and look through every Object Data (OBJD) looking for the NPCs, since this would take a long time and would require the game to be installed with expected registry entries. Instead, these SimIDs are hard-coded in the HoodChecker logic and then matched to the Sim Description (SDSC) instance number for each neighborhood.

Some sims, such as Multi-PollinationTechnician and Multi-Plantsim mods, have their user data stored in the Downloads folder. The only way to find this user data would be to read every single file within the Downloads folder and all of its subfolders each time that the program is run. At this time, I do not intend to do this.

There is also one valid sim who does not have a Sim Description (SDSC). The Mystery Sim has a special SimID but no other data; the HoodChecker recognizes this special ID.

The HoodChecker also "recognizes" the SimIDs for all of the sims that were shipped with EA neighborhoods or subhoods, but without either Sim Descriptions or User files. As with NPCs, these SimIDs are hardcoded and then matched to the Sim Descriptions at run-time. At this time, there's no indication that these sims are actually used and it may make sense to remove them. Until I can get some verification that these sims are not used and do not regenerate, I intend to leave them in.

Last edited by Mootilda : 27th Apr 2011 at 12:32 AM.

One of the Neighborhood memories is "Token - Urnstone", which should have a valid (dead) sim instance as the second piece of data in the memory's data array. In the original E001 Belladonna Cove, there is one of these Urnstone memories for each dead sim in the neighborhood. In the clean E001 neighborhood, most of these sims no longer exist, but their neighborhood urnstone memories still exist (because there is no way to clean them up at the present time).

This is interesting to me because when I play the cleaned E001, I get errors about urnstones. I suspect that these memories are the cause of the problem, although the errors are also associated with one of the MATY mods (my guess is that the MATY mod is using those urnstone memories and then choking when the sim doesn't exist).

I've also found Lot memories about non-existent pre-selected NPCs (ie, NPCs who have been assigned to a lot, but who no longer exist in the cleaned neighborhood).

So, it looks like this is a fertile area for cleanup.

Note: for memories, I am using the terminology Owner and Subject to match the display in SimPE.

Last edited by Mootilda : 29th Apr 2011 at 01:30 AM. Reason: Better title.

Very interesting. These are things I obviously didn't even know about. No one probably did until now.

I, too, got those urnstone errors in clean Belladonna Cove, but for some reason only in apartment lots. I can't remember if they were related to any mods, though. Which MATY mod are you refering to?

Would be very nice to be able to clean this stuff.

Sorry, I can't remember which MATY mod needs to be removed to make the urnstone error messages disappear. I believe that it's one of the ones in the director's cut, possibly the one which maintains some of the "married" family ties after death? If I find it again, I'll let you know.

I suppose that it's time to stop checking the memories and start removing some of them, so that I can see whether the error messages disappear when the urnstone memories are gone. I'm concerned about removing Sim memories until I have the logic to ensure that all gossip about those memories is also removed, but there should be no problem removing corrupt memories associated with the Neighborhood, Lots, or Families, since none of these generate gossip. This fact could explain why these memories haven't caused the inevitable corruption of the entire neighborhood.

All Sim memories can be fixed or removed using SimPE, so removing them is a lower priority.

Are you OK with my logic about the User files? Or, do you think that I should be searching the Downloads folder for any User info?

I agree, removing the sim memories is both a lower priority and more risky. Have you made the logic to check for corrupted sim memories? If the hood checker finds them, SimPE could be used to fix/remove them, at least for now.

I'm also curious to see if removing the urnstone memories stops the errors.

I can see how searching through some people's Downloads folder (especially on a slow machine) would take... quite a while, and would quickly become annoying if you're trying to check several 'hoods. As long as the user files aren't deleted, but just generate a warning, I think it's okay to not search the downloads. Just make sure there's a note on the final download that specifies that if you're using such hacks, the hood checker will find them, and it doesn't mean anything's wrong. For people who're at least a little bit aware of what they put in their game, it should be easy enough to make sure that the number of unknowns corresponds with the number of hack-related sims.

I'm working on the memory logic. Validating memories is a much harder task than the family ties and sim relations, since there is no obvious procedural method for determining whether the memory subject is supposed to be a sim. This means that I need to have a list of memory GUIDs and their associated category (see below).

There are potentially 3 sims who are associated with a memory:
1) The Sim / Family / Lot or Neighborhood who has the memory in their slot.
2) The Owner of the memory; usually the active participant (the person who retired, the person who had the baby, the person who got the secret lot membership) from data[4].
3) The Subject of the memory, usually the passive participant (the friend, lover, baby, enemy, etc.) either from the GUID in data[5] and [6], or from an optional instance in data[12]. The Subject may be a sim, or it may be some other type of object.

Gossip is just a memory where the Owner is not the Sim who has the memory. I haven't found a way to distinguish between a memory with an incorrect owner and gossip about that owner.

Right now, I'm categorizing individual memories as one of the following:
1) Memory about self. (Verify that the owner is the subject is a valid sim.)
2) Memory about another sim. (Verify that the owner is a valid sim and that the subject is a valid sim.)
3) Memory without a sim subject. (Verify that the owner is a valid sim; other checks may be added later.)
4) Inventory item. (No checks at the current time; checks may be added later.) In later EPs, inventory items have an inventory number. However, earlier EPs do not have that field, so I'm collecting a set of inventory memory GUIDs for debugging purposes.
5) Non-memory item, such as a game token or urnstone. (Checks performed depend upon the actual memory GUID. For example, urnstone memories have a valid dead sim as data[1]. Assigned (pre-selected) NPCs have the employee in data[1] and the employer in data[2], both must be valid sims.)

I'll add something to try to make it clear that the missing user file messages are warnings, not errors.

Last edited by Mootilda : 28th Apr 2011 at 04:14 PM. Reason: Clean up format a bit, to make things more clear

N003 (Veronaville) has a family memory slot for 7FFE (Townies) and 7FFF (Service NPCs), even though there is no FAMI entry for those families. Note that there are sims who belong to these families.

So, I believe that HoodChecker should recognize all of the known FAMI instances for these special families, even if the FAMI entries do not exist.

[Update:] I may have made a mistake in my testing. If there is no FAMI entry, there should also be no sims in that family, so the memory slot should be able to be deleted.

Last edited by Mootilda : 1st May 2011 at 09:20 PM. Reason: Test partially invalidated.

Removing the Urnstone memories which have invalid sim references does not remove the urnstone errors. So I decided to do a bit more investigation into the cause of these errors.

ALFixes.package is the mod which brings out the problem with the urnstone memories. It replaces the standard Global Token - Verify Lot/Neighborhood BHAV and adds some additional logic to check for apartment sublots.

The problem appears to be that most of the Urnstone memories in E001 have an invalid Lot ID (0x81). I believe that these Urnstone memories should be pointing to Gamesend Grounds, which has a Lot ID of 0x8A. So, I made that change and the urnstone errors disappeared, even with ALFixes installed. Attached you'll find a picture of Gamesend Grounds after this change was made. Leave it to EA to ship broken neighborhoods; only two of the urnstone memories associated with E001 are valid.

Unfortunately, if you delete these invalid Urnstone memories, you will lose the associated urn / tombstone forever. I believe that I'll look into a way to ask people where they want the urns / tombstones redirected. Alternatively, perhaps the HoodChecker could just select any community lot and use that one? It looks like that's what the game does: it just sends tombstones to the community lot with the lowest lot number.

Reference:
http://www.moreawesomethanyou.com/s...topic,72.0.html

especially post # 669and post # 670By the way, I'm having a lot of fun looking into these memories. If only EA had added some meta data to the memories, so that it would be obvious whether to expect a sim ID or a lot ID or whatever.

Last edited by Mootilda : 29th Apr 2011 at 05:37 AM.

How interesting about the urnstones! Wow. EA's failures never cease to amaze. did wonder why they had made a graveyard without putting any graves in it.

Hmm. But the HoodChecker should still delete urnstone memories that refer to invalid/deleted sims, right? Redirecting the rest would be a great option. Offering a clean Belladonna Cove with a full graveyard would be pretty cool.

Out of interest, which were the two valid memories? I'm guessing one is Olivia Riley, as her grave can be found at the museum community lot.

Also, about Veronaville, is that the (my) clean Veronaville or the original (or a different one)? I'm a little confused, as I'm sure the original Veronaville had FAMI instances for the townies and service NPCs, and I'm also sure I deleted both the families and all the belonging sims in the cleaned version... (I hadn't needed to delete the special family instances anyway as they are hardcoded and will regenerate, but well, I've learned a bit since then.)

That's right. The HoodChecker can't do anything useful to fix the problem if the sim is invalid. But, I hate to lose valid sims just because the graveyard lot number is invalid.

The other is Leroy Ejess at Watervista Palace.

It's your clean version. However, I might have gotten confused during testing, since I've been switching between the clean and original versions fairly frequently as I try to figure things out. It makes sense that the family memories would not have been deleted when you removed the families and sims, since SimPE does not give you access to those memories.

Except for testing the HoodChecker, I use all of your clean neighborhoods. Thanks so much for making them. I'm looking forward to your new improved versions.

Last edited by Mootilda : 30th Apr 2011 at 02:36 PM.

Agreed! I actually like to recreate the graves for premade dead sims when possible, this would make that a lot easier for BDC.

Hm, I've deleted Leroy Ejess, hopefully that means his grave is also gone... Will have to check that out.

Also, thank you. I'm looking forward to finishing the improved versions myself!

More about urnstones:

1) Urnstone tokens can be used to move tombstones to residential lots, not just community lots.

2) Urnstone tokens can be used to move tombstones between the primary neighborhood and subhoods by modifying the hood number in data[4]. However, the specified lot must be in the specified hood or the tombstone will remain in limbo.

And with that in mind, I found another bug. If everyone in an apartment dies, their tombstones are sent to the lot with the lowest lot number. If that lot is in a different subhood, the game tries to send the tombstones to that lot, but in the current neighborhood. This fails.

So, when Ramir and Ana Patel died in Belladonna Cove, the game sent their tombstones to lot #1 in Belladonna Cove. Unfortunately, lot #1 is a Downtown lot. Since there is no lot #1 in Belladonna Cove, their tombstones don't appear on any lot in any neighborhood.

Sigh.

Yet another reason to not use apartments, I guess. I don't understand why they couldn't just let the player decide what lot to send them to. It's not as if that functionality doesn't already exist. What would happen if lot #1 happened to be another apartment lot? With residents?

Good question. I haven't tested that one. My (educated) guess is that the urnstone would be delivered and belong to that family. If everyone on that lot dies, all of the urnstones would go back into the (unending?) funeral procession to the designated graveyard.

Question: does an occupied apartment ever get a lot number lower than the lot number of the apartment building? If not, then the real question might be: what happens if an urnstone is delivered to an occupied apartment building (base)? Does the urnstone get delivered? Do we end up with multiple copies of the urnstone for each occupied apartment (sublot) within the building? Ouch!

I also need to remove all of my CC, to ensure that this one isn't a failure of one of Pescado's mods, but is actually EA's problem. [Verified that this is standard EA behavior]

I've been trying to figure out exactly which lots to display when asking the user to choose a graveyard lot. I think that I may just display all sub/hoods and all lots and expect the user to use some common sense. After all, if the tombstone can't be delivered, it probably remains in limbo unless you use Pescado's batbox to delete it. I may also give people an option to 1) delete all urnstones 2) send all valid urnstones to the lowest numbered lot in each sub/hood 3) select a lot in each sub/hood to use as a graveyard or 4) select a global lot in one sub/hood to be used as the graveyard for all sub/hoods.

I was originally thinking that it might make sense for neighborhood creators to renumber the graveyard lots so that they are the lowest numbered lots in each neighborhood template. I renumbered all of the community lots in Brainania, so I know that it can be done. Unfortunately, that doesn't solve the problem of EA sending urnstones to an invalid lot in the current sub/hood.

There is a mod available which allows you to select your graveyard when a sim dies; I haven't tried it yet, but I may start recommending it to people (after thorough testing, of course):
http://www.modthesims.info/download.php?t=316426

These Urnstone errors seem to be a common occurrence and Pescado is telling people that they've destroyed their neighborhoods, when it's really EA who is causing the problem and the problem is fixable via the HoodChecker. If you aren't using Pes's Director's Cut, then you may just notice missing tombstones, without receiving any error. It's the same problem.

I'm also considering creating a mod (Urnstone Delivery Service) with several in-game menu options: select "all" to deliver all outstanding tombstones to the current lot, select "hood" to deliver all outstanding tombstones marked for this hood (regardless of lot), or select an individual tombstone to deliver to the current lot. Unfortunately, that would require a playable character, so it isn't a good solution for neighborhood creators or community lots.

The urnstone logic has really been holding up my release of the initial memory checking logic. I want the urnstone fix to do something reasonable, but it's difficult figuring out the correct solution until I do enough research to understand the nature of the problem.

Last edited by Mootilda : 3rd May 2011 at 12:55 AM. Reason: Verified that standard EA behavior creates invalid urnstone tokens.

I'm guessing that the apartment sublots will never be lot #1, so yeah, my concern was if the urnstone was sent to the base lot of an occupied apartment building. That just seems... bad. If possible, it might even be for the best to not make apartment lots available as a place to move urnstones.

That mod seems interesting, and your mod idea sounds good as well. I'd still like the option in the HoodChecker, though, at least for "lost" urnstones like in the case of Belladonna Cove.

I wonder if any of the other premade hoods has floating urnstone tokens like this? They all ship with a silly number of dead sims, after all. (Although most of the base game ones are broken.)

I also wonder if all of this could somehow be related to how urnstones sometimes mysteriously go missing without the player doing anything to them?

I once had this happen in a Strangetown/Pleasantview combo 'hood. All of the urnstones in the whole 'hood disappeared. I played Olive Specter's lot anyway, and saved. Later, the urnstones in Pleasantview (Goths, Michael Bachelor, a community lot graveyard I made myself) somehow were returned, but all the ones in Strangetown, both on Olive's lot and elsewhere, remained gone. I have no idea what happened, but it would seem like the urnstone memories got corrupted somehow...? That wouldn't be so odd, if it weren't for the Pleasantview urnstones returning.

I find this all to be very interesting, and I wish I had more time to do some research as well.