I had the same thing but luckily I had a toddler back at the house so it let me re-enter and my sims came home a few seconds later. If this happened to me in the scenario you've detailed, I know what I would try, it's up to you whether you would want to give it a go, it would mean losing the vacation lot that they were at and also losing their home, but I don't think there's a way around that. I use Insimenator, but you can use any summoning method, if you prefer to play without hacks you can use boolprop and spawn the Tombstone of L and D and it would work the same way, but just take longer.
If you don't have 'testing cheats' auto-on then before you enter a household at your neighboorhood screen open the cheat console and type 'boolprop testingcheatsenabled true' and enter. Enter another residential lot in the same hood (it needs to have at least one more slot for a new household member). Once in, shift and click on any household member and look for the pie menu entry 'spawn', click on that and then from the next menu choose 'Tombstone of L & D'. It will appear next to the sim and be in the guise of a gravestone. Click on the gravestone and choose 'Add neighbor to family...' this will then bring up a menu of ALL sims in the corresponding hood, NPC's, townies and playables. Click MORE and keep clicking MORE until you see the name of an adult member of the lost house. Beware, that it only lists by first name so hopefully the names of your sims are not a common name used by other NPC's and townies. When you see one of their names click on it and that member will be added to the household. Once you've done that you can just move them out, set them up on their own lot and then repeat the process for the rest of the lost household in the new lot.
I use the Insimenator, which would make it easier as the Summoner that is part of that hack splits the family into family name so you can go direct to the missing person/s without having to flick through 10 billion pages of pie menu, I'm not sure if Inge's telporter shrub would do the same as I've never used it, maybe someone else would know the answer to that.
This is only my take on this, if anyone else has a quicker, easier, better way of retriving lost sims I would love to know in case it ever happens to me again. Whatever you decide to do I hope you get your family back, it's most annoying when you lose sims through stupid glitches like this. Good luck !
Do you think that if I went to my self sims house it would force every one back. I only sent me and my wife as guests and I have 2 kids at the house. But Im not sure that it would save the host family.