I spent months doing nothing but building so I didn't discover a lot of things about the actual game play until quite recently. That includes aging off or on affecting the lot and not the family. I cannot recommend aging off in a household with a baby based on the following.
I moved a family of two parents (one pregnant) and a baby to a larger house; I had been playing them with aging on. I had moved the previous sim owner out of this house months ago, and hadn't played the lot since. Everything seemed fine for a bit, but I noticed the 12,000-hour "help with birthday" vigil did not start that night as it should have. I couldn't even "grow up" the baby on my own the next day. The pregnancy continued normally, though, and now there were two babies. Then I realised I hadn't heard the aging chime for the adults. I would have played the previous owner with aging off, so I figured the attribute had to apply to the lot and not the family.
I knew then I would be exiting the lot without saving, so I continued just to see what would happen. I turned aging on, the birthday vigil started that night, and then things got really grotty. The game would slow to almost a stop for 10 or 15 seconds (real time) then pick up, the parent holding the infant would dump him on the floor and continue to stand there, and the "help with birthday" action would enter and drop from the cue repeatedly. Sim parents dump babies on the floor quite frequently, but it's usually to do something far more important like play chess, look out the window, or empty the trash compactor because the maid never will. I've never seen them drop the baby then just stand there. I exited the lot (without saving) with as much haste as I've done anything.
I quit the game, re-booted my computer, and started it up again. When I re-entered the house, the first thing I did was turn aging on. After that everything progressed normally. I cannot say for certain that the aging being off was the sole or even a partial cause of the weirdness, but as I made no other changes between visits I have to wonder. I'm not anxious to try duplicating the situation, either!