I dunno. I've always assumed that there wasn't a lot of time when all of them were together.
Because their mother was hospitalized because of overworking and she wouldn't have to work after marrying Dr. Nakano (or even earlier, since he'd probably try to lessen her burden when he became interested in her).
It's possible that the marriage has happened a bit before she had to be hospitalized, but wouldn't Dr. Nakano being a doctor and all, try to put her in a sanatorium or something so she could recover?
Also, since quints never had a real father, support and wisdom from him would've been appreciated.