Well done on achieving such a financial achievement at 37!
Plenty of NCHDs have young children, myself included. Plenty of consultants have young kids too.
College will be grand with kids insofar as it would be no different than a job. They'd have to go to a childminder or family member, but the hours aren't horrific and as long as you can get a couple of hours of study in a day towards the exams you'll be grand.
As an intern it varies wildly. In my intern year (pre-strike and so still doing up to 36 hours), my weeks ranged from ~50 up to high 90s. I have never broken 100 hours in a 7 day period, which disappoints me in some ways. Intern hours have reduced down a bit now though so that should be better.
As an SHO it depends on your specialty. Surgery can be absolutely cat. Medicine in some places is similar. You could be back to the high 90s again. The EWTD isn't really a thing, despite what they say.
Short answer, you can make it work but you won't see your kids as much as you are now. You'll be working one weekend a month minimum in most jobs, if not two+. If that's acceptable to you, go for it. If not, I would have a long think about what you want to do.