Dexter pre-Harry

Ok, so I've been wondering about something. But before I get started, I'm only on episode 5 of Season 1, so please don't spoil anything for me as I'm really enjoying this show.

Harry adopted Dexter at a young age, right? But how young? In the episode "Let's Give the Boy a Hand", Dexter is looking through his photo album and comes across a baby photo(perhaps, I didn't see it too well). Are we to believe that Dexter was adopted as an infant?

The main reason behind this question is I think I remember Harry saying in episode 1/2 something along the lines of, "After what happened to you." So does that mean that something happened to Dexter as an infant to make him the way he is now? Or was that "something" not pre-Harry, and in fact post-adoption.