Gynecomastia and 'man boobs'

Hi, I am a 23 year old male and very slim in build. I lift some free weights at home just try keep myself a bit toned and I try to eat healthy and very little junk food. However I don't use a gym or anything so am obviously not a big fitness head. When I was a kid I was over weight and grew out of this when I was about 15. I began rowing around that age and build up some muscle in my upper body. Although I am very slim now I still have a big chest. I have sort of 'man boobs' only they are not saggy and are more muscle.

I think my chest sticks out a bit naturally with my posture so when I stand sideways my chest sticks out further than my stomach. I do push ups and some free weights every day but this just makes my moobs more toned and does not get rid of the fat around them. Lately I have been reading up on Gynecomastia which I am pretty sure is what I have. I think now that the reason I can not get rid of them is because it is tissue and not fat around them so the only way to get rid of them is through surgery. Since I definitely cannot afford to get surgery (and don't want to), is there anyway I can make them smaller?

Thanks!!