The fact the Haye found the need to catapult himself straight into the Wlad fight as opposing to earning it by beating any of the likes of Adamek, Arreloa, Povetkin, Chagaev or anyone who's been deemed half decent the last few years is always going to be detriment.
He hasn't fought anyone of worth in the division, and when he had his chance he fought cautious and almost scared of Wlad.
The right thing to do would be to come back, take on a few contenders, build himself up and take another crack at it, all the while becoming more comfortable at the weight, maybe building it up more gradually and learning from the experience of actually fighting some proper heavyweights.
He won't do that... so it's very hard to defend him, regardless of his physical attributes.
He talked himself into a heavyweight title fight with little merit, rode the British hype train right to the top. It went tits up for him, and he's still has preconceived notions of his standing in the boxing world.