In some respects black slaves had higher social capital than free but unskilled white labour. A black slave represented a capital investment; it made sense not to jeopardise your investment, or to devalue it. So really dangerous work was not assigned to black slaves, but to hired labour; if a free worker was injured or killed you suffered no loss; you could just hire another. Similarly you were stuck with your slaves - you couldn't fire them and hire others; you had to make the signficant capital investment of buying others. So that gave you a incentive to keep your slaves fit, healthy, with at least a basic level of morale and motivation; you had no similar incentive in relation to hired day-labourers who were easily replaced.
This left unskilled free labour in a very difficult situation - they got the worst work and could be treated worse than slaves so far as conditions of employment/relations with employers were concerned. But the possibility of starting at the bottom and rising to better conditions was cut off by the fact that, for most work of anything more than utterly minimal value, it made economic sense to employ slaves rather than free labour. Thus they perceived themselves as being at the bottom of the heap, and with the only way up blocked by the people second from the bottom, the slaves.
The result is that the interests of slave labour and free unskilled labour were fundamentally opposed, and this was the perfect environment for fostering bitter resentment between them which, because of the racialised nature of slavery, very easily turned into bitter racism.