That is the only reason why I can see the stereotype being justified in the States or Europe.
Another historical origin, if not justification, might be that (African) slaves were often sold nude, or at least viewed naked by prospective buyers to better appraise their physical condition and probably further cow, denigrate and impress upon the slaves the helplessness of their situation.
This would have been a socially acceptable occasion, for both men and ladies to look at other naked people and ogle them intensively. Slaves were probably also taken off ship while wearing little or nothing in the hot seasons, another occasion for respectable people of both genders to look and stare.
It might have indeed been one of the few occasions on which ladies might have been able/allowed to view completely naked males. Since whites in those times did not walk around naked, with many husbands not even doing so in their own homes or in the bedroom, this might have led to the general impression that African slaves were more generously endowed, there being no real comparison possible.
Add to that the scare stories put about on the horrors of servile insurrection leading to murder, rape and mayhem, add a little exaggeration to justify legal suppression and draconian slave laws and you have a perfect breeding ground for rumor, whispered exaggerations and sexual suggestive allegations and outright lies. You need to make monsters out of those you would suppress and exploit. Making them sexual monsters to boot, is a sort of icing to a cake of lies about the racial inferiority and innate bestiality of slaves.
Another facet might have to do with circumcision. In slave holding times, white American men were seldom circumcised, while many slaves were Muslims and hence cut. That could also possibly explain why slaves were thought to be larger in size.