Something to consider, which was somewhat touched upon a few lines up but I wish to expand upon, is where the attention is directed from the desire. Somewhere in this lies the misconception, and not a completely unfounded one, that many 'outsiders' have. The desire is towards the diaper itself, the object rather than the person who may be wearing it, yet in many cases the person wearing the diaper easily becomes an extension of this desire as well due to the nature of said desire.
Rather than feel I'm simply speaking without any base, I will turn your attention to the message boards themselves. Scour them here, there and elsewhere and it's always the same: baby diapers, the more babyish the better; why can't adult diapers be more like baby diapers?, etc, etc. Not only that, but often times it doesn't seem to matter the model sporting the goods, it doesn't matter if the model has a face at all, as long as that diaper is prominently on display. So, it is most definitely the object itself, the diaper. This is why the underwear on women analogy doesn't hold, because it is the woman you desire, not her underwear; if she were wearing a diaper, it would be the diaper you desire, the woman simply becomes an extension, a close second if you will.
Because of the nature of this fetish, paraphelia, what-have-you, those individuals who understand not what it means to desire an inanimate object in such a fashion have no idea how to distinguish between a person fantasizing about their child's diaper rather than their child. Sadly, there are many who don't give much credit in the favor of the innocent (I use this loosely) in this area by lusting over diaper ads sporting children in them, by placing up pictures of minors in diapers throughout various diaper picture sites, groups, even Myspace. There are those who would discredit the rest by chasing after underage children online or over cellphones, etc.
We are not more removed and against pedophilia than non enthusiasts, we are simply more aware of the implications of our own desires and therein lies a certain spark of doubt and perhaps guilt even. We know and understand the perceived correlation. Unfortunately, due to the precarious nature of this affliction (yes, affliction), it sometimes lends to a line most thin. Those guilty parties mentioned above are more than one person sullying the group's good name, and with such small numbers comparatively, more than one is a big number considering the charge.
Why do people draw these correlations, these stiff conclusions?, simply look at the PR that we represent ourselves with. Look at those who make the news and where those investigations lead. How many times, and under how many threads in this forum alone has that exploitation flick of the young girl (a minor) who wore diapers and shot her mom shown up? How many people constantly inquired about it that it showed up? How many stories, how many pictures, how many comments of a seriously incriminating nature are present? Then, to make it worse you constantly have those who would strut around in public and further rub it in the faces of non-consenting, non-enthusiasts and you ask why?