Unless you're just blatantly obvious about your diapers (so bulky that you can't keep your legs together, wearing tight sweatpants with your plastic pants sticking four inches up out of the back and your t-shirt tucked in and not at all covering them up), nobody is going to notice your diapers.
Trust me. On the few rare times when I go out in mine, I actually want to be noticed--not by kids and not by other men, but by women, especially middle-aged women because they always give you that “I know what you're wearing” look of amused amazement and that “I'm about to laugh out loud at you” smirk.
But unless I'm about as obvious as described above, nobody seems to pay one bit of attention. People are off in their own worlds, thinking their own thoughts, and generally just not being very observant about whats going on around them. (Case in point--when there's a robbery and the police are questioning witnesses and often they disagree about the suspect's description or sometimes can only remember “He had on a red shirt” but nothing more).
For me to really be noticed in my diapers, they have to be thick, soaking wet and drooping, under tight pants or shorts, I have to be walking like a toddler with all the bulk between my lets, and often that doesn't do it--I'll pull my plastic pants up out of the back of my shorts and act like I don't know this, and even sometimes I have to drop my keys and bend over to get them. And even then, most people pay NO attention.
So I'd say just wear your diapers, if you think they might fall down, wear suspenders or a onesie under your clothes, loose pants, and don't worry about it.