Most people don't give a damn what others do in their bedrooms.
I think you forgot to add 'anymore' - most people don't care anymore ... and even then it would be in the West.
That wasn't necessarily so in the recent past, as evidenced by the number of anti-homosexual laws still on the books when I was young. People in general did care what 2 men got up to together because they were brought up to care and worry and condemn it.
In the main it was indeed clergy and legislators pandering to religious bigotry who sort of stoked the fires of fear and loathing. Some of them seem to have made it their trademark signature, scaring people with tales of perdition and depravity having to do with gay men.
And while a majority of people in the West don't care much anymore, a large and often very vocal minority of people still do. Witness the number of people who end up making media splashes in the US because of their anti-gay bigotry: county clerks refusing to issue marriage licenses to gay couples, dissenting Supreme Court judges, privately owned companies airing their anti-gay views, legislators galore who yap on and on about the dangers of homosexuality, the number of organizations that make money by trying to 'cure' homosexuality by prayer or forceful deprogramming techniques, not to mention radio talk show hosts and preachers and pastors who really have a thing against gays. Group beatings up of gays that result in death are not unheard of either. Even the fact that there are differences in ages of consent between straight and gay sex is still a remnant of an anti-gay attitude, residual perhaps, but still present.
In the EU, people from the Eastern countries have a harder time accepting gays as well. Poland for instance, staunchly Catholic and conservative, is notorious for its anti-gay attitude. (On a personal note, I once knew a Polish handyman who refused to carry a plastic bag in public because it was printed with colored flowers - too gay he finally admitted ... ha ha. So I took it and carried the tools in the flowery gay bag for him, much to my amusement.) Even further to the East, strongman and buddy-boy to billionaires, Putin, makes no secret of his anti gay attitudes and the new laws promulgated in the Russian Federation.
In Belgium for instance, our language and cultural assimilation courses for new immigrants have a special section that expressly tries to explain the acceptance and legality of homosexuality. Why would that be necessary if not for anti-gay attitudes expressed by a large percentage of immigrants?
And then are other continents and cultures. Africa is a continent where several countries have drastic and draconian anti-gay laws on the books. They have stopped short just of the death penalty only because of international pressure and media outrage. Otherwise you would see that happening too: hangings or whatever of gays (like how 'witches' are killed in secret by lynching, burning or whatever) ...
And let's not even start about certain tendencies in Muslims that decry and condemn homosexuality. And this last is ironic in a way, since some hundred years ago, (affluent and rich or down and out artists and writers) American/European gays often went to Mid Eastern countries or North African colonies expressly in order to indulge their gay urges. Try that now.
Which shows that it is possible to influence entire groups of people for the worse.
And as a last item, I guess people have already forgotten the US Defense of Marriage Act, passed in 1996 (20 years ago) which at its core defended an anti-gay attitude.