But who decides how the label is defined so that there is a label? That is what I am wondering.
Unlike several other languages (like French or Dutch) there are no institutions that have any say in how the English language should be used, except through the example of their trustworthiness. They may issue learned and weighty pronouncements on how they think or believe English should be used, or how they have observed English being used.
But they have no authority whatsoever in enforcing much of anything they say. All they have is their reputation.
English usage is defined by consensus. As soon as there is a sizeable number of English speakers using words in this or that particular way, then that particular way becomes the correct usage.
Obviously this means that English is a fast evolving language, with usage and definitions changing within an average person's lifetime.
This also means that there is an awful lot of arguing going on over what some words mean. But with no one in authority to decide what is correct or not, these debates will go on and on until some sort of consensus is hopefully agreed on. Or not. In areas that concern fast changing mores and values, especially sexual, don't expect much of a consensus to ever really evolve. Especially since value judgements enter into definitions.
Btw, as an addendum, while the Academie Francaise or the Dutch Taalunie have the weight of actual national law behind them, they don't work much better at keeping the languages in stasis either. These evolve just like English, but perhaps at a somewhat slower pace. Their linguistic findings are however considered to have the force of law in that whatever they decree to be correct language usage, has to be implemented in schools, government and the civil service.