It does/did matter for different reasons at different times.
As I had been homophobic and not too fond of men touching me, I went with a young female dentist, as a 1st active professional health care choice. But I had to deal with my employer's (male) plant physician for annual physicals, and didn't need my own primary care because i was always in perfect health.
I went on to choose female gastroenterologists, a dermatologist, and eventually a primary care. But here's where I think it matters now....
My female primary care kind of insists i wear a gown, even though they tell me to keep my underwear on also. Even though I mentioned that I'm "old school" and didn't really think a gown was necessary. And also, she doesn't include genitals and hernia as part of the routine annual exam.
Now I'm thinking that if I were to instead have a male primary care, he would likely be cool with waiving the gown, and may include examining genitals &/or hernia as part of the annual physical (and I've since become comfortable with men touching me).
So in that regard, i would say the sex of the doctor would probably matter.
But does it really matter if I wear a gown or not, or have hernia/genitals included or not?
In the grand scheme of things, no. However, i would enjoy the experience more.
Perhaps if I had a male dermatologist, he would have checked my genitals during at least one of the "full body" semi annual skin exams that I have. So again, the sex of the dermatologist may matter in the thoroughness of the "full body" skin exams as well.