Only once, when I was 14.
My pediatrician did all of my exams the same way, with me only in my briefs. As I was a late bloomer, I looked like a little boy and was treated accordingly. He started the exam with me sitting on the table, talking the whole time more to my mom, who was sitting in the chair nearby, than he talked to me, and then he later had me lie down. At the end after palpating my abdomen, he always made the same lame joke about checking the plumbing, which was my mom's queue to stand up and look on as he pulled my undies down to above my knees.
At the exam at 14, he said he had good news that I finally had started puberty. My mom asked how he could tell, since I looked the same as last year. She was mostly right, as I still had the penis of a little boy and not even the start of pubic hair or darkening scrotum with rugae, or sculpturing of the glans. But the doctor pulled the orchidometer out of a drawer and showed her that my testicles had started growing and were already the size of the bead marked with a 5, which meant puberty definitely had started. He then asked her if I had started to have nocturnal emissions, and she said no, at which point he pulled my undies back up. The orchidometer exam didn't embarrass me, because I still was oblivious to sexuality at that age.
When I was 15, at the same part of the exam, I guess it was obvious to both of them that my testicles had grown, as the orchidometer did not come out again. But my mom asked when my penis would start growing, and the doctor said it had started, but I had gained enough weight that the extra length was obscured by the fat pad. She then mentioned how much darker my scrotum was, and he said it was natural and would be less obvious once the pubic hair that had just started filled in some more. He again asked about nocurnal emissions, and she said still no, and he said they could start any day, and pulled my undies back up. At that exam, the first when classmates were becoming men, and I was newly pudgy, I was more embarrassed about taking off my shirt than the doctor pulling down my undies.
Finally on my 16th birthday, when the doctor pulled my undies down to show my mom my nearly adult-sized penis and testicles with significant pubic hair, he pivoted to asking me questions instead of asking her. He asked if my penis had started to ejaculate semen, and I said yes. He asked when, and I said a few months earlier. Finally, he asked if I was awake or asleep, and I said awake (unwittingly admitting to masturbation). He turned to her, and she nodded yes, so I wasn't any good at keeping that secret, it seems. The doctor could see my little five-incher stiffening from all the attention, so he quickly pulled my undies back up over my erection and declared that I had caught up to my classmates. He then proceeded with the standing hernia check with my back to my mom for the first time ever. I guess he figured she had seen enough. Clearly there was no need for the orchidometer that year. I had mixed emotions. I had started the exam proud of how much progress I had made, having grown a few inches taller that year, as well as having lost my puffy chest and big belly. I also knew that it was good news that I had pubic hair and could make sperm, so I wouldn't need hormone treatment, as feared a couple of years before. I of course was embarrassed about them both seeing my uncontrollable erection, but I was relieved nobody said anything was wrong with it when they saw it. And because I finally was caught up, that was my last exam with the pediatrician. The next year, I got to see a GP, and I was examined by myself.
In retrospect, I guess the orchidometer is mostly useful at the first exam after puberty starts, as it's obvious without the beads both before and after that exam. I'm curious, @Kid89 why you needed the orchidometer multiple times. Please share!