I'm not a physician, but my clients are. I've been directly involved in HIPAA regulations for 22 years (since 1996, when it passed).
The original question has been well answered, but the nature of our shared interests led me to want to offer a bit more background on rights, provider obligations and what's practical.
There is a provision in HIPAA {remember, there's no "PP" in HIPAA - that's a different kink} that obligates providers to honor patients' requests to keep specific information confidential that might otherwise become part of a general record. For example, if I am tested and diagnosed with a genetic disorder that might affect my employment, I would not want my self-insured employer to know about that unless/until I decided to disclose it, nor do I want that information in a database (more on that below) affecting my insurability in the future. I might need to talk to my doctor about managing my hypertension or diabetes in the context of suspension play or bondage (positional syncope), breath play (hypoxia), enemas (M & M, soda/sugar, wine, beer) or other medical consequences, but do not want my kinks in my medical record. HIPAA provides that we may specify certain information cannot be shared - period - and the provider is obligated to honor that and could be legally/financially liable it that confidentiality is violated.
Suppose my wife is having an affair and becomes pregnant. My vasectomy removes me as a candidate, so her abortion would be something she could require her Gyn to keep confidential, even from the staff (or limited to specific staff). In a small town, that's a biggie!
There are also a few things that a provider may not agree to keep confidential, even if you insist. HIV/STI infections require notice to the Health Department, not matter what. Spousal and child abuse must be reported. Suicidal ideation or being a threat to others are two more.
Can you totally rely on your doctor to protect your "special information" as confidential? I think docinny45 covered that really well. Yeah, kinda, we hope, maybe, but not exactly, for the reasons he cited. A few years ago I worked with a practice commencing use of an EMR. One of the doctors had some patients with "special situations" (consensual heavy SM play, for example, another with marital sexual abuse). She kept a set of private, paper files locked in her office with a note in the EMR indicating that there were separate, confidential, "eyes only" files related to that patient. Think of someone just beginning to sort out being transgender in a small, rural town in Utah.
One solution is for YOU to maintain custody of your very private records, taking them to your doctor for each visit; NOTE: s/he may or may not agree for liability reasons. Once they've been updated, as needed, you take them home That allows your doctor to have them during the visit, note that there are "external records" being maintained, but with much less chance of an accidental disclosure. Unless YOU lose them, have a nosy spouse or kid(s) or need to quickly get them to a new doctor or ER. A flash drive might solve the problem - or not.
HIPAA requires that providers supply you with a complete copy of your medical records, on demand. They're also supposed to provide you with a copy of today's visit record, every time (part of the ACA requirements related to the mythical "interoperability"). Get your records and review them. Several of mine were wrong (R kidney vs. L kidney?) and I had to fight for correction. If your confidential information shows up, deal with it on-the-spot.
EVERY health insurance claim processed by EVERY insurer has some of the claim data exported. Your name, SSN#, DOB, date of services, billing codes and diagnosis codes go to the Medical Information Bureau. Why? So when I apply for a $5M life insurance policy and swear I'm healthier than an olympic athlete, MIB will report my history of cancer, obesity, drug abuse and depression to prevent insurance fraud. NO copies of your medical records go with the data sent to MIB - just codes, but parts or all of your medical records could end up in your insurer's files if medical necessity or other claim dispute arises.
Our digital age has positives and negatives. As noted, privacy is mostly a casualty. State, area and regional data exchanges are still mostly in uncharted (and not reliable) waters. Find out if your doctor, hospital, lab, pharmacy, etc. are participating, willingly or unwillingly. You have the right to totally bar sharing your information, or define what cannot be shared, but you have to ask the question "are you?" and you have to get an honest answer. If your doctor(s) are employees of a hospital, or the practice is owned by a hospital but still uses their original name, they may not have the authority to agree to your limits - that might be the domain of the hospital's Medical Director or Privacy Officer. Ask. Verify. Ask, every time ("have your privacy policies changed?").
Don't risk your health or life by not sharing information with the people who can keep you alive, but how you share that and how they keep records - or not - matter immensely.