I imagine this question was answered in the Human Genome Project that ran from 1990 to 2003.
The genes to give you a penis are from your father. The genes controlling its size are mixed from both parents.
Here are a few articles:
https://melmagazine.com/en-us/story/who-do-you-get-your-penis-size-from-mom-or-dad
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2816361/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3890219/
There are other articles on penis size and intersex disorders.
You will find them if you do a GOOGLE SCHOLAR search.
For you little dick AND big dick guys with low self confidence, see this: https://www.tandfonline.com/doi/full/10.1080/23311908.2016.1147933
The hand/penis size comparisons may actually have some truth because in all mammals except humans, there is a penis bone. https://link.springer.com/article/10.1007/s004410051262
Hox genes in digit development and evolution
J Zákány, D Duboule - Cell and tissue research, 1999 - Springer
… The blue staining in A and B indicates the expression of the lacZ reporter gene, which
replaced the Hoxd-11–12–13 genes in the HoxD-deficient allele … The size of the baculum,
the small bone in the penis, is proportional to the size of the digits …