Actually, Phelps' body is not that unusual for an elite male swimmer. He is slightly taller than average (6-4 compared to 6-2 or 6-3) and has a slightly longer armspan, but its not remarkable when compared to other elite males. (Same is true on the women;s side. The remarkable Katie Ledecky is six feet tall, but again, the average elite female swimmer is about 5-10, and elite female swimmers at six feet is not uncommon).
And, of course, at this date, only seven years after Phelps last competition, all but one of his world records have been broken by swimmers shorter than he.
But, the biggest problem with the "Phelps Gambit" (much has been written about this rationalization) is that it suggests that no sporting competition can possibly ever be truly "fair", because nature is infinitely diverse. Obviously, to accept that leads to the end of sport, Male and female categories exist because the range of abilities amongst males at the elite level is relatively small (swim races are typically won by tenths of a second)'; and the same is true for females, save for the unique Ledecky.
BUT, the difference in performance at the elite level between male and female is ENORMOUS, in comparison. So, where in London in 2012 Phelps defeated Chad LeClos in the 100 fly by 2 tenths of a second, he was 4.8 seconds faster than the winning female time of Dana Vollmer.
And in fact, for every female event, there is a freshman in high school somewhere who is faster than the female world record.
The differentials between the sexes is .... substantial, to say the least.