It is not a damaging notion that Jesus was just a man.
Not to me. That’s kind of the Muslim view, after all. A man imbued with prophethood.
But, there are no shortage of brilliant Christian theologians, past and present, that would disagree with your premise that it is “not damaging.” What you’re proposing is called Adoptionism, and it was really the first heresy the Church ever addressed as such:
Adoptionism - OrthodoxWiki
Adoptionism is a form of the heresy of Monarchianism that appeared in varying forms in the second and third centuries…
This link drives down into the theological reasoning which so concerned the early Church:
Adoptionism, in a broad sense, a christological theory according to which Christ, as man, is the adoptive Son of God…
The interesting thing about Adoptionism as that the early church jumped all over it even prior to the codification of Trinitarian doctrine. Therefore, it cannot be argued, as Unitarian might, that it was skewered because it conflicted with Trinitarianism; the heresy predates the Ecumenical Councils.
Then, there’s Arius, who delivered a variation on that theme. His view was that Jesus was not originate.
Arianism - Wikipedia
Arianism, in Christianity, is a Christological concept that asserts that Jesus Christ is the Son of God who was…
At any rate, this issue was not just huge to the early Church, it was in many ways the first important theological issue they wrestled with.