Collars have a seam in the same place as something like a t-shirt. I even find polos to be less confortable than a t-shirt around the neck.

Collars exist to keep neckties from being directly on the skin, and that would be uncomfortable.