Many avoid wearing the clerical collar for the same reason Pastor Torvik did in The Hammer of God.
"You must certainly understand that I want to come as an ordinary human being."
Pastor Bengtsson, an older and much sounder Lutheran pastor had this admonishment, which, in my opinion, is the best explanation of why a pastor should look like a pastor:
"Then you are sailing under false colors. You are no ordinary person. You have been ordained by the Church as a servant of the Word. You have been elected and called by the Christian congregation at Odesjo to be its pastor. You get support from the fields which godly forbears donated for the pastor's upkeep. It is pure dishonesty to take the money, if you want to be just and ordinary person."*
*The Hammer of God, pg. 254