It's hard to defend him," Smit said.
The 2007 World Cup-winning captain told the Rugby Union Daily podcast: "The way he has approached this is not right.
"Are you telling me Rassie is the only coach frustrated by a call that has gone the wrong way?
"Something has to be done. There has to be a line that has to be drawn, and he is making it difficult for his team. It's made us, as a rugby team, so easy to dislike."
mozart
Hall Of Fame
39927 posts
Men's game needs ego check'
Smit, meanwhile, urged South Africa to concentrate on playing rather than obsessing with referees, and wants to see a "reset" across the board in how officials are treated.
"I honestly believe there should be a penalty for any player who thinks they can help the referee referee the game," he said.
"The referees have to get control back. Everyone seems to think they have an influence over what the referee should be doing. When I was captain, when anyone else spoke they were disrespecting me, and the referee.
"Refereeing rugby is incredibly difficult. What we want is consistency and respect, and I think when we start giving that back to the referee the pressure will be off them [and they will] have more consistent performances.
"I would like referees to get back in charge and not have to defend themselves the whole time.
"Obviously we would all like referees to have perfect games every weekend. But I think it starts with the players. We have this very masculine, aggressive game in the men's game, and you watch the women's [World Cup] final and you don't see any of that going on.
"Maybe it is time for everyone across the board to do a little ego check, reset the clock, and get back to what made rugby great, which is being respectful."