Free Speech! You shouldn't say that.

Insults, even recommendations to be quiet, are generally not violations of your free speech, as opinions you should be silent are examples of free speech expression; although they are immature, so, too, are insults.
But if those recommendations become demands, and if you are treated with frequent abuse by those making them, during conversations, debates, etc., than they are, in principle, violations of your free speech rights, as such behavior prevents you from expressing yourself as readily as you should be able to.

There are differences in meaning between 'should' and 'cannot' or, conversely, 'have to.'

Limitations on government aside, the arguable exception to this forgiving generality is when the persons expressing these opinions of you have legal or well-recognized authority over you which you and others might reasonably see as adverse to your welfare and self interest to stand against, i. e., the person is abusing his authority over you, therefor he is preventing your free speech rights through intimidation. I say arguable because it must be asked if the person of authority should have his free speech limited by a concern about the recourse he might take against you, assumed by his status and capacity, as opposed to his character. But the generality inherently holds true in most cases. Afterall, you are not subordinate to everyone, are you?

As for which is worse: being told to shut up, for it proposes no speech at all, not even crass speech.

Occasional insults, etc., are not automatic violations of your free speech rights.