I think that Jefferson was saying that since voicing one's views cannot harm you, there is no need for government to interfere, so long as it isn't the government trying to extend the views of one particular sect or creed. Freedom of speech doesn't just include the right to speak your mind without fear of censorship, it also includes the freedom to turn off the radio, turn off the TV, or walk away from the protest rather than whining to your local politician about how your rights were violated because you had to listen to someone else's drivel