A slight majority of Americans oppose banning hateful and offensive speech—but mostly because we can’t agree on what speech is hateful and offensive.

That’s a key takeaway from the Cato Institute’s new survey report, “The State of Free Speech and Tolerance in America.” The findings in almost every category are distressing for those who abhor offensive speech but believe it should remain legal to express such sentiments in the public square.

According to the report, only 59 percent of Americans say people should be allowed to express unpopular opinions in public, even those that are deeply offensive to other people, while a substantial minority (40 percent) say government should prevent people from engaging in hate speech against certain groups in public.

The divide is mostly based on race, ethnicity, and partisan affiliation. While solid majorities of Republicans (72 percent) and independents (60 percent) oppose government banning hate speech, Democrats stand out with a slim majority in support (52 percent). However, African American and Latino Democrats largely drive these numbers with a majority (55 percent) of white Democrats saying government should allow public hate speech, but majorities of black Democrats (59 percent) and Hispanic Democrats (65 percent) saying it should prevent such speech in public.

Among college graduates, 64 percent say hate speech should be legal and a third (36 percent) say it should not. But current college and graduate students are equally split on the issue, with nearly half (49 percent) of current students saying government should ban hate speech and nearly half (49 percent) saying it should not.

Libertarians (82 percent) are the most opposed to hate speech laws, followed by Conservatives (75 percent) and a slim majority (53 percent) of Liberals. However, nearly two-thirds of Populists (64 percent) say government should prevent hate speech in public.

It seems the main thing holding back hates speech laws (aside from the First Amendment) is that American can’t agree on what constitutes hate speech:

59 percent of liberals and 17 percent of conservatives say it’s hate speech to say transgender people have a mental disorder39 percent of conservatives and 17 percent of liberal believe it’s hate speech to say the police are racist80 percent of liberals and 36 percent of conservatives say it’s hateful or offensive to say illegal immigrants should be deported87 percent of liberals and 47 percent of conservatives say it’s hateful or offensive to say women shouldn’t fight in bat roles90 percent of liberals and 47 percent of conservatives say it’s hateful or offensive to say homosexuality is a sin.

While there is disagreement on what counts as hate speech, you can find almost one-third of Americans who would support banning it for just about any group.

Consider the percentage of Americans who would ban hateful or offensive speech against the following groups: African Americans (46 percent), Jewish Americans (41 percent), immigrants (40 percent), armed service members (40 percent), Hispanics (39 percent), Muslims (37 percent), the police (37 percent), gays, lesbians, and transgender people (36 percent), Christians (35 percent), white people (32 percent).

In fact, you can find almost a third of Americans who consider it morally acceptable to use physical violence against Nazis as a reaction to their speech (32 percent), support banning Holocaust denial (35 percent), and believe revoking a person’s citizenship is a reasonable response to flag burning (39 percent).

Perhaps the most disturbing finding of the survey is that more than half of Americans (53 percent) say hate speech is an act of violence. While two-thirds (66 percent) of Democrats say hate speech is violence, 58 percent of Republicans say hate speech is not violence. Independents are split, with 51 percent who disagree hate speech is tantamount to violence.

African Americans (75 percent) and Latinos (72 percent) are nearly 30 points more likely than white Americans (46 percent) to believe hate speech is violence. Instead, a slim majority (53 percent) of white Americans believe it is not.

While nearly two-thirds (63 percent) of women believe hate speech is violence, a majority (56 percent) of men disagree.

Americans under 30 (60 percent) and seniors (57 percent) are also more likely than middle-aged Americans (35-64) to believe hate speech is violence (49 percent).

If speech is violence, how much longer will Americans allow it to be protected by law? And how long do we have before opposing banning “hateful” speech is considered a hate crime in America?