In December 2013, Judge Robert Spencer ordered the owner of Masterpiece Cake Shop to start baking cakes for gay weddings. According to undercover footage obtained by the ACLU the proprietor, Jack Phillips, agreed to bake a cake for a wedding between two dogs, but refused to bake a cake for a gay marriage. Some saw the issue in terms of free speech; Jack Phillips was refusing to participate in something he disagreed with. Others saw it as a question of discrimination; Jack Phillips was denying a service to gays that he granted to heterosexuals (including heterosexual dogs). While both views have merit, if you value civil rights you have to side with Jack Philips.

The basic free speech argument is straightforward, and easy to understand. Jack Phillips didn’t deny Charlie and David Mullins a cake because they were gay; he denied them a cake because he didn’t want to participate in a gay wedding. If they came in and tried to buy a cake for some other purpose he would have sold it to them, and he would have denied a cake to the heterosexual parents of Charlie and David; assuming they wanted it for a gay wedding. From this perspective the case is clearly a matter of free speech, and free association, not discrimination.

But wouldn’t this, in effect, allow private party discrimination? Couldn’t someone refuse to bake a cake for an interracial couple, because they don’t believe in interracial marriage? Would this open the door to all sorts of private party discrimination on free speech grounds? These objections can be answered two ways. First, the slippery slope runs both directions, if Jack Phillips has to bake a cake for a gay marriage why not a gay pride parade? We also have to consider the scope of the problem; if we are dealing with a handful of cranks, what do we gain by restricting their freedom?

While a bigot could attempt to justify his discrimination as speech, he would have to do so before a judge and jury. He would have to defend his actions as actual speech, and not a pretext for discrimination. Gay marriage is not recognized in Colorado, but anti-gay discrimination is illegal. Jack Phillips can plausibly claim to be expressing a political opinion, since gay marriage remains a live issue. If Jack Phillips refused to bake a cake for David and Charlie’s house warming party, on the basis that gays shouldn’t own homes; that would be harder to defend as an honest political statement.

In his superb essay on libertarianism and private party discrimination, Julian Sanchez hit upon the importance of scope. If discrimination amounts to a single “whites only” restaurant, legal action would be completely unnecessary; we could safely allow the “handful of cranks” to enjoy dinner in their “monochrome environment.” If “whites only” restaurants became a pervasive phenomenon, and it started to have a negative impact on non-whites, then it would be reasonable to have public accommodations laws. America confronted just such a problem in the segregated south, a problem that required the strong arm of the law to correct. Do we confront a similar problem regarding gays? This author can’t recall seeing any signs saying, “No gays allowed.” Nor does he recall reading any stories about businesses maintaining such policies. The only problem appears to be a handful of wedding-related business refusing to participate in gay weddings, a trivial cost.

In a free society, freedom of association is the general rule, and public accommodations laws the exception to that rule. As a general rule, it is perfectly legitimate to decline to participate in an activity one disagrees with. On occasion it becomes necessary to violate this principle, by forcing people to associate with those they dislike. That was the case in 1964, but that clearly isn’t the case in 2014. Twenty eight states do not have laws protecting gays from discrimination, but we don’t have segregated lunch counters, or “no gays allowed” signs. In the absence of an obvious need we are better off leaving people alone.