If nothing else, the Supreme Court’s decision in Students for Fair Admissions v. Harvard is a victory for the conservative vision of the so-called colorblind Constitution — a Constitution that does not see or recognize race in any capacity, for any reason.

As Chief Justice John Roberts wrote in his opinion for the court, “Eliminating racial discrimination means eliminating all of it.” Or as Justice Clarence Thomas put it in his concurrence, “Under our Constitution, race is irrelevant.”

The language of colorblindness that Roberts and Thomas use to make their argument comes directly from Justice John Marshall Harlan’s lonely dissent in Plessy v. Ferguson, the decision that upheld Jim Crow segregation. “There is no caste here. Our Constitution is colorblind, and neither knows nor tolerates classes among citizens,” wrote Harlan, who would have struck down a Louisiana law establishing “equal but separate” accommodations on passenger railways.

But there’s more to Harlan’s dissent than his most frequently cited words would lead you to believe. When read in its entirety, the dissent gives a picture of Harlan not as a defender of equality, but as someone who thinks the Constitution can secure hierarchy and inequality without the assistance of state law. It’s not that segregation was wrong but that, in Harlan’s view, it was unnecessary.

Today, Harlan’s dissent is among the most famous in the history of the Supreme Court. For much of the century after they were written, though, Harlan’s words were obscure, known mostly to opponents of white supremacy. For obvious reasons, the court ignored Harlan during its long period of indifference to racial segregation. But the court also ignored Harlan when, in the 1940s and ’50s, it began to undermine Jim Crow and strike down state laws requiring racial segregation.

You won’t find Harlan in 1944’s Smith v. Allwright, which invalidated the “white primary.” You won’t find him in 1948’s Shelley v. Kraemer, which prohibited states from enforcing racially restrictive housing covenants. And you won’t find him in 1954’s Brown v. Board of Education, in which the court overturned Plessy and declared mandatory school segregation illegal.

Harlan does not begin to show up in the opinions of the Supreme Court and its justices with any regularity until the 1980s, when Republican appointees began to take aim at the affirmative action policies of the previous decade.

In a dissent, joined by Justice William Rehnquist, to the court’s decision in Fullilove v. Klutznick in 1980, Justice Potter Stewart quoted Harlan’s assertion that the Constitution is “colorblind” to challenge the majority’s view that Congress could use its spending power to remedy past discrimination. “Under our Constitution,” he wrote, “any official action that treats a person differently on account of his race or ethnic origin is inherently suspect, and presumptively invalid.”

In City of Richmond v. Croson in 1989, the court struck down a municipal set-aside program for minority businesses as a violation of the equal protection clause of the 14th Amendment. In his concurrence, Justice Antonin Scalia quoted Harlan to argue that “the difficulty of overcoming the effects of past discrimination is as nothing compared with the difficulty of eradicating from our society the source of those effects, which is the tendency — fatal to a nation such as ours — to classify and judge men and women on the basis of their country of origin or the color of their skin.”

Thomas, as one of the most influential proponents of the colorblind Constitution, has quoted or invoked Harlan throughout his tenure on the Supreme Court.

What is interesting about the conservative use of Harlan’s dissent in Plessy is that it begins and ends with his rejection of legal caste and its assertion that the Constitution “neither knows nor tolerates classes among citizens.” Conservatives seem to be less interested in the words that immediately precede Harlan’s statement of principle.

“The white race deems itself to be the dominant race in this country. And so it is, in prestige, in achievements, in education, in wealth, and in power,” Harlan wrote. “So I doubt not, it will continue to be for all time if it remains true to its great heritage and holds fast to the principles of constitutional liberty. But in view of the Constitution, in the eye of the law, there is in this country no superior, dominant, ruling class of citizens.” What follows are the words with which we are most familiar.

If you read the most-quoted part of Harlan’s dissent in isolation, you might get the impression that this former slave owner was some kind of anti-racist pioneer. The context makes clear he was nothing of the sort.

“Whenever Harlan approvingly spoke of equality, it was always in its legal manifestations,” the legal scholar Phillip Hutchison observes in a 2015 analysis of the Plessy dissent. “Collectively, it is clear from all these references that, for Harlan, the legal equality of which he spoke operated in isolation, as in the opening sentences of the passage, he made the social inequality between Blacks and whites unambiguous.”

Harlan’s brief for the colorblind Constitution rested, within the text, on a belief in the inherent superiority of white Americans and the basic inferiority of their Black counterparts. “Blacks and whites could be ‘equals before the law,’” notes Hutchison, “but that did not mean they were equals in any other respect — in the social realm, racial inequality would rule ‘for all time’ if the nation remained blind to race ‘in view of the Constitution.’ ”

There’s another passage in Harlan’s dissent that seems to suggest an egalitarian worldview. “In respect of civil rights, all citizens are equal before the law,” wrote Harlan. “The humblest is the peer of the most powerful. The law regards man as man and takes no account of his surroundings or of his color when his civil rights as guaranteed by the supreme law of the land are involved.”

But, as Hutchison points out, this is just another statement of Harlan’s indifference to social realities as it relates to the law. “The same implication applies,” Hutchison writes, “just as people could be ‘equal before the law’ while stunningly unequal in every other respect, the law could still ‘regard man as man’ whether that man resided in a mansion or a cardboard box.”

Douglas S. Reed, a professor of government at Georgetown, writes that Harlan’s dissent “rests in part on a legal formalism that forces public actors to hold reality at bay, to ignore the lived realities of inequality within the American experience, and to grasp onto a strict separation between public, legal categories and private, social standing.”

It’s as if Harlan is restating, with no apparent irony, Anatole France’s quip that “the law, in its majestic equality, forbids rich and poor alike to sleep under bridges, to beg in the streets, and to steal their bread.” The law may not recognize social distinctions and material inequalities, but social distinctions and material inequalities still exist and shape the way the law is felt by both groups and individuals.

The crux of Harlan’s dissent in Plessy is a practical disagreement with the majority. Unlike the other justices and the State of Louisiana, he does not think that it is necessary to segregate Americans by race in public places to maintain white racial supremacy: “Social equality no more exists between two races when traveling in a passenger coach or a public highway than when members of the same races sit by each other in a streetcar or in the jury box.” The argument that social equality could exist “between the white and black races in this country,” Harlan says, “is scarcely worthy of consideration.”

Legal segregation, he adds, will only “keep alive a conflict of races” and delegitimize the United States in the eyes of the world: “We boast of the freedom enjoyed by our people above all other peoples. But it is difficult to reconcile that boast with a state of the law which, practically, puts the brand of servitude and degradation upon a large class of our fellow citizens, our equals before the law.”

As Harlan saw it, the “white race,” as long as it “remains true” to the “principles of constitutional liberty,” did not need Jim Crow. It would remain “the dominant race” “in prestige, in achievements, in education, in wealth, and in power.” The colorblind Constitution would do as much as legal segregation to sustain supremacy, without the risk to order or international prestige.

Conservatives will surely disagree with this assessment of Harlan’s dissent. So will others. Even if Harlan was no egalitarian, his words were used by generations of civil rights activists in their war against legal caste.

But the text is the text, and Harlan the sophisticated defender of white racial dominance is more consistent with his life, his views (as a younger man, he initially opposed the Reconstruction Amendments) and his jurisprudence (three years after Plessy, he would uphold a system of school segregation that taxed Black families for the exclusive benefit of white ones) than is Harlan the anti-racist.

More important, to read Harlan’s dissent in full is to see why it was so readily embraced in the age of opposition to efforts to redress racial inequality and past injustice. As Harlan knew, a colorblind Constitution could do as much or more to preserve a hierarchical and unequal society as laws designed for that purpose.