We can start by calling out the way you judge history by 21'st century standards. That’s daft. All history must be understood by the standards of the day. If a person owned slaves at a point in time when it was legal and considered moral to own slaves, then that person cannot fairly be judged immoral.
The concept of “originalism” does not have a solid definition; Scalia’s was just one. In its essence, “originalism” is simply the belief that the Framers of the Constitution, in a rare moment of time when a group of people who were knowledgeable of political history all got together to AGREE on a set of governing principles, provided guidance via a governing document which should not be changed or amended lightly. Ergo, it is the polar opposite of the view held in moonbat circles that the Constituion is some sort of “living document” which should be viewed as a quaint anachronism which can be put aside in order to obtain a “correct outcome”.
Kagan was right; everyone SHOULD be an “originalist” in that they believe that the Constituion contains valuable wisdom which is being transmitted down through the ages in order to avoid the collapse of the Republic.
You don’t have to try to divine what the Framers meant when they tell you, either through history or through The Federalist. Let’s start with the 3/5ths rule. The Framers had a problem when writing the Constitution, because they had to choose between having ONE nation, or see the result of their Revolutionary War efforts result in TWO nations, over the issues of slavery. The Southern states had created an economy which would collapse without slave labor, so they were incalcitrant about their demands that slavery be legal.
Ultimately, the people who were AGAINST slavery promoted, related to representation, the 3/5 rule, which prevented the slave states (who wanted each slave to count as a full person) from gaining majority representation. If not for the 3/5 rule, slavery probably would have persisted for another 20–30 years in the US.
Now, what’s the point behnd all that history? To point out what should be obvious — that there was complication behind political positions which complicates the ability to divine the “intent” of the Framers.
Trying to pigeonhole each into “racist” or “nonracist” categories is, most of the time, not possible.