academics (and all of us) should trust published research less and arguments more
you can't just say "this paper was published so it's true"
arguments over authority
I used to be a “trust the research” guy. Now, not so much. These days, I find arguments I find persuasive and trust those. Arguments should be intuitively plausible, have solid empirical backing from a variety of sources, and be robust to counterarguments or competing explanations.
I want to highlight an exchange on Twitter debating the extent to which NAFTA related job losses caused the rightward political shift. My goal here is not to wade into the debate (although I will offer something of an opinion at the end), but to talk about how we debate and evaluate evidence.
Matthew Yglesias registered some skepticism about the NAFTA hypothesis, and economist Arin Dube replied by saying there was an American Economic Review (AER) paper showing the link is real, as if that settled the debate. I’ve included some of the back and forth below here.
In my past life as a grad student, I would have been like “Oh, Arin is totally right there is an AER paper.” But now, I see an argument-from-authority like this as a point against the hypothesis. Ideally, when quoting academic research you should be able to summarize the argument for the point from the research and present that, with a link to supporting evidence. So, for instance, “according to this paper the rightward shift was only observed in X places” or something.
making the sausage
Academic papers certainly have a lot of words. But the arguments themselves can often be presented in a paragraph or two and the key empirical analysis in a few charts or tables. One of the skills you learn as a grad student is how to ignore most of the words in published papers and how to find the actual argument. When distilling articles down to these key points, the result is often something you could present in a medium-sized blog post. The code to generate the analyses may be quite large, though.
The reason academic articles are so long is twofold
To succeed as an academic, you need to convince other academics to let you publish your stuff, which comes with requirements to obey a norm of writing a lot and citing a lot of stuff
Academics are not like maintaining production code or anything, so the main way to do something is to make articles longer to signal deeper erudition
These norms are mostly orthogonal to an empirically true argument. You can certainly make a good argument that obeys these norms, or not. But succeeding in following these rules is not itself evidence of a true argument. We need to read the argument and evaluate it on its own merits, something academic peer review does not really select for—peer review selects for consistency with past arguments, agreement with current disciplinary norms/trends, and following the rules in terms of form and formatting.
Let’s zoom in on the paper cited by Arin Dube: “Local Economic and Political Effects of Trade Deals: Evidence from NAFTA” by Choi, Kuzimenko, Washington, and Wright (pdf link). The paper is 112 pages long. Certainly a lot of words, and I feel for the poor economists that had to put so much verbal padding around their core argument, which is a really plausible hypothesis that I’m happy they investigated.
Let’s think about the mechanics of a paper like this: contemporary social science paper like this is really a lot of coding, the core results are primarily expressed in math (theory) and code (execution). Academic peer review certainly scrutinizes the math, but nobody is checking the code—and certainly no programming expert is checking the code.
Let’s also think about the authorship: on this paper you have four economists who study a variety of things, including trade/job losses/politics. So, you have four non-coding-professionals who also study other things making an argument. By definition, work by academics is not directly used anywhere, so it’s not like these academics are maintaining statistical models of voting behavior that are used by political parties (which would be a strong signal of usefulness). Again, none of these things are evidence the argument is wrong, but it is evidence that we should reject Arin Dube’s argument from authority that because the paper was published in the AER it occupies a special position in terms of evidence. It’s one argument like any other, and we need to evaluate the argument on its merits.
One thing to be concerned about here is the code: none of the authors are coding experts (it’s not like one of them spent time as a coder before becoming an economist) and because coding errors are easy to make (particularly in the direction of your priors), hard to find, and have huge impacts—we should have some skepticism about all social science like this. In the course of my dissertation, I found some pretty substantial coding errors in sociology papers published in top journals.
This coding issue is the dark matter of contemporary social science—peer review does not check it at all, so large number of results-altering mistakes are likely just missed. Again, this does not mean the NAFTA argument is wrong but it does mean we should have some healthy skepticism.
So—we need to retire the idea that peer reviewed academic work is true by default. It is certainly a signal of a level of seriousness on the part of the researchers, but it is a mixture of seriousness of finding the right answer and a seriousness of complying with disciplinary norms/trends.
Just to throw one clearly false argument out there that got past peer review, this year Nature Communications published this transparently wrong argument by my favorite academic hack—Jason Hickel. As I detailed in past blog posts, Hickel and the entire degrowth movement are wrong on matters both big and small. In this particular paper, Hickel makes the truly insane argument that the “global north” is stealing labor from the “global south,” and the entire wrong argument rests on the idea that you can just classify all labor worldwide into “high skill” or not, and so a “high skill” person in India is identical in terms of economic output to one in the USA. Just understanding the most basic stuff about complentarities in economics will show you that this is wrong (the capital stock is a lot more advanced in the USA than India!), to say nothing of the validity of the skill classification itself.
the argument itself
So—back to NAFTA. When we hunt through the ~110 pages of this article for the argument itself, we find some persuasive evidence that NAFTA caused concentrated job losses in NAFTA-vulnerable areas. And we see some potentially persuasive evidence that the NAFTA-related job losses caused a rightward political shift, such as this plot.
But the analysis is ecological—at the county level. And the NAFTA exposure map looks like this.
So—are we telling a story about NAFTA related job losses in the upper midwest? Not really. We’re telling a story about the Old South and rural areas moving right. And like, this could be related to NAFTA or it could be related to contemporaneous cultural trends such as Southern Democrats and rural voters generally voting for Republicans on cultural grounds. The authors don’t really do much to disentangle this—they don’t test their explanation in a regression with competing cultural hypotheses.
This doesn’t mean the NAFTA story is wrong—it could be part of the puzzle. But it could also be a story that is mixed up with other trends. This paper provides evidence for one story. But it does not do work to rule out other stories. And so it’s not definitive evidence like Arin Dube claims even though it does provide some evidence that we should take seriously.
Just to lay my cards on the table here, I think the NAFTA story has some explanatory power but I don’t see a super compelling operationalization in this paper. Compared to what we do in professional, full-time political analysis, a county-level analysis that does not even attempt to control for competing political explanations is suggestive but far from definitive.
As I have tweeted many times—and been blocked by academics for many times—I would urge academics interested in doing political analysis to talk to full-time political practitioners in the exploratory phase of their research. Failure to do this is how you get stuff like Medieval studies PhDs writing books on how to do climate messaging that rehash ideas people have (unsuccessfully) tried to make work for 30 years.
The reality is that contemporary social science is going through a period of fusion—political, sociological, and economic are all coming together. Researchers need to embrace this breadth, draw on the expertise of each field, and check in with practitioners regularly. In the process, we need to have humility about the certainty of our conclusions and learn to weigh evidence from different sources (academic, industry, government, etc) effectively. It’s tough, but we can do it.