I am currently reading Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society by Jim Manzi. I found out about the book after reading philosopher Gary Gutting’s op-ed “How Reliable Are the Social Sciences?” from a few years ago. Gutting argues that social science research doesn’t match up to natural science research, and he quotes part of Manzi’s book. Here is the quote in full:
When it comes to deciding what policy actions to take, we should listen carefully to economists and other social scientists, but we should treat their assertions differently than we do scientific predictions. Their predictions should be subjected to useful cross-examination by laymen, weighing of technical and nontechnical opinions, introspection concerning human motivation, and all the rest. Beyond this, we should always keep in mind the unreliability of such predictions and treat the fog of uncertainty about the potential effects of our actions when considering what to do. I am not arguing that social science is valueless–I would no more advise a president to make a major economic decision without professional economic advice than I would suggest that he make a decision about war and peace without consulting relevant historians–but I am arguing that we should be extremely humble about our ability to make reliable, useful, and nonobvious predictions about the results of policy interventions.
These limitations are inherent to the methodology of nonexperimental social science.
I have no interest in undermining social science research, as I would like to continue working as a sociologist, but I think humility is important. Skeptics and atheists have two tendencies when it comes to social science research. They either dismiss it outright, holding to a narrow view of what counts as science, or they treat all research as if it has the same reliability, regardless of the discipline or the methods used. I found that many more skeptics/atheists held to the latter position than I would have assumed. On a number of occasions I have had to say that being skeptical of social science research is not the same as denying climate change science. It’s okay to wonder about the reliability of social science research and withhold an opinion.