Work in Progress: The Replication Crisis
Friday 04 March 2022
Studying civilization is one of the great interdisciplinary challenges. There are other objects of knowledge that pose similarly great challenges, but I am less familiar with these, so I will stick with what I know best. This was brought home to me after sending off last week’s newsletter, when I came up with a better formulation of what one of the points I wanted to make last week. In last week’s newsletter I wrote this:
The structural differentiation of the institutional structure of civilization means that the conceptual framework, economic infrastructure, and central project start pulling away from each other so that social institutions once bound up in each other are increasingly loosened from each other and begin to define themselves autonomously. Institutions like education tend to become independent industries and those who populate these institutions are entirely creatures of these institutions, identifying more with the institution than with the central project, which seems increasingly distant and irrelevant. The same thing is found with the particular institutions that constitute the economic infrastructure, which increasingly sees itself as the engine that moves society forward, and does not want to be subordinate to the central project of any of the institutions of the conceptual framework.
Another way, and perhaps a better way to express a related idea, would be like this:
Structural differentiation is intrinsically a form of detachment, i.e., the detachment of institutions from each other; it is likely that individuals who populate institutions undergoing structural differentiation will follow the institutional detachment from the individuals who populate the other institutions from which one’s own institution is becoming detached.
How might this idea be expressed with greater clarity and rigor? An adequate formulation of this idea would draw upon both sociology and psychology; what is needed is a sociological psychology or a psychological sociology — in other words, social psychology.
I’m not interested in social psychology for its own sake; I am interested in how social psychology could be a piece of the puzzle when putting together a picture of human life as large as civilization. If social psychology had produced some applicable concepts and reliable results, these would be incredibly helpful in a science of civilization. One wouldn’t need to re-invent the wheel, as it were. But there are serious problems with social psychology, and this serious problem has a name: the replication crisis. Other social sciences also suffer from the replication crisis, and psychology suffers more than the other social sciences, with social psychology suffering the most, with only 23 percent of published social science research being replicable. You have less than half of the likelihood of a coin toss that some piece of research in social psychology is replicable. This is not a situation to inspire any confidence in the work of social psychologists, and that leaves people like me, who would like to use the work of social psychologists, in a bind.
After coming to the above realization I did some casual reading in the replication crisis, especially as it has been discussed in psychology and particular in social psychology. Of the articles I skimmed, several suggested explanations of the source of the replication crisis, but none of them even came close to the explanations I have suggested in recent newsletters, viz. that psychology itself as a science is suffering the consequences of its original sin, which original sin was to have been conceived in conceptual confusion. (Quine once said that modal logic was conceived in sin — the sin of confusing use and mention.)
The Replication Crisis in Psychology gives four arguments for the replication crisis: 1) falsification of data (i.e., fraud), 2) small sample sizes, 3) shifts in attitudes due to a difference in time or location of the research in question, and 4) failure attributable to the attempted replication. Why is the replication crisis centered on social psychology? has a different take, with a number of observations on the research methods of psychology and its data sets, but, most importantly (as I see it), this: “hypotheses in psychology, especially social psychology, are often vague, and data are noisy. ” A comment on the above is also good: “small noisy experiments are tolerated and accepted.” The charge of vagueness comes close to recognizing conceptual confusion, while the charge of tolerating vagueness comes close to acknowledging corruption in scientific institutions, and also on the part of individual scientists.
However, none of these explanations of the replication crisis are especially foundational; one might acknowledge them, attempt to correct for them, and then go on with doing psychology in the conventional way, with barely a hesitation or a bump in the road. Vagueness can be clarified and noisy data can be sifted and narrowed down to paradigm cases. Perhaps this is the best course of action: business as usual, with some improvements. To mention Quine again, leaving psychology untouched means leaving most of what is near to the center of the web of belief undisturbed, and this is a reasonable scientific bias to have. Revolutions in science are only to be effected with good reason, and not on a whim. Just as Carl Sagan said that extraordinary claims demand extraordinary evidence, so too extraordinary changes to the web of belief demand some extraordinary exigency to be addressed. Very well. One could reasonably argue that the replication crisis is indeed an extraordinary exigency calling for extraordinary changes to the web of belief.
Even if we see a need for a reform of the whole of psychology, the reform of the whole of psychology will be an intellectual undertaking for the longue durée; it is not something that can be done by one man, nor could it be accomplished in one lifetime. Perhaps, after the initial age of the growth of scientific knowledge — say, the modern period from the scientific revolution, up to the present day, until some date in the future when this initial paradigm of science will have been exhausted, and nothing more is to be derived from the conventional scientific research program — there will be a grand revision of the whole of science, in which the mysteries of cosmology, the conflict between general relativity and quantum theory, and the replication crisis in psychology will all be subjected to the necessary radical critique, and the all the sciences will emerge purged and stronger for the critique afforded by this grand revision. But this will not take place in our lifetimes; no one reading this will live to see such a grand revision of the whole of science since the scientific revolution. This will be the work of a future civilization, different from our civilization, driven by motivations flowing from a central project that we cannot today imagine. (What I have called a properly scientific civilization — a civilization that takes science as its central project — would be capable of such a grand revision, but I cannot imagine any civilization short of properly scientific civilization undertaking the necessary and painful work of self-critique.)
Where does that leave us in the present, with our needs for a social psychology but the lack of reliable results in social psychology? Given what we have learned from the replication crisis, not only ought we to be skeptical of any results in social psychology, and be rightfully hesitant to appeal to these results, one has a moral duty to question these results, and to rejection them when they fail to reflect reality.
I employ a concept like structural differentiation because it expresses something that I was already getting at, so that I was predisposed to be sympathetic to an idea that already expressed what I was imperfectly converging upon. In other words, structural differentiation reflects reality in a way I was already trying to capture, when I found that it already had been better formulated by others; the concept was preadapted to a perspective I had already attained. Needless to say, the opportunities for going wrong in assessing the fidelity of a concept to reality are legion.
In surveying a field that is ridden with crisis and scandal, the best one can do is to make one’s own judgments of the material, treating it as suggestive rather than as definitive. Thus I can go to social psychology to mine it for ideas, but I am not bound by a highly integrated discipline that must be accepted as a whole or disregarded; one has a certain license to pick and choose. This could be understood as a license for eclecticism, and an invitation to unrepentant confirmation bias. Certainly this could be the case, and here is where we encounter the uncomfortable truth that, as far as science has advanced in its development up to the present day, it is still so imperfect that the character of the individual is in some cases more important than “settled science” and institutionalized knowledge, which latter are artifacts of social consensus and thus vulnerable to social demands and political pressure.
The concepts of social psychology must present themselves before the bar of common sense, however flawed and vulnerable common sense may be. We, all of us, since the scientific revolution, exercise a kind of augmented common sense that has its origins in an acute awareness of the failings of common sense, the truth of counter-intuitive ideas, the ever-present reality of cognitive bias, and all that science has done to point us in the direction of knowledge that transcends common sense. Common sense fully aware of its own limitations is an improvement over naïve common sense, but it is no more infallible than science of which we have learned to be skeptical.
But while we all have the opportunity to exercise this augmented common sense, not all are equal in its application. It is the moral character of the individual that is the decisive factor in such circumstances of uncertainty, unknowns, and unprecedented circumstances. And this decisive role of the moral character of the individual is not a fluke, not an arbitrary correlation. Science as an ideal (though not as an institution) requires of its practitioners a selfless effacement before the world that is of the same character as moral virtue. The most honorable action possible under given circumstances is not likely to be the easiest course of action, but if we understand why it is the honorable course of action, we understand why it is worthwhile to take on a difficult and demanding undertaking. Similarly, the highest ideals of science are likely to require of us more effort than the slipshod path, but if we understand why we are doing what we are doing, we understand why it is worth it to do the right thing. In both cases, egocentric bias must be overcome and the preference of Homo economicus to lighten one’s labor must be seen as the fons er origo of corruption.
