Testing Technology as a Scientific Research Program
A Practical Exercise in the Philosophy of Technology
Testing the claims made on behalf of particular technologies has become increasingly important to scientific civilization as human agency comes to rely more and more on technological innovations. Many technologies are introduced as a result of scientific research. Only a small handful of these technologies — those which prove themselves to be the most efficient and robust — are engineered for industrial application and mass adoption in the consumer marketplace. Thus testing what technologies work, and which work best of all, is not merely a question of whether or not they work, but also of how well they work, and how well a technology works may well be contingent upon knowing how exactly that technology works.
But do we always now how a technology works? I discussed this problem Tinkering with Artificial Intelligence: On the Possibility of a Post-Scientific Technology, in which I suggested the possibility of a post-scientific technology, growing out of the scientific research programs that ultimately have their origins in the scientific revolution, but having been pushed beyond this research program by the rapid development of technology, allowing us to do things with technology that we do not understand scientifically — or, at least, our understanding is not exhaustive and so our scientific explanations are not instances of ontological reduction.
I take it to be an open question as to whether we have yet produced a technology that we do not understand scientifically. Here we need to first make the distinction between technologies that are not understood by someone, and technologies that are not understood by anyone. Most people do not know how the technology they use every day works, and so these technologies in everyday application are not understood by someone. This is common, but the question I am getting at is whether there are, or could be, technologies that are not understood by anyone, including those who engineer the technologies in question. Obviously, if someone constructs some technology they will know something about that technology, so that their ignorance isn’t total, but at the same time their knowledge is not necessary exhaustive.
Let me give some examples of what I mean. In recent years we have seen scientific disputes emerge over the efficacy of the testing of technologies, ranging from quantum D-Wave computers — what exactly are they doing, how are they doing it, and are they really any faster than conventional computers? — to nuclear fusion, whether cold or hot. There has also been a great deal of disagreement over the efficacy of the EmDrive thruster, and especially over how it has been tested to determine whether or not it does what its inventors claims that it does. Those who assert that the EmDrive does not work and cannot work are straightforward on why it cannot work: it violates conservation of energy and conservation of momentum. Those who assert that the EmDrive can and does work have given various accounts of how it works (the original concept is due to Roger Shawyer, so his should be considered the definitive account), but they have not yet produced an entirely new physics in which conservation of energy and conservation of momentum are either modified to allow for the EmDrive or eliminated altogether.
Tests of EmDrive prototypes have yielded differing results, and these differing results have often been attributed to measurement errors in the test, but testing itself can be made the object of scientific methodology and scientific scrutiny. Scientific controls and scientific standards of measurement can be employed in the testing of technologies, whether or not we consider the activity of testing some technology to be science per se. We may choose to understand testing as a kind of science, in which case each test of technology is an experiment. The testing of technology would then be a scientific research program, vulnerable to Kuhnian paradigm shifts. There is nothing wrong with this point of view, even if it is unusual to think of such activities as constituting a science. Suppose we do think of testing a new technology as a kind of science, then. Is a scientist obligated to state exactly how a technology works in order to be able to effectively test that technology, to determine whether or not it functions as predicted? Or is it the function of the test of a technology simply to determine whether or not it works, whether or not it fulfills its stated function, and not how it works?
If we think about this question, I think that we need to admit that it is much more difficult to answer than it appears at first. When conflicting claims are made in regard to how exactly a technology does what it does, it would seem that the final judgment must be rendered on the basis of whether or not the technology actually does do what it claims to do. This ought to be very simple, as sophisticated numerical methods and quantitative measures are available to us whereby we ought to be able to make an objective judgement regarding the technology. Nevertheless, the testing of technologies is more difficult that it appears at first. It is, first of all, ambiguous to speak in terms of what a technology claims to do. We should be able to disambiguate this, however, without too much trouble. There is the end that the technology serves, and this is one thing that a technology “does,” while there is also the means by which a technology accomplishes its end, and this is another thing that a technology “does.”
This ends-means analysis ought to clear things up a bit, but rather it makes us see that the philosophy of technology can be as difficult and as problematic as ethics, since philosophers have been debating means and ends in ethics for thousands of years without any clear conclusions being reached. Given the different positions that can be taken vis-a-vis means and ends in ethics, we can distinguish teleological and deontological conceptions of technology, with the teleological conception uninterested in the means as long as the end is attained, while the deontological conception holds the end to be attained only when the means are known and are consonant with the end, and perhaps also conform to ideal moral principles.
A purely deontological scientific research program of testing technology would constitute what we would otherwise call a proof of concept, but where we do not know the concept involved, i.e., the concept to be tested, and we have only a device of which we must determine its fitness to some end, there can be no proof of concept, therefore no purely deontological test. What examples are there of technological artifacts whose means are not understood but there is a device available to test? There is the EmDrive thruster mentioned above, but here I will consider two other examples: the Koren helmet and nootropics.
There is an ongoing controversy over the efficacy of the Koren helmet, more commonly known as the “God helmet” (which I also discussed in Tinkering with Artificial Intelligence), designed by Stanley Koren in order to technologically induce altered states of consciousness, frequently experienced as religious forms of consciousness, in the mind of a test subject not otherwise prone to altered states of consciousness. Other scientists have been unable to replicate the claimed effects of the Koren helmet (failure of reproducibility is a common problem in science, not confined to the testing of technology, and should be addressed in any rigorous scientific methodology); different individuals report different experiences, and at least some cases seem to be an example of the placebo effect. We have good research to demonstrate that even when individuals are informed about cognitive biases, their cognition is still skewed by bias, and so we would expect that, even if a skeptical test subject submits to the Koren helmet fully knowing that donning the helmet may result in a placebo effect, they may still experience a placebo effect. Freud once wrote that it is difficult to deal scientifically with feelings; one might say with equal justification that it is difficult to deal scientifically with experiences.
Another area of research in which we must deal with feelings and reports of subjective experiences are the testing of nootropics, or “smart drugs.” An article on “smart drugs,” Your Friendly Guide to Nootropics by Spenser Davis, notes:
“…so-called ‘smart drugs’ are also known as nootropics. The psychologist and chemist C.C. Giurgea is credited with coining the term, which comes from the Greek nous, or ‘mind,’ and trepein meaning ‘to bend.’ Giurgea synthesized his first smart drug, called piracetam, in 1964. Although he did not claim to know the mechanism, he believed it boosted brain power and began exploring the idea of nootropics.”
Claims made on behalf of nootropics are notoriously vague, and if the mechanism by which a test subject is made “smarter” the vagueness escalates to the point where we have to ask if any substantive claim can be made or tested. If a drug makes a person smarter, this should be quantifiable, but many claims for smart drugs are of the kind such as, “increases well-being” or “eliminates jet lag.” Also, there is little confidence in the objectivity of testing, as in conducting IQ tests. If IQ tests were reliable and effective, one could administer a nootropic, taking an IQ test before and after the drug, and note the difference in the scores (of course, one would hopefully do this with a population, and not merely an individual, and also employ a control group). If there were a statistically valid improvement in IQ scores after the use of a nootropic, then this would constitute quantifiable evidence of the efficacy of such drugs — but only if we trust IQ scores.
IQ tests themselves might be considered a testing technology intended to measure general intelligence, but since we do not yet understand what general intelligence is, we are testing something we don’t understand. If IQ tests were administered to some artificial general intelligence that functioned by way of neural nets that had been trained for the task of taking IQ tests, then we would be using an IQ test to test a technology that we don’t fully understand (i.e., our understanding is not exhaustive, even if we know how to build a neural network). While a computer program can be exhaustively disassembled and its constituent parts laid bare (so that the performance of a program is subject to exhaustive ontological reduction), this is not the case with a neural network, which we can take apart, but the function of which can remain mysterious to us, just as we can dissect a brain and still no know exactly how the brain functions or how consciousness is related to brain function.
One can posit a conservative interpretation of technology testing according to which no technology should be tested until we can state exactly how it functions in accord with a fully elaborated scientific theory, so that the function of the technology can be tested against the theory that makes its function possible. If we pursued such a rigorous and conservative protocol for technology, we would refuse to test many technologies, essentially on moral grounds, and I think that this would strike many as absurd, or at very least as unhelpful. It seems a lot like those who refused to look through Galileo’s telescope on principle. One would certainly be accused of rigidity and being closed-minded. Tests would then be made by the less rigorous and the less competent, and the result would be further confusion rather than enlightenment.
A truly rigorous scientific research program in the testing of technology would probably also be attacked as elitist, since only those technologies would be tested that came out of the newest and most sophisticated advances in science, and the newest and most advanced science usually comes out of elite educational institutions. We have become so accustomed to the STEM cycle and the emergence of new technologies from scientific theory, that the very possibility of a technology emerging from tinkering, that the tinkerer cannot explain, other to say that it works, strikes us as improbable in the extreme, and very possibly a waste of time. That is to say, even to test the claim made by the tinkerer would be a waste of time. And yet, the steam engine, which was effectively the trigger of the industrial revolution, was more a product of tinkering than of science.
What is needed is a rigorous way to test non-rigorous claims, in so far as we can consider the claims of achieving some end without knowing how it is achieved is the definition of a non-rigorous claim. And this may not be possible, though to say why this may not be possible will not be easy, as it involves an implicit appeal to a principle with much more general import than any parochial concern of a technology testing program. I am not prepared with an exposition at this time, nor can I yet even formulate the elusive principle that I have in mind, so this will have to be an inquiry for another time.
