Monday, 19 August 2013

Defending Psychology in the Science Wars - Part 1: The Phantom Arguments

If Psychology is ever to become anything more than a mere aggregation of opinions, it can only be by the establishment of some datum universally agreed to.

– Herbert Spencer (1855, p. 8)

Yes, I will defend that psychology is a genuine scientific endeavour and more than a mere aggregation of opinions against people who recently claimed otherwise (e.g. here and here ). I will do so mostly by agreeing with their critiques in part 2 or part 3 of this Science Wars saga (there may even be droids and furry animals involved at some point).

More important right now is retreat, regroup and damage control: The defence against the claim of the hard sciences has been so poor in my opinion, these responses have only strengthened the proclamation by the math-o-philes that Psychology isn't science! 

For some reason the defence was led by a platoon of social psychologists (e.g. here and here and here and here), a subdiscipline of psychology that is not know for its calm and erudite response to fundamental criticism of their scientific claims (e.g. here and here).

I am ashamed, really ashamed to have to read the nonsense raised again and again as arguments in defence of psychological science. I will, in great detail, explain why those arguments are nonsense and why they are something to be ashamed about (Summary: do not use the “not presently testable” argument; Meehl, 1990)

First, to illustrate the origins of my objections against this crippling defensive strategy, a short fiction about the future of the discipline of psychology...


Not so long from now, in a barred spiral galaxy some 120.000 light-years in diameter...
Psychology lost the Science Wars in what is now known as "the great battle of versimilitude" in which the accumulated knowledge of all of science was at the disposal of  competing disciplines to prove the truth-likeness of their communities' scientific claims in battle. Soft science's arch-enemy, the natural sciences turned out to have developed theories and methodologies that actually allowed them to exploit the lawful structure beneath the chaotic and complex mess that makes up most of the observable universe. They could build weapons, spaceships and synchronise with their GPS satellites using general relativity, you know, the stuff you may need in a war. None of the soft sciences were able take advantage of that kind of knowledge, because it was written in a language they had heard about in high school, but never understood what it was for, you know, like German.
The soft sciences, psychology in particular, were taken by surprise. They had not realised the natural sciences (aka, "the nerds") had accumulated so much knowledge about the world that turned out to be very plausible, or at least, extremely useful. After all, scientists in all disciplines had used the same scientific method and the same rigorous methods of expert peer review to assure only the best theories were allowed to be added to the record of scientific knowledge. These procedures had shown many of the psychologists' hypotheses to be true, mind you, all their scientific knowledge consisted of true significant differences! It was even a known fact that much more hypotheses of the soft sciences had been found to be true when compared to the natural sciences. Apparently such high numbers of corroboration of theories in publications doesn't really give you an advantage when it comes to a clash of such claims with reality.
The psychologists fought bravely and found allies in the sociologists after the economists had finally given up the rational consumer model and joined the nerds in their masochistic math fetish to create more realistic models of extremely complex phenomena. The soft science alliance had thrown every plausible theory they had at those foul smelling nerds; from performing an n-back task for a couple of minutes before making complex strategic decisions, to priming the enemy troops with words and images associated with old-age in order to make them walk more slowly. As a last resort and in front of many stupefied enemy guards, a representative sample of soft science POWs attempted a replication of the Stanford prison experiment... to no avail. 
Many psychologists went into hiding after the war, taking on new low-profile identities such as novelist,  science-writer or tv-personality. Soon, an underground movement of veterans of the soft science alliance started to meet in secret. Here's what transpired at a typical private meeting of the SSA (Soft Scientists Anonymous): 
"My name is Fred Hasselman and I am a psychologist" 
    "Hi Fred!" Replied the room. 
"I have been using the scientific method to study psychological phenomena for more than 15 years now..." 
- applause - 
"... and me and my co-authors believe it's all got to change, because we produced a whole bunch of nothing as a discipline! Pseudo-intellectual garbage! Empiarrhea, clogging up the empirical record with so-called facts that have no say whatsoever at the level of appraising and amending theories! As long as we cannot decide between the veracity of a plethora of theories put forward to explain the same psychological phenomenon, we are in big trouble as a discipline of science!" 
 - silence -  
"You don't look like a psychologist to me!" shouted a prof. from Harvard. "Are you sure you didn't study physics or something? Spying for the nerds, eh?" 
- Soon the entire room turned against the speaker and his co-authors - 
"Or worse, maybe he's a philosopher!" a European PhD student yelled. "You obviously don't know anything about the complexity of human emotions... what about love? Appraise that with your formal system!"  
And all the speaker and his colleagues could do was to start their own "society for psychologists who think just like we do", together with a journal of approximately the same name in which they published their rebellious and provocative papers that none of the other psychologists ever cited.... until they were discovered by the nerds...
 [to be continued] 

What occurred at the fictional SSA meeting has occurred many times in the history of psychological science. It still occurs to right now, to scientists who submit manuscripts to scientific journals, but get told by reviewers and editors that their work is "too difficult" for the "average psychologist". This usually means that some maths, some formal language or some ontology unknown to the reviewers was used in the manuscript. This is the most unscientific rejection notice imaginable. If these scholars were submitting to a popular science journal, or were being interviewed for a newspaper article, sure, then you can be "too difficult", but in a genuine scientific journal? It should not be possible to be too difficult for science!

That's not even my main point, these same "average psychologists", when attacked by a physicist for not being scientific, will not hesitate to point out that psychology is waaaay more difficult than physics and they are just doing the best they can.

It's not true, psychological science is not doing the best it can and it hasn't been doing so for a long time. I'll get back to that story at a later point in the saga ("A tale of three principles").

In this part, I'll discuss the phantom arguments used to ward off the phantom menace that apparently are the natural sciences and hope they shall never be used again when someone criticises scientific psychology.

1. Is psychology the hardest science? - Newton's Curse and the Donders Legacy

There will not be a physicist who denies psychology is the hardest science.
Interestingly, ask a physicist "Why is psychology more difficult than physics?" and you'll get pretty much the same answer from each of them. Ask a psychologist and... well... most of the other arguments discussed here concern such answers.

The only correct answer is: Because physics studies dead matter systems and psychology studies living matter systems. The problem is the persistence of mainstream psychology to study the brain-body-environment system as predominantly linear, ergodic, component-dominant and open to efficient cause. Behaviour and cognition are conceived of as linear arrangements of specialised (invisible) causal components whose powers of causation (effects) are additive in nature and mostly located in the brain… Mainstream neuroscience is still just F.C. Donders’ mental chronometry conducted with more expensive equipment, so are most experimental designs in psychology.

In other words: Psychology is studying human behaviour based on a crude conception of Newtonian physics, completely detached from contemporary developments in physics, mathematics, biology and philosophy. Van Leeuwen (2009) recently dubbed this Newton’s Curse, which refers to:
 “[…] conceptualising causal primacy in terms of a reduction of wholes to parts, where the wholes are causally impotent epiphenomena, i.e. merely aggregates of microphysical constituents.” (pp. 38).
This is an untenable assumption for scientific inquiries into the behaviour of living systems capable of maintaining and increasing their internal complexity against the second law of thermodynamics. The disciplines of the hard sciences know it is an untenable position and when they question psychological science about it, we have to say, "you're right, so what do you think we should do?" You do not tell them to go away and shut up already.

Paradoxically, as will become clear from the arguments that follow, psychologists actually claim to study phenomena that are inherently non-Newtonian, massively context sensitive, closed to efficient cause (e.g., must be defined as non-well founded sets), but completely ignore all the impressive scientific progress that has been made to describe and study such systems and phenomena! In fact, psychologists do not even fully exploit one of Newton's important contributions to science, which is the mathematical description of change processes (difference and differential equations). Psychology does claim to study behavioural change, but those who dare to suggest the mathematics of change should be used have met a very similar fate as described in the fictional story above.

In a way the arguments listed here are a confession of a scientific discipline. It is saying: "We are using the wrong tools to study the phenomena we are interested in". Or worse: "We will continue to do so even though we know there is a science of physical complexity applied to living systems, robotic systems, artificial agents, ecosystems, complex physiological networks, massively context sensitive behavioural phenomena such as chaos and turbulence.... that far supersede the complexity of human behavioural modes. Leave us alone."

Yes, that's unscientific. I think that it is not unfair to say so, especially from the perspective of the complexity sciences where physics meets chemistry and mathematics meets biology and what not (well, not mainstream psychology obviously). It is unscientific to not question your assumptions about the object of study, to not question whether your measurement theory or rules of inference are adequate, or for that matter, to not question your philosophy of science, preferred ontology, epistemology.
Not impressed?

Ok, let me convince you this argument is really silly by making a simple observation: If psychology is serious about being more difficult than physics and serious about being a science, then why doesn't it teach the methods and tools of quantum physics and general relativity to its undergraduates? That would be a logical, scientific thing to do, because what this argument implies is that if psychology wants to advance as a science that produces knowledge about the behaviour of living systems, it will have to do better than physics and needs to learn everything about the scientific methods and tools that made that easy science about dead matter so extremely accurate and successful.

Next please.

2. "But there are also cases of fraud in the hard sciences"

Yes, that's true.

Please understand they are deeply offended by the nonsense Bem (2011) was allowed to publish as a product of state-of-the-art psychological science and subsequently deduced such a thing can only happen in a discipline that cannot distinguish between false claims even if they are based on pseudoscience or fictional data. There's also a report, drafted by psychologists that confirms this idea and I do not blame other disciplines to refuse to consider disciplines of science that allow sloppy science to continue, as a discipline of genuine science. (that's a lot of science for one sentence)

Next please.

3. Physicists also have problems measuring unobservable stuff: The interpretation fallacy
(This may be entirely about:

First, let's introduce this argument about the measurement of complex phenomena in psychology by quoting from the blog entry linked above (yes, this is literal):
"During the very first week of my Intro to Social Psychology class, I send my students home with one simple assignment — come back next class with an answer to the question, “What is Love?” I play the Haddaway song as they’re walking out the door and tell them to come back in 2 days with a way that they would define and measure “love” if they were creating their own experiments."
We learn that the students return with answers such as: rate love for someone on a scale; write a story about how much someone loves another and independently judge the story for amount of love felt by the author; use a physiological measure such as heart rate change when a loved one enters the room to index amount of love; and some surprising answers that were apparently not very memorable except for willingness to sacrifice for a loved one. The lesson learned by the students is that there is no single obvious answer to the assignment.

Now, aside from the ethics of polluting fresh young minds with 90s Eurodance (baby don't hurt me indeed!), a scientist from the "hard" disciplines and probably most philosophers of science, would point out that this assignment and/or the answers are incorrect for several reasons:

  1. The phenomenon to evidence in the experiment is love, but all the "definitions" are in terms of measurement outcomes that are attributed to the object of measurement (the participant) as a property. Property attribution by means of measurement outcomes is only unproblematic for classical physical measurements (as I mentioned earlier): Is measuring love assumed to be a classical physical measurement?  There will not be many physicists who will agree with this assumption. Moreover, apparently love is not a relational concept to psychologists? None of the students provided a formal relational definition of love.
  2. An experiment is a measurement context in which a phenomenon may be observed that was predicted by a theory about the phenomenon (+ auxiliary theories), an intricate part of the scientific method that psychology just does not care enough about. The formal scientific definition and theory of love should dictate the measurement context in which it may be observed, not the other way around. Hey, there even may be competing theories of love! For instance, using heart rate needs a postulate linking increased heart rate to love (and much more). It is reasonable to consider in advance that increased heart-rate may be a necessary condition to evidence love, but definitely not a sufficient condition as there are many things that are associated with increased heart rate. So why should this be a reasonable scientific suggestion for an experiment to evidence amount of love? Physicists seek to conduct the most risky tests of their claims about the workings of the universe they can think of. Measuring increased heart-rate can be considered as uninformative a priori, as I mentioned earlier: Strong inference is how you evidence your scientific claims (or see Hasselman, 2013).
  3. Not all the suggested measurement contexts concern a true experiment.

Ok, maybe you think that is all nitty gritty nerdy science stuff and this is just simplified for the sake of a blog post and in reality the psychologists know better. Well, this text follows the paragraphs about the answers of the students (Dr. Berezow is the evil physicist who dared to attack psychology):

"Each and every semester, my class of 100 sophomore-aged undergraduates immediately comes to understand what Dr. Berezow has apparently yet to learn:Measurement is complicated. No matter what you study. Period. It’s complicated if you’re trying to explain what exactly Schrodinger was trying to say with his infamous cat problem. It’s complicated if you’re trying to figure out why there are EIGHT DIFFERENT SCALES that measure the exact same physical concept. It’s complicated if you’re trying to create the technology that might let you detect a hypothetical particle that, if found, would validate an entire model of particle physics. So why exactly are we supposed to be surprised that it’s still complicated when you’re trying to measure something abstract? Like feeeeeeelings?"

As a well trained psychologist I'll start with providing positive feedback: Good to learn "feeeeeeelings" are considered abstract concepts, that means there is hope for a formal description of feelings.

But... Really?

Is psychology really comparing the difficulty it has to decide whether to measure love on a 5 point rating scale or by judging a love letter for its love-content, to evidencing the Higgs Boson by building the LHC?

Oh boy.

The damage caused here may be beyond repair. I'll just leave the discussion of the differences between the construction of the most complex scientific instrument, "one of the great engineering milestones of mankind" that constitutes a collaborative effort involving over 10,000 scientists from 100 different countries and a Likert rating scale to rest for now. Except for this question: Do you think psychology would be able to organise the same amount of resources, effort, devotion and collaboration from the community of psychological scientists and cousin disciplines, funding agencies and policymakers from so many different nations to evidence a theoretical construct proposed by a theory of psychology?

I'm open for suggestions on what that might be.

In the meantime, the other comments are not helping either, because they do not evidence measurement problems, but are solutions or corroborations of theories with high verisimilitude. For instance, how does the existence of different scales used to measure any quantity counter the critiques of measurement in psychology? There are more than 13 fundamentally different ways to calculate Avogadro's number and that fact alone was seen as evidence that atoms existed (e.g., J.B. Perrin's argument, 1909).  Temperature, like pressure and entropy, are thermodynamic quantities defined as collective variables observable at the level of an ensemble of particles (a single particle does not possess the property temperature or pressure). Incidentally, the thermodynamic laws and quantities that operate at the level of ensembles are linked to theories of particle mechanics through statistical mechanics. So it naturally follows from formal definitions and theoretical connections and unifications that there are many different ways to express the temperature of an ensemble of particles, just by using different kinds of matter that have different properties (e.g. phase transitions). These different scales can all be transformed into one another, they all measure the same thing! This corroborates the (3) scientific theories involved, strengthens their veracity, it doesn't contradict them!

Let's transform heartrate change into the Likert scale and back. That's difficult because love is not formally defined by a theory that is linked to a proper measurement theory that is able to deal with interaction between the object of measurement and the the measurement instrument. Interaction such as between a "true attitude" and asking a student to rate the attitude on a 5 point scale in a cubicle in a psychology laboratory. Physics created a theory of preparation and measurement that is able to deal with such things, why doesn't psychology do the same?

The thought experiment about Schrödingers cat is about the interpretation of measurement outcomes. Whether you interpret the outcomes in terms of descriptive explanations of sensory experiences, that is, your everyday experience of reality, is irrelevant for the precision and accuracy of a scientific theory. It does not matter whether Schrödingers cat "really" is in a state between life and death or not, as long as the universe behaves according to the theory that needs such a concept to make accurate predictions that can be verified empirically. To believe otherwise is to fall prey to the interpretation fallacy in theory evaluation and many psychologists do. There are over 15 different interpretations of quantum physics, but they are all interpretations of the most precise and accurate scientific theory about the structure of reality ever produced by human minds.

Heisenberg's uncertainty principle, entanglement, superposition, duality, they are all precisely, formally described theoretical concepts that were conceived by scientists in order to build better theories about the phenomena observed in the quantum world. The isomorphism of the mathematical description of light and matter as either waves or particles was one of the profound insights that gave rise to quantum physics: Waves and particles are two sides of the same structural coin and many more such dualities exist.

Still not convinced? 

Here is the most accurate confirmation of a prediction of a measurement outcome (!) by a scientific theory: The existence of an anomalous electron magnetic dipole moment (which is anomalous to the Dirac equation) by Quantum Electrodynamics (QED). According to the most accurate calculations the anomaly, when measured in a metal trap with a cylindrical cavity whose resonance structure is known, should have a value of: aµQED = 11659180.4(5.1) x 10-10 (Aoyama, Hayakawa, Kinoshita, & Nio, 2008; Hagiwara, Martin, Nomura, & Teubner, 2007). The most accurate empirical measurement of the predicted value of the anomaly in the described measurement context is: aµexp = 11659208.0(6.3) x 10-10 (Bennett et al., 2006). 

Such examples of empirical accuracy do not exist in any other discipline of science, moreover note that the theory predicts the entire measurement context in which the phenomenon may be observed to yield the predicted measurement outcomes. Don't you think it is worthwhile to study how the theories of physics got so accurate? How they deal with things that on closer examination may be just as complex and fuzzy as "feeeeeeelings"? That would be the scientific thing to do. If it turns out you can't use anything they came up with (which I doubt, see applications of quantum probability theory to psychological science and APA's advanced training institute in nonlinear methods), that's fine, but you cannot dismiss these successes just like that. Moreover, I am certain it would be of the utmost scientific importance to find out why the tools of physics would not work for psychological science. It would be win-win situation for science, no matter what the outcome of such an inquiry might be.

Theorising in a science should not depend on the ability of its practitioners to understand what the theory implies in terms of the way humans experience reality. The universe was not built to accommodate humans, we hitched a ride. If the theory is precise and accurate according to severe testing by repeated application of the scientific method, you have to accept it is like the theory says it is. Don't worry, history of science shows that eventually the theory will be discarded again...

That is science baby. Take it or leave it.

Next please.

4. "But psychology is a young science, don't have such high expectations of us."

This seems like a sensible argument.

It does assume two things:
  1. It is true that psychology is younger than physics as a modern empirical science.
  2. Psychological science has been frantically and rigorously attempting to improve the empirical accuracy of its theories in order to grow up to become a genuine empirical science.
I believe the veracity of these two premises can be questioned.

As for the first point, of course, Newton and Leibniz were esteemed scholars, but were they atrophysicists? They were much more than that and what to think of Decartes, was he a cognitive psychologist who studied consciousness?

I am not a historian, but I think it is not very controversial to suggest the contours of modern science, with its separate disciplines and use of the scientific method started to take form in the 1800s. We're talking about the century of Maxwell's unification of electricity and magnetism, the Lorenz transformation, the mathematical advances of Hilbert  and Poincaré, about Darwin's theory of evolution, the recognition of the cell as building block of life and its role in health and pathology (Virchow / Remak, 1855) and about the psychophysics of Wechner and Feber. There was a theory of "unconscious cerebration" that was fiercely debated in the Bristish Jounral of Psychiatry (see e.g. Browne, 1870; Davies, 1873; Ireland, 1875; Laycock, 1876). Also, there were two scholars who had produced a: "Principles of Psychology" (one by Herbert Spencer in 1855 and one by William James in 1890).

Scripture (1891) provides an insightful taxonomy of scientific disciplines and analysed what kind of science psychology should be, distinguishing it from psychophysics. He concludes:
"In the foregoing attempt to define the problem of Psychology- an establishment of the following statements was striven for:—
  1. Psychology is the science of mental processes, not mental products. 
  2. It is a mental science, not a physiology of the brain. 
  3. It is one of the special sciences, not a part of philosophy. 
  4. It is a descriptive and explanatory, not a critical science. 
  5. It is an indispensable auxiliary to the physical, other mental, philosophical and didactic sciences."
The quote mentions "the problem of Psychology" and this problem bears undeniable resemblances to what we are discussing today. Scripture provides conditions a young "would-be science" (as it was perceived in 1891!) had to meet. Here's the second condition (the first is about point 1 in the previous quote):
"The second condition which our science is to fulfill is that it must be a modern science. The subjects of science are facts and hypotheses. The distinguishing characteristics of modern science are the establishment of accurately determined facts and the founding of hypotheses upon such facts alone. A would-be science that neglects any possible means of developing these characteristics finds itself at once in disrepute.
The science of mental processes dare not begin with metaphysical hypotheses and twist the facts to suit them; it dare not rest contented with loose and insufficient methods of ascertaining facts." Scripture (1891, p. 316-317, emphasis added)
Still wondering why we find ourself in disrepute? This is a criticism, or rather a warning from the era before statistics which provided a whole new dimension to the meaning of resting contented with "loose and insufficient methods of ascertaining facts".

We cannot seriously continue to claim to be a young "would-be science", because we have apparently been doing so for at least 120 years. Is it really that hard to imagine other disciplines are starting to loose patience? 

About point 2, the parental advisory for the young science wasn't the only criticism of that era, there were more "directed attacks" that had to do with Psychology as So-Called “Natural Science” (Ladd, 1892). This is about the claims of Psychology to use the scientific methods of the natural sciences to study psychological phenomena when in fact: 
[…] psychology as a science, devoid of all postulating of "deeper-lying entities," does nothing of the kind. It assumes only the phenomena - the thoughts and feelings as actually known, and the possibility of ascertaining uniform relations among them.” Ladd (1892, pp. 29–30, reviewing William James’ “The Principles of Psychology”, emphasis added).
These criticisms, that psychological science is "dressing up" as a science, have resurfaced again and again (see e.g., Spence, 1944; Meehl, 1967; Feynman, 1974). Add to that all the warnings of the great statisticians like Cohen (1962, yes, we knew back then already that power was very important!) and philosophers of science:
"After reading Meehl (1967) [and other psychologists] one wonders whether the function of statistical techniques in the social sciences is not primarily to provide a machinery for producing phony corroborations and thereby a semblance of ‘scientific progress’ where, in fact, there is nothing but an increase in pseudo-intellectual garbage." -Lakatos (1978, pp. 88–89, emphasis added)
You cannot seriously maintain that psychological science has been doing everything it possibly can to outgrow its "would-be science" status.

I have used this quote by Meehl (a psychologist) on previous occasions, but it just sums up, in a very profound way, that this argument of a being the new kid on the block that needs to be treated as a young child whose mistakes should be overlooked, cannot be an argument against the recent (but in fact very ancient) critiques by the hard sciences:
"I am prepared to argue that a tremendous amount of taxpayer money goes down the drain in research that pseudotests theories in soft psychology and that it would be a material social advance as well as a reduction in what Lakatos has called “intellectual pollution” (Lakatos, 1970, fn. 1 on p. 176) if we would quit engaging in this feckless enterprise.  
I think that if psychologists would face up to the full impact of the above criticisms, something worthwhile would have been achieved in convincing them of it. Besides, before one can motivate many competent people to improve an unsatisfactory cognitive situation by some judicious mixture of more powerful testing strategies and criteria for setting aside complex substantive theory as “not presently testable,” it is necessary to face the fact that the present state of affairs is unsatisfactory.  
My experience has been that most graduate students, and many professors, engage in a mix of defense mechanisms (most predominantly, denial), so that they can proceed as they have in the past with a good scientific conscience. The usual response is to say, in effect, “Well, that Meehl is a clever fellow and he likes to philosophize, fine for him, it’s a free country. But since we are doing all right with the good old tried and true methods of Fisherian statistics and null hypothesis testing, and since journal editors do not seem to have panicked over such thoughts, I will stick to the accepted practices of my trade union and leave Meehl’s worries to the statisticians and philosophers.”  
I cannot strongly fault a 45-year-old professor for adopting this mode of defense, even though I believe it to be intellectually dishonest, because I think that for most faculty in soft psychology the full acceptance of my line of thought would involve a painful realization that one has achieved some notoriety, tenure, economic security and the like by engaging, to speak bluntly, in a bunch of nothing." 
-Meehl (1990, emphasis and markup added).

Next please.

No, I'm done for now.


Aoyama, T., Hayakawa, M., Kinoshita, T., & Nio, M. (2008). Revised value of the eighth-order QED contribution to the anomalous magnetic moment of the electron. Physical Review D, 77(5), 1–24. doi:10.1103/PhysRevD.77.053012

Bennett, G., Bousquet, B., Brown, H., Bunce, G., Carey, R., Cushman, P., Danby, G., et al. (2006). Final report of the E821 muon anomalous magnetic moment measurement at BNL. Physical Review D, 73(7), 1–41. doi:10.1103/PhysRevD.73.072003

Browne, J. H. B. (1870). On the Method of the Study of Mind. The British Journal of Psychiatry, 16(74), 233–247. doi:10.1192/bjp.16.74.233

Hagiwara, K., Martin, a, Nomura, D., & Teubner, T. (2007). Improved predictions for g−2g−2 of the muon and αQED(MZ2). Physics Letters B, 649(2-3), 173–179. doi:10.1016/j.physletb.2007.04.012

Davies, W. G. (1873). Consciousness and “Unconscious Cerebration.” The British Journal of Psychiatry, 19(86), 202–217. doi:10.1192/bjp.19.86.202

Feynman, R. P. (1974). Cargo cult science. Engineering and Science, 37(7), 10–13.

Ireland, W. W. (1875). Can Unconscious Cerebration be proved? The British Journal of Psychiatry, 21(95), 366–387. doi:10.1192/bjp.21.95.366

Ladd, G. (1892). Psychology as So-Called “ Natural Science .” The Philosophical Review, 1(1), 24–53. Retrieved from

Laycock, T. (1876). Reflex, Automatic, and Unconscious Cerebration: A History and a Criticism. The British Journal of Psychiatry, 21(96), 477–498. doi:10.1192/bjp.21.96.477

Meehl, P. E. (1967). Theory testing in psychology and physics: a methodological paradox. Philosophy of science, 34, 103–115.

Meehl, P. E. (1990). Why Summaries of Research on Psychological Theories Are Often Uninterpretable. Psychological Reports, 66(1), 195. doi:10.2466/PR0.66.1.195-244

Scripture, E. W. (1891). Psychology as philosophy. Mind, 63, 306-326.

Spence, K. W. (1944). The nature of theory construction in contemporary psychology. Psychological Review, 51(1), 47–68. doi:10.1037/h0060940

Spencer, H. (1855). The principles of psychology. London, UK: Longman, Brown, Green and Longmans. Retrieved from

van Leeuwen, M. (2009). Thinking Outside the Box: A Theory of Embodied and Embedded Concepts. Universal Press, Veenendaal, The Netherlands.