Doyen – PsyPhi

More responses, this time Daniel Simons/Dave Nussbaum

Daniel Simons
“The expectancy effects study is rhetorically powerful but proves little. In their Experiment 1, Doyen et al. tested the same hypothesis about priming stereotypes that Bargh tested. But in Experiment 2, Doyen et al. tested a hypothesis about experimenter expectancies. That is a completely different hypothesis. The second study tells us that experimenter expectancies can affect walking speed. But walking speed surely can be affected by more than one thing. So Experiment 2 does not tell us to what extent, if any at all, differences in walking speed were caused by experimenter expectancies in Bargh’s experiment (or for that matter, anywhere else in the natural world outside of Doyen’s lab). This is the inferential error of confusing causes of effects with effects of causes. Imagine that Doyen et al. had clubbed the subjects in the elderly-prime condition in the knee; most likely that would have slowed them down. But would we take that as evidence that Bargh et al. had done the same?”

I was waiting for this to be said. Thank you, sir.

“The inclusion of Experiment 2 served a strong rhetorical function, by planting in the audience’s mind the idea that the difference between Bargh vs Doyen Exp 1 was due to expectancy effects (and Ed Yong picked up and ran with this suggestion by referring to Clever Hans). But scientifically, all it shows is that expectancy effects can influence the dependent variable in the Bargh experiment. That’s not nothing, but anybody who already believes that experiments need to be double-blind should have seen that coming. If we had documentary evidence that in the actual 1996 studies Bargh et al. did not actually eliminate expectancy effects, that would be relevant. (We likely never will have such evidence; see next point.) But Experiment 2 does not shed nearly as much light as it appears to”

Which was an idea popularised by Ed Yong. Nevertheless, it is a free-standing experiment including assumptions that were not present in Bargh’s study.

“That is, just because the original study could reject the null and the replication could not, that doesn’t mean that the replication is significantly different from the original study. If you are going to say you failed to replicate the original result, you should conduct a test of that difference.

As far as I can tell neither Doyen et al. nor Pashler et al. did that. So I did. I converted each study’s effect to an r effect size and then comparing the studies with a z test of the difference between independent rs, and indeed Doyen et al. and Pashler et al. each differed from Bargh’s original experiments. So this doesn’t alter the present discussion. But as good practice, the replication reports should have reported such tests.”

Again an insightful view of current events.

Dave Nussbaum contributes greatly to the understanding of conceptual and direct replication and the importance of these concepts. It also comes closer to my own opinion on why the Perception-Behaviour Link-experiments would be hard to ‘replicate’.

It is motivating to see the discourse focuses more and more on underlying issues rather than the symptoms. I am inclined to, still, believe that the controversy surrounding the specific Bargh study has to do with the underlying theory. I am however leaving that in my literature review, time to focus on the presentation of it instead ^^.

Matthew Lieberman’s response and solution

There was a solution posted in a recently started blog by Matthew Lieberman that focuses on direct/conceptual replications. His solution is indeed a very interesting one; add to the curriculum of graduate students in their first or second year that they replicate findings of studies previously nominated to be so. While comments on it are pessimistic (with justified reasoning), I do hope it resonates within the scholarly psychology community.

A personal take on Lieberman’s response is that I probably would not have minded to see this added to my own curriculum. I may not be all too pleased but considering how much one would learn by replicating something that has worked before, I may not be too peeved about it. Also, getting a name on a publication would be a pretty sweet bonus. Of my severely limited insight into other universities ‘caring and nurturing’ of aspiring scientists, some are better (and some worse) in picking up their students and involving them in the ongoing research. Maybe Lieberman’s idea would go a decent amount of the way to attenuate this issue as well.

Ed Yong’s response and a few comments

Ed Yong’s initial coverage* of Doyen’s** and Bargh’s*** study was, in my opinion, quite brutal. I have been taught through my undergraduate to criticise constructively and I do not think the initial post has the depth to do so. For example, a close look at Doyen’s study indicates that one of the few last alternatives at explaining participant’s slower walking speed was experimenter expectation (and a very well conducted piece of research to demonstrate it). The difference in the walk-fast/expect-fast condition was explained by the difference between manual and automatic measuring, not so in the walk-slow/expect-slow condition. I wrote this in my previous blog entry too, but with a different emphasis. This finding means that an environmental stimuli (experimenter expectation manifested in subtle behaviour) was internalised by the participants and subsequently affected observable behaviour (walking slower). This entails that the Doyen study, in fact, supports the original proposition of the Perception-Behaviour Link. This mitigates my criticism of Bargh’s work, since, the theory from which he based his 1996 experiment was conceptually replicated in the Doyen study. The PBL is not mentioned in Ed Yong’s initial coverage.

In Ed Yong’s reply**** to Bargh, he mentions Doyen to “[have] timed volunteers with infrared censors rather than a stopwatch” But they timed both with sensors and manually. This was one of the central reasons that they came to the conclusion that experimenter expectation was the only alternative left to explain their result.

It does strike me from having reviewed large parts of the literature surrounding priming that the published articles are all conceptual replications. The studies following Bargh et al. (1996) have differences in methodology to that study. The issue that has been raised in comments to Bargh’s reply to Ed Yong is that “purer” replications that have not given the same results are subject to the file-drawer phenomenon. I.e. publishers have not accepted them and so they’ve been put in the file-drawer. The issue with this statement is obviously that it is very hard to know (for an outsider like myself) if publishers have denied these studies because they show null-results (not very exciting and from comments it seems there are other rather valid reasons for them not to publish these) or if they contain errors of various types (making them unpublishable).

In either case, I believe I argue in my literature review, strongly, for the theory underlying priming (the Perception-Behaviour Link) but at the same time believe that researchers are getting ahead of themselves and testing advanced hypotheses, when really what this theory needs is the grunt-work of establishing even its simplest tenets. Be that an actual replication of the methodology in Bargh et al. (1996), even though I believe there exist other research more suitable to exemplify the Perception-Behaviour Link.

I should have chosen another topic to do my 30-page literature review on.

*http://blogs.discovermagazine.com/notrocketscience/2012/01/18/primed-by-expectations-%e2%80%93-why-a-classic-psychology-experiment-isn%e2%80%99t-what-it-seemed/
**http://dx.doi.org/10.1371/journal.pone.0029081
***http://www.yale.edu/acmelab/articles/bargh_chen_burrows_1996.pdf
****http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-replication-bargh-psychology-study-doyen/

Bargh, Doyen and conclusions thereof.

I have just witnessed an interesting phenomenon.

I am concurrently to writing this blog-post, writing a literature review on priming and the Perception-Behaviour Link*. Another paper whom closely replicated the findings of Bargh et al. (1996)*, Doyen et al. (2012)**, were unable to replicate findings as well as provide an interesting demonstration of experimenter expectation. It is however presumptious to assume that the original literature* then also is explained by this bias. The reason is given in a reply by Bargh*** to the Doyen study, basically stating that the experiment was run as a blind study and so experimenter expectation can be safely ruled out. While Bargh’s reply unfortunately contains personal attacks and evidence towards being technologically unwilling, he is defended by others in that his experiment _has_ been replicated successfully and that one study cannot refute several made on the same topic. The studies given (by others than Bargh -he did not give any studies as support in his criticism) as support for this claim are these; Elderly prime effect on simulated driving speed, Gay male prime effect on hostility, elderly or youth prime effect on walking time and accessibility as precursor to goal-fulfilment and high vs. low self-conscious difference in being primed by an elderly stereotype on walking speed (xps 2 & 3). I have reviewed one of these in my literature review and it is at best a conceptual replication with several issues in methodology and statistical interpretation (specifics available on request).

In my book, it is not enough to conceptually replicate, since one is then stuck with reviewing another piece of research with its own flaws and fallacies. Granted, Doyen comes close to replicating but did differ on the point of blind experimenters (it was one of the manipulations in Doyen). The assumption was that Bargh’s work was non-blind (something I came to the conclusion of as well, although I’ve read the paper a gazillion times). This was not the case, and hence, it is also a conceptual replication. What _is_ interesting with the Doyen study is that it still supports the PBL, albeit unintentionally. The most important statistic presented is the believe-slow-walking speed comparison between automatic and manual measurement. Believe-fast-walking’s significant difference was removed when considering both automatic and manual, not so for believe-slow. A last alternative for this significant difference is then experimenter expectation. The thing is though, experimenter expectation is also an environmental stimuli that is unconsciously internalised and evidently had an effect on observable behaviour! Well, this is what the Preception-Behaviour Link strictly posits.

Again, it is a bit of a shame that many of the arguments Bargh uses in his criticism of the Doyen study are arbitrary, unsupported and, on occasion, false in light of other research (even some from the area of priming)****. It does however not reflect on his prior research. In conclusion, Doyen does not specifically cut Bargh’s research down, but rather, introduces another concept able to be accounted for within the Perception-Behaviour Link’s framework.

On a second note, if you have unpublished research on the replication of Bargh et al.’s 1996 study, I would very much like to read it. I believe it is of central importance to be open-minded to one’s own fallacies and others’ criticism (even if I very much like the Perception-Behaviour Link theory), the only way is forward and it is only obstructed when self-preserving opinions and values are set before empirical research.

*Bargh, J.A., Chen, M., & Burrows, L. (1996). Automaticity of social behaviour: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 72, 230-244.
**Doyen, S., Klein, O., Pichon, C., & Cleeremans, A. (2012). Behavioural priming: It’s all in the mind, but whose mind? doi:10.1371/journal.pone.0029081
***Bargh, J.A. (2012, March 5). The natural unconscious: Nothing in their heads. [Web log post]. Retrieved from http://www.psychologytoday.com/blog/the-natural-unconscious/201203/nothing-in-their-heads.´
****See comment section of Bargh’s blog post, specifically the one referring to Assimilation and Contrasting (which is found in Dijksterhuis, A., Spears, R., Postmes, T., Stapel, D.A., van Knippenberg, A., & Scheepers, D. (1998). Seeing one thing and doing another: Contrast effects in automatic behaviour. Journal of Personality and Social Psychology, 75, 862-871.)