Cognitive neuroscientist Dr. Itiel Dror joins Aaron Olson to discuss his groundbreaking and controversial research on cognitive bias in forensic science. Dr. Dror, a senior researcher at University College London with a PhD from Harvard, has spent nearly two decades challenging the forensic community's claims of objectivity and infallibility.
He explains why cognitive bias isn't something that distorts our thinking; it's built into the very fabric of human cognition, and why awareness alone can't control it.
The discussion delves into Dr. Dror's explosive research findings, including studies showing that forensic pathologists were five times more likely to rule homicide for Black children versus white children with identical medical findings.
His work demonstrates how irrelevant contextual information, such as a suspect's race, criminal history, or confession status, can influence forensic conclusions, even when examiners believe they remain objective. The response has been intense, with professional attacks, complaint filings, and attempts to have his papers retracted.
Despite fierce pushback, Dr. Dror notes encouraging progress as many forensic domains begin acknowledging bias and implementing changes. He warns that courts are increasingly excluding biased forensic evidence, giving the community a choice: change proactively "the easy way" or face judicial consequences "the hard way."
Clips
Related podcasts
When Science Meets Justice: Fred Whitehurst's Fight Against FBI Lab MisconductThe Accuracy of Standardized Field Sobriety Testing (SFST) with Dr. Greg Kane
Challenging the Validity of SFSTs and Drug Recognition Testing with David Rosenbloom
Shattering Accreditation Illusions: ISO 17025 Myths and Realities
Related links
Cognitive and Human Factors in Expert Decision Making: Six Fallacies and the Eight Sources of Bias
Biased and Biasing: The Hidden Bias Cascade and Bias Snowball Effects
Automated transcript (not checked for errors)
Introduction
00:00:00 - Podcast Introduction
Aaron Olson: Welcome back to the show everyone. Today I'm joined by Dr. Itiel Dror who has dedicated his career to exploring human cognition. He is a cognitive neuroscientist and a leading expert in the field of cognitive bias and human performance. He holds a PhD from Harvard University and is a senior researcher at University College of London. Itiel Dror, thanks so much for coming on the show today.
Itiel Dror: My pleasure.
Aaron Olson: So, I'm excited to have you on to dive into all the different aspects of your work and how cognitive bias affects our thinking and judgment and forensic science. But I want to dive a bit into your research.
Itiel Dror: I interrupt you already. It doesn't affect our thinking and judgment. It's part of our thinking and judgment. It's built in. It's not like we have our thinking and judgment and bias kicks in and distorts it. It's part of it and distorts it. So, I'm just jumping right in the water before you finish your introduction.
Aaron Olson: So it's it's it's part when you say it's part of it. Is is that part of the idea that we can't really divorce ourselves from our biases that they're kind of inbuilt in but we can maybe try to become more aware of them?
00:01:08 - The Nature of Cognitive Bias
Itiel Dror: Yes. Yes. I would say first of all that the very cognitive mechanism the way we perceive the world is built from a brain processes and those brain processes do not see the world the way it is. The human mind is not a camera. The very mechanism of perception depends on who we are and our experience and our expectations and a lot of cognitive factors that mediate what we see and what we hear and we can be aware of it but we can't control them. So mere awareness of our biases doesn't mean we can I wish we could we don't have an on/off switch. So awareness is nice, but you cannot by mere willpower and awareness of your biases and cognitive processes control them. In fact, sometimes when people try to control their biases, they're even more biased. The more I try not to think about something, the more I think about it, it's like a judge telling the jurors, "Ignore this information. Don't use this information." The more times the judge says to the jurors, even though they're trying to ignore it, the more it actually impacts their judgment and decision making.
00:02:28 - Implicit vs. Explicit Bias
Itiel Dror: So we have very limited control about what we think and how we think. So awareness is nice, but it's not a way to mediate and mitigate bias.
Aaron Olson: Okay. Okay. So, we'll get into some of your papers, but tell me more about that. If if awareness can't mitigate it, what what are the solutions to help us
Itiel Dror: Well, we haven't gotten to the details of the issues and problems in bias, but generally we if we have awareness and we acknowledge bias. A big part is people don't are not aware of their biases. When I say bias, I need to be clear. I'm not talking about sexism and racism and all those kind of things. So I'm not talking about explicit intentional discrimination. I'm talking about implicit cognitive bias. There are brain mechanisms and we're not aware of it. So people are not aware of it. When you tell them when you show them their biases, many of them don't see it. We call it the bias blind spot and they get very defensive and push back.
00:03:33 - Practical Solutions: Information Management
Itiel Dror: So first of all, we need to explain to people and for people to be aware and accept that they're being biased. A stage one for them to do something about it because if they're in denial, they're not going to do something about it. And what to do about it is not mere willpower is taking practical steps. For example, we'll take a DNA. So when a DNA examiner is working with a DNA mixture and it's supposedly, you know, some sexual assault or not, they should avoid information that they do not need to do the DNA analysis but can bias them. For example, has a suspect confessed to the crime? Does a suspect have a criminal record of being a rapist or pedophile? All of this is irrelevant. When a firearm examiner compares, you know, two breach faces on a bullet cartridge, they don't need to know that it's homicide and the person who died is white and the suspect is black. And the reality that DNA and fingerprint and firearm examiners in the United States and all over the world, they know a lot a lot of irrelevant potentially biasing information that they don't need.
Initial Resistance from the Forensic Community (00:04:48 - 00:09:21)
00:04:48 - The Simple Solution
Itiel Dror: So 101, you know, a kind of the most common sense low-hanging fruit examiners should know the information they need but not be exposed to irrelevant past irrelevant information that is potentially biasing. So this is an one example. The more sophisticated techniques that people can minimize bias, but this is something really simple. It's a simple takeaway message. Why do you know it? Is that relevant to you? And you're comparing bullets to know that the suspect is black and the person who died that it's homicide is white. No. And oh, I knew it, but it didn't affect me. Well, it did affect you and I can prove that it affected you. And even if I tell the examiners, even if you don't think it affected you, why have to defend it in court? Because it's coming up in court. Say it didn't affect me because I did not know rather than trying an impossible task of saying I knew about it, but it didn't influence me. Which is incorrect.
Aaron Olson: Interesting. Yeah, that's fascinating. What has been what has been your acceptance from the forensic community when you say things like this?
00:05:57 - The Fingerprinting Bombshell
Itiel Dror: So when I started it's now almost 20 years ago no one ever dared question forensic examiners for example fingerprinting where I started the most used forensic domain been used you know for over 100 years no one ever collected data is a fingerprint examiner influenced knowing that the suspect confessed to the crime. or not or the race of the suspect. No one did this and I did that and when I came out with a research it was like all hell broke loose. I was not aware because I'm a cognitive researcher. I don't read. I'm not into forensic you know journals. I do cognitive one and I published it and you know it was unbelievable. It was a huge push back and for example a push back not from individ only from individual examined but the chair of the international fingerprint examiner wrote in a letter to the editor in their professional journal. So I'm not talking about what people say in private emails or on blog talking about the official response of the international fingerprint.
00:07:14 - "Seek Employment at Disneyland"
Itiel Dror: He said well Dr. Dror must have found some bad apples because we are not we are never biased. we are not affected and if Dr. Dror and I quote him found examiners that are affected by bias they shouldn't be forensic examiners they should seek employment in Disneyland that's what he wrote you know the arrogance of publishing something like that this is when I started and you do have to understand and I understand the shock because no one ever questioned them the court loves them the jurist loves them oh I have a fingerprint mark you know that's the end of it suddenly somebody saying to them you've been doing for 20, 30 years. You're proud of yourself. You're helping fight crime. And I'm telling you, you're biased to maybe sending innocent people to jail. You may be letting off guilty people because you're biased. So there was a huge push back, very aggressive, personal, hate, nasty emails. Over the years, they've come to around. So I have to say, for example, a lot of the fingerprint community have gone forward and now they're accepting it.
00:08:21 - Progress and New Battles
Itiel Dror: and are taking step. So I do have to say that you know professional organization like in the United States OSAC by the NIJ national institute of Justice have really taken this forward and many forensic domain have taken this forward including firearm and fingerprint. However I recently published in forensic pathology and again no one ever dared ever to ask if they're biased or affected by non-medical information. And when this came out just a few years ago, all hell broke loose. Not only they wanted the paper retracted and wrote letters to the editor, but they wrote complaints, eight complaints, a complaint, you know, ethical approval against my collaborator. They filed personal complaints against me to my university and they and it's all quite unprofessional and emotional. the editor himself of the Journal of Forensic Science. He's been the editor for decades and he's never seen anything like that. He welcomes discussion, but some of the responses were so emotional. You know, we're talking about the professional domain, an important domain. We need to put our personal emotions aside and talk about the data and the facts and the science.
The Pathology Study - Racial Bias in Death Determinations (00:09:41 - 00:14:34)
00:09:41 - Introducing the Controversial Study
Itiel Dror: And sometimes you have to look in the mirror and see things that you don't like to see. So you make them better and even if you cannot be make them better be transparent to the jurors and the court about the limitations the weaknesses of your conclusion.
Aaron Olson: That's so fascinating. Yeah. So, instead of you know combating your actual paper, they it sounds like they did a lot of personal attacks which which makes me think that you were on to something and that you actually found something interesting that needed to be changed. I mean, in in that study, you found that pathologists were five times more likely to rule homicide for black children versus white children with identical medical findings. Can you tell go into that paper a little bit more and tell me more what you found and what was so controversial?
00:10:31 - Part One: Real Death Certificates
Itiel Dror: So I'll tell I will tell you very quickly and I will have you add a link in the podcast on the bottom for people who want to read all the details. So there's a number of papers. The first paper you're talking about had two parts and I'm doing it really really quickly only the icing on the cake. Part number one is looking at actual cases, not invented cases, actual cases. We got 200,000 death certificates and we took out the ones of children and compared whether on the death certificate it said we're not interested in children where it's you know natural death and they don't have suicide. So we're looking whether the on the death certificate it said that they died as a result of accident or as a result of homicide. And you have to remember if they're not sure they don't have to decide they can say undetermined inconclusive I don't know I cannot say clearly if it's homicide or accident. And to remind people if they say homicide, what can be more tragic that your child died and now they didn't die because they fell down the stairs, someone murdered them and the police are investigating. So it's a lot a lot of power when a child died by accident and the result is saying no, it's homicide or vice versa, your child was murdered, but actually they say it's an accident.
00:12:02 - The Base Rate Bias Discovery
Itiel Dror: So if they're not sure, they say undetermined. So when we looked at the death certificates. We found that homicide appeared on black children more than on white children relative to accidents. So white children relative to black kids have much more homicide and accidents versus white kids have more accidents and homicide. Now as we write in the paper, it could be true. It could be that black children are more murdered than white children. But this is already a bias because this can be happening now or 10 years ago and it changed. So it used to be like this but now because of different reasons it's not like that anymore. But we get a base rate bias because they had experience that black children were associated with homicide more and white kids to accident. When they see a white child or a black child, they're already expecting accident or homicide based on their experience. So that's one part of the study. The other part of the study, but you know, first part of the study is very strong because it's real cases. However, every case is different.
00:13:11 - Part Two: Controlled Experiments
Itiel Dror: So we wanted to see what happens when it's exactly the same case except we change non-medical information. It's the same medical information which a medical examiner is a medical examiner. They make their information based on the medical examiner. However, some cases the child is black and was brought to the hospital by the mother's boyfriend and sometimes the child is white and brought to the hospital by the grandmother. If the child is black or the child is white, who brought him in the hospital? If they come from a, you know, wealthy affluent neighborhood or they come from the inner city, shouldn't matter. They're examining how the bones were broken, the medical information and saying the child died from accident like was reported in the emergency room or no, this is impossible. It cannot be an accident. It must be homicide. And again, if they're not sure, they can say undetermined. And we found a huge impact. Same identical medical information. We change the context of a non-medical information context and they change their decisions. And this is only the first paper. We have a second paper where there's a dead body, one bullet, a single gunshot to the head and they have to decide if it's suicide or homicide.
00:14:34 - Context Overriding Medical Evidence
Itiel Dror: And again, we get huge effects about what they determine based on context. And not only that, we have a condition when we give them no context and they decide homicide or suicide and then we give them context of the opposite non-medical and they flip they will change their mind made based on the medical information to fit the context. So it's not that the context is helping when the medical information is ambiguous. It will override the actual medical finding to fit the context. So that's the second paper that we've published and I'll give you the links to both of them so readers can read all the details.
Dealing with Personal Attacks and Scientific Criticism (00:14:34 - 00:18:05)
Aaron Olson: That's fascinating. And how did you deal with all the push back that you received, especially the emotional and all the personal criticisms? Were was that difficult? And were there any criticisms that were legitimate?
00:14:49 - Acknowledging Limitations
Itiel Dror: First of all about the legitimate criticism. Every study, every research has limitations, has vulnerabilities. Just like I tell forensic examiners, be transparent about the limitation, the scope of your findings. The same goes to me. So when you read the article, then I ask people to read the actual article, you will see there are limitations and they're stated in the article. So there's no hiding them in science. Every time we get one answer, we have two new questions. Right? This is the way science evolves. But the vast majority of the criticisms are attackable personal, emotional and nasty. I can send you copies of the complaint just nasty. You know, you can say things in a way I don't think you know. For example, they were complaining about the statistical analysis that was checked and then rechecked by the editor send it to statistician and they all said the stats is perfect. They could say when they complain to my university we think the statistical analysis is not good. They say oh the statistical analysis is appalling an undergraduate student wouldn't make it. It's an embarrassment to your university that a professor makes such you know this kind of language. I'd like to say didn't affect me, but I'm human. It affected me. It upset me. And they were launching such a coordinated attack personal against me.
00:17:03 - The Fight and the Beach in Mexico
Itiel Dror: So, it's not one or two. It's a professional bodies, dozens of people complaining and signing letters and whatever. I didn't know if they were going to win or not. It was such a nasty. I'm a one person trying to defend the signs. And at some stage I said to friends and colleagues I don't know if I will win. I know two things. One, I'm going to give it a good fight. So if I'm going to lose it, I know I gave it the best fight I can. I'll go down kicking and screaming. One, and two, if they win, maybe it will be good for me. I'll get fired from the university. I can't do more research. I'll retire. I'll move to Mexico. And I'll be sitting right now in 2025 on a beach in Mexico, drinking tequila and enjoying life, thanking them for the good life rather than continuing to do work and doing day and night more research in this domain. So I said, you know, I give it the best fight and if I lose, I lose.
00:18:05 - What Really Matters
Itiel Dror: But again, me and my emotions are not important because we're talking about determining if a child died by accident or homicide. This is such a big important issue. So me feeling upset by the personal attacks, you know, when I buy a computer, a laptop, I always need a big hard drive to copy all the hate mail I get. And this is a personal hate mail that I get. But we're talking also letters to the editors and and the complaints, you know, public. And you see in other places where people publicly make such nasty personal comments, right? You know, this is all a supposedly scientific discussion.
Why Forensics is Different from Other Expert Domains (00:18:05 - 00:25:26)
Aaron Olson: Yeah. Yeah. Fascinating. Um, have you seen So that was in around 2021-22.
Itiel Dror: I'm happy I'm happy you're speaking British English and saying fascinating. I think it's disgraceful. It's disgraceful that professional body of forensic examiners have professional scientific responsibility take criticism so personally and cannot look at the data and see if it's legitimate or not and improve if and when needed and make it all personal like a professional you know you know employers movement protecting employers right and salaries fighting for them rather than a professional scientific body that justice to the people.
Aaron Olson: Yeah. Yeah. What do you attribute that to? Is is is that a bias in itself or is that something different?
00:19:35 - The Feedback Problem
Itiel Dror: I have to tell you I have sympathy to the forensic examiner. Now I do not only look at bias and error in forensic examiner. I look at pilot civil aviation, US Air Force pilots. I look at medical doctor. I look at banker. I look at a whole range of expert domains and all the expert domains that they work are never nasty. They love me. They want me. It's only the forensic examiner. Why? Because they know when they make a mistake. When my when a my or a financial advisor tells you buy stock A and sell stock B, a week later you look at the Dow Jones and you know if they made a good decision or a bad decision and they know it too. the medical doctor, they give you antibiotics and you come after a week and say, "I don't feel any better or they amputate you the wrong leg. They hear about it and so on and so forth." So, they know they make mistakes, right? Forensic examiners don't know if they make mistakes. They say this child died from accident. The police doesn't investigate. The fingerprint examiner match a fingerprint. Then the person confesses to the crime even if they didn't commit it. Right at 98 95% is plea bargain. Your lawyer said to you they matched your fingerprint. They're going to convict you. They're offering you a great plea bargain. Just confess. You go to jail for a very short time. You're released on good behavior. You'll be out very soon. If you don't confess, they're going to convict you. The fingerprint DNA evidence. And we have many false confessions because of forensic flawed evidence. And again, we have research showing people who were exonerated that forensic evidence contributed not only to to the conviction, but that they confessed to crimes they have not done. So the forensic people don't know when they make a mistake, if they make a mistake, and the court loves them, and they work in the adversarial system.
00:21:47 - The Infallibility Myth
Itiel Dror: So they're not aware of their mistakes. So they get very defensive because no one questions them. And here and they don't know they make mistakes if and when they make a mistake. Relative to medical doctors pilots when the plane crashes we know a mistake was made right. And in the military when you shoot your own forces you know about it usually later. So almost in every expert mistake the experts get feedback. They know they make mistakes and they welcome somebody said I have some insight why you make mistakes. It's not because you're a bad person. It doesn't mean hardworking, competent, motivated experts still make mistakes and we understand why and we can minimize it. They welcome me in the forensic domain. I never make mistakes and they used to say before I started not only to me in court we have zero error rate. The FBI fingerprint manual said fingerprint examiners are are you sitting down? They would say it in court are infallible. Yeah, humans are not infallible. And when I would submit my articles, then some of the reviewers who were forensic examiners would say two and two equals four.
00:23:04 - The Math Analogy
Itiel Dror: If an examiner makes mistake, it means they don't know their job and they're saying two and two is five. Fingerprint is perfect. So if they make a mistake, it's the examiner not knowing. If you follow the procedure, there's never an error. And of course, this is mistaken. And this is what I had to deal with. And initially, there was a huge push back. Some of them still do, but they're coming around. It takes time. It's a shock when no one ever told you that you're biased, that you make mistakes. You're proud of your work. Your family is proud of you. you're fighting crime, putting bad people in jail, and someone says, you know, you may be putting innocent people in jail because you're biased. In one of the first training, I do training all over the US and the world in fingerprints. I was giving this big keynote talk and this fingerprint examiner really got annoyed when I was talking about that they interpret the fingerprint differently based on nonrelevant information. And he got up and was screaming at me.
00:24:05 - "The Fingerprint Doesn't Lie"
Itiel Dror: He said, "The fingerprint doesn't lie." And I said to him, you know, it took me a minute. I said, "The fingerprint doesn't lie, but the fingerprint doesn't speak. It's a human examiner that has to look at the friction range and determine minutia and decide are they relatively sufficiently similar to one another to determine a match." And this criteria what is similar enough which is not only in fingerprints but in firearms in handwriting in many forensic domain is subjective. When you ask him how similar do they need to be they never tell you not the fingerprint examiners not the people who train them what is sufficiently similar when talking about your firearm fingerprint handwriting. So you have a subjective criteria that you have to see and interpret. So of course it's going to be affected by context. But this is kind of part of the first discussion. Discussion is a nice word of the community. But I do want to emphasize again this was in the beginning. I think the vast majority of the forensic community has moved ahead huge steps forward relative to never hearing about bias, never talking about bias.
Progress and Current State (00:25:26 - 00:30:56)
00:25:26 - Significant Progress Made
Itiel Dror: We are objective. We are infallible. who have zero error rate to hearing the bad news initially very defensive responses initially and they've moved more and more to being open and taking steps some forensic domain some laboratories are really open they want training they want understanding some are semiopen and some are still resistant but are getting it's a big pill to swallow I always try you know a spoonful of sugar help the medicine go down but still the medicine is unpleasant. You have to look in the mirror and not see necessarily what you want to see. But I do have to say and emphasize that relative to the beginning I think the forensic community and I take my hat to them have made huge huge steps about bias. Now I don't think you can find many forensic examiners that would say bias I don't know what it is I never heard. They may disagree but at least it's on the table and they talk about it and this is the first step and some of them have taken a second and third step. So we we're we're doing quite well relatively speaking.
Aaron Olson: Yes. Um that's that's great that that people are taking steps. I I found that what what you just said is that looking in the mirror is very difficult and whenever I point out any errors or or biases in my papers I I get a lot of push back beyond you know we talked about acknowledging it and then some other sort of engineering steps that you can take to try to mitigate bias. What about this idea you know that forensic examiners are sort of separate and they don't want to look at it. Um you said they've made great strides but I see that there's still a lot of laboratories that don't want to look at their biases or errors. Uh do you have any other suggestions as far as moving forward is how we can get that feedback that you see in other domains.
00:26:37 - The Easy Way vs. The Hard Way
Itiel Dror: So all I have to say that forensic laboratories and I say it's you know what you see is what you get. I'm not a politician you know I say what I think they can change the easy way or the hard way. The easy way to be proactive rather than reactive to get proper training understanding change the SOP the standard operating procedures to take reasonable steps to minimize bias. The other way the hard way is to get the kick in the ass in the backside and this is happening. So until now when I appeared in court and I don't appear in court a lot and I appear for the prosecution as well as the defense when the issue of bias came up then one side well we don't need expert on bias the jurors know what bias is and the other said I want the evidence excluded because it's so biased and until now the judges you know they didn't want to deal we said you can present the issue of bias to the jurors and I will allow the forensic evidence and the jurors and decide what to do with it. But now we've seen case after case in the US, in the United Kingdom, where the judges exclude forensic evidence because of bias. The judges the bias is so strong the forensic laboratory, the crime lab is not allowed to present this evidence to the jury.
00:28:46 - Court Precedents
Itiel Dror: Recently in a case in Oregon in firearms, it happened. So this is the hard way. So if the forensic examiners are no proactive and they want to come to court and they want to be told off in court and the judge says I'm not willing to even have your evidence presented and I'm excluding it because you were biased well then we can do that also prefer to work with them but sometimes you know some of them may need the encouragement the extra encouragement but you know now evidence forensic evidence is starting
Aaron Olson: Yes.
Itiel Dror: to be excluded in the courts. And you know the courts work on precedent. Once it's done once or twice, it may happen again and again.
Final Thoughts and Resources (00:29:14 - 00:30:56)
00:29:14 - Closing Remarks
Aaron Olson: Well, Itiel Dror, thank you so much for taking time out of your day to come and talk with us and or talk with me and give a high-level overview of your work and I'll put links in the show notes where people can read more of your papers and you'll send me a few links that I can put in the show notes. Are there any final thoughts you want to leave?
00:29:47 - "We're Just Warming Up"
Itiel Dror: Are you going to tell me? Yeah. Are you telling me we're done? We're just warming up. You know, I haven't said even the beginning. This is the icing on the cake. I understand very cognitively correct of you to keep the podcast short and to the point, not to increase cognitive load. So, it's the right thing to do. So, I would just say I will put with you the links for people who want to read more or have more questions and people can even email me directly if they want. I have 150 over 150 articles. So we'll only put two or three. But if people are interested specifically in DNA or in blood stain pattern analysis or CSI, we have articles looking at bias in all of these domains. And I want to thank you for the opportunity to talk to you. And this is part of the change to get the word out there for them to come on board and to work together to improve forensic science.
00:30:47 - Future Possibilities
Aaron Olson: Well, thank you so much and and maybe if people have questions and leave comments on this, we'll have you back for round
Itiel Dror: I'd be happy to do
