Why Scientists Hate Jury Duty: A Ray of Hope from the NAS

I hate jury duty. No, not because it adds yet more stuff into a fairly busy schedule as a parent-scientist. I love the theory of jury duty and the opportunity for public service and engagement of citizens in our public life. I do. What I hate with a burning vengeance is the system whereby truth seeking and the establishment of fact are a distant eleventh to the nonfactual, emotional manipulation of the jury to see the matter of guilt in a particular direction.
A panel convened by the National Research Council of the National Academies of Science has concluded that the forensic sciences need a serious overhaul in the US (full report).
What took us so long?

Listening to National Public Radio yesterday I was barraged throughout the day with this story (audio).
It resonated with the clarity of the purest crystal goblet. You see, my jury experiences have completely convinced me that the system is utterly broken and has no resemblance to anything rational or logical. (Yes, yes, my experiences are a tiny, microscopic minority of all the jury experiences in the US but this is blogging. )
Here’s the first thing that offends my sensibilities as a scientist person with at least a half a brain. There are two reference instructions about evidence that are particularly important. By the preponderance of the evidence and beyond a reasonable doubt. Scientists will recognize the former. It is our stock and trade. We draw tentative conclusions about the state of nature based on the currently available evidence. When evidence is in conflict, we attempt a thinking-person’s synthesis of the evidence to estimate the most likely truth of the matter.
Scientists do not recognize “beyond a reasonable doubt”.
This standard is foreign to us. So when faced with an instruction from the judge, we are being asked to be even more strict than we are in daily professional life! Even if we accept the “reasonable doubt” standard as being hyperbolic and meant for a lay audience, it should at the very least be a standard that is as strong as our scientific standard, right? Otherwise what does it mean?
My problem with juries and trials is that once the instructions have been laid down, the prosecution and defense try as hard as they bloody can to convince the jury that their evidence is in fact stronger than it actually is by playing on emotional strategies to trick the jury members. The supposed adherence to fact (and for that matter rules of evidence and exclusion, etc) is a total sham.
This brings me back to the NAS/NRC finding. At the risk of insulting your intelligence, the establishment of factual matters in a criminal case looks nothing like the TV version. Evidence is rarely convincing on the face, can be technically complicated to conduct or to explain and may rest on firmer or weaker methodological foundations. The NAS/NRC report emphasizes the latter.

A congressionally mandated report from the National Research Council finds serious deficiencies in the nation’s forensic science system and calls for major reforms and new research. Rigorous and mandatory certification programs for forensic scientists are currently lacking, the report says, as are strong standards and protocols for analyzing and reporting on evidence. And there is a dearth of peer-reviewed, published studies establishing the scientific bases and reliability of many forensic methods. Moreover, many forensic science labs are underfunded, understaffed, and have no effective oversight.

You know. Fingerprint matching (when was Pudd’n Head Wilson written?). Hair and fiber identification. Eyewitness testimony quality. Observer bias. DNA matches. Etc. The teevee would convince us this stuff is duck soup. Turn key. Infallible. Except when the antagonist of the drama intentionally puts a finger on the scale.
My experiences suggest this is total nonsense. I’ve seen situations in which the prosecution spent an hour bragging on the qualifications of an expert witness, how many training classes attended, etc so that said expert could get up and state that “in their professional opinion” the evidence pointed to a certain conclusion. Without putting up the data and showing the jury how the conclusions were reached. While briefly mentioning the caveat that no, there were not really any scientifically established standards for the field. So in the jury room let me tell you the scientist is looking around in jawdropping disbelief to find people accepting the expert statement but not even wanting to examine the weak-ass excuse for “evidence” that was entered but not actually presented by the expert!!!!!
Did I mention that I hate the jury process?
Okay, well perhaps I am too hard on issues of credentialed “expertise”. What did the report have to say?

Many professionals in the forensic science community and the medical examiner system have worked for years to achieve excellence in their fields, aiming to follow high ethical norms, develop sound professional standards, and ensure accurate results in their practice. But there are great disparities among existing forensic science operations in federal, state, and local law enforcement agencies. The disparities appear in funding, access to analytical instruments, and availability of skilled and well-trained personnel; and in certification, accreditation, and oversight. This has left the forensic science system fragmented and the quality of practice uneven. Except in a few states, forensic laboratories are not required to meet high standards for quality assurance, nor are practitioners required to be certified. These shortcomings pose a threat to the quality and credibility of forensic science practice and its service to the justice system, concluded the committee

Yeah, I love being right, big w00tang for me! Except….this forensic science business is kinda important.
More from the report that is totally congruent with my limited experiences:

Nuclear DNA analysis has been subjected to more scrutiny than any other forensic discipline, with extensive experimentation and validation performed prior to its use in investigations. This is not the case with most other forensic science methods, which have evolved piecemeal in response to law enforcement needs, and which have never been strongly supported by federal research or closely scrutinized by the scientific community.

but…but…this stuff has been in use for decades in many cases. Convicted, what, hundreds of thousands of alleged criminals. Are you saying nobody has bothered to figure out if it is complete bullshit?

As a result, there has been little rigorous research to investigate how accurately and reliably many forensic science disciplines can do what they purport to be able to do. In terms of a scientific basis, the disciplines based on biological or chemical analysis, such as toxicology and fiber analysis, generally hold an edge over fields based on subjective interpretation by experts, such as fingerprint and toolmark analysis. And there are variations within the latter group; for example, there is more available research and protocols for fingerprint analysis than for bitemarks.

Are you depressed yet?

The report points out the critical need to standardize and clarify the terms used by forensic science experts who testify in court about the results of investigations. The words commonly used — such as “match,” “consistent with,” and “cannot be excluded as the source of” — are not well-defined or used consistently, despite the great impact they have on how juries and judges perceive evidence.
In addition, any testimony stemming from forensic science laboratory reports must clearly describe the limits of the analysis; currently, failure to acknowledge uncertainty in findings is common. The simple reality is that interpretation of forensic evidence is not infallible — quite the contrary, said the committee.

Right. Another way to put this is “The system flat out lies about the quality of the evidence to sway the jury to make a decision on the basis of something other than the best interpretation of the evidence”.
Can we fix things by nudging the existing structures to do better? Hell no.

The existing forensic science enterprise lacks the necessary governance structure to move beyond its weaknesses, the report says. The recommended new National Institute of Forensic Science could take on its tasks in a manner that is as objective and free of bias as possible — one with the authority and resources to implement a fresh agenda designed to address the problems found by the committee. The institute should have a full-time administrator and an advisory board with expertise in research and education, the forensic science disciplines, physical and life sciences, and measurements and standards, among other fields.
The committee carefully considered whether such a governing body could be established within an existing agency, and determined that it could not. There is little doubt that some existing federal entities are too wedded to the current forensic science community, which is deficient in too many respects. And existing agencies have failed to pursue a strong research agenda to confirm the evidentiary reliability of methodologies used in a number of forensic science disciplines.

You know we ponder the role of science in society a lot around the science blogosphere. We talk about communicating the scientific thought process to nonscientists and frequently brag on how our approaches can make much of public life better.
Isn’t the finding of fact in criminal cases in which an individual is facing incarceration an issue which reaches the highest standard of importance?
The system is clearly doing it wrong.
Science can help.