Report to the President. Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods
Downloads
Abstract
PCAST concluded that there are two important gaps: (1) the need for clarity about the scientific standards for the validity and reliability of forensic methods and (2) the need to evaluate specific forensic methods to determine whether they have been scientifically established to be valid and reliable. Our study aimed to help close these gaps for a number of forensic “feature-comparison” methods—specifically, methods for comparing DNA samples, bitemarks, latent fingerprints, firearm marks, footwear, and hair. Our study, which included an extensive literature review, was also informed by inputs from forensic researchers at the Federal Bureau of Investigation Laboratory and the National Institute of Standards and Technology as well as from many other forensic scientists and practitioners, judges, prosecutors, defense attorneys, academic researchers, criminal-justice-reform advocates, and representatives of Federal agencies. The findings and recommendations conveyed in this report, of course, are PCAST’s alone. Our report reviews previous studies relating to forensic practice and Federal actions currently underway to strengthen forensic science; discusses the role of scientific validity within the legal system; explains the criteria by which the scientific validity of feature-comparison forensic methods can be judged; and applies those criteria to the selected feature-comparison methods. Based on our findings concerning the “foundational validity” of the indicated methods as well as their “validity as applied” in practice in the courts, we offer recommendations on actions that could be taken by the National Institute of Standards and Technology, the Office of Science and Technology Policy, and the Federal Bureau of Investigation Laboratory to strengthen the scientific underpinnings of the forensic disciplines, as well as on actions that could be taken by the Attorney General and the judiciary to promote the more rigorous use of these disciplines in the courtroom.References
Alosh, M., Fritsch, K., Huque, M., Mahjoob, K., Pennello, G., Rothmann, M., Russek-Cohen, E., Smith, F., Wilson, S. y Yue, L. (2015). Statistical considerations on subgroup analysis in clinical trials. Statistics in Biopharmaceutical Research, 7, p. 286-303.
Association of Firearm and Tool Mark Examiners. (2011). Theory of Identification as it Relates to Tool Marks: Revised. AFTE Journal, 43(4).
Balding, D.J. y Nichols, R.A. (1994). DNA profile match probability calculation: how to allow for population stratification, relatedness, database selection and single bands. Forensic Science International, 64, p. 125-140.
Bieber, P. (2014). Fire investigation and cognitive bias. En Wiley Encyclopedia of Forensic Science. onlinelibrary.wiley.com/doi/10.1002/9780470061589.fsa1119/abstract.
Bieber, F.R., Buckleton, J.S., Budowle, B., Butler, J.M. y Coble M.D. (2016). Evaluation of forensic DNA mixture evidence: protocol for evaluation, interpretation, and statistical calculations using the combined probability of inclusion. BMC Genetics, 17. www.bmcgenet.biomedcentral.com/articles/10.1186/s12863-016-0429-7.
Bouchet, C., Guillemin, F. y Braincon, S. (1996). Nonspecific effects in longitudinal studies: impact on quality of life measures. Journal of Clinical Epidemiology, 49(1).
Boroditsky, L. (2007). Comparison and the development of knowledge. Cognition, 102.
Bodziak, W. J. (2000). Footwear Impression Evidence: Detection, Recovery, and Examination (2nd ed.). CRC Press-Taylor & Francis.
Budowle, B., Buscaglia, J. y Perlman, R.S. (2006). Review of the scientific basis for friction ridge comparisons as a means of identification: committee findings and recommendations. Forensic Science Communications, 8.
Budowle, B., Moretti, T.R., Keys, K.M., Koons, B.W. y Smerick, J.B. (1997). Validation studies of the CTT STR multiplex system. Journal of Forensic Sciences, 42(4).
Budowle, B., Moretti, T.R., Baumstark, A.L., Defenbaugh, D.A. y Keys, K.M. (1999). Population data on the thirteen CODIS core short tandem repeat loci in African Americans, U.S. Caucasians, Hispanics, Bahamians, Jamaicans, and Trinidadians. Journal of Forensic Sciences, 44(6).
Budowle, B., Shea, B., Niezgoda, S. y Chakraborty, R. (2001). «CODIS STR loci data from 41 sample populations», Journal of Forensic Sciences, 46(3), p. 453-489. En julio de 2015 se informó de errores encontrados en la base de datos original. (Erratum (2015). Journal of Forensic Sciences, 60(4), p. 1114-6, pueden encontrase en www.fbi.gov/about-us/lab/biometric-analysis/codis/ expanded-fbi-str-final-6-16-15.pdf.)
Buckleton, J.S., Curran, J.M. y Gill, P. (2007). Towards understanding the effect of uncertainty in the number of contributors to DNA stains. Forensic Science International Genetics, 1(1).
Butler, J.M. (2015). Advanced Topics in Forensic DNA Typing: Interpretation. Elsevier/ Academic.
Butler, J.M. (2015). The future of forensic DNA analysis. Philosophical Transactions of the Royal Society B, 370(1674). https://doi.org/10.1098/rstb.2014.0252.
Butler, J.M. (2015). DNA Error Rates. Presentado en International Forensics Symposium a Washington, D.C. www.cstl.nist.gov/strbase/pub_pres/Butler-ErrorManagement-DNAError.pdf.
Bracht, G.H. y Glass, G.V. (1968). The external validity of experiments. American Educational Research Journal, 5(4), p. 437-474.
Bush, M.A., Cooper, H.I. y Dorion, R.B. (2010). Inquiry into the scientific basis for bitemark profiling and arbitrary distortion compensation. Journal of Forensic Sciences, 55(4).
Bush, M.A., Miller, R.G., Bush, PJ. y Dorion, RB. (2009). Biomechanical factors in human dermal bitemarks in a cadaver model. Journal of Forensic Sciences, 54(1).
Carroro, P. y Plebani M. (2007). Errors in a stat laboratory: types and frequencies 10 years later. Clinical Chemistry, 53.
Champod, C. (2014). Research focused mainly on bias will paralyse forensic science. Science & Justice, 54.
Clayton, T.M., Whitaker, J.P., Sparkes, R. y Gill, P. (1998). Analysis and interpretation of mixed forensic stains using DNA STR profiling. Forensic Science International, 91(1), p. 55-70.
Coble, M.D., Bright, J.A., Buckleton, J.S. y Curran, J.M. (2015). Uncertainty in the number of contributors in the proposed new CODIS set. Forensic Science International Genetics, 19.
Cole, S.A. (2004). Grandfathering evidence: Fingerprint admissibility rulings from Jennings to Llera Plaza and back again. American Criminal Law Review, 41(1189).
Cole, S.A. (2005). More than zero: Accounting for error in latent fingerprint identification. Journal of Criminal Law and Criminology, 95(3), p. 985-1078.
Conney, L. (2010). Latent Print Training to Competency: Is it Time for a Universal Training Program? Journal of Forensic Identification, 60, p. 223-58.
Dror, I.E., Charlton, D. y Peron A.E. (2006). Contextual information renders experts vulnerable to making erroneous identifications. Forensic Science International, 156.
Dror, I.E. y Hampikian G. (2011). Subjectivity and bias in forensic DNA mixture interpretation. Science & Justice, 51(4).
Dror, I.E. (2016). A hierarchy of expert performance. Journal of Applied Research in Memory and Cognition, 5, p. 121-127.
Faigman, D.L., Cheng, E.K., Mnookin, J.L., Murphy, E.E., Sander, J. y Slobogin, C. (Eds.). (2016). Modern Scientific Evidence: The Law and Science of Expert Testimony. Thomson/West Publishing.
Fregeau, C.J., Bowen, K.L. y Fourney, R.M. (1999). Validation of highly polymorphic fluorescent multiplex short tandem repeat systems using two generations of DNA sequencers. Journal of Forensic Sciences, 44(1), p. 133-166.
Gaudette, B.D. y Keeping, E.S. (1975). An attempt at determining probabilities in human scalp hair comparisons. Journal of Forensic Sciences, 19.
Garrett, B.L. y Neufeld, P.J. (2009). Invalid forensic science testimony and wrongful convictions. Virginia Law Review, 91(1), p. 1-97.
Giannelli, P.C. (1980). The admissibility of novel scientific evidence: Frye v. United States, a halfcentury later. Columbus Law Review, 80(6).
Giannelli, P.C. (2003). The Supreme Court’s ‘Criminal’ Daubert Cases. Seton Hall Law Review, 33(1096).
Giannelli, P.G. (2010). Independent crime laboratories: The problem of motivational and cognitive bias. Utah Law Review, 2, p. 247-266.
Gill, P., Jeffreys, A.J. y Werrett, D.J. (1985). Forensic application of DNA ‘fingerprints’. Nature, 318(6046).
Goode, M. (2002). Some observations on evidence of DNA frequency. Adelaide Law Review, 23, p. 45-77.
Goldstone, R. L. (1994). The role of similarity in categorization: Providing a groundwork. Cognition, 52, p. 125–157.
Gross S.R., y Shaffer, M. (2012). Exonerations in the United States, 1989-2012. National Registry of Exonerations. www.law.umich.edu/special/exoneration/Documents/exonerations_us_1989_2012_full_report.pdf.
Hassin, R. (2001). Making features similar: comparison processes affect perception. Psychonomic Bulletin & Review, 8.
Hofgartner, W.T. y Tait, J.F. (1999). Frequency of problems during clinical molecular-genetic testing. American Journal of Clinical Pathology, 112.
Houck, M.M. y Budowle, B. (2002). Correlation of microscopic and mitochondrial DNA hair comparisons. Journal of Forensic Sciences, 47(5).
Kassin, S.M., Dror I.E. y Kakucka, J. (2013). The forensic confirmation bias: Problems, perspectives, and proposed solutions. Journal of Applied Research in Memory and Cognition, 2(1), p. 42-52.
Kaye D.H. (1993). DNA Evidence: Probability, Population Genetics, and the Courts. Harv. J.L. & Tech, 7, p. 101-172.
Keijser, J.W., Malsch, M., Luining, E.T., Kranenbarg, M.W. y Lenssen, D.J.H.M. (2016). Differential reporting of mixed DNA profiles and its impact on jurists’ evaluation of evidence: An international analysis. Forensic Science International: Genetics, 23.
Kim, J., Novemsky, N. y Dhar, R. (2012). Adding small differences can increase similarity and choice. Psychological Science, 24.
Kieser, J.A., Bernal, V., Neil Waddell, J. y Raju, S. (2007). The uniqueness of the human anterior dentition: a geometric morphometric analysis. Journal of Forensic Sciences, 52.
Koehler, J.J. (2016). Intuitive error rate estimates for the forensic sciences. Jurimetrics, 57, p. 153-168. papers.ssrn.com/sol3/papers.cfm?abstract_id=2817443
Koehler, J.J. (2016). Forensics or fauxrensics? Ascertaining accuracy in the forensic sciences. Arizona State Law Journal, 49, p. 1369-1416. papers.ssrn.com/sol3/papers.cfm?abstract_id=2773255
Kimpton, C.P., Oldroyd, N.J., Watson, S.K., Frazier, R.R., Johnson, P.E., Millican, E.S., Urguhart, A., Sparkes, B.L. y Gill, P. (1996). Validation of highly discriminating multiplex short tandem repeat amplification systems for individual identification. Electrophoresis, 17(8).
Kloosterman, A., Sjerps, M. y Quak, A. (2014). Error rates in forensic DNA analysis: Definition, numbers, impact and communication. Forensic Science International: Genetics, 12.
Koppl, R. y Krane, D. (2016). Minimizing and leveraging bias in forensic science. En Robertson C.T., y A.S. Kesselheim (Eds.), Blinding as a solution to bias: Strengthening biomedical science, forensic science, and law. Elsevier.
Krane, D.E., Ford, S., Gilder, J., Iman, K., Jamieson, A., Taylor, M.S. y Thompson, W.C. (2008). Sequential unmasking: A means of minimizing observer effects in forensic DNA interpretation. Journal of Forensic Sciences, 53(4).
Krimsky, S. y Simoncelli, T. (2011). Genetic Justice: DNA Data Banks, Criminal Investigations, and Civil Liberties. Columbia University Press.
Lander, E.S. (1994). DNA fingerprinting on trial. Nature, 339, p. 501-505.
Lander, E.S. y Budowle, B. (1994). DNA fingerprinting dispute laid to rest. Nature, 371, p. 735- 738.
Langleben, D.D. y Moriarty, J.C. (2013). Using brain imaging for lie detection: Where science, law, and policy collide. Psychology, Public Policy, and Law, 19(2).
Langenburg, G. (2009). A performance study of the ACE-V process: A pilot study to measure the accuracy, precision, reproducibility, repeatability, and biasability of conclusions resulting from the ACE-V process. Journal of Forensic Identification, 59(2).
Larkey, L.B., y Markman, A.B. (2005). Processes of similarity judgment. Cognitive Science, 29.
Lygo, J.E., Johnson, P.E., Holdaway, D.J., Woodroffe, S., Whitaker, J.P., Clayton, T.M., Kimpton, C.P. y Gill, P. (1994). The validation of short tandem repeat (STR) loci for use in forensic casework. International Journal of Legal Medicine, 107(2).
Mangione-Smith, R., Elliott, M.N., McDonald, L. y McGlynn, E.A. (2002). An observational study of antibiotic prescribing behavior and the Hawthorne Effect. Health Services Research, 37(6), p. 1603-1623.
McCabe, J. (1996). DNA fingerprinting: The failings of Frye. Northern Illinois University Law Review, 16, p. 455-82.
McCarney, R., Warner, J., Iliffe, S., van Haselen, R., Griffin, M. y Fisher, P. (2007). The Hawthorne Effect: a randomized, controlled trial. BMC Medical Research Methodology, 7(30).
Medin, D.L., Goldstone, R.L. y Gentner, D. (1993). Respects for similarity. Psychological Review, 100.
Medin, D.L., Goldstone, R.L. y Markman, A.B. (1995). Comparison and choice: Relations between similarity processes and decision processes. Psychonomic Bulletin and Review, 2, p. 1–19.
Miller, L.S. (1987). Procedural bias in forensic examinations of human hair. Law and Human Behavior, 11.
Moretti, T.R., Baumstark, A.L., Defenbaugh, D.A., Keys, K.M., Smerick, J.B. y Budowle, B. (2001). Validation of Short Tandem Repeats (STRs) for forensic usage: performance testing of fluorescent multiplex STR systems and analysis of authentic and simulated forensic samples. Journal of Forensic Sciences, 46(3), p. 647-660.
Morrison, G.S., Zhang, C. y Rose, P. (2011). An empirical estimate of the precision of likelihood ratios from a forensic-voicecomparison system. Forensic Science International, 208, p. 59–65.
Mujis, D. (2006). Measuring teacher effectiveness: Some methodological reflections. Educational Research and Evaluation, 12(1), p. 53–74.
Nosofsky, R. M. (1986). Attention, similarity, and the identification categorization relation. Journal of Experimental Psychology, General, 115, p. 39–57.
Mnookin, J.L., Cole, S.A., Dror, I.E., Fisher, B.A.J., Houck, M.M., Inman, K., Kaye, D.H., Koehler, J.J., Langenburg, G., Risinger, D.M., Rudin, N., Siegel, J. y Stoney D.A. (2011). The need for a research culture in the forensic sciences. UCLA Law Review, 725.
Neufeld, P.J. y Colman, N. (1991). When science takes the witness stand. Scientific American, 262, p. 46-53.
Page, M., Taylor, J. y Blenkin, M. (2011). Forensic identification science evidence since Daubert: Part II—judicial reasoning in decisions to exclude forensic identification evidence on grounds of reliability. Journal of Forensic Sciences, 56(4).
Pavese, F. (2009). An Introduction to Data Modelling Principles in Metrology and Testing. En F. Pavese y A. B. Forbes (Eds.), Data Modelling for Metrology and Testing in Measurement Science. Birkhäuser.
Perlin, M.W., Hornyak, J.M., Sugimoto, G. y Miller, K.W.P. (2015). TrueAllele genotype identification on DNA mixtures containing up to five unknown contributors. Journal of Forensic Sciences, 60(4).
Peterson, J.L., Lin, G., Ho, M., Chen, Y. y Gaensslen, R.E. (2003). The feasibility of external blind DNA proficiency testing. II. Experience with actual blind tests. Journal of Forensic Science, 48(1).
Petraco, N.D., Shenkin, P., Speir, J., Diaczuk, P., Pizzola, P.A., Gambino, C. y Petraco, N. (2012). Addressing the National Academy of Sciences’ Challenge: A Method for Statistical Pattern Comparison of Striated Tool Marks. Journal of Forensic Sciences, 57.
Plebani, M. y Carroro, P. (1997). Mistakes in a stat laboratory: types and frequency. Clinical Chemistry, 43.
Pocock, S.J. (1983). Clinical trials: a practical approach. Wiley.
Pretty, I.A. (2011). Resolving Issues in Bitemark Analysis. En R. B. J. Dorian (ed.), Bitemark Evidence: A Color Atlas. CRC Press.
Pretty, I.A. y Sweet, D. (2010). A paradigm shift in the analysis of bitemarks. Forensic Science International, 201.
Risinger, D.M., Thompson, W.C., Jamieson, A., Koppl, R., Kornfield, I., Krane, D., Mnookin, J.L., Rosenthal, R., Saks, M.J. y Zabell, S.L. (2014). Regarding Champod, editorial: «Research focused mainly on bias will paralyse forensic science. Science and Justice, 54.
Riva, F. y Christope, C. (2014). Automatic comparison and evaluation of impressions left by a firearm on fired cartridge cases. Journal of Forensic Sciences, 59.
Roberts, L. (1991). Fight erupts over DNA fingerprinting. Science, 254, p. 1721-1723.
Saks, M.J. y Koehler, J.J. (2005). The coming paradigm shift in forensic identification science. Science, 309(5736), p. 892-895.
Saks, M.J. y Koehler J.J. (2008). The individualization fallacy in forensic science evidence. Vanderbilt Law Review, 61(1), p. 199-218.
Stacey, R.B. (2005). Report on the erroneous fingerprint individualization in the Madrid train bombing case. Forensic Science Communications, 7.
Stahl, M., Lund, E.D. y Brandslund, I. (1998). Reasons for a laboratory’s inability to report results for requested analytical tests. Clinical Chemistry, 44.
Tangen, J.M., Thompson, M.B. y McCarthy, D.J. (2011). Identifying fingerprint expertise. Psychological Science, 22(8).
Thompson, W.C. y Ford, S. (1990). Is DNA fingerprinting ready for the courts? New Scientist, 125, p. 38-43.
Thompson, W.C., Taroni F. y Aitken, C.G.G. (2003). How the Probability of a False Positive Affects the Value of DNA Evidence. J Forensic Sci, 48(1).
Thompson, W.C. (2009). Painting the target around the matching profile: the Texas sharpshooter fallacy in forensic DNA interpretation. Law, Probability and Risk, 8(3).
Thompson, W.C. (2013). The Myth of Infallibility. En S. Krimsky & J. Gruber (Eds.), Genetic Explanations: Sense and Nonsense. Harvard University Press.
Thompson, S.G. (2015). Cops in Lab Coats: Curbing Wrongful Convictions through Independent Forensic Laboratories. Carolina Academic Press.
Tversky, A. (1977). Features of similarity. Psychological Review, 84, p. 327–52.
Ulery, B.T., Hicklin, R.A., Buscaglia, J. y Roberts, M.A. (2011). Accuracy and reliability of forensic latent fingerprint decisions», Proceedings of the National Academy of Sciences, vol. 108(19).
Weech, T.L. y Goldhor, H. (1982). Obtrusive versus unobtrusive evaluation of reference service in five Illinois public libraries: A pilot study. Library Quarterly: Information, Community, Policy, 52(4), p. 305-324.
Wertheim, K. (2002). Letter re: ACE-V: Is it scientifically reliable and accurate? Journal of Forensic Identification, 52.
Wilson, H.D. (2012). Comparison of the individual characteristics in the outsoles of thirty-nine pairs of Adidas Supernova Classic shoes. Journal of Forensic Identification, 62(3).
Informes
FDA Guidance (2016). Adaptive Designs for Medical Device Clinical Studies. www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ ucm446729.pdf.
Forensic Science Commission (2016). Forensic bitemark comparison complaint filed by National Innocence Project on behalf of Steven Mark Chaney – Final Report. www.fsc.texas.gov/sites/default/files/FinalBiteMarkReport.pdf.
Innocence Project. DNA Exonerations in the United States. www.innocenceproject.org/dna-exonerations-in-the-united-states.
Ministerio del Interior Español (2018). La relevancia del título oficial del perito criminalístico nombrado por el juez en la jurisdicción penal española
National Academies of Sciences, Engineering, and Medicine (2015). Support for Forensic Science Research: Improving the Scientific Role of the National Institute of Justice. The National Academies Press.
National Commission on Forensic Science (2015). Defining forensic science and related terms. www.justice.gov/ncfs/file/786571/download.
National Commission on Forensic Science (2016). Recommendations to the Attorney General Regarding Use of the Term ‘Reasonable Scientific Certainty’. www.justice.gov/ncfs/file/839726/download.
National Institute of Justice (2006). Status and Needs of Forensic Science Service Providers: A Report to Congress. www.ojp.usdoj.gov/nij/pubs-sum/213420.htm.
National Physical Laboratory (2010). A Beginner’s Guide to Measurement. www.npl.co.uk/upload/pdf/NPL-Beginners-Guide-to-Measurement.pdf.
National Research Council (1996). The Evaluation of Forensic DNA Evidence. The National Academies Press.
National Research Council (2004). Forensic Analysis: Weighing Bullet Lead Evidence. The National Academies Press.
National Research Council (2008). Ballistic Imaging. The National Academies Press.
National Research Council (2009). Strengthening Forensic Science in the United States: A Path Forward. The National Academies Press.
National Research Council (2010). Biometric Recognition: Challenges and Opportunities. The National Academies Press.
Organization for Scientific Area Committees (OSAC) (2015). Research Needs Assessment Form, «Study to Assess the Accuracy and Reliability of Firearm and Toolmark». www.nist.gov/forensics/osac/upload/FATM-Research-NeedsAssessment_Blackbox.pdf.
U.S. Department of Justice, Office of Justice Programs, National Institute of Justice (1996). Convicted by Juries, Exonerated by Science: Case Studies in the Use of DNA Evidence to Establish Innocence after Trial (xxviii).
U.S. Department of Justice (2006). Office of the Inspector General (Oficina del Inspector General). A review of the FBI’s handling of the Brandon Mayfield case. oig.justice.special/s0601/final.pdf.
U.S. Department of Justice (2015). New NIST Center of Excellence to Improve Statistical Analysis of Forensic Evidence. www.nist.gov/forensics/center-excellence-forensic052615.cfm.
U.S. Department of Justice. National Institute of Justice (2016). Report Forensic Science: Fiscal Year 2015 Funding for DNA Analysis, Capacity Enhancement and Other Forensic Activities.
