In this paper, we’ll review the major activities and results in biometric testing and evaluation at the U.S. National Biometric Test Center over the past year.  Testing and standards activities sponsored by other governments and independent groups will also be briefly discussed.

1.0  The U.S. National Biometric Test Center

Organized biometric device evaluation and testing at San Jose State University began in 1995 under funding from the Federal Highway Administration.  The Department of Defense (DoD) took over support in 1997, conferring the title “National Biometric Test Center” (NBTC) on the effort.  This funding is set to expire in the fall of this year, requiring us to broaden our base of support to additional governmental agencies.

Under DoD funding and the leadership of Dr. John Colombi, we have defined three general tasks in biometric testing and evaluation: 1) development of mathematical and statistical methods for test design and evaluation;  2) evaluation of technologies for specific applications of interest to the DoD;  3) device testing.  The Department of Defense has no interest in device testing, so it is the first two tasks that have been the core of our efforts, in particular the development of mathematical and statistical methods to lower the cost of testing and to allow for testing from operational data.

1.1  Substitution of “Inter-template” for “Impostor” Distribution

In our study three years ago of the hand geometry system in the Immigration and Naturalization Service Passenger Accelerated Service System (INSPASS) application, we had available both the “genuine” and “inter-template” distance distributions.  Because sample vectors were not stored by the system, we had no way of making random assignments of samples to non-matching templates to compute the “impostor” distribution.  We wondered if a Receiver Operating Characteristic (ROC) curve could be approximated using the “inter-template” distribution.  Simulation models based on the available data show that this approach over-estimates errors.  Figure 1 shows simulation results and the disparity of ROC curves based on “inter-template” and “impostor” distributions.

During the past year, considerable progress has been made in understanding this phenomenon.  In reference [1], we have been able to explain the mathematical basis for the difference in these two curves.  In theory, the “impostor” distribution is a higher-dimensional convolution of the “genuine” and “inter-template” distributions.  In reference [2], commissioned by the NBTC, methods for convolving and de-convolving these distributions under limiting assumptions are presented.

Lacking information on the variation and correlation between fingers of partitions, previous work [6] has assumed independence. Table 1, also taken from reference [5], shows variation of penetration and bin error rates across fingers.  Table 2 shows effects of correlations on penetration and bin error rates for a two-finger system based on thumbs, index, middle or ring fingers.

TABLE 1:  SINGLE FINGER BINNING STATISTICS

TABLE 2:  TWO-FINGER BINNING STATISTICS      

1.2.1  New Fingerprint Tests

Our original fingerprint benchmark test database consisted of 4080 “training” or “enrollment” prints and 4128 “test” prints.  Actually, we collected three images of each finger in the “training” database.  At the start of 1999, we released the additional two images of each training print.  These additional images, now available on CD-ROM, should allow vendors requiring three enrollment prints to test using this database, although no vendor has yet taken this test. 

We are hoping to acquire copies of our original 4080x4128 images after standard WSQ compression and re-expansion at 15-to-1, 30-to-1 and 45-to-1 compression ratios.  These new images will allow us to assess the effects of WSQ compression on the performance of the various AFIS algorithms.

1.2.2  Error Bounds

Methods for establishing error bounds on the ROC are not well understood.  Each point on the ROC curve is calculated by integrating “genuine” and “impostor” distributions between zero and some threshold, t [7,8].  Traditionally, error bounds for the ROC at each threshold, t, have been found through a summation of the binomial distribution. The confidence, b, given a non-varying probability p, of K sample/template comparison scores, or fewer, out of N independent comparison scores being in the region of integration would be

                               (1)

This equation can be used to determine the required size of a biometric test, N, for a given level of confidence, b, if the error probability, p, is known in advance.  Of course, the purpose of the test is to determine the error probability, so, in general, the required number of comparison scores (and test subjects) cannot be predicted prior to testing.  To deal with this, “Doddington’s Law[1]” is to test until 30 errors have been observed.  If the test is large enough to produce 30 errors, we will be about 95% sure that the “true” value of the error rate lies within about 40% of that measured.  A nice paper on “Doddington’s Law”[9], done internally at ITT Technologies by Jack Porter, is now available from the NBTC.

Equation (1) will not be applicable to biometric systems if: 1) comparison scores are not independent; 2) the error probability varies across the population.  If cross-comparisons (all samples compared to all templates except the matching one) are used to establish the “impostor distribution”, the comparisons will not be independent and (1) will not apply. An equation for error bounds in this case has been given by Prof. Peter Bickel of the University of California Berkeley and is summarized in [1].   The varying error probability across the population (“goats” with high false non-match errors and “sheep” with high false match errors) similarly invalidates (1) as an appropriate equation for developing error bounds.  Consequently, we do not report error bounds on our experimental results. Developing appropriate equations for analytic computation of error bounds under “real-world” conditions of non-independence of the comparisons and non-stationarity of the error probabilities, and application to this problem of “re-sampling” techniques, is an important part of our current research.

The real tragedy in the break-down of equation (1) is in our inability to predict even approximately how many comparisons will be required to have “statistical confidence” in our results.  We currently have no way of estimating how large a test will be necessary to adequately characterize any biometric device in any application.

1.3  Test of Iris Scanning

We have begun data collection and testing of iris scanning technology.  Our goal to have a database of 100 enrollment/test iris image pairs irises available on CD-ROM has been achieved.  Our plan is to make this database available for development and testing.   We further plan to have an evaluative report on iris scanning available in the near future.

1.4  Additional Papers and Publications

We have written several other papers of possible interest to the biometrics community.  Challenges regarding the feasibility of large-scale identification systems, originally raised in reference [10], have continued [11].  In response, references [5, 12] were published to establish the feasibility of large-scale systems when multiple measures and database partitioning are used. 

Several papers [11,13,14] promoting “degrees of freedom” as a measure of biometric device performance have been published.  Although we recognize the importance of “entropy” as measured in the space of the samples and templates, we reject the notion that “degrees of freedom” in the reduced one-dimensional space of distance measures has any meaning in the assessment of biometric devices.  Our reasoning is put forth in reference [15].

Last summer (1998), several articles (deliberately not referenced) were published in mainstream newspapers alleging a U.S. government conspiracy to create a national identification card based on biometric information.  In response to this nonsense, we reviewed in reference [16] U.S. federal legislation pertaining to biometric identification.

2.0  Other Government Sponsored Activities
2.1  National Institute of Standards and Technology

The U.S. government Biometric Consortium (www.biometrics.org) is now co-chaired by Dr. Fernando Podio of the National Institute of Standards and Technology.  In February of 1999, NIST held a one-day meeting to discuss the interest in and feasibility of developing a standard format for fingerprint minutiae templates.  A follow-up meeting is scheduled during this CTST conference. 

The highly-respected Facial Recognition Technology (FERET) program, begun originally at the U.S. Army Research Laboratory, continues at NIST under Dr. Jonathan Philips[17-22]. The most recent study [20] compared the performance of eight algorithms against a standardized facial image database. Dr. Phillips has also recently co-edited a book on facial image recognition [21].  These test results are among the most interesting and thorough ever publicly reported in biometrics.

2.2  American National Standards Institute

The American National Standards Institute (ANSI) does not develop standards itself, but rather administrates and coordinates voluntary standardization efforts undertaken by the private sector (http://web.ansi.org).  At least two such efforts are currently underway:

X9F4 “Biometrics Management and Security for the Financial Services Industry” committee has been meeting to consider “adequate controls and proper procedures for using biometrics as an identification mechanism and/or authentication mechanism for secure remote electronic access or local physical access controls for the financial industry”.  The draft standard contains a glossary and adopts the biometric system description of reference [1] as normative.  Point of contact is Ms. Cynthia Fuller of the American Bankers Association.

The B10.8 Committee on Drivers’ Licensing Identity Documents Standards voted in June last year to establish two-index fingerprints as the “best practices” standard for use of biometrics on drivers’ licensing documents.  “Best practices” would also include “Appendix G” image quality standard and 15-to-1 WSQ image compression.  Point of contact is Mr. Geoff Slagle of the American Association of Motor Vehicle Administrators.

2.2  European Union -- BIOTEST

BIOTEST, under the direction of Dr. Tony Mansfield of the National Physical Laboratory (NPL), was an 18 month project for biometric testing funded by 12  “partners” and “associates” from 5 European countries. The project objectives were to: 1) define relevant metrics; 2) develop practical methodologies; 3) build databases of biometric samples; 4) measure device performance; 5) monitor market demands; 6) set up testing centers.  The project tested a commercially-available fingerprint system and a prototype hand geometry system and collected fingerprint and signature databases. No publicly available reports have been issued by the project.  Since the conclusion of BIOTEST,  NPL has maintained an active interest in biometric system evaluation under Dr. Mansfield’s direction.

2.3  United Kingdom -- Communication Electronic Security Group

The Communication Electronic Security Group (CESG) of the British Ministry of Defense has taken the lead in creating a Biometric Working Group to advise British the government on biometric technologies. Dr. Philip Statham is the point of contact for the project. The working group will be recognized as the “official” peer review process for governmental departments wishing to implement biometric solutions. Meetings will be held at two or three month intervals with the goal of developing advice documents and, perhaps, future testing and performance standards.  Both NPL and NBTC will participate in the working group sessions.  One excellent publication giving an overview of biometric technology has already been produced [23].

2.3  Japan – National Project of Test and Evaluation for Biometric Technologies

The Japanese Ministry of International Trade and Industry (MITI), through the Information-technology Promotion Agency, Japan, is sponsoring a one-year effort headed by Hitachi to study biometric technologies.  The goals of the project are: 1) to develop requirement guidelines for typical applications; 2) standardize measures of biometric device performance; 3) conduct experimental tests. The project is scheduled to conclude in December 1999. Project head is Dr. Yoichi Seto of the Hitachi Systems Development Laboratory. MITI is also sponsoring a second effort for API development.

2.4  China – The Biometrics Technology Center

The Biometrics Technology Center is supported by the Hong Kong Government at Hong Kong Polytechnic University to perform research on integrated biometric technologies. The Center aims to: 1) transfer multiple biometric technologies from university to industry; 2) provide a biometrics knowledge base for industry and technological advancement; 3) explore integrated biometric solutions to practical industrial applications. Dr. David Zhang, the center director, has attempted over the past year to create an international journal devoted to biometric authentication technology and will publish a book on biometrics later this year[24].  

2.5  Israel – BASEL Project

The Israeli government, as part of the BASEL project to use biometric access control at Gaza Strip border crossings, has conducted benchmark tests of several hand and finger imaging devices.  At this writing, the project is still in the procurement stage, so no discussion of results is available.  We hope that a description of the BASEL project will appear elsewhere in these CTST’99 proceedings.

2.6  Connecticut, U.S. -- Biometrics in Human Services Users’ Group

The State of Connecticut sponsors the Biometrics in Human Services Users’ Group, whose primary mission is to collect and disseminate information on the use of biometric authentication in social service applications. The primary point of contact is David Mintie. The group publishes a free, on-line newsletter six times per year which can be downloaded from www.dss.state.ct.us/digital.htm.  Several articles involving testing have appeared over the last year.  The web site also has a page devoted to an expanding collection of "Biometric Tutorials" and a page with updates on the status of active and pending state projects in biometric identification.

3.0 Non-Government Testing and Reporting Activities

The International Biometric Group (IBG), a for-profit organization, has completed comparative commercial device testing using eight fingerprint and two facial imaging systems. The 240 volunteers, chosen from four demographic groups (elderly, Asians, construction workers and artists, and a control group), enrolled and tested at a six-week interval.  The failure-to-enroll rate for some groups was significant for several devices.  The study was commissioned by the banking industry, but the final report is commercially available from the IBG (www.bio1.com).

The International Biometric Industry Association (IBIA) has been recently formed and incorporated as a legally-recognized trade industry association.  Although the IBIA neither sponsors nor performs testing, a recent draft policy paper encourages independent testing, stating,  “As further assurance that the devices work as promised, each member adheres to a strict Code of Ethics and attests that any stated product performance claims are accurate and can be independently verified by a competent authority.”  

4.0 Conclusions

 While everyone seems to agree that independent testing and evaluation of biometric devices is important and interesting, the high cost of testing with human subjects limits the number and size of tests that can be conducted each year.  Nonetheless, the past 12 months have yielded a number of important new results that can be used to design cost-effective operational systems.  

5.0 References

[1] J.L. Wayman, “Technical Testing and Evaluation of Biometric Identification Devices” in A. Jain, etal (eds), Biometrics: Personal Identification in a Networked Society, (Boston, Kluwer Academic Press, 1999)

[2] C. Frenzen, “Convolution Methods for Mathematical Problems in Biometrics”, Naval Postgraduate School Technical Report, NPS-MA-99-001, January 1999

[3] J.L. Wayman, “Biometric Identifier Standards Research Final Report”, College of Engineering, San Jose State University, October, 1997, sponsored by the Federal Highway Administration, downloadable from www.engr.sjsu.edu/biometrics/fhwa.html.

[4] J.L. Wayman, “Benchmarking Large-Scale Biometric System: Issues and Feasibility”, Proc. CTST Government’97, Sept. 1997

[5] J.L. Wayman, “Multi-Finger Penetration Rate and Roc Variability for Automatic Fingerprint Identification Systems”, National Biometric Test Center, May 1999.

[6] J.L. Wayman, “Error Rate Equations for the General Biometric System”, IEEE Automation and Robotics Magazine, March 1999

[7] J.L. Wayman, “Introduction To Biometric Authentication Technologies”, Proc. CTST’99, May 1999.

[8] J.L. Wayman, ““Testing and Evaluating Biometric Technologies:  What the Customer Needs To Know”, Proc. CTST’98, May 1998.

[9] J.Porter, “On the ’30 errors’ criterion”, ITT Industries/Speaker-Key internal document, April 1997, available from the National Biometric Test Center.

[10] R. Hopkins, “Benchmarking very large-scale identity systems”, Proc. CTST’97, Vol.II, pg. 313-332

[11] J. Daugman, “Recognizing Persons by Their Iris Patterns” in A. Jain, etal (eds), Biometrics: Personal Identification in a Networked Society, (Boston, Kluwer Academic Press, 1999)

[12] J.L. Wayman, “Continuing Controversy Over The Technical Feasibility Of Large-Scale Systems”, Biometrics in Human Services Users’ Group Newsletter #11, volume 2, no.5 ,November 1998, downloadable from www.dss.state.ct.us/digital.html

[13] G.O. Williams, “Iris Recognition Technology”, IEEE AES Systems Magazine, April 1997, pg. 23-29

[14] C. Wu, “Private Eyes”, Science News, Vol.153, No.14, April 4, 1998

[15] J.L. Wayman, “’Degrees of Freedom’ as a measure of biometric device performance”, AVANTI newsletter, Issue 2, Number 1, January 1999, available from the National Biometric Test Center.

[16] J.L. Wayman, “The State Of Biometrics: Standards, Alliances and Applications”, Proc. CTST Government, September 1999.

[17]P.J. Phillips, et al, “FERET (Face-Recognition Technology) Recognition Algorithm Development and Test Results”, Army Research Laboratory, ARL-TR-995, October 1996

[18] P.J. Rauss, et al, “FERET (Face-Recognition Technology) Recognition Algorithms”, Proceedings of ATRWG Science and Technology Conference, July 1996

[19] P.J. Phillips, et al, “The FERET Evaluation Methodology for Face-Recognition Algorithms”, Proc. IEE Conf.on Comp.Vis.and Patt. Recog., San Juan, Puerto Rico, June 1997

[20] S.A. Rizvi, etal, “The FERET Verification Testing Protocol for Face Recognition Algorithms”, NIST, NISTIR 6281, October 1998

[21] P.J. Phillips, etal, “The FERET Evaluation” in H. Wechsler, etal (eds) Face Recognition: From Theory to Applications (Springer-Verlag, Berlin, 1998)

[22] P.J. Phillips, “The FERET Database and Evaluation Procedure for Face-Recognition Algorithms”, Image and Vision Computing Journal, Vol. 16, No.5, 1998, pg. 295-306 

[23] Communications Electronic Security Group, “Potential for the Use of Biometric Authentication Technology in UK Government Systems”, Issue 1, September 1998

[24] D.Zhang, Automated Biometrics: Technologies & Systems, (Kluwer, 1999)