Detection and matching of facial marks in face images

Detección y correspondencia de marcas faciales en imágenes de rostros

Fabiola Becerra-Riera^1*, Annette Morales-González¹

¹ Advanced Technologies Application Center. 7a No. 21812, Siboney, Playa, CP 12200, Havana, Cuba {fbecerra,  amorales}@cenatav.co.cu

*Autor para la correspondencia: fbecerra@cenatav.co.cu ]]>  

ABSTRACT

Soft biometrics traits (e.g.  gender, ethnicity, facial marks) are complementary information in face recognition. Although they are not fully distinctive by themselves, recent studies have proven that they can be combined with classical facial recognition techniques to increase the accuracy of the process. Facial marks, in particular, have proven useful in reducing the search for the identity of individuals, although they do not uniquely identify them. Facial marks based systems provide specific and more significant evidence about the similarity between faces. In this paper we propose the use of facial marks (e.g.   moles, freckles, warts) to improve the face recognition process.  To that end, we implemented an algorithm for automatic detection of facial marks and we proposed two matching algorithms: one based on Histograms of Oriented Gradients (HoG) to represent the marks and the other based on the intensities of the pixels contained in each mark bounding box. Experimental results based on a set of 530 images (265 subjects) with manually annotated facial marks, show that the combination of traditional face recognition techniques with facial marks, increases the accuracy of the process.

Key words: Soft biometrics, facial marks, face recognition

RESUMEN

Los soft biometrics (e.g. género, raza, marcas faciales) constituyen información complementaria en el proceso de reconocimiento de rostros. Si bien no son totalmente discriminativos por sí solos, estudios recientes han comprobado que pueden ser combinados con técnicas clásicas de reconocimiento facial para incrementar la eficacia de dicho proceso. Las marcas faciales, de manera particular, han demostrado ser útiles en la reducción de la búsqueda de la identidad de individuos, pese a no identificarlos unívocamente.  Los sistemas basados
en marcas faciales proporcionan evidencia aún más específica y significativa de la similitud entre rostros. En el presente trabajo se propone el empleo de marcas faciales (e.g.  lunares, pecas, verrugas) en beneficio del reconocimiento.  Para tales fines se implementó un algoritmo de detección automática de marcas faciales y se propusieron dos algoritmos de correspondencia de marcas:  uno basado en Histogramas de Gradientes Orientados (HoG) para establecer la representación de las marcas y el otro en las intensidades de los píxeles contenidos en la región rectangular correspondiente a cada marca.   Los resultados experimentales basados en un conjunto de 530 imágenes (265 sujetos) con marcas faciales anotadas manualmente, muestran que la combinación de técnicas clásicas de reconocimiento de rostros (e.g.  LBP) con marcas faciales, aumenta la eficacia del proceso.

Palabras clave: Soft biometrics, marcas faciales, reconocimiento de rostros

INTRODUCTION

Facial recognition is one of the most employed biometric application in recent years, becoming an active research area that covers various disciplines such as image processing, pattern recognition and computer vision.

Soft biometrics (e.g. gender, ethnicity, facial marks) (JAIN A. K., 2010) provide additional information in images of faces and are not fully discriminative by themselves; however, it has been found recently that they may be properly combined with classical facial recognition techniques to increase the accuracy in the verification or identification of individuals. In particular, facial marks have proven to be extremely useful: despite not uniquely identifying an individual, they can be used to narrow down the search of his identity.

Several approaches for automatically detecting and matching facial marks are proposed in the literature. There are approaches for the detection of moles prominent enough to be used (by themselves) in the process of identification (PIERRARD J. S., 2007), for the detection of marks covered by cosmetics (CHOUDHURYZ. H., 2012), others that introduce scars, marks and tattoos in image retrieval systems (LEE J. E., 2008) and authors who focus on the automatic classification of acne scars (RAMLI R., 2011; NIRMAL B., 2013). Unlike these works, based on the identification of only one particular type of facial mark, (PARK U., 2010) proposes a method which differs significantly from previous studies. It detects all types of facial marks that appear as locally prominent regions, and focuses on the detection of semantically significant facial marks, instead of the extraction of textural patterns that implicitly include marks. Facial marks matching has been addressed from different perspectives: there are algorithms that rely on a weighted Euclidean distance as a basis for the matching (ZHANG Z., 2009), others employ the intersection between histograms (PARK U., 2011), and others are based on a weighted bipartite graph (SRINIVAS N., 2011) to establish the similarity.

In the present work we implemented an algorithm for automatic detection of facial marks based on (PARK U., 2010), changing some steps to improve the process, and we propose two new methods to establish the similarity between the marks in face images. Our validation on a manually annotated face image set showed that including facial marks for face recognition reduces the final error of the process.

The paper is structured as follows. First, we present the algorithm for the detection of facial marks, followed by the description of our two proposals for facial marks matching between two face images. Next, we de- scribe the process of experimentation and the results obtained, and finally we present the conclusions and recommendations of our investigation.

Facial mark detection algorithm

Facial marks are usually manifested as locally prominent regions, thus a second-order derivative edge detector can be used for their detection. However, the direct application of a detector of this type on the face image can generate a large number of false positive, mostly because of the presence of primary facial features (e.g. eyes, nose, mouth). The location of such features for their subsequent extraction of the facial area, is a necessary step for the successful detection of facial marks.

Primary facial feature detection and mapping to mean shape

Mask construction

Following the procedure described in (PARK U., 2010), a general mask Mg (Figure. 1 (a)) was built from T_µ to suppress possible false positive generated by primary facial features, during the process of detection of regions. Since M_gdoes not cover the particular characteristics of each individual (e.g. beard, small wrinkles around eyes and mouth), that could be wrongly detected as potential facial marks, a user-specific mask M_s (Figure. 1 (c)) was constructed as the sum of M_g and the edges that are connected to M_g. The Canny edge detector (CANNY, 1986) was used for the detection of these peculiarities (Figure. 1 (b)) instead of the Sobel operator, proposed in (PARK U., 2010). Characterized by its good immunity to noise and the ability to detect true edge points with less error, Canny has shown better results than the Sobel operator.  As can be seen in Figure. 1, the use of M_s in (e) largely reduces the false positives that appear near the eyes and mouth in (d), where only M_g was employed.

Detection of facial marks

Since facial marks usually appear as isolated and salient regions corresponding to notable changes in intensity, a second-order derivative edge detector was selected for their detection, in this case, the Laplacian-of-Gaussian (LoG) filter (MARR, 1980), as proposed in (PARK U., 2010).

The LoG filtered image subtracted with the user-specific mask (Ms) undergoes a binarization process with a series of threshold values t_i (i = 1, ..., K) in a decreasing order. The threshold t_i is successively applied until the number of resulting connected components is greater than a pre-set value cc. After the detection of a minimum cc = 10, the facial marks candidates whose size do not exceed two pixels of width and height are discarded, in order to eliminate pixels or noisy areas that are false positives. Finally, the detected marks are identified by means of a bounding rectangle, indicating its position and size. The complete facial marks detection procedure is illustrated in Figure. 2.

Facial mark matching process

One of the objectives proposed in this research was the development of an algorithm to determine the similarity of two face images, based on their facial marks. Consequently, a representation of the marks is necessary as a first step before starting the matching process.

Facial mark representation

Despite the fact that a representation of the appearance of the marks is necessary, it is not discriminative enough by itself: two marks can be very similar in appearance and, however, they can be located in different regions of the face. It is essential that, for example, a mark located in the forehead is not wrongly considered similar to one located on the chin. The spatial distribution of marks within the face is an important factor to consider. For the spatial representation of a mark we employed the x and y central coordinates of its bounding box, normalizing their values in the range [0, 1] through the division between the width and height of the image, respectively. Finally, for the second approach, a facial mark is represented by the vector resulting from concatenating its central coordinates with the appearance features represented by the 8-dimensional HoG. This is a very compact representation, that will allow later for a faster comparison between marks.

Matching facial marks

Given two images l₁ and l₂ , and given N₁ and N₂ as the sets of their detected facial marks, respectively, the similarity between l₁ and l₂ (S) was established following two different approaches. A first concept of similarity was defined only taking into consideration the facial marks detected in l₁ and trying to make them correspond in l₂. For this purpose we employed the first representation of marks based only on the intensity values of pixels. In this case, the similarity between l₁ and l₂ can be computed as shown in Eq. 1,

where, for each mark ni ∈ N₁ a rectangular region R_i was built around its central coordinates in l₂ , as an area of potential matching. B represents the score of similarity associated with the best matching of the mark n_i in region R_i , established by a Normalized Cross Correlation (NCC) (BRIECHLE K., 2001). Scores of B below a certain threshold t, established in an empirical way as 0.5, were not taken into account for the final computation.

A second approach, for which it was used the criterion of representation based on HoG, was developed through a more explicit matching between the facial marks sets N₂ and N₂. It is shown in Eq. 2,

where, for each mark nj ∈ N₂, x_j and y_j  are its spatial central coordinates. In this case, the region R_i was used so that only those nj ∈ N₂ , contained in R_i, were considered in the process of matching. This ensures a spatial coherence in the matching of the marks, i.e, very distant facial marks are not verified.

D is a measure of the distance between the marks, in this case computed with the Bhattacharyya distance (BHATTACHARYYA, 1943), which showed the best results for compare histograms during the experimental process. For the score using Bhattacharyya, low values indicate good correlation while high values correspond to little similarity.   Being d₁₂  the distance between two representatives vectors v1  and v2,  the final score corresponds to 1 − d₁₂, so higher values respond to greater similarity.

EXPERIMENTS AND RESULTS

The process of experimentation was divided into two fundamental parts: the evaluation of the detection of facial marks and the face image verification using facial marks.

To validate our results, a set of images with manually annotated facial marks was created, since we were not able to find in the related literature references to international public databases to perform validation of facial mark detection. As a result of the annotation process, we obtained a collection of 530 images (265 subjects) with an average of 5 facial marks each. Moles, freckles, grains and warts were the most frequently found marks; few images of faces were found with birthmarks, enlightened, darkened areas and pockmarks; and there were no scars or tattoos.

Evaluation of the facial marks detection

The validation of the accuracy of the detection algorithm proposed for the defined set of 530 images, was conducted taking into account the measures of Precision and Recall.

We extrapolated the standard criterion to evaluate face detection (JAIN V., 2010) to the problem of facial marks detection, in order to establish the similarity between a detected facial mark and an annotated one. Given an image I with A annotated marks, the detection of a facial mark ni was considered correct if ∃n_a ∈ A such that:

The threshold t₀  was established empirically as 0.4.

It was not possible to find available implementations of existing facial marks detection algorithms in the literature, therefore, in order to compare our detection results with those of other facial marks detection algorithms on the same image set, we employed the Viola and Jones object detector (VIOLA P., 2001). This detector, although initially created for face detection, can be trained to detect a variety of objects. In order to detect facial marks, it was trained with a set of 150 positive samples (images of facial marks) and 500 negative samples (images of areas of skin without marks). The results obtained are shown in Tab. 1. As can be seen, the proposed detection algorithm achieves higher values of precision and recall than the Viola and Jones detector, which means that it is able to detect more correct marks (superiority in recall) and less false positives (superiority in precision).

In order to evaluate the accuracy of the facial marks (on their own) in the verification process, we established a comparison between the results obtained using a nearest neighbor classier with Local Binary Patterns (LBP) (OJALA T., 1996) features, one of the most popular facial descriptors, and the proposed facial marks matching algorithms: one based on the representation of the marks through the intensity of their pixels (we will call it Pixel FM) and the other using HoG (we will call it HoG FM).

Tab. 2 shows the results in terms of Equal Error Rate (EER) and False Recognition Rate (FRR) when the Operation Point (OP) is set to a False Acceptance Rate (FAR) of 0.1 %. As can be seen in the first three rows of 2, by means of a verication based only in facial marks (Pixel FM and HoG FM), it is not possible to achieve good results, which is an indication that soft biometrics are not discriminative by themselves.

In order to reduce the errors, the results of the proposed facial marks matching algorithms were combined with the results of the LBP comparison. The combination was performed with the output scores of each process, which is possible since they are in the same range ([0, 1]) and they share the same behavior: higher values correspond to more similar faces. The combination was established by the weighted sum of the scores obtained in the following way: represent the weights associated with the score achieved by making use of the LBP (p1) and the proposed facial marks matching algorithm (p2), respectively. As shown in Tab. 2, facial marks, combined with other classical facial recognition

techniques (LBP in this case), improve the accuracy in the verication process. The best combination LBP - HoG FM was able to decrease the EER with respect to the LBP alone, but more signicant to this is that both combinations were able to reduce the rate of FRR when the FAR = 0.1%. This means that it was possible to reduce the rate of false rejected for very low values of false accepted, i.e., true genuine subjects that before were wrongly classied as impostors (false rejected) were correctly classied now thanks to their facial marks.

CONCLUSIONS

Making use of the proposed algorithms for facial marks detection and matching we were able to confirm that facial marks, used on their own for recognition, are not discriminative enough, as it was already stated in the literature. However, by combining the algorithms of facial marks matching with classical techniques for face recognition, we were able to achieve lower error rates in our face verification experiments, which shows the usefulness of this type of trait as complementary information. Of the two proposed facial marks matching approaches, the one which uses the representation based on HoG (HoG FM) obtained the best results in terms of EER, than the one based only on the intensity of pixels (Pixel FM).

As future work we propose to incorporate new types of facial marks to achieve a better description of individuals and we also plan to develop algorithms for the classification of marks, thus allowing to obtain images of interest through the filtering of large databases, making use of semantic queries.

REFERENCIAS BIBLIOGRÁFICAS

BHATTACHARYYA, A. (1943). On a measure of divergence between two statistical populations defined by their probability distributions. Bulletin of Cal. Math. Soc., 35(1):99–109.

BRIECHLE K., H. U. D. (2001). Template matching using fast normalized cross correlation. In Proceedings of SPIE: Optical Pattern Recognition XII, volume 4387, pages 95–102.

CANNY, J. (1986). A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell., 8(6):679–698.

CHOUDHURY Z. H., M. K. M. (2012). Robust facial marks detection method using aam and surf. International Journal of Engineering Research and Applications (IJERA), 2(6).

JAIN A. K., K. A. (2010). Biometrics of Next Generation: An Overview. Springer.

JAIN V., L.-M. E. (2010). Fddb:  A benchmark for face detection in unconstrained settings. Technical report.

MARR, D., H. E. (1980). Theory of edge detection. In Proceedings of the Royal Society of London. Series B,Biological Sciences, volume 207, pages 187–217.

ME´ NDEZ N., CHANG L., P. Y. M. H. (2013). Facial landmarks detection using extended profile lbp-based active shape models. In Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications- 18th Iberoamerican Congress, CIARP 2013, Havana, Cuba.

NAVNEET D., B. T. (2005). Histograms of oriented gradients for human detection. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) - Volume 1 - Volume 01, CVPR ’05, pages 886–893. IEEE Computer Society.

NIRMAL B., DEY B. C., G. R. R. (2013). Automatic detection of acne scars: Preliminary results. In IEEE Point-of-Care Healthcare Technologies (PHT), Bangalore, India.

OJALA T., PIETIKAINEN M., H. D. (1996). A comparative study of texture measures with classification based on featured distributions. Pattern Recognition, 29(1):51–59.

PARK U., J. A. K. (2010). Face matching and retrieval using soft biometrics. IEEE Transactions on Infor- mation Forensics and Security, 5(3):406–415.

PARK U., LIAO S., K. B. V. J. J. A. K. (2011). Face finder: Filtering a large face database using scars, marks and tattoos. Technical Report MSU-CSE-11-15, Department of Computer Science, Michigan State University.

PIERRARD J. S., V. T. (2007). Skin detail analysis for face recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society.

RAMLI R., MALIK A. S., H. A. F. M. Y. F. B. B. (2011). Segmentation of acne vulgaris lesions. In International Conference on Digital Image Computing: Techniques and Applications (DICTA), Noosa, QLD, Australia, pages 335–339. IEEE Computer Society.

SRINIVAS N., AGGARWAL G., F. P. J. B. R. W. V. (2011). Facial marks as biometric signatures to distinguish between identical twins. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

VIOLA P., J. M. (2001). Robust real-time object detection. In International Journal of Computer Vision.

ZHANG Z., TULYAKOV S., G. V. (2009). Combining facial skin mark and eigenfaces for face recognition. In ICB, volume 5558 of Lecture Notes in Computer Science. Springer.

Recibido: 12/10/2015
Aceptado: 15/12/2015