Examining the Effects of Supervised Descent Method as 3D Facial Landmark Localization Method on 3D Face Recognition, and Introduction of the Bosphorus 3D Face Dataset

Beneficiary: Necati Cihan Camgöz, Master’s Student at Boğaziçi University
Host:  Assist. Prof. Dr. Vitomir Struc, University of Ljubljana, Faculty of Electrical Engineering
Period: 27/04/2015 to 01/05/2015

Purpose of the STSM

First and far most important goal of this short term scientific mission to Ljubljana University was to establish a fruitful collaboration between two institutes while gaining further knowledge on my research field from my esteemed host. Our goals further included to share knowledge on facial landmark localization methods, analyze experimental results which were conducted on Bosphorus 3D face dataset, discuss possible improvements over the previously developed Supervised Descent Method (SDM) based approaches, and examine the effects of having better facial landmark localization on 3D face recognition.

Description of the work carried out during the STSM

By working with Assist. Prof. Dr. Vitomir Struc and his post-doctoral scholar colleague Dr. Janez Krizaj, who are experts on 3D facial landmark localization and had their own approach based on SDM, we were able assess the performance of our methods and discuss the possible ways to improve 3D facial landmark localization. Since the host was already working on the Bosphorus 3D face dataset there was no need for an introduction. We can categorize the work carried out during the STSM in to the following topics:

Creating the Experimental Setup:

Taking state-of-the art methods on 3D Facial Landmark Localization into consideration, we have created our experimental setup on Bosphorus 3D face dataset; so that we would be able to compare our results with them. In our setup we divided the dataset in to two approximately equal, user exclusive parts and did two fold cross-validation for our experiments.

Comparing 3D Facial Landmark Localization Methods and Analyzing the Experimental Results:

Since both parties had their own 3D facial landmark localization approach based on SDM, we experimented with both of the approaches. Our experiments showed us that the approach proposed by the host had higher 3D facial landmark localization accuracy. Furthermore, by comparing with the state-of-the-art methods we have realized that this approach had state-of-the-art performance while working with 10 and 22 most commonly used landmarks on Bosphorus 3D face dataset.

To examine the effects of the features which have been used to describe 3D facial landmarks we have experimented with various sizes of Histogram of Oriented Gradients (HOG) and Scale Invariant Feature Transform (SIFT). These experiments showed us that using HOG features generally resulted in higher accuracy of 3D facial landmark localization. Also finding the optimal size for the features affected the performance drastically. We also experimented with changing the size of the features in each step of the SDM, which gave the approach a coarse-to-fine alignment effect and increased the performance.

Discussing ways to improve the current 3D Facial Landmark Localization Method:

By examining the experimental results we came to the conclusion that there are two ways we can improve the current approach. First is by coming up with a better structure for SDM, i.e. a cascaded approach, which will allow the landmark localization to be more precise as it iterates through its stages. The second idea is to use or create better features for describing 3D facial landmarks, which will exploit the 3D information of the data.

Examining the Effects of Better Facial Landmark Localization Method on 3D Face Recognition:

A naïve 3D face recognition method was applied to faces that were aligned using facial landmarks found by both parties’ approaches. Although the face recognition performances were lower than the state-of-the-art, experiments showed us that the approach with better facial landmark localization resulted in more successful 3D face recognition.

Description of the main results obtained

Our experiments showed us that the approach proposed by the host had state-of-the-art 3D facial landmark localization performance while working with 10 and 22 most commonly used facial landmarks on Bosphorus 3D face dataset.

We have also realized that working with different features can affect the 3D landmark localization performance drastically (HOG vs. SIFT). To this extent we started on experimenting with different features, which may be more suitable for 3D data.

To examine the effects of having better facial landmark localization on 3D face recognition, we have experimented with a naïve 3D face recognition approach which used the facial landmark locations from both parties’ approaches to align faces. We have seen that better facial landmark localization results in a higher 3D face recognition accuracy, thus proving the 3D facial landmark localization being crucial for biometric applications that work on 3D data.

Future collaboration with host institution

The experiments during the STSM showed that the method in our hands has the state-of-the-art 3D facial landmark localization performance. Currently we are experimenting on ways to improve this method by trying out cascaded structures and new features which will exploit the 3D information. Our goal is to turn this work into a publication as our collaboration continues.

Building a Robust and Informative System for SPN-based Image Forgery Detection

Beneficiary:  Xufeng Lin, University Of Warwick, xufeng.lin@warwick.ac.uk

Host:  Prof.  Andreas Uhl, University of Salzburg, uhl@cosy.sbg.ac.at

Period: 01/05/2015 to 31/05/2015

Place: Salzburg (Austria)

1.      Purpose of the STSM

 

The purpose of this STSM is to build a robust and informative system of SPN-based image forgery detection and enhance the collaboration between digital forensics and biometrics.

 

Although there are many approaches proposed to improve the performance of SPN-based image forgery detection, each of them only focuses on one or several components of the detection framework. An integrated approach for assembling the existing methods to provide superior performance is still lacking.  Therefore, we try to build a robust and informative system of SPN-based image forgery detection by combining the existing state-of-the-art methods. It would be also very interesting and reasonable to apply our experience of camera sensor identification to the analysis of biometric datasets in WaveLab.

2.      Description of the work carried out during the STSM

 

During the STSM, the following work has been carried out:

 

      • I first implemented several state-of-the-art SPN extraction, preprocessing and enhancement algorithms. Different combinations of these algorithms have been tried and evaluated aiming at achieving the “best” quality of camera fingerprint, which lies the foundation for SPN-based image forgery detection. The combination of the algorithms makes the SPN estimation more robust against the interferences introduced from the difference procedures in the image acquisition pipeline.

 

      • After obtaining the fingerprint of camera, image forgery is exposed by comparing the normalized cross correlation (NCC) between two patches from the same location of camera fingerprint and the noise residual of the suspicious image with a pre-defined threshold. As the computational cost of NCC, straightforwardly localizing the pixel-level forgery becomes computationally prohibitive. Therefore, a fast pixel-level NCC calculation method was proposed by pre-calculating the local mean and variance of the camera fingerprint and the noise residual of the suspicious image.

 

      • One problem of existing SPN-based image forgery detection algorithms is that the output is a binary image indicating the pixels have been forged, which will easily lead to omissions and false positives. By incorporating the prior knowledge about the quality of SPN as well as the theoretical analysis of the distribution of correlation values, I converted the detection results into a probability map, which can provide the forensic investigators with more rich and instructive information for decision-making.

 

      • Collaborated with Wavelab group, we frequently discussed and shared experience in the area of image clustering based on SPN. In the later phase of this STSM, we jointly conducted image clustering experiments on CASIA iris dataset and achieved very promising results.

 

      • Collaborated with Wavelab group, we devised a fast correlation-based image alignment algorithm using the noise residuals extracted from iris images. This alignment algorithm serves as a preprocessing step for sensor aging rate estimation.

 

 

 

3.      Future collaboration with the host institution (if applicable)

 

During the STSM, I worked with the members of Wavelab group and tried to apply the sensor pattern noise technology to analyze the biometric datasets. The outcome of the collaborative work shows the effectiveness of sensor pattern noise in analyzing the Iris databases. So the future collaboration will focus on extending the applications of sensor pattern noise in the area of digital biometrics.