FaceAlignment : A Machine Learning Model For Recognizing Key Points On a Face

This is an introduction to「FaceAlignment」, a machine learning model that can be used with ailia SDK. You can easily use this model to create AI applications using ailia SDK as well as many other ready-to-use ailia MODELS.

Overview

FaceAlignment takes a face image as input and outputs 68 keypoints. The input resolution is (1,3,256,256) and the output of the model is a heatmap of (1,68,64,64). For each of the 68 keypoints, a confidence of (64,64) resolution is output. The input images are normalized to (0–1.0) in BGR order.

How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial…Abstract This paper investigates how far a very deep neural network is from attaining close to saturating performance…www.adrianbulat.com

FaceAlignment results

Considering the input image shown below.

Source：https://pixabay.com/ja/photos/%E3%83%95%E3%82%A1%E3%83%83%E3%82%B7%E3%83%A7%E3%83%B3-%E3%82%A2%E3%82%B8%E3%82%A2-%E6%97%A5%E6%9C%AC-3179178/

FaceAlignment works on the face area, so it first cuts out the face area and then performs the recognition process.

Cut out of the face

FaceAlignment can extract 2D keypoints with high accuracy even for profile faces.

FaceAlignment output in 2D

You can also use the 3D mode to extract key points in 3D.

FaceAlignment output in 3D

The output of the model will be a heat map like the one below.

FaceAlignment heatmap output

The coordinates of the key points are computed by detecting the maximum value in each heatmap.

For the computation of the 3D keypoints, the 2D keypoints are first calculated from the heatmaps. Then the 3 channels of the input image and the 68 channels of the 2D keypoint heatmap are concatenated, making a (71,256,256) input, and then fed to the depth estimation model. The output of the depth estimation model will be (1,68) z-values.

The assignment of the 68 keypoints conforms to the Multi-PIE format.

The 68 Multi-PIE landmarks scheme and the landmarks selected for our method marked by the circles.（Source:https://www.researchgate.net/publication/311741971_Automatic_cheek_detection_in_digital_image)s）

Architecture

FaceAlignment uses The Face Alignment Network (FAN), which is a stack of HG (Hourglass) in structure.

Source：https://www.adrianbulat.com/downloads/FaceAlignment/FaceAlignment.pdf

Usage

The following sample demonstrates how to use FaceAlignement with ailia SDK.

ailia-ai/ailia-models(from https://github.com/1adrianb/face-alignment/tree/master/test/assets) Ailia input shape : (1, 3, 256, 256) Range …github.com

You can use the following command to get the 2D keypoints of a face for any image.

$ python3 face_alignment.py -i input.png -s output.png

You can use the following command to get the 3D keypoints.

$ python3 face_alignment.py -i input.png -s output.png — active-3d

Overview

FaceAlignment results

Architecture

Usage

Related topic