Cross Domain Facial Expression Recognition Benchmark

Implementation of papers:

Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)，2021.
Tianshui Chen*, Tao Pu*, Hefeng Wu, Yuan Xie, Lingbo Liu, Liang Lin.
Adversarial Graph Representation Adaptation for Cross-Domain Facial Expression Recognition
ACM International Conference on Multimedia (ACM MM), 2020. (Oral Presentation)
Yuan Xie, Tianshui Chen, Tao Pu, Hefeng Wu, Liang Lin.

Environment

Ubuntu 16.04 LTS, Python 3.5, PyTorch 1.3

Note: We also provide docker image for this project, click here. (Tag: py3-pytorch1.3-agra)

Datasets

To apply for the AFE, please complete the AFE Database User Agreement and submit it to [email protected] or [email protected].

Note:

The AFE Database Agreement needs to be signed by the faculty member at a university or college and sent it by email.
In order to comply with relevant regulations, you need to apply for the image data of the following data sets by yourself, including CK+, JAFFE, SFEW 2.0, FER2013, ExpW, RAF.

Pre-Train Model

You can download pre-train models in Baidu Drive (password: tzrf) and OneDrive.

Note: To replace backbone of each methods, you should modify and run getPreTrainedModel_ResNet.py (or getPreTrainedModel_MobileNet.py) in the folder where you want to use the method.

Usage

Before run these script files, you should download datasets and pre-train model, and run getPreTrainedModel_ResNet.py (or getPreTrainedModel_MobileNet.py).

Run ICID

cd ICID
bash Train.sh

Run DFA

cd DFA
bash Train.sh

Run LPL

cd LPL
bash Train.sh

Run DETN

cd DETN
bash TrainOnSourceDomain.sh     # Train Model On Source Domain
bash TransferToTargetDomain.sh  # Then, Transfer Model to Target Domain

Run FTDNN

cd FTDNN
bash Train.sh

Run ECAN

cd ECAN
bash TrainOnSourceDomain.sh     # Train Model On Source Domain
bash TransferToTargetDomain.sh  # Then, Transfer Model to Target Domain

Run CADA

cd CADA
bash TrainOnSourceDomain.sh     # Train Model On Source Domain
bash TransferToTargetDomain.sh  # Then, Transfer Model to Target Domain

Run SAFN

cd SAFN
bash TrainWithSAFN.sh

Run SWD

cd SWD
bash Train.sh

Run AGRA

cd AGRA
bash TrainOnSourceDomain.sh     # Train Model On Source Domain
bash TransferToTargetDomain.sh  # Then, Transfer Model to Target Domain

Result

Souce Domain: RAF

Methods	Backbone	CK+	JAFFE	SFEW2.0	FER2013	ExpW	Mean
ICID	ResNet-50	74.42	50.70	48.85	53.70	69.54	59.44
DFA	ResNet-50	64.26	44.44	43.07	45.79	56.86	50.88
LPL	ResNet-50	74.42	53.05	48.85	55.89	66.90	59.82
DETN	ResNet-50	78.22	55.89	49.40	52.29	47.58	56.68
FTDNN	ResNet-50	79.07	52.11	47.48	55.98	67.72	60.47
ECAN	ResNet-50	79.77	57.28	52.29	56.46	47.37	58.63
CADA	ResNet-50	72.09	52.11	53.44	57.61	63.15	59.68
SAFN	ResNet-50	75.97	61.03	52.98	55.64	64.91	62.11
SWD	ResNet-50	75.19	54.93	52.06	55.84	68.35	61.27
Ours	ResNet-50	85.27	61.50	56.43	58.95	68.50	66.13

Methods	Backbone	CK+	JAFFE	SFEW2.0	FER2013	ExpW	Mean
ICID	ResNet-18	67.44	48.83	47.02	53.00	68.52	56.96
DFA	ResNet-18	54.26	42.25	38.30	47.88	47.42	46.02
LPL	ResNet-18	72.87	53.99	49.31	53.61	68.35	59.63
DETN	ResNet-18	64.19	52.11	42.25	42.01	43.92	48.90
FTDNN	ResNet-18	76.74	50.23	49.54	53.28	68.08	59.57
ECAN	ResNet-18	66.51	52.11	48.21	50.76	48.73	53.26
CADA	ResNet-18	73.64	55.40	52.29	54.71	63.74	59.96
SAFN	ResNet-18	68.99	49.30	50.46	53.31	68.32	58.08
SWD	ResNet-18	72.09	53.52	49.31	53.70	65.85	58.89
Ours	ResNet-18	77.52	61.03	52.75	54.94	69.70	63.19

Methods	Backbone	CK+	JAFFE	SFEW2.0	FER2013	ExpW	Mean
ICID	MobileNet V2	57.36	37.56	38.30	44.47	60.64	47.67
DFA	MobileNet V2	41.86	35.21	29.36	42.36	43.66	38.49
LPL	MobileNet V2	59.69	40.38	40.14	50.13	62.26	50.52
DETN	MobileNet V2	53.49	40.38	35.09	45.88	45.26	44.02
FTDNN	MobileNet V2	71.32	46.01	45.41	49.96	62.87	55.11
ECAN	MobileNet V2	53.49	43.08	35.09	45.77	45.09	44.50
CADA	MobileNet V2	62.79	53.05	43.12	49.34	59.40	53.54
SAFN	MobileNet V2	66.67	45.07	40.14	49.90	61.40	52.64
SWD	MobileNet V2	68.22	55.40	43.58	50.30	60.04	55.51
Ours	MobileNet V2	72.87	55.40	45.64	51.05	63.94	57.78

Souce Domain: AFE

Methods	Backbone	CK+	JAFFE	SFEW2.0	FER2013	ExpW	Mean
ICID	ResNet-50	56.59	57.28	44.27	46.92	52.91	51.59
DFA	ResNet-50	51.86	52.70	38.03	41.93	60.12	48.93
LPL	ResNet-50	73.64	61.03	49.77	49.54	55.26	57.85
DETN	ResNet-50	56.27	52.11	44.72	42.17	59.80	51.01
FTDNN	ResNet-50	61.24	57.75	47.25	46.36	52.89	53.10
ECAN	ResNet-50	58.14	56.91	46.33	46.30	61.44	53.82
CADA	ResNet-50	72.09	49.77	50.92	50.32	61.70	56.96
SAFN	ResNet-50	73.64	64.79	49.08	48.89	55.69	58.42
SWD	ResNet-50	72.09	61.50	48.85	48.83	56.22	57.50
Ours	ResNet-50	78.57	65.43	51.18	51.31	62.71	61.84

Methods	Backbone	CK+	JAFFE	SFEW2.0	FER2013	ExpW	Mean
ICID	ResNet-18	54.26	51.17	47.48	46.44	54.85	50.84
DFA	ResNet-18	35.66	45.82	34.63	36.88	62.53	43.10
LPL	ResNet-18	67.44	62.91	48.39	49.82	54.51	56.61
DETN	ResNet-18	44.19	47.23	45.46	45.39	58.41	48.14
FTDNN	ResNet-18	58.91	59.15	47.02	48.58	55.29	53.79
ECAN	ResNet-18	44.19	60.56	43.26	46.15	62.52	51.34
CADA	ResNet-18	72.09	53.99	48.39	48.61	58.50	56.32
SAFN	ResNet-18	68.22	61.50	50.46	50.07	55.17	57.08
SWD	ResNet-18	77.52	59.15	50.69	51.84	56.56	59.15
Ours	ResNet-18	79.84	61.03	51.15	51.95	65.03	61.80

Methods	Backbone	CK+	JAFFE	SFEW2.0	FER2013	ExpW	Mean
ICID	MobileNet V2	55.04	42.72	34.86	39.94	44.34	43.38
DFA	MobileNet V2	44.19	27.70	31.88	35.95	61.55	40.25
LPL	MobileNet V2	69.77	50.23	43.35	45.57	51.63	52.11
DETN	MobileNet V2	57.36	54.46	32.80	44.11	64.36	50.62
FTDNN	MobileNet V2	65.12	46.01	46.10	46.69	53.02	51.39
ECAN	MobileNet V2	71.32	56.40	37.61	45.34	64.00	54.93
CADA	MobileNet V2	70.54	45.07	40.14	46.72	54.93	51.48
SAFN	MobileNet V2	62.79	53.99	42.66	46.61	52.65	51.74
SWD	MobileNet V2	64.34	53.52	44.72	50.24	55.85	53.73
Ours	MobileNet V2	75.19	54.46	47.25	47.88	61.10	57.18

Mean of All Methods

Souce Domain: RAF

Backbone	CK+	JAFFE	SFEW2.0	FER2013	ExpW	Mean
ResNet-50	75.87	54.30	54.49	54.82	62.09	59.51
ResNet-18	69.43	51.88	47.94	51.72	61.26	56.45
MobileNet V2	60.78	45.15	39.59	47.92	56.46	49.98

Souce Domain: AFE

Backbone	CK+	JAFFE	SFEW2.0	FER2013	ExpW	Mean
ResNet-50	65.41	57.93	47.04	47.26	57.87	55.10
ResNet-18	60.23	56.25	46.95	47.57	58.34	53.87
MobileNet V2	63.57	48.46	40.14	44.91	56.34	50.68

Citation

@article{chen2020cross,
  title={Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning},
  author={Chen, Tianshui and Pu, Tao and Wu, Hefeng and Xie, Yuan and Liu, Lingbo and Lin, Liang},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2021},
  pages={1-1},
  doi={10.1109/TPAMI.2021.3131222}
}

@inproceedings{xie2020adversarial,
  title={Adversarial Graph Representation Adaptation for Cross-Domain Facial Expression Recognition},
  author={Xie, Yuan and Chen, Tianshui and Pu, Tao and Wu, Hefeng and Lin, Liang},
  booktitle={Proceedings of the 28th ACM international conference on Multimedia},
  year={2020}
}

Contributors

For any questions, feel free to open an issue or contact us:

Adversarial Graph Representation Adaptation for Cross-Domain Facial Expression Recognition (AGRA, ACM 2020, Oral)

Related tags

Overview

Cross Domain Facial Expression Recognition Benchmark

Environment

Datasets

Pre-Train Model

Usage

Run ICID

Run DFA

Run LPL

Run DETN

Run FTDNN

Run ECAN

Run CADA

Run SAFN

Run SWD

Run AGRA

Result

Souce Domain: RAF

Souce Domain: AFE

Mean of All Methods

Souce Domain: RAF

Souce Domain: AFE

Citation

Contributors

Owner

Code for our EMNLP 2021 paper "Learning Kernel-Smoothed Machine Translation with Retrieved Examples"

OpenLT: An open-source project for long-tail classification

Repository for the paper "From global to local MDI variable importances for random forests and when they are Shapley values"

A PyTorch implementation of Mugs proposed by our paper "Mugs: A Multi-Granular Self-Supervised Learning Framework".

Motion planning algorithms commonly used on autonomous vehicles. (path planning + path tracking)

Evaluation suite for large-scale language models.

A quantum game modeling of pandemic (QHack 2022)

A torch.Tensor-like DataFrame library supporting multiple execution runtimes and Arrow as a common memory format

An end-to-end framework for mixed-integer optimization with data-driven learned constraints.

An open-source Kazakh named entity recognition dataset (KazNERD), annotation guidelines, and baseline NER models.

Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology (LMRL Workshop, NeurIPS 2021)

This package contains deep learning models and related scripts for RoseTTAFold

i3DMM: Deep Implicit 3D Morphable Model of Human Heads

In this project we investigate the performance of the SetCon model on realistic video footage. Therefore, we implemented the model in PyTorch and tested the model on two example videos.

Prompts - Read a textfile of prompts and import into anki via ankiconnect

The code release of paper 'Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization' NIPS 2020.

The authors' official PyTorch SigWGAN implementation

Paddle implementation for "Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation" (NAACL 2021)

Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs, ICCV 2021

Empirical Study of Transformers for Source Code & A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code