Awesome Transformer Architecture Search:

To keep track of the large number of recent papers that look at the intersection of Transformers and Neural Architecture Search (NAS), we have created this awesome list of curated papers and resources, inspired by awesome-autodl, awesome-architecture-search, and awesome-computer-vision. Papers are divided into the following categories:

General Transformer search
Domain Specific, applied Transformer search (divided into NLP, Vision, ASR)
Insights on Transformer components or searchable parameters
Transformer Surveys

This repository is maintained by the AutoML Group Freiburg. Please feel free to pull requests or open an issue to add papers.

General Transformer Search

Title	Venue	Group
UniNet: Unified Architecture Search with Convolutions, Transformer and MLP	arxiv [Oct'21]	SenseTime
Analyzing and Mitigating Interference in Neural Architecture Search	arxiv [Aug'21]	Tsinghua, MSR
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search	ICCV'21	Sun Yat-sen University
Memory-Efficient Differentiable Transformer Architecture Search	ACL-IJCNLP'21	MSR, Peking University
Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition	arxiv [Aug'20]	Google Research
AutoTrans: Automating Transformer Design via Reinforced Architecture Search	arxiv [Sep'20]	Fudan University
NAT: Neural Architecture Transformer for Accurate and Compact Architectures	NeurIPS'19	Tencent AI
The Evolved Transformer	ICML'19	Google Brain

Domain Specific Transformer Search

Vision

Title	Venue	Group
AutoFormer: Searching Transformers for Visual Recognition	ICCV'21	MSR
GLiT: Neural Architecture Search for Global and Local Image Transformer	ICCV'21	University of Sydney
Searching for Efficient Multi-Stage Vision Transformers	ICCV'21 workshop	MIT
HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers	CVPR'21	Bytedance Inc.
Vision Transformer Architecture Search	arxiv [June'21]	SenseTime, Tsingua University

Natural Language Processing

Title	Venue	Group
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models	ACL'21	MIT
NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search	KDD'21	MSR, Tsinghua University
AutoBERT-Zero: Evolving the BERT backbone from scratch	arxiv [July'21]	Huawei Noah’s Ark Lab
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing	ACL'20	MIT

Automatic Speech Recognition

Title	Venue	Group
LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search	ICASSP'21	MSR
Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR	arxiv [Aug'21]	NPU, Xi'an
Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search	arxiv [April'21]	Chinese Academy of Sciences
Evolved Speech-Transformer: Applying Neural Architecture Search to End-to-End Automatic Speech Recognition	INTERSPEECH'20	VUNO Inc.

Insights on Transformer components and interesting papers

Title	Venue	Group
Patches are All You Need ?	ICLR'22 under review	-
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows	ICCV'21 best paper	MSR
Rethinking Spatial Dimensions of Vision Transformers	ICCV'21	NAVER AI
What makes for hierarchical vision transformers	arxiv [Sept'21]	HUST
AutoAttend: Automated Attention Representation Search	ICML'21	Tsinghua University
Rethinking Attention with Performers	ICLR'21 Oral	Google
LambdaNetworks: Modeling long-range Interactions without Attention	ICLR'21	Google Research
HyperGrid Transformers	ICLR'21	Google Research
LocalViT: Bringing Locality to Vision Transformers	arxiv [April'21]	ETH Zurich
NASABN: A Neural Architecture Search Framework for Attention-Based Networks	IJCNN'20	Chinese Academy of Sciences
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned	ACL'19	Yandex

Transformer Surveys

Title	Venue	Group
Transformers in Vision: A Survey	arxiv [Oct'21]	MBZ University of AI
Efficient Transformers: A Survey	arxiv [Sept'21]	Google Research

Misc resources

Awesome Visual Transformer

A curated list of awesome resources combining Transformers with Neural Architecture Search

Related tags

Overview

Awesome Transformer Architecture Search:

General Transformer Search

Domain Specific Transformer Search

Vision

Natural Language Processing

Automatic Speech Recognition

Insights on Transformer components and interesting papers

Transformer Surveys

Misc resources

Owner

Yash Mehta

Focal Loss for Dense Rotation Object Detection

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

PyTorch implementation of DeepLab v2 on COCO-Stuff / PASCAL VOC

TensorFlow implementation of Deep Reinforcement Learning papers

Code and data accompanying our SVRHM'21 paper.

Deep learning image registration library for PyTorch

Generate text captions for images from their CLIP embeddings. Includes PyTorch model code and example training script.

Official PyTorch code of Holistic 3D Scene Understanding from a Single Image with Implicit Representation (CVPR 2021)

Fortuitous Forgetting in Connectionist Networks

Visual Question Answering in Pytorch

Unified MultiWOZ evaluation scripts for the context-to-response task.

OBBDetection: an oriented object detection toolbox modified from MMdetection

"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

A generator of point clouds dataset for PyPipes.

ObjectDrawer-ToolBox: a graphical image annotation tool to generate ground plane masks for a 3D object reconstruction system

Omnidirectional camera calibration in python

Laplace Redux -- Effortless Bayesian Deep Learning

Python PID Tuner - Based on a FOPDT model obtained using a Open Loop Process Reaction Curve

Implementation of various Vision Transformers I found interesting

Python scripts using the Mediapipe models for Halloween.