Neural Scene Flow Prior (NeurIPS 2021 spotlight)

Last update: Jan 03, 2023

Overview

Neural Scene Flow Prior

Xueqian Li, Jhony Kaesemodel Pontes, Simon Lucey

Will appear on Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS, 2021) as spotlight.

arXiv link: https://arxiv.org/pdf/2111.01253.pdf

Scene flow results on Argoverse
Point cloud integration (left: densified point cloud; right: sparse point cloud)

Prerequisites

This code is based on PyTorch implementation, and tested on torch=1.6.0 with CUDA 10.1 OR torch=1.7.1 with CUDA 10.2.

For a detailed installation guide, please go to requirements.txt.

Dataset

We provide four datasets we used in our paper. You may download datasets used in the paper from these anonymous links:

KITTI (266MB)
Argoverse (5.3GB)
nuScenes (436MB)
FlyingThings3D (436MB)

After you download the dataset, you can create a symbolic link in the ./dataset folder as ./dataset/kitti, ./dataset/argoverse, ./dataset/nuscenes, and ./dataset/flyingthings.

Optimization

Since we use neural scene flow prior for runtime optimization, our method does not include any "training".

Just run following lines for a simple optimization on a small KITTI Scene Flow dataset (only 50 testing samples)

python optimization.py \
--dataset KITTISceneFlowDataset \
--dataset_path dataset/kitti \
--exp_name KITTI_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 70 \
--visualize

You can then play with these configurations. We provide commands we used to generate results in the small point coud (2048 points) experiments and large point cloud (all points included) experiments.

1. small point cloud (2048 points)

KITTI Scene Flow

python optimization.py \
--dataset KITTISceneFlowDataset \
--dataset_path dataset/kitti \
--exp_name KITTI_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 70 \
--visualize

Argoverse Scene Flow

python optimization.py \
--dataset ArgoverseSceneFlowDataset \
--dataset_path dataset/argoverse \
--exp_name Argoverse_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 30 \
--visualize

nuScenes Scene Flow

python optimization.py \
--dataset NuScenesSceneFlowDataset \
--dataset_path dataset/nuscenes \
--exp_name Argoverse_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 30 \
--visualize

FlyingThings3D

python optimization.py \
--dataset FlyingThings3D \
--dataset_path dataset/flyingthings \
--exp_name FlyingThings_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--num_points 2048 \
--hidden_units 128 \
--lr 0.008 \
--backward_flow \
--early_patience 30 \
--visualize

2. dense point cloud (all points included)

KITTI Scene Flow

python optimization.py \
--dataset KITTISceneFlowDataset \
--dataset_path dataset/kitti \
--exp_name KITTI_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--use_all_points \
--hidden_units 128 \
--lr 0.001 \
--early_patience 100 \
--visualize

Argoverse Scene Flow

python optimization.py \
--dataset ArgoverseSceneFlowDataset \
--dataset_path dataset/argoverse \
--exp_name Argoverse_2048_points \
--batch_size 1 \
--iters 5000 \
--compute_metrics \
--use_all_points \
--hidden_units 128 \
--lr 0.003 \
--backward_flow \
--early_patience 100 \
--visualize

Contributing

If you find the project useful for your research, you may cite,

@article{li2021neural,
  title={Neural scene flow prior},
  author={Li, Xueqian and Pontes, Jhony Kaesemodel and Lucey, Simon},
  journal={arXiv preprint arXiv:2111.01253},
  year={2021}
}

Neural Scene Flow Prior (NeurIPS 2021 spotlight)

Related tags

Overview

Neural Scene Flow Prior

Scene flow results on Argoverse

Point cloud integration (left: densified point cloud; right: sparse point cloud)

Prerequisites

Dataset

Optimization

1. small point cloud (2048 points)

KITTI Scene Flow

Argoverse Scene Flow

nuScenes Scene Flow

FlyingThings3D

2. dense point cloud (all points included)

KITTI Scene Flow

Argoverse Scene Flow

Contributing

Owner

Lilac Lee

This repository contains tutorials for the py4DSTEM Python package

Visualizing lattice vibration information from phonon dispersion to atoms (For GPUMD)

This is the repository of shape matching algorithm Iterative Rotations and Assignments (IRA)

Code samples for my book "Neural Networks and Deep Learning"

This code provides a PyTorch implementation for OTTER (Optimal Transport distillation for Efficient zero-shot Recognition), as described in the paper.

HIVE: Evaluating the Human Interpretability of Visual Explanations

Modifications of the official PyTorch implementation of StyleGAN3. Let's easily generate images and videos with StyleGAN2/2-ADA/3!

A Fast Sequence Transducer Implementation with PyTorch Bindings

Implementation of CVAE. Trained CVAE on faces from UTKFace Dataset to produce synthetic faces with a given degree of happiness/smileyness.

Generic U-Net Tensorflow implementation for image segmentation

Official implementation of "GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators" (NeurIPS 2020)

Multimodal commodity image retrieval 多模态商品图像检索

A library for performing coverage guided fuzzing of neural networks

[CVPR'21] FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space

DeepLab is a state-of-art deep learning system for semantic image segmentation built on top of Caffe.

PyTorch implementation of Lip to Speech Synthesis with Visual Context Attentional GAN (NeurIPS2021)

Neurons Dataset API - The official dataloader and visualization tools for Neurons Datasets.

This is the code for the paper "Contrastive Clustering" (AAAI 2021)

PyTorch implementation of the WarpedGANSpace: Finding non-linear RBF paths in GAN latent space (ICCV 2021)

PyTorch implementation of Hierarchical Multi-label Text Classification: An Attention-based Recurrent Network