Summary:

• We add video transforms in TorchVision: https://github.com/pytorch/vision/pull/1306/
• In ClassyVision, we add default transforms for training / test stage. Alternatively, user can also explicitly provide transform config in json config input. See an example in the unit test.
• Video data transforms supports audio modality in the video dataset.
• Compared with image transforms which only returns a torch.Tensor, video transforms return a dict where key is the modality name (e.g. {"video", "audio"}) and value is a torch.Tensor for the modality data.

Reviewed By: taylorgordon20

Differential Revision: D16999453

i use ucf-101 example but get this problem

Traceback (most recent call last):
  File "/home/sucom/hdd_1T/project/video_rec/my_video_rec/self_video_train.py", line 160, in <module>
    trainer.train(task)
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/site-packages/classy_vision/trainer/local_trainer.py", line 27, in train
    super().train(task)
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/site-packages/classy_vision/trainer/classy_trainer.py", line 45, in train
    task.on_phase_start()
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/site-packages/classy_vision/tasks/classification_task.py", line 945, in on_phase_start
    self.advance_phase()
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/site-packages/classy_vision/tasks/classification_task.py", line 847, in advance_phase
    self.create_data_iterator()
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/site-packages/classy_vision/tasks/classification_task.py", line 900, in create_data_iterator
    self.data_iterator = iter(self.dataloaders[self.phase_type])
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 719, in __init__
    w.start()
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 59, in _launch
    cmd, self._fds)
  File "/home/sucom/.conda/envs/classy_vision/lib/python3.6/multiprocessing/util.py", line 417, in spawnv_passfds
    False, False, None)
ValueError: bad value(s) in fds_to_keep

Summary: Average Precision (AP) summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight.

See AP formal definition (https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html).

Mean Average Precision is an average of AP over different classes.

In this diff, we implement Mean Average Precision meter that is useful for multi-label classification task.

Differential Revision: D18715190

Summary: Added a normalize_inputs argument to FullyConnectedHead and VisionTransformerHead.

Added simple test cases for both heads

Differential Revision: D25213285

Summary:

• Implement Global Reasoning Networks (glore) (https://arxiv.org/abs/1811.12814). The reference implementation is on Github (https://github.com/facebookresearch/GloRe)
• Rename the json configs to tell whether glore is on or off. Therefore
  • train_kinetics400_glore_off_i3d50 means glore is off, but we use the baseline I3D-50 model in glore paper.
  • train_kinetics400_glore_on_i3d50 means glore is on and glore units are inserted into baseline I3D-50 model.
  • Similar changes are made to SlowFast sf json config. Since we only implement baseline I3D model used in SF, which is a slightly different I3D model from I3D model in glore, it is always off.

Differential Revision: D18344566

Summary: Bringing in fairscale to provide an optional state sharded optimizer in Classy, which should help in situations bounded by memory pressure. No new communication backend, this is using vanilla torch.distributed

Differential Revision: D22518768

Summary:

• Implement R(2+1)D unit (https://arxiv.org/abs/1711.11248).
• Compared with vanilla R3D model, in R(2+1)D we replace a 3D conv with a 2D spatial conv and a 1D temporal conv, while the total No. of model parameters are kept the same.
• We use R(2+1)D unit in ResNeXt3DStem and BasicTransformation.

Differential Revision: D19084922

Summary: FX Graph Mode Quantization needs to know whether an fx node is a floating point Tensor before it can decide whether to insert observer/fake_quantize module or not, since we only insert observer/fake_quantize module for floating point Tensors. Currently we have some hacks to support this by defining some rules like NON_OBSERVABLE_ARG_DICT (https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/fx/utils.py#L496), but this approach is fragile and we do not plan to maintain it long term in the pytorch code base.

As we discussed in the design review, we'd need to ask users to provide sample args and sample keyword args so that we can infer the type in a more robust way. This PR starts with changing the prepare_fx and prepare_qat_fx api to require user to either provide example arguments thrugh example_inputs, Note this api doesn't support kwargs, kwargs can make https://github.com/pytorch/pytorch/pull/76496#discussion_r861230047 (comment) simpler, but it will be rare, and even then we can still workaround with positional arguments, also torch.jit.trace(https://pytorch.org/docs/stable/generated/torch.jit.trace.html) and ShapeProp: https://github.com/pytorch/pytorch/blob/master/torch/fx/passes/shape_prop.py#L140 just have single positional args, we'll just use a single example_inputs argument for now.

If needed, we can extend the api with an optional example_kwargs. e.g. in case when there are a lot of arguments for forward and it makes more sense to pass the arguments by keyword

BC-breaking Note: Before: m = resnet18(...) m = prepare_fx(m, qconfig_dict)

After: m = resnet18(...) m = prepare_fx(m, qconfig_dict, example_inputs=(torch.randn(1, 3, 224, 224),))

Reviewed By: vkuzo, andrewor14

Differential Revision: D35984526

When I load a pretrained imagenet model by pytorch using a finetune task, an AssertionError: Checkpoint does not contain classy_state_dict was raised. So I want to know, how to load a imagenet model to initialize the backbone of a class model? Thank you.

Summary: For OSS, we use pyav video backend. It is different from video_reader backend in that it can not spatially rescale videos during video decoding.

Therefore, for testing video, we need to rescale it first before doing Fully-Convolutional Network (FCN) style evaluation.

With this diff, the users do not need to pre-rescale the video datasets ahead of model training.

Differential Revision: D18409505

Summary: Add Pytorch AMP support. A follow up for FairScale would be to add ShardedGradScaler, so that we support mixed precision with ShardedDDP and ShardedOptimizer

Differential Revision: D25383305

🐛 Bug

One can't import classy_vision.dataset with the current version of torchvision==0.14.0.

To Reproduce

To reproduce the behavior run:

from classy_vision import dataset

the command should fail with

ImportError: cannot import name Kinetics400 from 'torchvision.datasets.kinetics' (/usr/local/lib/python3.8/dist-packages/torchvision/datasets/kinetics.py)

This is most likely because torchvision deprecated Kinetics400 in 0.12.0: https://pytorch.org/vision/0.12/generated/torchvision.datasets.Kinetics400.html?highlight=kinetics#torchvision.datasets.Kinetics400

Expected behavior

The command above should work flawlessly. Downgrading torchvision to 0.12.0 is a workaround for this issue - a constraint such as torchvision<= 0.12.0 may be added as fix in https://github.com/facebookresearch/ClassyVision/blob/74e83dd97afbc9371b336304af09548f5080fa9c/requirements.txt#L4

Environment

• What commands did you use to install Classy Vision (conda/pip/build from source)? pip
• What does classy_vision.__version__ print? 0.6.0

python collect_env.py gives me:

PyTorch version: 1.13.0+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.6
Libc version: glibc-2.27

Python version: 3.8.16 (default, Dec  7 2022, 01:12:13)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.133+-x86_64-with-glibc2.27
Is CUDA available: False
CUDA runtime version: 11.2.152
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.13.0+cu116
[pip3] torchaudio==0.13.0+cu116
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.14.0
[pip3] torchvision==0.14.0+cu116

Summary: We want to check whether training XRayVideo with simply video resizing (in addition to other existing transformation like horizontal flipping and normalization) without random corp is sufficient.

The resize dimension is used as 224*224.

workflow: f362077622 (Note: in the workflow fcc_mvit_dataset_v4p2_arkc.yaml is used which I renamed to fcc_mvit_dataset_v4p2_onlyresize.yaml in this diff.)

As can be seen, the validation MAP goes to around .422 as opposed to 0.46 when random resized crop is used (f355567669) and rest of the configuration is kept the same. Hence, it is better to keep random resized crop.

Differential Revision: D38522980

📚 Documentation

efficientnet.py

It's a clean and well documented code. Kudos to the authors and thank you for making it open source :)

On the line 195, I think it's just an Excitation layer. Squeeze is being done by AdaptiveAvgPool2d.

❓ Questions and Help

I have created the data folder with the following structure

train/label_1/aa.png train/label_2/bb.png val/label_1/cc.png

and so on

The config used is

{ "name": "classification_task", "num_epochs": 95, "loss": { "name": "CrossEntropyLoss" }, "dataset": { "train": { "name": "image_path", "batchsize_per_replica": 32, "num_samples": null, "use_shuffle": true, "transforms": [{ "name": "apply_transform_to_key", "transforms": [ {"name": "RandomResizedCrop", "size": 224}, {"name": "RandomHorizontalFlip"}, {"name": "ToTensor"}, { "name": "Normalize", "mean": [0.485, 0.456, 0.406], "std": [0.229, 0.224, 0.225] } ], "key": "input" }], "image_folder": "/Users/nitesh/Workspace/MSon/classification-models/classy-vision/xx-project/data/train" }, "test": { "name": "image_path", "batchsize_per_replica": 32, "num_samples": null, "use_shuffle": false, "transforms": [{ "name": "apply_transform_to_key", "transforms": [ {"name": "Resize", "size": 256}, {"name": "CenterCrop", "size": 224}, {"name": "ToTensor"}, { "name": "Normalize", "mean": [0.485, 0.456, 0.406], "std": [0.229, 0.224, 0.225] } ], "key": "input" }], "image_folder": "/Users/nitesh/Workspace/MSon/classification-models/classy-vision/xx-project/data/val" } }, "meters": { "accuracy": { "topk": [1, 5] } }, "model": { "name": "resnext3d", "frames_per_clip": 1, "input_planes": 3, "clip_crop_size": 224, "skip_transformation_type": "postactivated_shortcut", "residual_transformation_type": "postactivated_bottleneck_transformation", "num_blocks": [3, 4, 6, 3], "stem_name": "resnext3d_stem", "stem_planes": 64, "stem_temporal_kernel": 1, "stem_spatial_kernel": 7, "stem_maxpool": true, "stage_planes": 256, "stage_temporal_kernel_basis": [[1], [1], [1], [1]], "temporal_conv_1x1": [true, true, true, true], "stage_temporal_stride": [1, 1, 1, 1], "stage_spatial_stride": [1, 2, 2, 2], "num_groups": 1, "width_per_group": 64, "num_classes": 1000, "zero_init_residual_transform": true, "heads": [ { "name": "fully_convolutional_linear", "unique_id": "default_head", "pool_size": [1, 7, 7], "activation_func": "softmax", "num_classes": 1000, "fork_block": "pathway0-stage4-block2", "in_plane": 2048, "use_dropout": false } ] }, "optimizer": { "name": "sgd", "param_schedulers": { "lr": { "name": "multistep", "num_epochs": 95, "values": [0.1, 0.01, 0.001, 0.0001], "milestones": [30, 60, 90] } }, "weight_decay": 0.0001, "momentum": 0.9, "nesterov": true } }

However while running the training its throwing an error. "IsADirectoryError: Caught IsADirectoryError in DataLoader worker process 0."

Could you please help me to solve this ? I am quite new to ML

Summary: Exponential parameter scheduler for fvcore/ClassyVision. This type of scheduler is useful as a LR scheduler when using EMA.

Differential Revision: D27369474