Hi. I am getting the following error:

from upit.data.unpaired import *
from upit.models.cyclegan import *
from upit.train.cyclegan import *
from fastai.vision.all import *

horse2zebra = untar_data('https://people.eecs.berkeley.edu/~taesung_park/CycleGAN/datasets/horse2zebra.zip')


folders = horse2zebra.ls().sorted()
trainA_path = folders[2]
trainB_path = folders[3]
testA_path = folders[0]
testB_path = folders[1]

dls = get_dls(trainA_path, trainB_path,num_A=100)
cycle_gan = CycleGAN(3,3,64)
learn = cycle_learner(dls, cycle_gan,show_img_interval=1)
learn.show_training_loop()
learn.lr_find()

AttributeError: 'Learner' object has no attribute 'pred'

I opened an issue (#7) on 03/09/2020. The problem and the solution are mentioned below with the colab files as well.

On 03/09/2020, a commit was made in fastai/fastcore/uitls.py (fastai/[email protected]) (line 86) which made sure that we need not use "self" when we call the method store_attr. ( separate the names using commas)

I made a small change in the code ie. in the upit/train/cyclegan.py such that it follows the changed fastcore/utils.py structure. I removed the self component for store_attr in cyclegan.py.

The error can be seen here

The colab link: https://colab.research.google.com/drive/1lbXhX-bWvTsQcLb2UUFI0UkRpvx_WK2Y?usp=sharing

After the change, the error does not occur as shown here.

The Colab link: https://colab.research.google.com/drive/13rrNLBgDulgeHclovxJNtQVbVYcGMKm0?usp=sharing

Hey there,

I think it would be a good idea to expose the tfms of the Datasets in the get_dls-Function, so users can set their own. https://github.com/tmabraham/UPIT/blob/c6c769cf8cedeec42865deddba26c2e413772303/upit/data/unpaired.py#L30

Same goes for dataloaders batch_tfms, (e.g. if I want to disable Flipping I have to rewrite the dataloaders)

https://github.com/tmabraham/UPIT/blob/c6c769cf8cedeec42865deddba26c2e413772303/upit/data/unpaired.py#L33

I would like to see what fake_A and fake_B look like at this step in the process saved as images. I can't seem to figure out how to convert them properly.

def forward(self, output, target): """ Forward function of the CycleGAN loss function. The generated images are passed in as output (which comes from the model) and the generator loss is returned. """ fake_A, fake_B, idt_A, idt_B = output #Save and look at png images of fake_A and fake_B here

https://github.com/tmabraham/UPIT/blob/1f272eac299181348c31988289c1420936cb580b/upit/train/cyclegan.py#L132

Was this intended or was this line supposed to be: self.learn.loss_func.D_B_loss = loss_D_B.detach().cpu() ?

Hey,

me again. Sorry to bother you again!

So I am trying to do inference on a trained model (with default values). I exported the Generator with the export_generatorFunction. Now I try to load my generator as shown in the Web App Example.

But I get errors in loading the state_dict. The state dict seems to have extra key for nine extra layers, if I understand the error message correctly:

model.load_state_dict(torch.load("generator.pth", map_location=device))
  File "/Applications/Utilities/miniconda3/envs/ml/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Sequential:
        Missing key(s) in state_dict: "10.conv_block.5.weight", "10.conv_block.5.bias", "11.conv_block.5.weight", "11.conv_block.5.bias", "12.conv_block.5.weight", "12.conv_block.5.bias", "13.conv_block.5.weight", "13.conv_block.5.bias", "14.conv_block.5.weight", "14.conv_block.5.bias", "15.conv_block.5.weight", "15.conv_block.5.bias", "16.conv_block.5.weight", "16.conv_block.5.bias", "17.conv_block.5.weight", "17.conv_block.5.bias", "18.conv_block.5.weight", "18.conv_block.5.bias". 
        Unexpected key(s) in state_dict: "10.conv_block.6.weight", "10.conv_block.6.bias", "11.conv_block.6.weight", "11.conv_block.6.bias", "12.conv_block.6.weight", "12.conv_block.6.bias", "13.conv_block.6.weight", "13.conv_block.6.bias", "14.conv_block.6.weight", "14.conv_block.6.bias", "15.conv_block.6.weight", "15.conv_block.6.bias", "16.conv_block.6.weight", "16.conv_block.6.bias", "17.conv_block.6.weight", "17.conv_block.6.bias", "18.conv_block.6.weight", "18.conv_block.6.bias".

import torch
from upit.models.cyclegan import resnet_generator
import torchvision.transforms
from PIL import Image


device = torch.device("cpu")
model = resnet_generator(ch_in=3, ch_out=3)
model.load_state_dict(torch.load("generator.pth", map_location=device))
model.eval()


totensor = torchvision.transforms.ToTensor()
normalize_fn = torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
topilimage = torchvision.transforms.ToPILImage()


def predict(input):
    im = normalize_fn(totensor(input))
    print(im.shape)
    preds = model(im.unsqueeze(0)) / 2 + 0.5
    print(preds.shape)
    return topilimage(preds.squeeze(0).detach().cpu())


im = predict(Image.open("test.jpg"))
im.save("out.jpg")

Thanks again for your support!

Hey, thanks for your awesome work. If I want to set l_idt of the CycleGANLoss to zero, how would I do this? Can I pass this some argument to the cycle_learner? On a quick look this seems to be hardcoded in the cycle_learner, right ? So I would have to right "my own" cycle_learner?

Thanks for the answers to my - most likely - stupid questions!

Hi. The learner has a method learn.progress.show_cycle_gan_imgs. However, how to plot it with matplotlib's plt.show() if I use python repl. There is an argument event_name in learn.progress.show_cycle_gan_imgs. I would like to do it after fit:

>>> learn.fit_flat_lin(1,1,2e-4)
epoch     train_loss  id_loss_A  id_loss_B  gen_loss_A  gen_loss_B  cyc_loss_A  cyc_loss_B  D_A_loss  D_B_loss  time    
/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastprogress/fastprogress.py:74: UserWarning: Your generator is empty.
  warn("Your generator is empty.")
0         10.809340   1.621755   1.690260   0.420364    0.452442    3.359353    3.504819    0.370692  0.370692  00:08     
1         9.847495    1.283465   1.510985   0.353303    0.349454    2.682495    3.135504    0.253919  0.253919  00:07 

https://github.com/tmabraham/UPIT/blob/020f8e2d8dbab6824cd4bef2690ea93e3ff69a6e/upit/train/cyclegan.py#L167-L176

With image classifier I usually do:

test_dl = object.dls.test_dl("n02381460_1052.jpg") # object is model/learner
predictions = object.get_preds(dl = test_dl)

However, it throws:

  TypeError: 'NoneType' object cannot be interpreted as an integer 

Traceback (most recent call last):
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 155, in _with_events
    try:       self(f'before_{event_type}')       ;f()
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 161, in all_batches
    for o in enumerate(self.dl): self.one_batch(*o)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 179, in one_batch
    self._with_events(self._do_one_batch, 'batch', CancelBatchException)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 155, in _with_events
    try:       self(f'before_{event_type}')       ;f()
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 133, in __call__
    def __call__(self, event_name): L(event_name).map(self._call_one)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastcore/foundation.py", line 226, in map
    def map(self, f, *args, gen=False, **kwargs): return self._new(map_ex(self, f, *args, gen=gen, **kwargs))
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastcore/basics.py", line 537, in map_ex
    return list(res)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastcore/basics.py", line 527, in __call__
    return self.fn(*fargs, **kwargs)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 137, in _call_one
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 137, in <listcomp>
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/callback/core.py", line 44, in __call__
    if self.run and _run: res = getattr(self, event_name, noop)()
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/upit/train/cyclegan.py", line 112, in before_batch
    self.learn.xb = (self.learn.xb[0],self.learn.yb[0]),
IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/torch_core.py", line 268, in to_concat
    try:    return retain_type(torch.cat(xs, dim=dim), xs[0])
TypeError: expected Tensor as element 0 in argument 0, but got NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 235, in get_preds
    self._do_epoch_validate(dl=dl)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 188, in _do_epoch_validate
    with torch.no_grad(): self._with_events(self.all_batches, 'validate', CancelValidException)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 157, in _with_events
    finally:   self(f'after_{event_type}')        ;final()
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 133, in __call__
    def __call__(self, event_name): L(event_name).map(self._call_one)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastcore/foundation.py", line 226, in map
    def map(self, f, *args, gen=False, **kwargs): return self._new(map_ex(self, f, *args, gen=gen, **kwargs))
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastcore/basics.py", line 537, in map_ex
    return list(res)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastcore/basics.py", line 527, in __call__
    return self.fn(*fargs, **kwargs)
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 137, in _call_one
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/learner.py", line 137, in <listcomp>
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/callback/core.py", line 44, in __call__
    if self.run and _run: res = getattr(self, event_name, noop)()
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/callback/core.py", line 123, in after_validate
    if not self.save_preds: self.preds   = detuplify(to_concat(self.preds, dim=self.concat_dim))
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/torch_core.py", line 270, in to_concat
    for i in range_of(o_)) for o_ in xs], L())
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastai/torch_core.py", line 270, in <listcomp>
    for i in range_of(o_)) for o_ in xs], L())
  File "/home/turgut/.local/share/r-miniconda/envs/r-reticulate/lib/python3.6/site-packages/fastcore/basics.py", line 425, in range_of
    return list(range(a,b,step) if step is not None else range(a,b) if b is not None else range(a))
TypeError: 'NoneType' object cannot be interpreted as an integer

I opened an issue (https://github.com/tmabraham/UPIT/issues/7) on 03/09/2020. The problem and the solution are mentioned below with the colab files as well.

On 03/09/2020, a commit was made in fastai/fastcore/uitls.py (https://github.com/fastai/fastcore/commit/ea1c33f1c3543e6e4403b1d3b7702a98471f3515) (line 86) which made sure that we need not use "self" when we call the method store_attr. ( separate the names using commas).

I made a small change in the code ie. in the upit/train/cyclegan.py such that it follows the changed fastcore/utils.py structure. I removed the self component for store_attr in cyclegan.py.

The error can be seen here

The colab link : https://colab.research.google.com/drive/1lbXhX-bWvTsQcLb2UUFI0UkRpvx_WK2Y?usp=sharing

After the change, the error does not occur as shown here.

The Colab link: https://colab.research.google.com/drive/13rrNLBgDulgeHclovxJNtQVbVYcGMKm0?usp=sharing

Pretrained models to be available on HuggingFace Hub, as well as allowing users to save their own models to HuggingFace Hub.

Relevant links:

• https://huggingface.co/docs/hub/adding-a-model
• https://huggingface.co/docs/hub/adding-a-model#using-the-huggingface_hub-client-library

This is really great work! I noticed you appear to be using this code for creating new examples of pathology images. I am doing something similar, just not pathology. Let's say I already have a separate model trained to classify these types of images and I want to run it against a validation set every epoch and determine how well the cyclegan is performing in generating new examples that fool an existing classifier. I'm trying to figure out where the best place might be to add code which does that. I can see a few places where it would probably work, but was curious if you have already thought about adding this functionality to monitor training progress?

Thanks,

Bob

Here are some tests of interest:

• [ ] Test the loss (example fake and real images for the reconstruction loss and discriminator)

• [ ] Check the batch independence and the model parameter updates

• [ ] Test successful overfitting on a single batch

• [ ] Test for rotation invariance and other invariance properties

I need to check if fastai's multi-GPU support work with my package, and if not, what needs to be modified to get it to work. Additionally, I may need to add a simpler interface for DDP or at least clear examples/documentation. This will enable for quicker model training on multi-GPU servers, like those at USF.

I want to add support for metrics, and even potentially include some common metrics, like FID, mi-FID, KID, and segmentation metrics (for paired) etc.

Additionally, monitoring the losses and metrics, I want to be able to use fastai's built-in callbacks for saving best model, early stopping, and reducing LR on plateau.

This shouldn't be too hard to include. A major part of this feature is finding good PyTorch/numpy implementations of some of these metrics and getting it to work.