Dear authors , Thx 4 your Great work ! But I'm working with AudioClip of which the embeddings are 1024 dims, But the models U've released have most 768 dims, Could U pls kindly release a model that can produce 1024 dims embedding ? Here is AudioClip : https://github.com/AndreyGuzhov/AudioCLIP With my best wish ! Looking forward to hearing from U !

Merge this only after doing this:

when you create a "Release x.y.z" it will release

you need to add a secret called PYPI_PASSWORD in the github repo and put inside a token you create at https://pypi.org/manage/account/token/

https://github.com/FreddeFrallan/Multilingual-CLIP/settings/secrets/actions/new

Choose the option "Squash and merge" on github when merging to create a single commit

Amazing repo! I'd love to cite it. Do you have a desired bibtex by chance?

Perhaps:

@misc{multilingual-clip, author = {Carlsson, Fredrik}, title = {Multilingual CLIP}, year = 2021, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/SajjjadAyobi/CLIPfa}}, }

Hey, Thanks for providing this awesome multilingual clip-aligned text encoder. We used it to filter the 3 billions of (image, text) pairs of laion5B https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/ and it worked well. I'm also using this model to provide a multilingual search in https://rom1504.github.io/clip-retrieval/. For laion400m we used the ViT-B/32 model of openai to produce the index, but for laion5B we went with ViT-L/14 which is much more powerful. To provide the same multilingual search feature, it would be really helpful if I had a clip ViT-L/14 aligned multilingual text encoder.

Would you advise running https://github.com/FreddeFrallan/Multilingual-CLIP#training-a-new-model (and now I'm writing it, I guess I could use a subset of the multilingual set of laion5B for this) to align such a text encoder ?

Hi,

Great repo! Are you planning to release a model with XLMR (with ViT-B) anytime soon? It was better for small-resource-languages than the multilingual BERT.

when you create a "Release x.y.z" it will release

you need to add a secret called PYPI_PASSWORD in the github repo and put inside a token you create at https://pypi.org/manage/account/token/

https://github.com/FreddeFrallan/Multilingual-CLIP/settings/secrets/actions/new

Hello! According to XTD-10 repo, the test set contains 800 images from MSCOCO train set. During training you also use MSCOCO train set – it seems you have data leak. Or may be I don't understand something.

from transformers import AutoConfig

kwargs = {'_from_auto': True}
pretrained_model_name_or_path = 'M-CLIP/XLM-Roberta-Large-Vit-L-14'
config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)

Hi, I installed required transformers==4.8.1, and run the above code to get following error.

    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/anaconda3/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 448, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
KeyError: 'M-CLIP'

seems like model_type 'M-CLIP' is not in the CONFIG_MAPPING, can anyone help to figure it out?

I tried the following piece of code present in the repo at location https://github.com/FreddeFrallan/Multilingual-CLIP/blob/main/src/multilingual_clip.py

The only changes I made is that I added print statements in between.

` import pickle

import torch import transformers

AVAILABLE_MODELS = { 'M-BERT-Distil-40': { 'model_name': 'M-CLIP/M-BERT-Distil-40', 'tokenizer_name': 'M-CLIP/M-BERT-Distil-40', 'head_name': 'M-BERT Distil 40 Linear Weights.pkl' },

'M-BERT-Base-69': {
    'model_name': 'M-CLIP/M-BERT-Base-69',
    'tokenizer_name': 'M-CLIP/M-BERT-Base-69',
    'head_name': 'M-BERT-Base-69 Linear Weights.pkl'
},

'Swe-CLIP-500k': {
    'model_name': 'M-CLIP/Swedish-500k',
    'tokenizer_name': 'M-CLIP/Swedish-500k',
    'head_name': 'Swedish-500k Linear Weights.pkl'
},

'Swe-CLIP-2M': {
    'model_name': 'M-CLIP/Swedish-2M',
    'tokenizer_name': 'M-CLIP/Swedish-2M',
    'head_name': 'Swedish-2M Linear Weights.pkl'
},

'M-BERT-Base-ViT-B': {
    'model_name': 'M-CLIP/M-BERT-Base-ViT-B',
    'tokenizer_name': 'M-CLIP/M-BERT-Base-ViT-B',
    'head_name': 'M-BERT-Base-69-ViT Linear Weights.pkl'
},

}

class MultilingualClip2(torch.nn.Module): def init(self, model_name, tokenizer_name, head_name, weights_dir='data/weights/'): super().init() self.model_name = model_name self.tokenizer_name = tokenizer_name self.head_path = weights_dir + head_name

    self.tokenizer = transformers.AutoTokenizer.from_pretrained(tokenizer_name)
    self.transformer = transformers.AutoModel.from_pretrained(model_name)
    self.clip_head = torch.nn.Linear(in_features=768, out_features=640)
    self._load_head()

def forward(self, txt):
    txt_tok = self.tokenizer(txt, padding=True, return_tensors='pt').to(device)
    embs = self.transformer(**txt_tok)[0]
    print('embs_text')
    print(embs.size())
    att = txt_tok['attention_mask']
    print('att_text')
    print(att.size())
    embs = (embs * att.unsqueeze(2)).sum(dim=1) / att.sum(dim=1)[:, None]
    print('embs_text')
    print(embs.size())
    p =  self.clip_head(embs)
    print('clip head obj')
    print(self.clip_head)
    print('cliphed_text')
    print(p.size())
    return p

def _load_head(self):
    with open(self.head_path, 'rb') as f:
        lin_weights = pickle.loads(f.read())
    self.clip_head.weight = torch.nn.Parameter(torch.tensor(lin_weights[0]).float().t())
    self.clip_head.bias = torch.nn.Parameter(torch.tensor(lin_weights[1]).float())
    print('ok')
    print(self.clip_head.weight.size())
    print(self.clip_head.bias.size())

def load_model2(name): config = AVAILABLE_MODELS[name] return MultilingualClip2(**config)

mod = load_model2('M-BERT-Base-ViT-B') z = mod(Query[0]) `

Output for this code : ok torch.Size([512, 768]) torch.Size([512]) embs_text torch.Size([1, 6, 768]) att_text torch.Size([1, 6]) embs_text torch.Size([1, 768]) clip head obj Linear(in_features=768, out_features=640, bias=True) cliphed_text torch.Size([1, 512])

This output suggest that the file 'M-BERT-Base-69-ViT Linear Weights.pkl' doesn't have the size of 640 X 768 but 512 X 768

Is there any issue with the config then ?

i have finetune the text_encode use 300000 texts and its embedding,but i find the result is so bad ,could you give me some advertise to improve the result

if i have <other language text , image, label> pair data, can i directly use 'distilbert-base-multilingual-cased' to pre-train clip model? Why re-train a model for englist to other languages ?