spacy-huggingface-hub: Push your spaCy pipelines to the Hugging Face Hub
This package provides a CLI command for uploading any trained spaCy pipeline packaged with spacy package to the Hugging Face Hub. It auto-generates all meta information for you, uploads a pretty README (requires spaCy v3.1+) and handles version control under the hood.
🤗
About the Hugging Face Hub
The Hugging Face Hub hosts Git-based repositories which are storage spaces that can contain all your files. These repositories have multiple advantages: versioning (commit history and diffs), branches, useful metadata about their tasks, languages, metrics and more, browser-based visualizers to explore the models interactively in your browser, as well as an API to use the models in production.
🚀
Quickstart
You can install spacy-huggingface-hub from pip:
pip install spacy-huggingface-hub
To check if the command has been registered successfully:
python -m spacy huggingface-hub --help
Hugging Face uses Git Large File Storage (LFS) to handle files larger than 10mb. You can find instructions on how to download it here.
You can then upload any pipeline packaged with spacy package. Make sure to set --build wheel to output a binary .whl file. The uploader will read all metadata from the pipeline package, including the auto-generated pretty README.md and the model details available in the meta.json.
huggingface-cli login
python -m spacy package ./en_ner_fashion ./output --build wheel
cd ./output/en_ner_fashion-0.0.0/dist
python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl
The command will output two things:
- Where to find your repo in the Hub! For example, https://huggingface.co/spacy/en_core_web_sm
- And how to install the pipeline directly from the Hub!
pip install https://huggingface.co/spacy/en_core_web_sm/resolve/main/en_core_web_sm-any-py3-none-any.whl
Now you can share your pipelines very quickly with others. Additionally, you can also test your pipeline directly in the browser!
⚙️
Usage and API
If spaCy is already installed in the same environment, this package automatically adds the spacy huggingface-hub commands to the CLI. If you don't have spaCy installed, you can also execute the CLI directly via the package.
push
python -m spacy huggingface-hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose]
python -m spacy_huggingface_hub push [whl_path] [--org] [--msg] [--local-repo] [--verbose]
| Argument | Type | Description |
|---|---|---|
whl_path |
str / Path |
The path to the .whl file packaged with spacy package. |
--org, -o |
str | Optional name of organization to which the pipeline should be uploaded. |
--msg, -m |
str | Commit message to use for update. Defaults to "Update spaCy pipeline". |
--local-repo, -l |
str / Path |
Local path to the model repository (will be created if it doesn't exist). Defaults to hub in the current working directory. |
--verbose, -V |
bool | Output additional info for debugging, e.g. the full generated hub metadata. |
Usage from Python
Instead of using the CLI, you can also call the push function from Python. It returns a dictionary containing the "url" of the published model and the "whl_url" of the wheel file, which you can install with pip install
from spacy_huggingface_hub import push
result = push("./en_ner_fashion-0.0.0-py3-none-any.whl")
print(result["url"])
295 Dec 26, 2022
36 Nov 15, 2022
35 Dec 30, 2022
45 Feb 06, 2022
79 Dec 19, 2022
8 Oct 26, 2022
1.5k Jan 02, 2023
51 Sep 22, 2022
62 Dec 27, 2022
69 Dec 12, 2022
375 Dec 30, 2022
27 Nov 23, 2022
518 Dec 12, 2022
12 Nov 08, 2022
1 Jan 12, 2022
0 Aug 17, 2021
621 Jan 08, 2023
393 Dec 22, 2022
0 Oct 29, 2021
643 Dec 30, 2022