Pre-commit hook for upgrading type hints

Overview

supported python versions code coverage pre-commit

PEP585 Upgrade

This is a pre-commit hook configured to automatically upgrade your type hints to the new native types implemented in PEP 585.

This will work for any Python version above 3.7, though if you're not using 3.9 you will need to run the hook with futures-imports=true.

A complete type list is shown below.

See the complete list
Used to be Will be upgraded to
typing.Tuple tuple
typing.List list
typing.Dict dict
typing.Set set
typing.FrozenSet frozenset
typing.Type type
typing.Deque collections.deque
typing.DefaultDict collections.defaultdict
typing.OrderedDict collections.OrderedDict
typing.Counter collections.Counter
typing.ChainMap collections.ChainMap
typing.Awaitable collections.abc.Awaitable
typing.Coroutine collections.abc.Coroutine
typing.AsyncIterable collections.abc.AsyncIterable
typing.AsyncIterator collections.abc.AsyncIterator
typing.AsyncGenerator collections.abc.AsyncGenerator
typing.Iterable collections.abc.Iterable
typing.Iterator collections.abc.Iterator
typing.Generator collections.abc.Generator
typing.Reversible collections.abc.Reversible
typing.Container collections.abc.Container
typing.Collection collections.abc.Collection
typing.Callable collections.abc.Callable
typing.AbstractSet collections.abc.Set
typing.MutableSet collections.abc.MutableSet
typing.Mapping collections.abc.Mapping
typing.MutableMapping collections.abc.MutableMapping
typing.Sequence collections.abc.Sequence
typing.MutableSequence collections.abc.MutableSequence
typing.ByteString collections.abc.ByteString
typing.MappingView collections.abc.MappingView
typing.KeysView collections.abc.KeysView
typing.ItemsView collections.abc.ItemsView
typing.ValuesView collections.abc.ValuesView
typing.ContextManager contextlib.AbstractContextManager
typing.AsyncContextManager contextlib.AbstractAsyncContextManager
typing.re.Pattern re.Pattern
typing.re.Match re.Match

I'm a visual learner

In a nutshell, this code

from typing import List, Tuple, Dict, Set, FrozenSet

def do_thing(x: List[Tuple[str, ...]], y: Dict[str, Set[str]]) -> FrozenSet:

becomes this

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

or this, if you're running python < 3.9 or enable the futures option

from __future__ import annotations

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

Features:

  • Performs in-line substitution for new types
  • Adds new imports for upgrade types which need them
  • Adds __futures__ imports if the futures flag is enabled
  • Removes no longer needed typing imports

Note: even though we remove and add imports reasonably well, I would recommend running this in tandem with hooks like isort to aggregate and sort your imports, and flake8 to discover any unused imports neither were able to remove. Otherwise you risk needing to do some manual cleanup from time to time (though it should be pretty rare).

Config

To use this with pre-commit, simply add this to your config file:

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints

and while futures imports are added automatically if you're running Python older than 3.9, you can also enable them explicitly, by adding a --futures arg.

This is required, e.g., when maintaining code that needs to support older Python versions.

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints
    args: [ '--futures=true' ]

For more information about available arguments, see the function definitions.

Running this once on my codebase

If you wish to run this once on your codebase, that's not easy to do without pre-commit, as it piggybacks on that process quite a bit.

However, installing pre-commit and configuring the hook to run will take you less than a minute. These are the steps:

  • Run pip install pre-commit
  • Run touch .pre-commit-config.yaml
  • Copy the configuration shown above into the file
  • Run pre-commit run --all-files

Running this once on a single file

To run the upgrade on a single file, you can clone the repo and run python -m upgrade_type_hints from the src folder, or something equivalent.

Known imperfections

  • We have a hard time removing common import typing imports, since we don't have a full inventory of all possible places typing could be used. Seeing something like this, you might think this is easy to handle

    import typing
    
    x: typing.List

    but extending this example to a thousand-line file, the way we've structured the logic, there is no way to know whether there is a valid typing.Optional somewhere in the file.

  • We might remove typing imports in a file where you needed them for more than just type annotations. An example of this is custom type declarations:

    from typing import List
    
    x: List  # this will be upgraded and the import will be removed
    y = List[str]  # this will be left without its required import

    The reason for this is that custom type declarations are not a part of the ast objects we look at.

Both points are resolved by running flake8.

Supporting the project

Please leave a ✭ if this project helped you 👏 and contributions are always welcome!

Comments
  • Don't upgrade generics - part 2

    Don't upgrade generics - part 2

    I observe the same issue as #22, using v1.0.1.

    Starting with the file

    from __future__ import annotations
    
    from typing import Any, Callable, Mapping
    
    MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    def func(x: Mapping[str, Any], y: Any) -> None:
        return None
    
    callback: MessageCallback
    callback = func
    

    This originally runs without errors, but not after running pep585-upgrade has run:

    $ python -V
    Python 3.8.6
    $ python test.py
    $ pre-commit run
    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    - files were modified by this hook
    
    Fixing src/workflows/test.py
    
    isort....................................................................Failed
    - hook id: isort
    - files were modified by this hook
    
    $ git diff
    index 72539fc..56bc282 100644
    --- a/test.py
    +++ b/test.py
    @@ -1,6 +1,7 @@
     from __future__ import annotations
    
    -from typing import Any, Callable, Mapping
    +from collections.abc import Mapping
    +from typing import Any, Callable
    
     MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    $ python test.py
    Traceback (most recent call last):
      File "test.py", line 6, in <module>
        MessageCallback = Callable[[Mapping[str, Any], Any], None]
    TypeError: 'ABCMeta' object is not subscriptable
    
    opened by Anthchirp 11
  • Support for typing.cast call

    Support for typing.cast call

    The code-base I'm modernizing has the following code:

    project_acl_list = cast(Optional[List[str]], project.acl)
    

    Unfortunately, this is not transformed.

    I'm willing to fix that, but I don't know yet how...

    Could we discuss how to implement the transformation for this code?

    opened by rominf 9
  • end_lineno error while running python 3.7

    end_lineno error while running python 3.7

    Adding hook via:

      - repo: https://github.com/sondrelg/pep585-upgrade
        rev: ''
        hooks:
          - id: upgrade-type-hints
            args: [ '--futures=true' ]
    

    When running in a python 3.9 virtualenv everything works, but switching to python 3.7 causes this error:

    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    
    opened by ArcLightSlavik 8
  • pre-commit autoupdate warning

    pre-commit autoupdate warning

    Running pre-commit autoupdate leads to the following result: Updating https://github.com/sondrelg/pep585-upgrade ... updating v1.0.1 -> v1.

    Running pre-commit autoupdate again will result in the following warning:

    [WARNING] The 'rev' field of repo 'https://github.com/sondrelg/pep585-upgrade' appears to be a mutable reference (moving tag / branch).
    Mutable references are never updated after first install and are not supported. 
    See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details. 
    Hint: `pre-commit autoupdate` often fixes this.
    

    It would be cool if pre-commit autoupdate would work on this hook as expected. I suspect that the problem is that all 3 tags point to the same commit.

    opened by alkatar21 7
  • Consider module docstring while inserting future import

    Consider module docstring while inserting future import

    Summary

    When the python file consists of module docstring, the formatter inserts the from __future__ import annotation on the top of the docstring.

    Previous Behavior

    Source

    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    from typing import List
    
    a: List[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
        pass
    

    Generated

    from __future__ import annotations
    
    
    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    
    a: list[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
    

    The PR adds a function to find the module docstring. It will insert future import after the module docstring.

    opened by kracekumar 4
  • Shebang is not respected

    Shebang is not respected

    Input:

    #!/usr/bin/env python3
    from typing import AbstractSet
    
    s: AbstractSet
    

    Output:

    from collections.abc import Set
    #!/usr/bin/env python3
    
    s: Set
    

    I didn't prepare the fix yet, but I think that the best solution would be to save the first occurrence of non-comment, non-docstring expressions while scanning AST and put the imports before that. What do you think?

    opened by rominf 4
  • Create/push a tag for the package

    Create/push a tag for the package

    Hi guys, Thanks a lot for this great package, it makes the code much more pythonic! I have one favor: could you create/push a tag for the package? Else I have this annoying warning:

    [WARNING] The 'rev' field of repo 'https://github.com/snok/pep585-upgrade' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.
    
    opened by MRigal 3
  • don't add another import line from same package

    don't add another import line from same package

    For an input file like this:

    from collections.abc import Callable
    
    from typing import Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    when applying pep585 hook I get:

    from collections.abc import Collection
    from collections.abc import Callable
    
    def a(a: Callable, b: Collection):
        pass
    

    while I expect:

    from collections.abc import Callable, Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    It could be argued this is the job of e.g. isort, however - pep585-upgrade already tries to be nice in terms of imports - e.g. in a larger import set - this new import line gets added nearby other import from same package and not at the end (which would be simplest I guess)

    opened by kretes 1
  • Don't Upgrade Generics

    Don't Upgrade Generics

    The typing package supports generics, whereas collections.abc does not. Hence, when this tool replaces typing types that are used as generics with their equivalent types in collections.abc, it breaks the build. I found it helpful to run this tool as a one-off and then manually undo these cases though, so thank you!

    opened by Kurt-von-Laven 1
  • Imports removal corrupts imports

    Imports removal corrupts imports

    Minimal working example:

    Input:

    from typing import Any, Type, TypeVar
    
    t: Type
    

    Output:

    from typing import AnyVar
    
    t: type
    

    I have a fix for this already. Please wait for a PR with a fix and tests.

    opened by rominf 1
  • Correct name of .pre-commit-config.yaml

    Correct name of .pre-commit-config.yaml

    I got an error from pre-commit which suggested it was looking for a .pre-commit-config.yaml file, when I changed the filename, as in this commit.

    Possibly OS specific, this was on MacOS Big Sur, Python 3.9.

    opened by james-d-f 1
Releases(v1.0.1)
Owner
snok
Open source collaboration organization made by @jonasks & @sondrelg
snok
🤞 Website-Survival-Detection

- 🤞 Website-Survival-Detection It can help you to detect the survival status of the website in batches and return the status code! - 📜 Instructions

B0kd1 4 Nov 14, 2022
Ultimate Microsoft Edge Uninstaller!

Ultimate Microsoft Edge Uninstaller

1 Feb 08, 2022
Data on COVID-19 (coronavirus) cases, deaths, hospitalizations, tests • All countries • Updated daily by Our World in Data

COVID-19 Dataset by Our World in Data Find our data on COVID-19 and its documentation in public/data. Documentation Data: complete COVID-19 dataset Da

Our World in Data 5.5k Jan 03, 2023
A compiler for ARM, X86, MSP430, xtensa and more implemented in pure Python

Introduction The PPCI (Pure Python Compiler Infrastructure) project is a compiler written entirely in the Python programming language. It contains fro

Windel Bouwman 277 Dec 26, 2022
A python package to adjust the bias of probabilistic forecasts/hindcasts using "Mean and Variance Adjustment" method.

Documentation A python package to adjust the bias of probabilistic forecasts/hindcasts using "Mean and Variance Adjustment" method. Read documentation

1 Feb 02, 2022
An event-based script that is designed to improve your aim

Aim-Trainer Info: This is an event-based script that is designed to improve a user's aim. It was built using Python Turtle and the Random library. Ins

Ethan Francolla 4 Feb 17, 2022
Python Function to manage users via SCIM

Python Function to manage users via SCIM This script helps you to manage your v2 users. You can add and delete users or groups, add users to groups an

4 Oct 11, 2022
A full-featured, hackable tiling window manager written and configured in Python

A full-featured, hackable tiling window manager written and configured in Python Features Simple, small and extensible. It's easy to write your own la

Qtile 3.8k Dec 31, 2022
A Python program for calculating the 95%CI for GNSS-derived site velocities

GNSS_Vel_95%CI A Python program for calculating the 95%CI for GNSS-derived site velocities Function_GNSS_95CI.py is a Python function for calculating

<a href=[email protected]"> 4 Dec 16, 2022
Telegram bot to search quotes from brainyquote.com

Brainy Quote Bot @BrainQuoteBot A star ⭐ from you means a lot to us! Telegram bot to search quotes from brainyquote.com Usage Deploy to Heroku Tap on

21 Nov 24, 2022
Wrapper around anjlab's Android In-app Billing Version 3 to be used in Kivy apps

IABwrapper Wrapper around anjlab's Android In-app Billing Version 3 to be used in Kivy apps Install pip install iabwrapper Important ( Add these into

Shashi Ranjan 8 May 23, 2022
Notebooks for computing approximations to the prime counting function using Riemann's formula.

Notebooks for computing approximations to the prime counting function using Riemann's formula.

Tom White 2 Aug 02, 2022
Play tic-tac-toe in PowerPoint

The presentation has around 6,000 slides representing every possible game state (and some impossible ones, since I didn't check for wins or ties). You play by clicking on the squares, which are hyper

Jesse Li 3 Dec 18, 2021
Scientific color maps and standardization tools

Scicomap is a package that provides scientific color maps and tools to standardize your favourite color maps if you don't like the built-in ones. Scicomap currently provides sequential, bi-sequential

Thomas Bury 14 Nov 30, 2022
Manually Install Python 2.7 pip without any problem !

Python2.7_install_pip Manually Install Python 2.7 pip without any problem ! Download installPip.py to your system and Run the code using this Command

Ali Jafari 1 Dec 09, 2021
This is a method to build your own qgis configuration packages using osgeo4W.

This is a method to build your own qgis configuration packages using osgeo4W. Then you can automate deployment in your organization with a controled and trusted environnement.

Régis Haubourg 26 Dec 05, 2022
Snack Rice - A Rice University servery finder, customized for your needs!

Snack Rice - A Rice University servery finder, customized for your needs!

Aidan Gerber 3 Sep 25, 2022
Advanced Developing of Python Apps Final Exercise

Advanced-Developing-of-Python-Apps-Final-Exercise This is an exercise that I did for a python advanced learning course. The exercise is divided into t

Alejandro Méndez Fernández 1 Dec 04, 2021
Grimoire is a Python library for creating interactive fiction as hyperlinked html.

Grimoire Grimoire is a Python library for creating interactive fiction as hyperlinked html. Installation pip install grimoire-if Usage Check out the

Scott Russell 5 Oct 11, 2022
An extension module to make reaction based menus with disnake

disnake-ext-menus An experimental extension menu that makes working with reaction menus a bit easier. Installing python -m pip install -U disnake-ext-

1 Nov 25, 2021