Experiments with Tox plugin system

For some of my tox projects, I use a single virtualenv for all test commands. The config looks something like this:

[tox]
envlist = flake8, pylint, docs, unit

[testenv]
envdir = {toxworkdir}/env
deps =
    -rrequirements-test.txt
commands =
    flake8,lint,test: flake8
    pylint,lint,test: pylint
    docs: sphinx-build
    unit,test: pytest

So even though I have different tox environments, they all share the same virtualenv.

What I run into with tox-battery is that the initial tox build ends up recreating each tox env for a single run of $ tox because there doesn't exist a requirements.previous file yet for each of the envs. Subsequent runs are fine though. For my local dev side, this isn't too much of a problem; however, for CI servers, the build times are longer than they need to be due to the venv recreation since the CI build always starts with a fresh checkout.

My question is whether you would consider having an option to name the previous requirements file based on the actual venv path instead of the tox env name or possibly a setting under the env configuration that explicitly defines the name used in previous requirements file. Something like:

[testenv:flake8]
# would be used to build filename "requirements.txt.{tox_battery_basename}.previous"
tox_battery_basename = env

[testenv:pylint]
tox_battery_basename = env

and maybe a global option which would apply to all envs unless they overwrote the value in the env specific config:

[testenv]
tox_battery_basename = env

Thoughts? If you are open to this idea, I could submit a PR for it.

Thanks!

This patch implements the support for nested requirements.

The use case for this is when a project contains a requirements.txt file pointing to another requirements file (i.e.: production-requirements.txt) which also points to other requirement files.

The patch achieves this by compiling the list of all the requirements, hashing it, and comparing this hash to the hash of the previous tox run (if any).

Hello, would you consider moving the project under the tox-dev umbrella? See documentation under https://tox.readthedocs.io/en/rewrite/plugins.html#adoption-of-a-plugin-under-tox-dev-github-organization

Furthermore, tox v4 is getting ready and we'd like to make sure this plugin is supported from day 1, we're collecting feature gaps for this under https://github.com/tox-dev/tox/issues/1974. Would be great if you could join our development chat under https://discord.gg/tox so we can assist with this. If you do so please drop in a line in the #plugin chat with the name of the repository you maintain. Thanks!

Python 3 uses a different default behavior for hash seeds than Python 2 (https://docs.python.org/3/using/cmdline.html#cmdoption-R). While with Python 2 the default is a static seed, with Python 3 it's a random one. That causes tox-battery under Python 3 to generate different seeds for multiple invocations, because of the use of the hash function in https://github.com/signalpillar/tox-battery/blob/master/toxbat/requirements.py#L101. That results in a re-creation of the environment, as the compared hashs always differ.

A workaround is to set PYTHONHASHSEED=0 when running tox. That'll cause a static hash seed to be used for tox-battery, while still using a random one per test environment. A better solution would be of course to replace the hash function with something which returns deterministic results.

I have a project that has some pip flags in the requirements.txt (specifically --no-binary). I'd love to use this project, but I'm getting a parser error for the file. Specifically, it seems that using pkg_resources doesn't support parsing that option.

I get this error on Appveyor on WIndows:

Traceback (most recent call last):
  File "c:\python36-x64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\python36-x64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Python36-x64\Scripts\tox.exe\__main__.py", line 7, in <module>
  File "c:\python36-x64\lib\site-packages\tox\session\__init__.py", line 44, in cmdline
    main(args)
  File "c:\python36-x64\lib\site-packages\tox\session\__init__.py", line 64, in main
    config = load_config(args)
  File "c:\python36-x64\lib\site-packages\tox\session\__init__.py", line 80, in load_config
    config = parseconfig(args)
  File "c:\python36-x64\lib\site-packages\tox\config\__init__.py", line 252, in parseconfig
    pm = get_plugin_manager(plugins)
  File "c:\python36-x64\lib\site-packages\tox\config\__init__.py", line 71, in get_plugin_manager
    pm.load_setuptools_entrypoints("tox")
  File "c:\python36-x64\lib\site-packages\pluggy\manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "c:\python36-x64\lib\site-packages\importlib_metadata\__init__.py", line 94, in load
    module = import_module(match.group('module'))
  File "c:\python36-x64\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\python36-x64\lib\site-packages\toxbat\requirements.py", line 44, in <module>
    from pip._internal.download import PipSession
ModuleNotFoundError: No module named 'pip._internal.download'

But it doesn't happen on Travis. I'm not sure what the OS has to do with it, probably the ordering of installation?

pip 10 was recently released, making some backwards-incompatible changes to internal APIs that tox-battery uses:

$ tox
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.14/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tox/session.py", line 39, in main
    config = prepare(args)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tox/session.py", line 27, in prepare
    config = parseconfig(args)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tox/config.py", line 205, in parseconfig
    pm = get_plugin_manager(plugins)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/tox/config.py", line 44, in get_plugin_manager
    pm.load_setuptools_entrypoints("tox")
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pluggy/__init__.py", line 397, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2405, in load
    return self.resolve()
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2411, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/toxbat/requirements.py", line 38, in <module>
    from pip.download import PipSession
ImportError: No module named download

pip-tools got around this by vendoring a copy of pip: https://github.com/jazzband/pip-tools/issues/580

The requirements.txt starts with:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#

argparse==1.2.1

The traceback is:

Traceback (most recent call last):
  File "/usr/local/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/usr/local/lib/python3.6/site-packages/tox/session.py", line 38, in main
    config = prepare(args)
  File "/usr/local/lib/python3.6/site-packages/tox/session.py", line 26, in prepare
    config = parseconfig(args)
  File "/usr/local/lib/python3.6/site-packages/tox/config.py", line 246, in parseconfig
    pm.hook.tox_configure(config=config)
  File "/usr/local/lib/python3.6/site-packages/pluggy.py", line 745, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy.py", line 339, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy.py", line 334, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/usr/local/lib/python3.6/site-packages/pluggy.py", line 614, in execute
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.6/site-packages/toxbat/requirements.py", line 38, in tox_configure
    return _ensure_envs_recreated_on_requirements_update(config)
  File "/usr/local/lib/python3.6/site-packages/toxbat/requirements.py", line 48, in _ensure_envs_recreated_on_requirements_update
    requires_recreation = are_requirements_changed(env)
  File "/usr/local/lib/python3.6/site-packages/toxbat/requirements.py", line 70, in are_requirements_changed
    for reqfile in requirement_files if reqfile and os.path.isfile(reqfile))
  File "/usr/local/lib/python3.6/site-packages/toxbat/requirements.py", line 70, in <genexpr>
    for reqfile in requirement_files if reqfile and os.path.isfile(reqfile))
  File "/usr/local/lib/python3.6/site-packages/toxbat/requirements.py", line 108, in is_changed
    previous_requirements_hash = int(content)
ValueError: invalid literal for int() with base 10: '#\n# This file is autogenerated by pip-compile\n# To update, run:\n#\n#    pip-compile --output-file requirements.txt requirements.in\n#\n\nargparse==1.2.1\n…

For now, it's possible to downgrade to tox-battery~=0.2.0 to work around this.

I have a project where tox.ini includes "-r{toxinidir}/requirements-test.txt", and requirements-test.txt includes "-r requirements.txt". When I try to run tox with tox-battery installed, I get the following error.

Traceback (most recent call last):
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 90, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 1137, in parseString
    raise exc
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 1127, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 2412, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 2669, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 1001, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 2395, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 1005, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/pyparsing.py", line 1785, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2788, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 94, in __init__
    requirement_string[e.loc:e.loc + 8]))
pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'-r requi'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/callisto/.venv/bin/tox", line 11, in <module>
    sys.exit(cmdline())
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/tox/session.py", line 38, in main
    config = prepare(args)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/tox/session.py", line 26, in prepare
    config = parseconfig(args)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/tox/config.py", line 236, in parseconfig
    pm.hook.tox_configure(config=config)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pluggy.py", line 724, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pluggy.py", line 338, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pluggy.py", line 333, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pluggy.py", line 596, in execute
    res = hook_impl.function(*args)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/toxbat/requirements.py", line 36, in tox_configure
    return _ensure_envs_recreated_on_requirements_update(config)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/toxbat/requirements.py", line 46, in _ensure_envs_recreated_on_requirements_update
    requires_recreation = are_requirements_changed(env)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/toxbat/requirements.py", line 69, in are_requirements_changed
    for reqfile in requirement_files
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/toxbat/requirements.py", line 70, in <genexpr>
    if reqfile and os.path.isfile(reqfile)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/toxbat/requirements.py", line 89, in is_changed
    changed = not are_equal_requirement_files(fpath, prev_version_fpath)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/toxbat/requirements.py", line 101, in are_equal_requirement_files
    reqs1 = sorted(parse_requirements(content_of(fpath1)), key=hash_cmp)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/toxbat/requirements.py", line 111, in parse_requirements
    return list(pkg_resources.parse_requirements(file_content))
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2781, in parse_requirements
    yield Requirement(line)
  File "/home/david/callisto/.venv/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2790, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'-r requi'"

Because the tox deps list is mapped through the parse_requirements_fname function. Some values in the initial list can be none. Skip over those instead of throwing an error.

Looks like pip broke tox-battery again:

Traceback (most recent call last):
  File "c:\python27-x64\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27-x64\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27-x64\Scripts\tox.exe\__main__.py", line 7, in <module>
  File "c:\python27-x64\lib\site-packages\tox\session\__init__.py", line 44, in cmdline
    main(args)
  File "c:\python27-x64\lib\site-packages\tox\session\__init__.py", line 64, in main
    config = load_config(args)
  File "c:\python27-x64\lib\site-packages\tox\session\__init__.py", line 80, in load_config
    config = parseconfig(args)
  File "c:\python27-x64\lib\site-packages\tox\config\__init__.py", line 267, in parseconfig
    pm.hook.tox_configure(config=config)  # post process config object
  File "c:\python27-x64\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\python27-x64\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\python27-x64\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "c:\python27-x64\lib\site-packages\pluggy\callers.py", line 208, in _multicall
    return outcome.get_result()
  File "c:\python27-x64\lib\site-packages\pluggy\callers.py", line 81, in get_result
    _reraise(*ex)  # noqa
  File "c:\python27-x64\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\python27-x64\lib\site-packages\toxbat\requirements.py", line 60, in tox_configure
    return _ensure_envs_recreated_on_requirements_update(config)
  File "c:\python27-x64\lib\site-packages\toxbat\requirements.py", line 72, in _ensure_envs_recreated_on_requirements_update
    requires_recreation = are_requirements_changed(env)
  File "c:\python27-x64\lib\site-packages\toxbat\requirements.py", line 96, in are_requirements_changed
    if reqfile and os.path.isfile(reqfile)])
  File "c:\python27-x64\lib\site-packages\toxbat\requirements.py", line 126, in is_changed
    new_requirements = parse_pip_requirements(fpath)
  File "c:\python27-x64\lib\site-packages\toxbat\requirements.py", line 109, in parse_pip_requirements
    session=PipSession())
  File "c:\python27-x64\lib\site-packages\toxbat\requirements.py", line 110, in <genexpr>
    if r.req
AttributeError: 'ParsedRequirement' object has no attribute 'req'

https://github.com/signalpillar/tox-battery/blob/110c931c4cb7b803f2637f8f4a0feae1593f5522/toxbat/requirements.py#L158-L160

is great, but doesn't handle the long-form option:

[testenv:test]
deps =
    --no-deps
    --requirement deps/test.txt

This would be a good issue for someone to handle for Hacktoberfest 😄

I would like to know if there are any reasons for adding this outside tox itself? To me this looks like a product bug that needs to be fixed.

The correct bug seems to be https://github.com/tox-dev/tox/issues/149

Consider the scenario where install_requires changes in setup.py but isn't reflected in any requirements file (e.g. maybe the requirements file only lists test dependencies and package dependencies only exist in setup.py).

Currently, this does not trigger a venv recreate since tox-battery only checks deps defined in tox.ini, and tox itself doesn't trigger a recreate either in this case (at least in my initial testing I didn't see this). This can result in reusing a venv with an old dependency installed from setup.py.

What if tox-battery used those dependencies in addition to the ones in the requirements files to generate its requirements hash? It's possible to parse the contents of <pkgname>.egg-info/requires.txt to get a list of dependencies defined in setup.py so that approach may work in most cases.

Thoughts?