ModuleNotFoundError when using `--doctest-modules`, a `src/` layout, and `--import-mode=importlib` (and no editable mode install)

[flying-sheep]

reproducer

We’d like to run our tests on an installed package instead of the development version, as we delete some files in our build process. We therefore don’t use editable installs.

I think the problem is that there seems to be no way to configure the import root used with the importlib import mode. The project rootdir gets passed here instead of something user configurable:

pytest/src/_pytest/doctest.py

Lines 545 to 549 in c34eaaa

	module = import_path(
	self.path,
	root=self.config.rootpath,
	mode=self.config.getoption("importmode"),
	)

Which means that the following code will run module_name = module_name_from_path('<rootdir>/src/anndata/_types.py', '<rootdir>'), i.e. module_name = 'src.anndata._types' when it should be 'anndata._types'

pytest/src/_pytest/pathlib.py

Lines 524 to 525 in 9f3bdac

	if mode is ImportMode.importlib:
	module_name = module_name_from_path(path, root)

Without src/ layout, this works accidentally, as the module_name happens to match the real module name, and the module only gets imported once.

Pytest 7.4.2

[richardxia]

I just wanted to add that we are hitting this issue as well, but we are using editable installs, so it looks like this issue affects both the case with editable and non-editable installs of the project source files.

Just to be clear, our project is:

• an editable installation
• has a src/ layout
• puts non-doctest tests outside the application code, in a tests/ directory that is a sibling to the src/ directory
• uses --import-mode=importlib
• (attempts to) use --doctest-modules

We more or less exactly follow the suggested layout and settings in https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code, and everything was working fine until attempting to run doctests within the application code using --doctest-modules.

[flying-sheep]

Yeah, could this please be added to the 8.0 milestone?

This might require a new setting or some breaking change, so this is the ideal moment

[bluetech]

Does it work better if you use pytest --doctest-modules --import-mode=importlib --pyargs my.package?

[flying-sheep]

Actually yes! It’s getting really complicated though:

[tool.pytest.ini_options]
addopts = [
    "--strict-markers",
    "--doctest-modules",
    "--pyargs",
]
testpaths = [
    "anndata", # doctests in docstrings (module name due to --pyargs)
    "./src/anndata/tests", # unit tests
    "./docs/concatenation.rst", # further doctests
]

and then in src/anndata/tests/conftest.py a collect_ignore = ["helpers.py"]. But it works.

[bluetech]

In the issue subject you write --import-mode=importlib but you are not using that right?

If you give me a bit of instructions how to reproduce this locally, I can take a look.

[flying-sheep]

Sorry, I muddled my project (scverse/anndata#1151) too much with this issue. I’m trying to be clear now.

Adding --import-mode=importlib to the above results in this:

E   ModuleNotFoundError: No module named
  'home.phil..local.share.hatch.[…].site-packages.anndata._core';
  'home.phil..local.share.hatch.[…].site-packages.anndata' is not a package

What needs to work for this issue to be considered fixed:

+ src/my_module/__init__.py  # containing a doctest
+ tests/test_basic.py

[build-system]
build-backend = "hatchling.build"
requires = ["hatchling", "hatch-vcs"]

[project]
name = "my_module"
version = "0.1.0"

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
    "--strict-markers",
    "--doctest-modules",
    "--pyargs",
]
testpaths = [
    "my_module",  # or without `--pyargs`: "./src/my_module"
    "./tests",
]

What would be nice if it worked for migrating our project:

+ src/my_module/
  + __init__.py  # containing a doctest
  + tests/
    + conftest.py  # containing `collect_ignore = ["helpers.py"]`
    + helpers.py
    + test_basic.py

...

[tool.pytest.ini_options]
addopts = [
    "--import-mode=importlib",
    "--strict-markers",
    "--doctest-modules",
    "--pyargs",
]
testpaths = [
    "my_module",
    "./src/my_module/tests",
]

[flying-sheep]

Very weird, I just can’t find a good reproducer. In our project, it fails: https://dev.azure.com/scverse/anndata/_build/results?buildId=5213&view=logs&jobId=5ea502cf-d418-510c-3b5f-c4ba606ae534&j=5ea502cf-d418-510c-3b5f-c4ba606ae534&t=5091afe0-1787-5ef7-c6a1-ce6dc06b30a7

platform linux -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/vsts/work/1/s
configfile: pyproject.toml
testpaths: anndata, ./src/anndata/tests, ./docs/concatenation.rst
plugins: xdist-3.5.0, memray-1.5.0, anyio-4.2.0, cov-4.1.0
collected 3028 items / 20 errors

==================================== ERRORS ====================================
__________________________ ERROR collecting utils.py ___________________________
/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/anndata/utils.py:13: in <module>
    from ._core.sparse_dataset import BaseCompressedSparseDataset
E   ModuleNotFoundError: No module named 'opt.hostedtoolcache.Python.3.11.7.x64.lib.python3.11.site-packages.anndata._core'; 'opt.hostedtoolcache.Python.3.11.7.x64.lib.python3.11.site-packages.anndata' is not a package
__________________ ERROR collecting _core/aligned_mapping.py ___________________
/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/anndata/_core/aligned_mapping.py:23: in <module>
    from ..utils import deprecated, dim_len, ensure_df_homogeneous, warn_once
E   ImportError: cannot import name 'deprecated' from 'opt.hostedtoolcache.Python.3.11.7.x64.lib.python3.11.site-packages.anndata.utils' (/opt/hostedtoolcache/Python/3.11.7/x64/lib/python3.11/site-packages/anndata/utils.py)
...

but using the “reproducer” I so confidently outlined above, everything works.

[flying-sheep]

OK, I minified the repo somewhat. I assume the problem is that even with --pyargs, pytest tries to import the module with weird names while running doctest.

https://github.com/flying-sheep/pytest-doctest-import-mismatch

I see errors like this:

____ ERROR collecting _core/merge.py ____
/home/phil/.local/share/hatch/env/virtual/anndata/kX3YdB0h/anndata/lib/python3.11/site-packages/anndata/_core/merge.py:28: in <module>
   from ..compat import _map_cat_to_str
E   ModuleNotFoundError: No module named 'home.phil..local.share.hatch.env.virtual.anndata.kX3YdB0h.anndata.lib.python3.11.site-packages.anndata.compat'; 'home.phil..local.share.hatch.env.virtual.anndata.kX3YdB0h.anndata.lib.python3.11.site-packages.anndata' is not a package
____ ERROR collecting _core/raw.py ____
/home/phil/.local/share/hatch/env/virtual/anndata/kX3YdB0h/anndata/lib/python3.11/site-packages/anndata/_core/raw.py:9: in <module>
   from .aligned_mapping import AxisArrays
E   ImportError: cannot import name 'AxisArrays' from 'home.phil..local.share.hatch.env.virtual.anndata.kX3YdB0h.anndata.lib.python3.11.site-packages.anndata._core.aligned_mapping' (/home/phil/.local/share/hatch/env/virtual/anndata/kX3YdB0h/anndata/lib/python3.11/site-packages/anndata/_core/aligned_mapping.py)

The reproducer is by no means minimal yet, and I can make it smaller for sure, but maybe it’s helpful before i get to that already.

[bluetech]

(First, consider editing the original post to point to the reproduction; the ImportPathMismatchError error stated there cannot happen in --import-mode=importlib so it's quite confusing).

Thanks for creating a reproduction. I tried it now. I was able to reproduce with pytest 7.4, but with pytest main it works (only if using venv name venv instead of .venv -- see below). I bisected the fix to commit 194a782 which is also included in 8.0.0rc1.

However, there's an issue I noticed: If the path contains dots, e.g. .venv and python3.11 in this case, then --import-mode=importlib fails like this:

Dot problem

__________________________________________________ ERROR collecting .venv/lib/python3.11/site-packages/anndata/__init__.py ___________________________________________________
.venv/lib/python3.11/site-packages/anndata/__init__.py:11: in <module>
    from ._core.anndata import AnnData
E   ValueError: Empty module name
________________________________________________ ERROR collecting .venv/lib/python3.11/site-packages/anndata/_core/access.py _________________________________________________
/usr/lib/python3.11/importlib/__init__.py:121: in import_module
    raise TypeError(msg.format(name))
E   TypeError: the 'package' argument is required to perform a relative import for '.venv.lib.python3.11.site-packages'
____________________________________________ ERROR collecting .venv/lib/python3.11/site-packages/anndata/_core/aligned_mapping.py ____________________________________________
/usr/lib/python3.11/importlib/__init__.py:121: in import_module
    raise TypeError(msg.format(name))
E   TypeError: the 'package' argument is required to perform a relative import for '.venv.lib.python3.11.site-packages'
________________________________________________ ERROR collecting .venv/lib/python3.11/site-packages/anndata/_core/anndata.py ________________________________________________
/usr/lib/python3.11/importlib/__init__.py:121: in import_module
    raise TypeError(msg.format(name))
E   TypeError: the 'package' argument is required to perform a relative import for '.venv.lib.python3.11.site-packages'
_________________________________________________ ERROR collecting .venv/lib/python3.11/site-packages/anndata/_core/merge.py _________________________________________________
/usr/lib/python3.11/importlib/__init__.py:121: in import_module
    raise TypeError(msg.format(name))
E   TypeError: the 'package' argument is required to perform a relative import for '.venv.lib.python3.11.site-packages'
_____________________________________________________ ERROR collecting .venv/lib/python3.11/site-packages/anndata/tests ______________________________________________________
/usr/lib/python3.11/importlib/__init__.py:121: in import_module
    raise TypeError(msg.format(name))
E   TypeError: the 'package' argument is required to perform a relative import for '.venv.lib.python3.11.site-packages.anndata.tests'

Python module names (individual components) cannot contain dots, and the --import-mode=importlib code does stuff like module_name.split(".").

I tried "escaping" the dots by replacing them with _dot_, which allows things to continue, however then it fails like this:

Exec module problem

self = <DoctestModule __init__.py>

    def collect(self) -> Iterable[DoctestItem]:
        import doctest
    
        class MockAwareDocTestFinder(doctest.DocTestFinder):
            """A hackish doctest finder that overrides stdlib internals to fix a stdlib bug.
    
            https://github.com/pytest-dev/pytest/issues/3456
            https://bugs.python.org/issue25532
            """
    
            def _find_lineno(self, obj, source_lines):
                """Doctest code does not take into account `@property`, this
                is a hackish way to fix it. https://bugs.python.org/issue17446
    
                Wrapped Doctests will need to be unwrapped so the correct
                line number is returned. This will be reported upstream. #8796
                """
                if isinstance(obj, property):
                    obj = getattr(obj, "fget", obj)
    
                if hasattr(obj, "__wrapped__"):
                    # Get the main obj in case of it being wrapped
                    obj = inspect.unwrap(obj)
    
                # Type ignored because this is a private function.
                return super()._find_lineno(  # type:ignore[misc]
                    obj,
                    source_lines,
                )
    
            def _find(
                self, tests, obj, name, module, source_lines, globs, seen
            ) -> None:
                if _is_mocked(obj):
                    return
                with _patch_unwrap_mock_aware():
                    # Type ignored because this is a private function.
                    super()._find(  # type:ignore[misc]
                        tests, obj, name, module, source_lines, globs, seen
                    )
    
            if sys.version_info < (3, 13):
    
                def _from_module(self, module, object):
                    """`cached_property` objects are never considered a part
                    of the 'current module'. As such they are skipped by doctest.
                    Here we override `_from_module` to check the underlying
                    function instead. https://github.com/python/cpython/issues/107995
                    """
                    if isinstance(object, functools.cached_property):
                        object = object.func
    
                    # Type ignored because this is a private function.
                    return super()._from_module(module, object)  # type: ignore[misc]
    
            else:  # pragma: no cover
                pass
    
        if self.path.name == "conftest.py":
            module = self.config.pluginmanager._importconftest(
                self.path,
                self.config.getoption("importmode"),
                rootpath=self.config.rootpath,
            )
        else:
            try:
>               module = import_path(
                    self.path,
                    root=self.config.rootpath,
                    mode=self.config.getoption("importmode"),
                )

../../src/pytest/src/_pytest/doctest.py:569: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

p = PosixPath('/home/ran/tmp/pytest-doctest-import-mismatch/.venv/lib/python3.11/site-packages/anndata/__init__.py')

    def import_path(
        p: Union[str, "os.PathLike[str]"],
        *,
        mode: Union[str, ImportMode] = ImportMode.prepend,
        root: Path,
    ) -> ModuleType:
        """Import and return a module from the given path, which can be a file (a module) or
        a directory (a package).
    
        The import mechanism used is controlled by the `mode` parameter:
    
        * `mode == ImportMode.prepend`: the directory containing the module (or package, taking
          `__init__.py` files into account) will be put at the *start* of `sys.path` before
          being imported with `importlib.import_module`.
    
        * `mode == ImportMode.append`: same as `prepend`, but the directory will be appended
          to the end of `sys.path`, if not already in `sys.path`.
    
        * `mode == ImportMode.importlib`: uses more fine control mechanisms provided by `importlib`
          to import the module, which avoids having to muck with `sys.path` at all. It effectively
          allows having same-named test modules in different places.
    
        :param root:
            Used as an anchor when mode == ImportMode.importlib to obtain
            a unique name for the module being imported so it can safely be stored
            into ``sys.modules``.
    
        :raises ImportPathMismatchError:
            If after importing the given `path` and the module `__file__`
            are different. Only raised in `prepend` and `append` modes.
        """
        mode = ImportMode(mode)
    
        path = Path(p)
    
        if not path.exists():
            raise ImportError(path)
    
        if mode is ImportMode.importlib:
            module_name = module_name_from_path(path, root)
            with contextlib.suppress(KeyError):
                return sys.modules[module_name]
    
            for meta_importer in sys.meta_path:
                spec = meta_importer.find_spec(module_name, [str(path.parent)])
                if spec is not None:
                    break
            else:
                spec = importlib.util.spec_from_file_location(module_name, str(path))
    
            if spec is None:
                raise ImportError(f"Can't find module {module_name} at location {path}")
            mod = importlib.util.module_from_spec(spec)
            sys.modules[module_name] = mod
>           spec.loader.exec_module(mod)  # type: ignore[union-attr]

../../src/pytest/src/_pytest/pathlib.py:540: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_frozen_importlib_external.SourceFileLoader object at 0x7feac0a280d0>
module = <module '_dot_venv.lib.python3_dot_11.site-packages.anndata' from '/home/ran/tmp/pytest-doctest-import-mismatch/.venv/lib/python3.11/site-packages/anndata/__init__.py'>

>   ???

<frozen importlib._bootstrap_external>:940: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

f = <built-in function exec>
args = (<code object <module> at 0x7feac0c24ff0, file "/home/ran/tmp/pytest-doctest-import-mismatch/.venv/lib/python3.11/site...__file__': '/home/ran/tmp/pytest-doctest-import-mismatch/.venv/lib/python3.11/site-packages/anndata/__init__.py', ...})
kwds = {}

>   ???

<frozen importlib._bootstrap>:241: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """Annotated multivariate observation data."""
    from __future__ import annotations
    
    # Allowing notes to be added to exceptions. See: https://github.com/scverse/anndata/issues/868
    import sys
    
    if sys.version_info < (3, 11):
        # Backport package for exception groups
        import exceptiongroup  # noqa: F401
    
>   from ._core.anndata import AnnData
E   ModuleNotFoundError: No module named '_dot_venv'

.venv/lib/python3.11/site-packages/anndata/__init__.py:11: ModuleNotFoundError

This is possibly another problem with --import-mode=importlib; if I swap the order of the exec_module and the insert_missing_modules statements then it does work, but I'm not familiar enough the code to know if it's correct.

@nicoddemus Do you have any idea about this?

[flying-sheep]

First, consider editing the original post to point to the reproduction; the ImportPathMismatchError error stated there cannot happen in --import-mode=importlib so it's quite confusing

fixed

Thanks for creating a reproduction. I tried it now. I was able to reproduce with pytest 7.4, but with pytest main it works

Well, unless I’m missing something, the same package is still imported twice with this approach, right?

Which might cause problems similar to #11306 when the two versions try to access some shared resource. (e.g. let’s say they both open a database connection or write a file on import time – not super good style, but possible)