Running all Pytest tests no longer respects pytest.ini testpaths

[MetRonnie]

Environment data

• VS Code version: 1.59.1
• Extension version (available under the Extensions sidebar): v2021.9.1246542782
• OS and version: Linux x64 3.10.0-1160.41.1.el7.x86_64
• Python version (& distribution if applicable, e.g. Anaconda): 3.9.0 (conda-forge)
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Relevant/affected Python packages and their versions: Pytest 6.2.5
• Relevant/affected Python-related VS Code extensions and their versions: N/A
• Value of the python.languageServer setting: Pylance

[NOTE: If you suspect that your issue is related to the Microsoft Python Language Server (python.languageServer: 'Microsoft'), please download our new language server Pylance from the VS Code marketplace to see if that fixes your issue]

Expected behaviour

Clicking the "run all tests" button in the testing sidebar (a.k.a. test explorer) should run pytest without specifying a path, respecting the testpaths setting from pytest.ini.

From looking at backups of previous Python logs before the recent update to the extension, this is what would get called

> python -m pytest --rootdir ~/myproject

(I've removed extraneous absolute paths and options)

Actual behaviour

It passes the root dir path to pytest. In the Python log:

> python -m pytest --rootdir ~/myproject ~/myproject

This is a problem because I can't run the tests successfully like this. I get ModuleNotFoundErrors as it tries to collect stuff it doesn't need.

Steps to reproduce:

1. Have a Python project with testpaths set in pytest.ini
2. Click the "run all tests" button

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~/miniconda3/envs/<env-name>/bin/python -m pytest --rootdir <$HOME>/myproject --override-ini junit_family=xunit1 --junit-xml=/var/tmp/tmp-<rnd>.xml "-n 4" --disable-warnings -vv <$HOME>/myproject

[karthiknadig]

@MetRonnie Can you share your workspace settings?

[MetRonnie]

{
    "python.testing.pytestArgs": [
        "-n 4",
        "--disable-warnings",
        "-vv"
    ],
    "python.linting.flake8CategorySeverity.A": "Warning", // flake8-builtins
    "python.linting.flake8CategorySeverity.B": "Warning", // flake8-bugbear
    "python.linting.flake8CategorySeverity.C": "Warning", // flake8-comprehensions
    "python.linting.flake8CategorySeverity.M": "Warning", // flake8-mutable
    "python.linting.flake8CategorySeverity.N": "Warning", // flake8-broken-line
    "python.linting.flake8CategorySeverity.S": "Warning", // flake8-simplify
    "python.linting.flake8CategorySeverity.T": "Warning", // flake8-debugger
    "python.pythonPath": "<$HOME>/miniconda3/envs/<env-name>/bin/python",
}

[karthiknadig]

@MetRonnie I will need the pytest.ini. Looks like the above info is not enough to repro.

[MetRonnie]

The project is https://github.com/cylc/cylc-flow

pytest.ini: https://github.com/cylc/cylc-flow/blob/bf010514f8bd7bcce2529dc45a7c72d5709de417/pytest.ini#L17-L34

[karthiknadig]

It seems to be working fine for me:

[MetRonnie]

What if you invoke pytest on the command line with a path argument? This is what happens for me (same thing happens when clicking "run all tests" button in VSCode, except without the -x option it proceeds to run the tests and a lot of them fail):

$ cd ~/github/cylc-flow
$ pytest --disable-warnings -x .
================================== test session starts ==================================
platform linux -- Python 3.9.0, pytest-6.2.5, py-1.9.0, pluggy-0.13.1 -- ~/miniconda3/envs/cylc8/bin/python3.9
cachedir: .pytest_cache
rootdir: ~/github/cylc-flow, configfile: pytest.ini
plugins: forked-1.3.0, env-0.6.2, mock-3.4.0, xdist-2.1.0, cov-2.12.1, asyncio-0.15.1, anyio-3.3.0, tornasync-0.6.0.post2
[gw0] linux Python 3.9.0 cwd: ~/github/cylc-flow
[gw0] Python 3.9.0 | packaged by conda-forge | (default, Nov 26 2020, 07:57:39)  -- [GCC 9.3.0]
collecting 0 items / 1 error                                                            
======================================== ERRORS =========================================
_____ ERROR collecting tests/f/jinja2/08-local-lib-python/Jinja2Filters/qualify.py ______
tests/f/jinja2/08-local-lib-python/Jinja2Filters/qualify.py:19: in <module>
    import local_lookup
E   ModuleNotFoundError: No module named 'local_lookup'
================================ short test summary info ================================
ERROR tests/f/jinja2/08-local-lib-python/Jinja2Filters/qualify.py - ModuleNotFoundErro...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!
=================================== 1 error in 9.81s ====================================

I get the same ModuleNotFoundError if I disable xdist using -n0 --dist no

[karthiknadig]

Ok. I am able to repo this now.

[karthiknadig]

@MetRonnie Can you check if this build helps with this: https://github.com/microsoft/vscode-python/suites/3957288208/artifacts/99373426?

I tested it locally, I still need to run it through a bunch of other test scenarios, but let me know.