python.testing.cwd ignored when discovering or running tests

[botsman]

Environment data

• VS Code version: 1.40.1
• Extension version (available under the Extensions sidebar): 2019.11.49689
• OS and version: MacOS Catalina 10.15.1
• Python version (& distribution if applicable, e.g. Anaconda): 3.7.4
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Relevant/affected Python packages and their versions: pytest==5.3.0
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi

Expected behaviour

Tests are displayed in the test explorer according to python.testing.cwd setting and I am able to run single test or tests from single file

Actual behaviour

python.testing.cwd is ignored for displaying tests and I am unable to run single test or tests from single file because I am getting an error file not found.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Sample project: https://github.com/pbotsman/sample_project

1. Open project
2. Open tests/py/some_test.py to start exploring tests
3. Try to execute single test test_something or tests in some_test.py file.
4. Get the error ERROR: file not found: ./tests/py/some_test.py::test_something
   (Sometimes in order to reproduce the problem I had to relaunch my VSCode)

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~/.virtualenvs/sample_project/bin/python -m pytest --rootdir ~/projects/sample_project --junitxml=/var/folders/0_/6w94lk1171vc_fpwbjqx02nw0000gn/T/tmp-4746yxdv7Lw3muwB.xml -v -s --log-cli-level INFO ./tests/py/some_test.py::test_something
cwd: ~/projects/sample_project/tests/py

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

console.ts:137 [Extension Host] Info Python Extension: 2019-11-20 14:06:31: getActivatedEnvironmentVariables, Class name = b, completed in 1ms, Arg 1: <Uri:/Users/pavel/projects/sample_project>, Arg 2: undefined, Arg 3: undefined
console.ts:137 [Extension Host] Info Python Extension: 2019-11-20 14:06:31: > ~/.virtualenvs/sample_project/bin/python -m pytest --rootdir ~/projects/sample_project --junitxml=/var/folders/0_/6w94lk1171vc_fpwbjqx02nw0000gn/T/tmp-5598WI4lpp4kFEsC.xml -v -s --log-cli-level INFO ./tests/py/some_test.py::test_something
console.ts:137 [Extension Host] Info Python Extension: 2019-11-20 14:06:31: cwd: ~/projects/sample_project/tests/py

Additional info

Looks like something has changed in the last release:

1. Previously tests would be displayed in the test explorer just as some_test.py would be root test. I mean that tests/py would be omitted in the test explorer since I have "python.testing.cwd": "tests/py" in my settings.
   Currently directories tests/py are displayed in the test explorer even though I have this setting.
2. I can't run single test because I am always getting an error file not found. It looks like something is wrong with the file paths.
   When the command
   ~/.virtualenvs/sample_project/bin/python -m pytest --rootdir ~/projects/sample_project --junitxml=/var/folders/0_/6w94lk1171vc_fpwbjqx02nw0000gn/T/tmp-5598WI4lpp4kFEsC.xml -v -s --log-cli-level INFO ./tests/py/some_test.py::test_something
   is called my current working directory is set to python.testing.cwd and that is why I am getting the error.
   So in the command mentioned above valid file path should be ./some_test.py::test_something, not ./tests/py/some_test.py::test_something.

[brettcannon]

Thanks for the bug report! We just wanted to quickly acknowledge we received it and we will triage this as soon as we can.

[kimadeline]

Hi @pbotsman 👋 Thank you for reaching out and thoroughly filling out the issue template 👍

I can confirm that I could reproduce your issue in the latest release of the extension (2019.11.49689), and that it doesn't happen in the previous release (2019.10.44104).

Seems like we don't take python.testing.cwd into account anymore when discovering tests, or running single tests using pytest.

A possible workaround while waiting for a fix would be to move the content of python.testing.cwd in the list of pytest arguments, let me know if the following works for you:

{
    "python.pythonPath": "/path/to/python",
    "python.testing.pytestArgs": [        
        "./tests/py",
        "-v",
        "-s",
        "--log-cli-level", "INFO",
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}