PyTest optional args don't support wildcards

[schlagercollin]

Type: Bug

Behaviour

Expected vs. Actual

Expected: pytest's positional arguments support wildcards to specify what test directories/files you want to run. E.g. pytest mydir/**/*_test.py.

Actual: Something about the VSCode-Python extension interprets these wildcards literally and results in an ERROR: file not found "mydir/**/*_test.py"

Steps to reproduce:

1. Add wildcard file into optional args for Python>PyTest
2. Refresh tests in the Testing pane
3. Watch as pytest discovery fails because the harness can't find the literal file name (i.e. doesn't interpret the wildcard characters correctly)

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.8.6
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Conda
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

> ~/miniconda3/bin/conda run -n myenv --no-capture-output python ~/.vscode/extensions/ms-python.python-2022.15.12711056/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.15.12711056/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear --continue-on-collection-errors platform/**/test_*.py
cwd: .
[ERROR 2022-8-28 16:31:18.838]: Error discovering pytest tests:
 [n [Error]: ============================= test session starts ==============================
platform darwin -- Python 3.8.6, pytest-7.1.2, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/cschlager/work/src2, configfile: setup.cfg
plugins: ddtrace-1.3.0, anyio-3.6.1, benchmark-3.4.1, env-0.6.2, timestamper-0.0.9, mock-3.8.2, hypothesis-6.54.4, hydra-core-1.2.0, cov-3.0.0
collected 0 items

========================= no tests collected in 0.01s ==========================
ERROR: file or directory not found: platform/**/test_*.py

User Settings

languageServer: "Pylance"

linting
• flake8Enabled: true
• mypyEnabled: true

formatting
• provider: "black"

testing
• cwd: "<placeholder>"
• pytestArgs: "<placeholder>"
• pytestEnabled: true
• unittestArgs: "<placeholder>"

Extension version: 2022.15.12711056
VS Code version: Code 1.71.2 (Universal) (74b1f979648cc44d385a2286793c226e611f59e7, 2022-09-14T21:05:37.721Z)
OS version: Darwin arm64 21.6.0
Modes:
Sandboxed: No

System Info

Item	Value
CPUs	Apple M1 Pro (10 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	4, 5, 5
Memory (System)	32.00GB (0.11GB free)
Process Argv	--crash-reporter-id 58741217-6e7d-4429-9190-74aeb72b3dd3
Screen Reader	no
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411cf:30557515
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vscaat:30438848
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
azure-dev_surveyone:30548225
2144e591:30553903
pyindex848cf:30577861
fc301958:30573243

[eleanorjboyd]

Hello, we are aware of buggy behavior on pytest discovery and running in vscode. We are currently work on a rewrite of how pytest works in vscode and hope this rewrite will fix this and similar bugs. The main issue which will be working on is here: #17242, and progress will be updated on this issue. Thank you!

[eleanorjboyd]

Hi! Currently we run pytest discovery using ChildProcess spawn function from nodejs. The spawn function handles the args as an array of strings and therefore does not expand wildcards in args. I think for this reason this is currently not possible with the extension.

@karthiknadig, would this be feasible at some point? Looked and it seems hard to switch from spawn to exec which is the recommended way around it. Also saw that pytest.main does not expand wildcards (but would need to check with maintainers on this) so unsure if we could switch to calling a script that then called pytest.main.

[eleanorjboyd]

opened pytest-dev/pytest#11319 to confirm

[eleanorjboyd]

Yes it is true they don't support it on pytest.main but I was reminded that we could handle it via glob from the stdlib to expand before passing it in.

@karthiknadig thoughts on if this would be worthwhile/doable?

[eleanorjboyd]

@karthiknadig, following up on this. Confirming it is not something we want to work on right?

[karthiknadig]

This is now handled via test explorer. If you want to run a subset of the test, you can use the search field in the test explorer and select the test or test files that you want to run.

Confirming it is not something we want to work on right?

That is correct. PyTest doesn't provide us with an API to do this. Given that this is shell dependent, and can get really complex to support generally,

[schlagercollin]

Thanks @eleanorjboyd and @karthiknadig for following up on this.