`pytest.warns()` asserts when `pytest.skip()` is called

[mgorny]

When pytest.skip() is used inside pytest.warns() (and no warning is emitted), the test fails. I'm not sure if this is intentional but it feels a bit off, given that the test would normally be skipped instead. I've hit this in the wild in urllib3's test suite.

Quick reproducer:

import pytest


def test_foo():
    with pytest.warns(UserWarning):
        pytest.skip("skipping")

Yields:

$ python -m pytest
========================================================= test session starts =========================================================
platform linux -- Python 3.11.7, pytest-8.1.0.dev116+g4546d5445, pluggy-1.4.0
rootdir: /tmp/test
collected 1 item                                                                                                                      

test_foo.py F                                                                                                                   [100%]

============================================================== FAILURES ===============================================================
______________________________________________________________ test_foo _______________________________________________________________

    def test_foo():
        with pytest.warns(UserWarning):
>           pytest.skip("skipping")
E           Skipped: skipping

test_foo.py:6: Skipped

During handling of the above exception, another exception occurred:

    def test_foo():
>       with pytest.warns(UserWarning):
E       Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
E        Emitted warnings: [].

test_foo.py:5: Failed
======================================================= short test summary info =======================================================
FAILED test_foo.py::test_foo - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
========================================================== 1 failed in 0.02s ==========================================================

pip list:

Package    Version
---------- -----------------------
iniconfig  2.0.0
packaging  23.2
pip        23.3.2
pluggy     1.4.0
pytest     8.1.0.dev116+g4546d5445
setuptools 69.0.3

---

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

[bluetech]

Hmm this is from 15524f3. One might argue that the test should indeed fail because it actually did not warn, but I'd say the previous behavior is the more expected one. Probably pytest exceptions (fail, skip, xfail, exit) need to be special-cased here.

cc @Cheukting @Zac-HD

[Zac-HD]

I agree that we should add a special case for these exceptions.

[bluetech]

Posted a fix in #11920. I opted to let all base exceptions (e.g. keyboard interrupt, sys.exit, etc.) propagate, not only pytest ones.