Skip to content

chore: typing: PEP 563, PEP 585, PEP 604, PEP 613 #6218

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 6, 2024
Merged

chore: typing: PEP 563, PEP 585, PEP 604, PEP 613 #6218

merged 3 commits into from
Oct 6, 2024

Conversation

bastimeyer
Copy link
Member

@bastimeyer bastimeyer commented Oct 4, 2024 •
edited
Loading

In preparation for the upcoming py38 removal, this updates all typing annotations to modern syntax.

Before I'm going to merge this, I will have to double and triple check everything first, which I haven't done yet. I will have a look at this on the weekend. The changes were all done manually, and I tried to translate everything 1-to-1, but some minor things were also changed and fixed in the process. Not everything though, but this wasn't the goal here. The goal is to not introduce any new typing bugs.

$ git diff master --shortstat
 123 files changed, 2286 insertions(+), 2080 deletions(-)

As a requirement, currently, without dropping py38 just yet, PEP 563 is required with its from __future__ import annotations, to postpone evaluation of annotations. This could've been done a long time ago already, but never was, because it didn't really matter. Now we have one giant commit that updates everything and will need to be added to .git-blame-ignore-revs.

There are three commits. Commit two and three don't belong into one, so they had to be split up.

    1. Add PEP 563 imports: from __future__ import annotations
      This can be removed again (where possible) when dropping py38 at the end of October. Union types (PEP 604) still require it, as it's a py310 feature and requires dropping py39 support.
    2. Remove all previously quoted forward-references in type annotations (due to PEP 563).
    3. Switch to PEP 585
      This replaces generics of the builtin base classes like type, list, tuple, set, dict, etc., which previously had to be done by imported their respective aliases from typing. This also imports stuff like Callable, Mapping, Iterable, etc. from collections.abc or deque from collections or Pattern from re, rather than their deprecated aliases from typing.
      https://docs.python.org/3/library/typing.html#deprecated-aliases
    4. Switch to PEP 604
      This replaces typing.Union/typing.Optional with union types and their a | b syntax. a | None should always come last, as a suffix rather than prefix or somewhere in-between a chain of union types.
    5. Add remaining missing TypeAlias annotations (PEP 613), and always import from typing_extensions in TYPE_CHECKING mode, as TypeAlias is a py310 feature which requires this compat import. Use annotation strings, to allow modern syntax via postponed evaluation (PEP 563).
    6. Fix and use non-invariant types in places where it was previously an obvious mistake to use the invariant type (only did this for Mapping/dict).
    7. Fix some other minor stuff (one of those minor changes was an incorrect import, which always caused typing issues in PyCharm and its mypy plugin).
  2. Add the scripts directory to the mypy config and fix typing issues. Both the GH release script and CDP generator script required some further changes which didn't belong into commit one, as well as the config update.
  3. Regenerate the CDP modules using the same parameters from their previous update.

Two exceptions which currently can't be updated yet are streamlink.plugin.plugin._MCollection(list[MType]), because it's a runtime typing requirement and can't be evaluated on py38, and streamlink.plugin.plugin.Matches(_MCollection[re.Match | None]), which requires py310 due to the union type.

$ git grep 'from typing' | column -ts: -Nfile,import -H-
file                                                 import
build_backend/__init__.py                            from typing import Any
build_backend/onbuild.py                             from typing import Any, Generic, TypeVar
build_backend/plugins_json.py                        from typing import TYPE_CHECKING, Any, ClassVar, Generic, TextIO, TypeVar
build_backend/plugins_json.py                            from typing_extensions import TypeAlias
build_backend/test_plugins_json.py                   from typing import Any
docs/ext_html_template_vars.py                       from typing import Any
docs/ext_plugins.py                                  from typing import Any
script/generate-cdp.py                               from typing import cast
script/generate-cdp.py                               from typing import Any
script/generate-cdp.py                               from typing import TYPE_CHECKING, Any
script/generate-cdp.py                                   from typing_extensions import TypeAlias
script/github-release.py                             from typing import IO, Any, Literal, NewType
script/update-user-agents.py                         from typing import Any
src/streamlink/cache.py                              from typing import Any
src/streamlink/compat.py                             from typing import Any
src/streamlink/logger.py                             from typing import IO, TYPE_CHECKING, Literal
src/streamlink/options.py                            from typing import Any, ClassVar, Literal, TypeVar
src/streamlink/plugin/api/validate/_schemas.py       from typing import Any, Literal
src/streamlink/plugin/api/validate/_validate.py      from typing import Any
src/streamlink/plugin/api/validate/_validators.py    from typing import Any, Literal
src/streamlink/plugin/api/websocket.py               from typing import Any
src/streamlink/plugin/plugin.py                      from typing import TYPE_CHECKING, Any, ClassVar, List, Literal, NamedTuple, Type, TypeVar, Union
src/streamlink/plugins/twitch.py                     from typing import ClassVar
src/streamlink/plugins/ustreamtv.py                  from typing import Any
src/streamlink/session/http.py                       from typing import Any
src/streamlink/session/http.pyi                      from typing import Any
src/streamlink/session/http.pyi                      from typing_extensions import TypeAlias
src/streamlink/session/options.py                    from typing import TYPE_CHECKING, Any, ClassVar
src/streamlink/session/plugins.py                    from typing import TYPE_CHECKING, Literal
src/streamlink/session/plugins.py                        from typing import TypeAlias, TypedDict  # type
src/streamlink/session/plugins.py                        from typing_extensions import TypeAlias, TypedDict
src/streamlink/session/session.py                    from typing import Any
src/streamlink/stream/dash/dash.py                   from typing import Any
src/streamlink/stream/dash/manifest.py               from typing import TYPE_CHECKING, Any, ClassVar, Literal, TypeVar, overload
src/streamlink/stream/dash/manifest.py                   from typing_extensions import TypeAlias
src/streamlink/stream/ffmpegmux.py                   from typing import Any, ClassVar, Generic, TextIO, TypeVar
src/streamlink/stream/hls/hls.py                     from typing import Any, ClassVar
src/streamlink/stream/hls/m3u8.py                    from typing import ClassVar, Generic, TypeVar
src/streamlink/stream/hls/m3u8.py                        from typing import Self  # type
src/streamlink/stream/hls/m3u8.py                        from typing_extensions import Self
src/streamlink/stream/hls/segment.py                 from typing import NamedTuple
src/streamlink/stream/segmented/segmented.py         from typing import ClassVar, Generic, TypeVar
src/streamlink/stream/segmented/segmented.py             from typing import TypeAlias  # type
src/streamlink/stream/segmented/segmented.py             from typing_extensions import TypeAlias
src/streamlink/utils/args.py                         from typing import Any, Generic, TypeVar
src/streamlink/utils/cache.py                        from typing import Generic, TypeVar
src/streamlink/utils/data.py                         from typing import Any
src/streamlink/utils/formatter.py                    from typing import Any
src/streamlink/utils/module.py                       from typing import TYPE_CHECKING
src/streamlink/utils/processoutput.py                from typing import BinaryIO
src/streamlink/utils/times.py                        from typing import Generic, TypeVar
src/streamlink/webbrowser/cdp/client.py              from typing import Any
src/streamlink/webbrowser/cdp/client.py                  from typing import Self, TypeAlias  # type
src/streamlink/webbrowser/cdp/client.py                  from typing_extensions import Self, TypeAlias
src/streamlink/webbrowser/cdp/connection.py          from typing import Generic, TypeVar, cast
src/streamlink/webbrowser/cdp/connection.py              from typing import Self, TypeAlias  # type
src/streamlink/webbrowser/cdp/connection.py              from typing_extensions import Self, TypeAlias
src/streamlink/webbrowser/cdp/devtools/browser.py    from typing import Any
src/streamlink/webbrowser/cdp/devtools/debugger.py   from typing import Any
src/streamlink/webbrowser/cdp/devtools/dom.py        from typing import Any
src/streamlink/webbrowser/cdp/devtools/emulation.py  from typing import Any
src/streamlink/webbrowser/cdp/devtools/fetch.py      from typing import Any
src/streamlink/webbrowser/cdp/devtools/input_.py     from typing import Any
src/streamlink/webbrowser/cdp/devtools/inspector.py  from typing import Any
src/streamlink/webbrowser/cdp/devtools/io.py         from typing import Any
src/streamlink/webbrowser/cdp/devtools/network.py    from typing import Any
src/streamlink/webbrowser/cdp/devtools/page.py       from typing import Any
src/streamlink/webbrowser/cdp/devtools/runtime.py    from typing import Any
src/streamlink/webbrowser/cdp/devtools/security.py   from typing import Any
src/streamlink/webbrowser/cdp/devtools/target.py     from typing import Any
src/streamlink/webbrowser/cdp/devtools/util.py       from typing import TYPE_CHECKING, Any
src/streamlink/webbrowser/cdp/devtools/util.py           from typing_extensions import TypeAlias
src/streamlink_cli/argparser.py                      from typing import Any
src/streamlink_cli/compat.py                         from typing import TYPE_CHECKING, BinaryIO
src/streamlink_cli/console.py                        from typing import Any, TextIO
src/streamlink_cli/main.py                           from typing import Any
src/streamlink_cli/output/file.py                    from typing import BinaryIO
src/streamlink_cli/output/player.py                  from typing import ClassVar, TextIO
src/streamlink_cli/utils/progress.py                 from typing import TYPE_CHECKING, TextIO
src/streamlink_cli/utils/progress.py                     from typing_extensions import TypeAlias
tests/cli/main/test_check_file_output.py             from typing import TYPE_CHECKING
tests/cli/test_argparser.py                          from typing import Any
tests/conftest.py                                    from typing import Any
tests/plugins/__init__.py                            from typing import TYPE_CHECKING
tests/plugins/__init__.py                                from typing_extensions import TypeAlias
tests/session/test_plugins.py                        from typing import cast
tests/stream/hls/test_hls.py                         from typing import NamedTuple
tests/test_plugin.py                                 from typing import Any
tests/webbrowser/cdp/test_client.py                  from typing import TYPE_CHECKING, cast
tests/webbrowser/cdp/test_client.py                      from typing_extensions import TypeAlias
tests/webbrowser/test_chromium.py                    from typing import Any

@bastimeyer bastimeyer force-pushed the typing/PEP-563+585+604 branch from 083e27d to 57031a4 Compare October 4, 2024 14:42
@bastimeyer

This comment was marked as outdated.

Sign in to view
@bastimeyer bastimeyer marked this pull request as draft October 4, 2024 14:46
@bastimeyer
Copy link
Member Author

I now understand the issue.

The broken dataclasses in the mentioned plugins were not caused by anything special in the dataclasses themselves. I got something wrong after reading a different and unrelated issue with a similar error message on the cpython issue tracker. This issue is caused by Streamlink's plugin module loader.

So, as explained, when applying PEP 563, all of the module's type annotations are turned into strings for postponed evaluation. The dataclasses implementation therefore needs to look up the loaded/cached module of that dataclass-object from sys.modules, because of the annotation namespaces that need to be resolved. This failed in the previous version of this PR with the modified twitch/ustreamtv modules, which have been reverted by me in the current version. But it only failed while building the docs and shell completions, and not while running the tests, because in those cases, plugins are not loaded lazily due to the editable install, which always loads all plugins.

  File "/opt/hostedtoolcache/Python/3.12.6/x64/lib/python3.12/dataclasses.py", line 749, in _is_type
    ns = sys.modules.get(cls.__module__).__dict__
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '__dict__'. Did you mean: '__dir__'?

The reason for the error is that Streamlink's plugin loader currently doesn't add the module to sys.module (and also doesn't look up cached modules). This was done on purpose when it was implemented because the plugin loader needs to be able to override custom user plugin modules (sideloading), so it always loaded and executed the plugin modules without touching the module runtime cache in sys.modules.

Simply looking up the cache and only adding new modules doesn't work, because it breaks the sideloading implementation, so this needs to be fixed by adding an override parameter to streamlink.utils.{load,exec}_module().

While investigating this, I also ran into issues with the freezegun dependency for freezing datetime objects. Already loaded/executed and cached modules need to be returned from cache, or there's a caching mismatch in freezegun's own module cache.

@bastimeyer bastimeyer mentioned this pull request Oct 5, 2024
Merged
@bastimeyer bastimeyer force-pushed the typing/PEP-563+585+604 branch 3 times, most recently from c7c4a38 to 093d1b3 Compare October 6, 2024 09:01
@bastimeyer bastimeyer marked this pull request as ready for review October 6, 2024 09:01
@bastimeyer bastimeyer force-pushed the typing/PEP-563+585+604 branch from 093d1b3 to aa01d4b Compare October 6, 2024 09:20
@bastimeyer bastimeyer force-pushed the typing/PEP-563+585+604 branch from aa01d4b to e5683f5 Compare October 6, 2024 09:39
@bastimeyer bastimeyer changed the title chore: typing: PEP 563, PEP 585, PEP 604 chore: typing: PEP 563, PEP 585, PEP 604, PEP 613 Oct 6, 2024
@bastimeyer bastimeyer merged commit 045166b into streamlink:master Oct 6, 2024
25 checks passed
@bastimeyer bastimeyer deleted the typing/PEP-563+585+604 branch October 6, 2024 09:48
@bastimeyer bastimeyer mentioned this pull request Oct 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
cleanup typing
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@bastimeyer