Skip to content

plugins.youtube: rewrite and fix matchers #5137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 6, 2023
Merged

plugins.youtube: rewrite and fix matchers #5137

merged 1 commit into from
Feb 6, 2023

Conversation

bastimeyer
Copy link
Member

Fixes #5136

This splits up the ugly verbose matcher regex into separate matchers for each URL type, and it adds the new @-URLs and /live/video-id URLs.

Before merging, please help me validate all the supported URLs, because splitting up the matchers required some refactoring, and not all self.match[key] keys are available anymore, which can lead to an unintended IndexError being raised.

Btw, this unrelated issue with the muxed adaptive streams and missing audio bitrates needs separate fixing:

$ streamlink 'https://www.youtube.com/embed/aqz-KE-bpKQ'
[cli][info] Found matching plugin youtube for URL https://www.youtube.com/embed/aqz-KE-bpKQ
Traceback (most recent call last):
  File "/home/basti/venv/streamlink-311/bin/streamlink", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/basti/repos/streamlink/src/streamlink_cli/main.py", line 933, in main
    handle_url()
  File "/home/basti/repos/streamlink/src/streamlink_cli/main.py", line 549, in handle_url
    streams = fetch_streams(plugin)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/home/basti/repos/streamlink/src/streamlink_cli/main.py", line 443, in fetch_streams
    return plugin.streams(stream_types=args.stream_types,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/basti/repos/streamlink/src/streamlink/plugin/plugin.py", line 420, in streams
    ostreams = self._get_streams()
               ^^^^^^^^^^^^^^^^^^^
  File "/home/basti/repos/streamlink/src/streamlink/plugins/youtube.py", line 376, in _get_streams
    streams.update(self._create_adaptive_streams(adaptive_formats))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/basti/repos/streamlink/src/streamlink/plugins/youtube.py", line 236, in _create_adaptive_streams
    if best_audio_itag is None or self.adp_audio[itag] > self.adp_audio[best_audio_itag]:
                                  ~~~~~~~~~~~~~~^^^^^^
KeyError: 328

@bastimeyer bastimeyer added the plugin issue A Plugin does not work correctly label Feb 1, 2023
@bastimeyer
Copy link
Member Author

bastimeyer commented Feb 1, 2023 •
edited
Loading

Apparently, youtube.com/user/... doesn't seem to work anymore. This should be removed then, I guess. Can anyone confirm (different channels)?

default matcher

$ streamlink 'youtube.com/live/21X5lGlDOfg'
[cli][info] Found matching plugin youtube for URL youtube.com/live/21X5lGlDOfg
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/v/21X5lGlDOfg'
[cli][info] Found matching plugin youtube for URL youtube.com/v/21X5lGlDOfg
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/watch?v=21X5lGlDOfg'
[cli][info] Found matching plugin youtube for URL youtube.com/watch?v=21X5lGlDOfg
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

channel matcher

$ streamlink 'youtube.com/NASA'
[cli][info] Found matching plugin youtube for URL youtube.com/@NASA
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/NASA/live'
[cli][info] Found matching plugin youtube for URL youtube.com/@NASA/live
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/@NASA'
[cli][info] Found matching plugin youtube for URL youtube.com/@NASA
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/@NASA/live'
[cli][info] Found matching plugin youtube for URL youtube.com/@NASA/live
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/c/NASA'
[cli][info] Found matching plugin youtube for URL youtube.com/c/NASA
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/c/NASA/live'
[cli][info] Found matching plugin youtube for URL youtube.com/c/NASA
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/channel/UCLA_DiR1FfKNvjuUpBHmylQ'
[cli][info] Found matching plugin youtube for URL youtube.com/channel/UCLA_DiR1FfKNvjuUpBHmylQ
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/channel/UCLA_DiR1FfKNvjuUpBHmylQ/live'
[cli][info] Found matching plugin youtube for URL youtube.com/channel/UCLA_DiR1FfKNvjuUpBHmylQ
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

embed matcher

$ streamlink 'youtube.com/embed/21X5lGlDOfg'
[cli][info] Found matching plugin youtube for URL youtube.com/embed/21X5lGlDOfg
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

$ streamlink 'youtube.com/embed/live_stream?channel=UCLA_DiR1FfKNvjuUpBHmylQ'
[cli][info] Found matching plugin youtube for URL youtube.com/embed/live_stream?channel=UCLA_DiR1FfKNvjuUpBHmylQ
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

shorthand matcher

$ streamlink 'youtu.be/21X5lGlDOfg'
[cli][info] Found matching plugin youtube for URL youtu.be/21X5lGlDOfg
Available streams: 144p (worst), 240p, 360p, 480p, 720p (best)

@bastimeyer bastimeyer mentioned this pull request Feb 1, 2023
Closed
4 tasks
@bastimeyer bastimeyer force-pushed the plugins/youtube/rewrite-and-fix-matchers branch from 6293401 to 1cfc047 Compare February 2, 2023 08:16
@bastimeyer bastimeyer mentioned this pull request Feb 2, 2023
Merged
@bastimeyer bastimeyer force-pushed the plugins/youtube/rewrite-and-fix-matchers branch from 1cfc047 to a3fe96f Compare February 2, 2023 12:36
@gravyboat
Copy link
Member

Do you want people to test this prior to merge?

@bastimeyer
Copy link
Member Author

If you take a look at the diff of the force-pushes, you can see that I already removed the /user/... URLs.

@gravyboat
Copy link
Member

I was talking about this portion:

Before merging, please help me validate all the supported URLs, because splitting up the matchers required some refactoring, and not all self.match[key] keys are available anymore, which can lead to an unintended IndexError being raised.

since you asked the reporter to take a look in #5136 (comment)

@bastimeyer
Copy link
Member Author

Changes should be fine, I guess. If there are no reviews, then so be it, and if there are any issues after merging, then those will need to get fixed afterwards.

@gravyboat
Copy link
Member

Let's give the original reporter one more day to test then consider this good. I tried a few URLs last night for the various matchers and they were fine.

@bastimeyer bastimeyer merged commit 9fa2d82 into streamlink:master Feb 6, 2023
@bastimeyer bastimeyer deleted the plugins/youtube/rewrite-and-fix-matchers branch February 6, 2023 21:41
@vstavrinov
Copy link
Contributor

@bastimeyer :

Apparently, youtube.com/user/... doesn't seem to work anymore. This should be removed then, I guess. Can anyone confirm (different channels)?

These are still working:

https://www.youtube.com/user/Bloomberg
https://www.youtube.com/user/FRANCE24English
https://www.youtube.com/user/channelnewsasia
https://www.youtube.com/user/aljazeeraenglish
https://www.youtube.com/user/twit

This list could be continued. Some /user/ and @ endpoints with the same id point to the same channel while others to different. Some where I saw redirection. What endpoints works depends on channel. So the migration could take a long time if not endless

@bastimeyer
Copy link
Member Author

The @-URLs are not mapped to /user/-URLs, because @-URLs had a first-come-first-serve registration phase. I was under the impression that /user/-URLs had been completely removed, because the channels I had tested were not working anymore, but apparently that's not the case for all channels. This is typical YouTube inconsistency. A revert of the removal is trivial though because it only affects the matcher regex and nothing else. I'll open a PR in a bit.

@bastimeyer bastimeyer mentioned this pull request Feb 8, 2023
Merged
@jenlunds
Copy link

I was under the impression that /user/-URLs had been completely removed, because the channels I had tested were not working anymore, but apparently that's not the case for all channels. This is typical YouTube inconsistency.

How did you decide the correct /user/-URL for the channels you tested? The NASA one for example is not /user/nasa but /user/NASAtelevision. Can you share the URLs of the other channels that seemingly didn't work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
plugin issue A Plugin does not work correctly
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

plugins.youtube: New URL formats are unsupported
4 participants
@bastimeyer @gravyboat @vstavrinov @jenlunds