plugins.twitch: refactor plugin #3227

bastimeyer · 2020-10-06T02:27:15Z

Opening this as a draft PR for now, because it's still WIP.
As already said in #3206, I wanted to refactor the entire Twitch plugin, as it hasn't been done in years and there's a ton of cruft.

I've split this up into multiple commits for reviewing reasons, but this should be squashed if it gets merged. Commits are therefore probably not 100% optimal and also don't have a proper commit message.

I can rebase this into fewer proper commits though, if you feel like it's too many changes for a single commit if squashed.

My goals with the code-refactor are the following:

Move most from the module's global context to class attributes, where it belongs
Remove old and now unsupported stuff (also affects plugin's URL regex)
- Remove different video types
  There's only one type now
- Remove VOD playlists
  Removed in favor of "collections" in 2017, but doesn't matter, since Streamlink can output only a single stream anyway and videos in a collection have their own canonical URL.
- Remove FLV stuff
  Which year is it again?
Improve Twitch plugin class
- Split up _get_hls_streams and _access_token by stream type ("live" and "video")
  Simplifies logic and removes dead code paths
- Improve hosted channel logic
  This is a private API call which can break at any time. It's currently being used on each live stream and doesn't have a try/except block. Also currently calls _get_hls_streams multiple times instead of just resolving the hosted channel chain in a loop.
  Now it also automatically updates the metadata on channel switch.
- Refactor rerun logic and its tests
- Refactor metadata code (already done that in plugins.twitch: fix metadata API response handling #3223)
  This PR moves it to its correct position within the plugin class.
- WIP: Improve main API calls
  Validation schemas should be added and redundancies removed.
- Various minor changes
Improve TwitchAPI class
- Remove unused API calls and API versions/subdomains
  Keep the new Twitch API in mind
- Add GQL API calls
- WIP: move validation schemas here and add methods for each request type, not for each API endpoint
Add tests
At least for the stuff that doesn't require mocking every network request

src/streamlink/plugins/twitch.py

gravyboat · 2020-10-07T02:08:53Z

One minor comment regarding qualities, but in general the commits look good to me.

bastimeyer · 2020-10-08T18:43:42Z

I'm not 100% happy with the changes, but overall, I think it should be much better now and the rest can be improve later on if really needed.

As an improvement, all validation schemas should ideally be created lazily on the first access, because there are a lot of schemas now and most of them are not needed in most cases. I was also thinking about adding a new method to the validation module, unionBy, to simplify the schema definitions with validate.union((validate.get(...), validate.get(...), ...)) calls, but since it doesn't belong into this PR, I left it out.

I've also decided to remove the custom stream weights, as they should not be needed anymore. I haven't seen a channel with old quality names in ages.

Codecov still seems to be dead. What a joke. The PR check requirement needs to be removed. I think it's best to also reduce the config to a minimum and let it always pass. This didn't turn out the way I thought it would, sorry about that. 😞 😠

Regarding the failed test run on py39, this is a bit surprising. I didn't touch the TwitchHLS logic here (also seen a failed test run yesterday IIRC) and it looks like a still remaining issue with the threads of the writer thread's ThreadPoolExecutor. Or could this be because of a change in 3.9?

streamlink/src/streamlink/stream/segmented.py

Lines 107 to 109 in cb5cee6

    
           self.executor.shutdown(wait=False) 
        
           if concurrent.futures.thread._threads_queues: 
        
               concurrent.futures.thread._threads_queues.clear()

https://docs.python.org/3/whatsnew/3.9.html#concurrent-futures

bastimeyer · 2020-10-10T03:57:33Z

I've rebased the PR onto the new master HEAD with the latest HLS bugfixes.
Unless you'd like to wait until after the next release, or unless you want me to rebase this into larger commits that could then not be squashed, this PR should be good to go.

One thing to mention though: did Twitch remove rerun streams? They were labeled as "stream_type=playlist" on the API before, but most channels didn't use that feature and just set a "rerun" stream title and used the regular live stream format. That would make the --twitch-disable-reruns parameter superfluous.

https://dev.twitch.tv/docs/v5/reference/streams#get-live-streams

$ curl -sSL \
  -H "Accept: application/vnd.twitchtv.v5+json" \
  -H "Client-ID: pwkzresl8kj2rdj6g7bvxl9ys1wly3j" \
  "https://api.twitch.tv/kraken/streams?stream_type=playlist&limit=1" \
  | jq
{
  "streams": []
}

Everything else should be working fine:

live

$ streamlink -l debug --title '{author} - {category} - {title}' twitch.tv/shroud best
[cli][debug] OS:         Linux-5.9.0-rc8-1-git-x86_64-with-glibc2.2.5
[cli][debug] Python:     3.8.5
[cli][debug] Streamlink: 1.6.0+46.gac5f010
[cli][debug] Requests(2.24.0), Socks(1.7.1), Websocket(0.56.0)
[cli][info] Found matching plugin twitch for URL twitch.tv/shroud
[plugin.twitch][debug] Getting live HLS streams for shroud
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 720p60, 936p60 (best)
...
[cli.output][debug] Opening subprocess: mpv "--title=shroud - Squad - :)| Follow @shroud on socials"

channel VOD

$ streamlink -l debug --title '{author} - {category} - {title}' twitch.tv/shroud/video/765624188 best
...
[cli][info] Found matching plugin twitch for URL twitch.tv/shroud/video/765624188
[plugin.twitch][debug] Getting video HLS streams for shroud
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: audio, 160p (worst), 360p, 480p, 720p, 720p60, 936p60 (best)
...
[cli.output][debug] Opening subprocess: mpv "--title=shroud - Just Chatting - :)| Follow @shroud on socials"

global VOD

$ streamlink -l debug --title '{author} - {category} - {title}' twitch.tv/videos/765624188 best
...
[cli][info] Found matching plugin twitch for URL twitch.tv/videos/765624188
[plugin.twitch][debug] Getting video HLS streams for shroud
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: audio, 160p (worst), 360p, 480p, 720p, 720p60, 936p60 (best)
...
[cli.output][debug] Opening subprocess: mpv "--title=shroud - Just Chatting - :)| Follow @shroud on socials"

channel clip

$ streamlink -l debug --title '{author} - {category} - {title}' twitch.tv/shroud/clip/StrongSillyGazelleMoreCowbell best
...
[cli][info] Found matching plugin twitch for URL twitch.tv/shroud/clip/StrongSillyGazelleMoreCowbell
[cli][info] Available streams: 360p (worst), 480p, 720p, 936p (best)
...
[cli.output][debug] Opening subprocess: mpv "--title=shroud - Just Chatting - How to clean your keyboard by Shroud"

global clip

$ streamlink -l debug --title '{author} - {category} - {title}' clips.twitch.tv/StrongSillyGazelleMoreCowbell best
...
[cli][info] Found matching plugin twitch for URL clips.twitch.tv/StrongSillyGazelleMoreCowbell
[cli][info] Available streams: 360p (worst), 480p, 720p, 936p (best)
...
[cli.output][debug] Opening subprocess: mpv "--title=shroud - Just Chatting - How to clean your keyboard by Shroud"

player live

$ streamlink -l debug --title '{author} - {category} - {title}' 'https://player.twitch.tv/?channel=shroud' best
...
[cli][info] Found matching plugin twitch for URL https://player.twitch.tv/?channel=shroud
[plugin.twitch][debug] Getting live HLS streams for shroud
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 720p60, 936p60 (best)
...
[cli.output][debug] Opening subprocess: mpv "--title=shroud - Squad - :)| Follow @shroud on socials"

player VOD

$ streamlink -l debug --title '{author} - {category} - {title}' 'https://player.twitch.tv/?video=765624188' best
...
[cli][info] Found matching plugin twitch for URL https://player.twitch.tv/?video=765624188
[plugin.twitch][debug] Getting video HLS streams for shroud
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: audio, 160p (worst), 360p, 480p, 720p, 720p60, 936p60 (best)
...
[cli.output][debug] Opening subprocess: mpv "--title=shroud - Just Chatting - :)| Follow @shroud on socials"

hosted channel

$ streamlink -l debug twitch.tv/esl_joindotared best
...
[cli][info] Found matching plugin twitch for URL twitch.tv/esl_joindotared
[plugin.twitch][info] esl_joindotared is hosting esl_dota2
[plugin.twitch][info] switching to esl_dota2
[plugin.twitch][debug] Getting live HLS streams for esl_dota2

hosted channel - disabled

$ streamlink -l debug --twitch-disable-hosting twitch.tv/esl_joindotared best
...
[cli][info] Found matching plugin twitch for URL twitch.tv/esl_joindotared
[cli][debug] Plugin specific arguments:
[cli][debug]  --twitch-disable-hosting=True (disable_hosting)
[plugin.twitch][info] esl_joindotared is hosting esl_dota2
[plugin.twitch][info] hosting was disabled by command line option
error: No playable streams found on this URL: twitch.tv/esl_joindotared

gravyboat · 2020-10-10T04:26:06Z

did Twitch remove rerun streams?

They definitely didn't remove them from the site, but after the discussion in #3206 it's possible that maybe they just dropped them from the API?

Unless you'd like to wait until after the next release

How about rebasing a few of them, then we can do a rebase and merge so the history is preserved without being too extensive for a single plugin?

bastimeyer · 2020-10-10T04:29:43Z

so the history is preserved without being too extensive for a single plugin?

Yes, that was basically the idea. I'll clean this up a bit later.

There's only one video type supported by Twitch, namely the old "v" one, which is implicit.

- refactor TwitchAPI class - extend TwitchAPI class with GQL API calls - replace API query methods with logical methods - move all validation schemas to TwitchAPI class - add validation schemas for channel ID API calls - remove requests.prepare_request availability check

- split up _get_hls_streams into live and video - re-implement hosted channel logic and add tests - refactor rerun check

- move URL regex - remove custom stream weights - reorder methods

bastimeyer · 2020-10-10T18:48:01Z

Completely rebased it again because I decided to move the validation schemas to the TwitchAPI class and add methods for each type of API query instead of each endpoint, like I've mentioned above. This eleminates the need for lazy schema definitions and makes the overall plugin code more clean and easier to read.

This also reduces the number of commits by one and the individual commits should be much better now (the API related one is a bit big though because it moves a lot of validation schema code).

gravyboat · 2020-10-10T19:10:06Z

That works for me and this looks good. I'm going to rebase and merge.

bastimeyer added the cleanup label Oct 6, 2020

bastimeyer force-pushed the plugins/twitch/big-refactor branch from 334fc1a to ccdbbfb Compare October 6, 2020 02:33

gravyboat reviewed Oct 7, 2020

View reviewed changes

src/streamlink/plugins/twitch.py Outdated Show resolved Hide resolved

bastimeyer force-pushed the plugins/twitch/big-refactor branch from ccdbbfb to 07aeb0e Compare October 8, 2020 18:12

bastimeyer marked this pull request as ready for review October 8, 2020 18:12

bastimeyer mentioned this pull request Oct 9, 2020

tests: fix early writer close in stream_hls mixin #3237

Merged

bastimeyer added the PR: squash commits Commits need to be squashed as a single commit - eg. multiple commits for a single component label Oct 9, 2020

bastimeyer force-pushed the plugins/twitch/big-refactor branch from 07aeb0e to ac5f010 Compare October 10, 2020 03:08

bastimeyer force-pushed the plugins/twitch/big-refactor branch from ac5f010 to f79f24f Compare October 10, 2020 08:26

bastimeyer added PR: merge commit Commits can be included as is on the base branch via a merge commit and removed PR: squash commits Commits need to be squashed as a single commit - eg. multiple commits for a single component labels Oct 10, 2020

bastimeyer added 4 commits October 10, 2020 20:28

plugins.twitch: remove video-type distinction

d16d77f

There's only one video type supported by Twitch, namely the old "v" one, which is implicit.

plugins.twitch: refactor _get_hls_streams

f4d444c

- split up _get_hls_streams into live and video - re-implement hosted channel logic and add tests - refactor rerun check

plugins.twitch: remove stream weights and clean up

3bb019f

- move URL regex - remove custom stream weights - reorder methods

bastimeyer force-pushed the plugins/twitch/big-refactor branch from f79f24f to 3bb019f Compare October 10, 2020 18:39

gravyboat merged commit 0a79e5b into streamlink:master Oct 10, 2020

bastimeyer deleted the plugins/twitch/big-refactor branch October 13, 2020 09:22

bastimeyer mentioned this pull request Nov 18, 2020

Twitch 404 Client Error (2020-11-18) #3343

Closed

1 task

bastimeyer mentioned this pull request May 18, 2021

plugins.crunchyroll: plugin fails with Crunchyroll Beta #3674

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

plugins.twitch: refactor plugin #3227

plugins.twitch: refactor plugin #3227

Uh oh!

bastimeyer commented Oct 6, 2020

Uh oh!

Uh oh!

gravyboat commented Oct 7, 2020

Uh oh!

bastimeyer commented Oct 8, 2020

Uh oh!

bastimeyer commented Oct 10, 2020

Uh oh!

gravyboat commented Oct 10, 2020

Uh oh!

bastimeyer commented Oct 10, 2020

Uh oh!

bastimeyer commented Oct 10, 2020

Uh oh!

gravyboat commented Oct 10, 2020

Uh oh!

Uh oh!

Uh oh!

plugins.twitch: refactor plugin #3227

plugins.twitch: refactor plugin #3227

Uh oh!

Conversation

bastimeyer commented Oct 6, 2020

Uh oh!

Uh oh!

gravyboat commented Oct 7, 2020

Uh oh!

bastimeyer commented Oct 8, 2020

Uh oh!

bastimeyer commented Oct 10, 2020

Uh oh!

gravyboat commented Oct 10, 2020

Uh oh!

bastimeyer commented Oct 10, 2020

Uh oh!

bastimeyer commented Oct 10, 2020

Uh oh!

gravyboat commented Oct 10, 2020

Uh oh!

Uh oh!