Skip to content

plugins.twitch: webbrowser-based client-integrity #5414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 11, 2023
Merged

plugins.twitch: webbrowser-based client-integrity #5414

merged 2 commits into from
Jul 11, 2023

Conversation

bastimeyer
Copy link
Member

@bastimeyer bastimeyer commented Jul 5, 2023 •
edited
Loading

Resolves #5370
Also see #5380

This PR implements the client-integrity token acquirement for the Twitch access token API endpoint, based on Streamlink's newly added webbrowser API (#5380).

Not fully ready yet, hence why I'm opening this as a draft. I am primarily looking for early feedback from users.
Before I'm going to properly open this, I'll have to update the commit message body and add some tests.

Overview

  • It adds the --twitch-purge-client-identity plugin argument
  • If available, it uses the device-id and client-integrity token from the plugin cache (client-integrity tokens expire after 16 hours)
  • If no cached device-id + client-integrity token were found or if they expired, then it generates a new device-id and then imports and calls the webbrowser API
  • In order to acquire the CI token, it accesses and intercepts https://www.twitch.tv/CHANNEL with an empty HTML document, and then executes JS code that implements the CI token acquirement
  • The acquired token is then decoded and checked for the is_bad_bot key.
  • If is_bad_bot is "false", then it caches the newly generated device-id and acquired client-integrity token
  • On any error or timeout along the way, or if is_bad_bot is not "false", then it returns None and nothing will be cached. Errors get logged.
  • When getting the access token and the results were not None, then the device-id and client-integrity HTTP headers get set, which should result in a valid access token

Currently not implemented

  • Initial attempt of getting the access token without a client-integrity token (so no web browser has to be launched while Twitch has disabled the restrictions, like right now)
  • Authentication data and other API headers set by the user currently don't get passed to the webbrowser implementation
  • Tests
    I don't think it's worth testing the CI token acquirement and mocking all the API calls. I've updated the important tests though.

Notes

  • It doesn't check whether the chosen channel is currently live, so it'll always acquire a client-integrity token if it's not cached. The live check is done by getting an access token. We'd have to check the live status via the GQL API, but that is delayed and thus not ideal (see the empty stream metadata issues that have popped up over the past months)
  • I have no idea about VODs and clips. The CI token currently only gets acquired for live channels, which might be incorrect

And as a reminder:
a Chromium-based web browser is required in order for the webbrowser API to work. See the available options:
https://streamlink.github.io/latest/cli.html#web-browser-options

How to test

Install Streamlink from the branch of this PR (and re-install once changes are pushed).
Be aware that there are breaking changes on this and Streamlink's master branch for the upcoming 6.0.0 release.

Guide:
https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#pull-request-feedback

Via pip:

$ python -m pip install git+https://github.com/bastimeyer/streamlink.git@plugins/twitch/client-integrity

Example

Note: --twitch-force-client-integrity is added temporarily and needs to be added to get the CI token. See my comment below why.

$ streamlink --twitch-force-client-integrity --twitch-purge-client-integrity twitch.tv/CHANNEL
[cli][info] Found matching plugin twitch for URL twitch.tv/CHANNEL
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/chromium
[plugins.twitch][info] Is bad bot? False
Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 720p60, 1080p60 (best)

edit:
--twitch-force-client-integrity removed again

@bastimeyer bastimeyer added plugin issue A Plugin does not work correctly API: webbrowser labels Jul 5, 2023
This was referenced Jul 5, 2023
Closed
Closed
@bastimeyer bastimeyer force-pushed the plugins/twitch/client-integrity branch 2 times, most recently from 7fe3c52 to d8ac534 Compare July 7, 2023 13:01
@bastimeyer
Copy link
Member Author

Auth data now gets passed to the web browser.

I've also changed the logic so that it tries to get an access token without a client-integrity token first. The web player did the same when the CI token was required.

In order for anyone to test the webbrowser stuff, I've temporarily added the --twitch-force-client-integrity plugin argument which disables that behavior, so it always attempts to get a client-integrity token when it's not already cached. Before this PR gets merged, I will remove that plugin argument again.

Please test this and give feedback. Nobody has bothered so far, which is disapponting. The Twitch issue thread exploded after a couple of minutes when the plugin broke, and now with an implementation available which fixes the issue (unless Twitch makes further changes), nobody even cares.

@Hakkin
Copy link
Contributor

Hakkin commented Jul 7, 2023 •
edited
Loading

Tested on a headless Debian server, works well with chromium.

[cli][debug] OS:         Linux-6.1.0-10-amd64-x86_64-with-glibc2.36
[cli][debug] Python:     3.11.2
[cli][debug] Streamlink: 5.1.1+306.gd8ac5346
[cli][debug] Dependencies:
[cli][debug]  certifi: 2023.5.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.18.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.31.0
[cli][debug]  trio: 0.22.1
[cli][debug]  trio-websocket: 0.10.3
[cli][debug]  typing-extensions: 4.7.1
[cli][debug]  urllib3: 2.0.3
[cli][debug]  websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug]  url=twitch.tv/twitchmedia4
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --output=/dev/null
[cli][debug]  --twitch-purge-client-integrity=True
[cli][debug]  --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL twitch.tv/twitchmedia4
[plugins.twitch][debug] Getting live HLS streams for twitchmedia4
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/chromium
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.twitch][debug] Client-Integrity token: {"client_id":"kimne78kx3ncx6brgo4mv6wki5h1ko","client_ip":"","device_id":"nXqBvL00ceFYED3eoHyLuoK6iP0goKYZ","exp":"2023-07-08T05:54:40Z","iat":"2023-07-07T13:54:40Z","is_bad_bot":"false","iss":"Twitch Client Integrity","nbf":"2023-07-07T13:54:40Z","user_id":""}
[plugins.twitch][info] Is bad bot? False

Somewhat obviously fails when you disable headless (no DE installed)

[cli][debug] OS:         Linux-6.1.0-10-amd64-x86_64-with-glibc2.36
[cli][debug] Python:     3.11.2
[cli][debug] Streamlink: 5.1.1+306.gd8ac5346
[cli][debug] Dependencies:
[cli][debug]  certifi: 2023.5.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.18.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.31.0
[cli][debug]  trio: 0.22.1
[cli][debug]  trio-websocket: 0.10.3
[cli][debug]  typing-extensions: 4.7.1
[cli][debug]  urllib3: 2.0.3
[cli][debug]  websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug]  url=twitch.tv/twitchmedia4
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --output=/dev/null
[cli][debug]  --webbrowser-headless=False
[cli][debug]  --twitch-purge-client-integrity=True
[cli][debug]  --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL twitch.tv/twitchmedia4
[plugins.twitch][debug] Getting live HLS streams for twitchmedia4
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/chromium
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
error: Unable to open URL: http://127.0.0.1:54657/json/version (HTTPConnectionPool(host='127.0.0.1', port=54657): Max retries exceeded with url: /json/version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f996d0cac10>: Failed to establish a new connection: [Errno 111] Connection refused')))

Wonder if it's worth adding some kind of extra error reporting if the browser acts up somehow? Chromium throws Missing X server or $DISPLAY in this case.

@sw2719
Copy link

sw2719 commented Jul 7, 2023

Works on a Raspberry Pi 4 running Ubuntu 22.04 LTS Server using snap chromium.

[cli][debug] OS:         Linux-5.15.0-1033-raspi-aarch64-with-glibc2.35
[cli][debug] Python:     3.10.6
[cli][debug] Streamlink: 5.1.1+306.gd8ac5346
[cli][debug] Dependencies:
[cli][debug]  certifi: 2023.5.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.18.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.31.0
[cli][debug]  trio: 0.22.1
[cli][debug]  trio-websocket: 0.10.3
[cli][debug]  typing-extensions: 4.7.1
[cli][debug]  urllib3: 2.0.3
[cli][debug]  websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug]  url=twitch.tv/[some random channel)
[cli][debug]  --loglevel=debug
[cli][debug]  --twitch-purge-client-integrity=True
[cli][debug]  --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL twitch.tv/[some random channel)
[plugins.twitch][debug] Getting live HLS streams for [some random channel)
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /snap/bin/chromium
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.twitch][debug] Client-Integrity token: {"client_id":"[REDACTED]","client_ip":"[REDACTED]","device_id":"[REDACTED]","exp":"2023-07-08T06:08:24Z","iat":"2023-07-07T14:08:24Z","is_bad_bot":"false","iss":"Twitch Client Integrity","nbf":"2023-07-07T14:08:24Z","user_id":""}
[plugins.twitch][info] Is bad bot? False
[plugins.twitch][debug] {'adblock': False, 'geoblock_reason': '', 'hide_ads': False, 'server_ads': True, 'show_ads': True}
[utils.l10n][debug] Language code: en_US
Available streams: audio_only, 160p (worst), 360p, 480p, 720p60 (best)

@Hakkin
Copy link
Contributor

Hakkin commented Jul 7, 2023

The header passthrough works for auth, but it seems a bit broken if you pass in your own Client-ID or Device-ID. When you pass in a Client-ID (--twitch-api-header "Client-Id=abc123"), it will end up passing in both the default streamlink Client-ID and the one you specified, so you get {"client_id":"kime ..., abc123"} in the token. The Device-ID doesn't seem to work at all if you pass it in as a header, it just returns a blank value in the token.

@Chiitoo
Copy link

Chiitoo commented Jul 7, 2023

Is this due to something specific to Chromium or something else?

I see this with Falkon and Qutebrowser (which are using qtwebengine (6) which uses chromium):

[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/falkon
error: Unable to open URL: http://127.0.0.1:33781/json/version (HTTPConnectionPool(host='127.0.0.1', port=33781): Max retries exceeded with url: /json/version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f68e23a0d50>: Failed to establish a new connection: [Errno 111] Connection refused')))

@bastimeyer
Copy link
Member Author

extra error reporting

I decided against passing through stderr from the spawned child process, because users can choose arbitrary executables, so there's no validation at all. Passing through output streams is not always a good idea. In cases like you've described, this is a bit different though and I could see this being useful. I don't want to add another CLI arg for that though. This needs a decision.

Falkon
Qutebrowser

Chromium forks needs to have the CDP built in. NW.js for example has two different builds, one with devtools and one without. It's possible that the builds of these browsers you're using have the CDP disabled in their build flags.

It's also possible that they deliberately don't support it. I've never used these browsers, so I don't know. As explained in the other threads, I don't care if this doesn't cover every existing browser. This is an impossible task and I want to get the base thing working. Thanks for checking these browsers though.

@Chiitoo
Copy link

Chiitoo commented Jul 7, 2023

Yeah, saw the comment on various browsers and definitely understand not wanting (or being able to) support everything there is.

I compile the browsers (and qtwebengine) myself, and am in touch with their developers, so I can certainly look into it deeper on that side. That hint on CDP should definitely help!

Thank you!

@bastimeyer bastimeyer force-pushed the plugins/twitch/client-integrity branch from d8ac534 to 1934225 Compare July 7, 2023 17:57
@bastimeyer
Copy link
Member Author

ImportError: cannot import name 'Self' from 'typing_extensions'
[cli][debug] typing-extensions: 3.7.4.3

Out of date. You'll have to upgrade to >=4.0.0.
I'll add a min-version requirement later.

@bastimeyer bastimeyer mentioned this pull request Jul 7, 2023
Merged
@bastimeyer bastimeyer force-pushed the plugins/twitch/client-integrity branch from 1934225 to 8300a79 Compare July 7, 2023 23:05
@hauk92
Copy link

hauk92 commented Jul 8, 2023

Works on Windows as well.

Default Edge

streamlink --loglevel=debug --twitch-force-client-integrity --twitch-purge-client-integrity https://www.twitch.tv/twitch
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.11.1
[cli][debug] Streamlink: 5.1.1+312.g8300a795
[cli][debug] Dependencies:
[cli][debug]  certifi: 2022.12.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.18.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.28.2
[cli][debug]  trio: 0.22.1
[cli][debug]  trio-websocket: 0.10.3
[cli][debug]  typing-extensions: 4.7.1
[cli][debug]  urllib3: 1.26.14
[cli][debug]  websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug]  url=https://www.twitch.tv/twitch
[cli][debug]  --loglevel=debug
[cli][debug]  --default-stream=['best']
[cli][debug]  --ffmpeg-ffmpeg=C:\Program Files\Streamlink\ffmpeg\ffmpeg.exe
[cli][debug]  --twitch-purge-client-integrity=True
[cli][debug]  --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL https://www.twitch.tv/twitch
[plugins.twitch][debug] Getting live HLS streams for twitch
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.twitch][debug] Client-Integrity token: {"client_id":"x","client_ip":"x","device_id":"x","exp":"2023-07-09T02:18:25Z","iat":"2023-07-08T10:18:25Z","is_bad_bot":"false","iss":"Twitch Client Integrity","nbf":"2023-07-08T10:18:25Z","user_id":""}
[plugins.twitch][info] Is bad bot? False
[plugins.twitch][debug] {'adblock': False, 'geoblock_reason': '', 'hide_ads': False, 'server_ads': True, 'show_ads': True}
[utils.l10n][debug] Language code: de_DE
[cli][info] Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 1080p (best)
[cli][info] Opening stream: 1080p (hls)

Caching

streamlink --loglevel=debug --twitch-force-client-integrity https://www.twitch.tv/twitch
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.11.1
[cli][debug] Streamlink: 5.1.1+312.g8300a795
[cli][debug] Dependencies:
[cli][debug]  certifi: 2022.12.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.18.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.28.2
[cli][debug]  trio: 0.22.1
[cli][debug]  trio-websocket: 0.10.3
[cli][debug]  typing-extensions: 4.7.1
[cli][debug]  urllib3: 1.26.14
[cli][debug]  websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug]  url=https://www.twitch.tv/twitch
[cli][debug]  --loglevel=debug
[cli][debug]  --default-stream=['best']
[cli][debug]  --ffmpeg-ffmpeg=C:\Program Files\Streamlink\ffmpeg\ffmpeg.exe
[cli][debug]  --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL https://www.twitch.tv/twitch
[plugins.twitch][debug] Getting live HLS streams for twitch
[plugins.twitch][info] Using cached client-integrity token
[plugins.twitch][debug] {'adblock': False, 'geoblock_reason': '', 'hide_ads': False, 'server_ads': True, 'show_ads': True}
[utils.l10n][debug] Language code: de_DE
[cli][info] Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 1080p (best)
[cli][info] Opening stream: 1080p (hls)

Chrome

streamlink --loglevel=debug --twitch-force-client-integrity --twitch-purge-client-integrity --webbrowser-executable "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" https://www.twitch.tv/twitch
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.11.1
[cli][debug] Streamlink: 5.1.1+312.g8300a795
[cli][debug] Dependencies:
[cli][debug]  certifi: 2022.12.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.18.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.28.2
[cli][debug]  trio: 0.22.1
[cli][debug]  trio-websocket: 0.10.3
[cli][debug]  typing-extensions: 4.7.1
[cli][debug]  urllib3: 1.26.14
[cli][debug]  websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug]  url=https://www.twitch.tv/twitch
[cli][debug]  --loglevel=debug
[cli][debug]  --default-stream=['best']
[cli][debug]  --ffmpeg-ffmpeg=C:\Program Files\Streamlink\ffmpeg\ffmpeg.exe
[cli][debug]  --webbrowser-executable=C:\Program Files (x86)\Google\Chrome\Application\chrome.exe
[cli][debug]  --twitch-purge-client-integrity=True
[cli][debug]  --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL https://www.twitch.tv/twitch
[plugins.twitch][debug] Getting live HLS streams for twitch
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: C:\Program Files (x86)\Google\Chrome\Application\chrome.exe
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.twitch][debug] Client-Integrity token: {"client_id":"x","client_ip":"x","device_id":"x","exp":"2023-07-09T02:27:19Z","iat":"2023-07-08T10:27:19Z","is_bad_bot":"false","iss":"Twitch Client Integrity","nbf":"2023-07-08T10:27:19Z","user_id":""}
[plugins.twitch][info] Is bad bot? False
[plugins.twitch][debug] {'adblock': False, 'geoblock_reason': '', 'hide_ads': False, 'server_ads': True, 'show_ads': True}
[utils.l10n][debug] Language code: de_DE
[cli][info] Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 1080p (best)
[cli][info] Opening stream: 1080p (hls)

@hanacchi
Copy link

hanacchi commented Jul 8, 2023

Thank you for the hard work!
It's working without any issues for me on a headless raspberry pi 4 with chromium.

log 
(.venv) pi@pi4:~/streamlink-test $ chromium --version
Chromium 114.0.5735.198 built on Debian 11.7, running on Debian 11.7
(.venv) pi@pi4:~/streamlink-test $ streamlink --loglevel debug --twitch-force-client-integrity --twitch-purge-client-integrity https://twitch.tv/[redacted]
[cli][debug] OS:         Linux-6.1.21-v8+-aarch64-with-glibc2.31
[cli][debug] Python:     3.9.2
[cli][debug] Streamlink: 5.1.1+312.g8300a795
[cli][debug] Dependencies:
[cli][debug]  certifi: 2023.5.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.18.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.31.0
[cli][debug]  trio: 0.22.1
[cli][debug]  trio-websocket: 0.10.3
[cli][debug]  typing-extensions: 4.7.1
[cli][debug]  urllib3: 2.0.3
[cli][debug]  websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug]  url=https://twitch.tv/[redacted]
[cli][debug]  --loglevel=debug
[cli][debug]  --twitch-purge-client-integrity=True
[cli][debug]  --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL https://twitch.tv/[redacted]
[plugins.twitch][debug] Getting live HLS streams for [redacted]
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/chromium
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.twitch][debug] Client-Integrity token: {"client_id":"[redacted]","client_ip":"[redacted]","device_id":"[redacted]","exp":"2023-07-09T04:15:00Z","iat":"2023-07-08T12:15:00Z","is_bad_bot":"false","iss":"Twitch Client Integrity","nbf":"2023-07-08T12:15:00Z","user_id":""}
[plugins.twitch][info] Is bad bot? False
[plugins.twitch][debug] {'adblock': False, 'geoblock_reason': '', 'hide_ads': False, 'server_ads': True, 'show_ads': True}
[utils.l10n][debug] Language code: en_GB
Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 720p60, 1080p60 (best)
(.venv) pi@pi4:~/streamlink-test $ streamlink --twitch-force-client-integrity https://twitch.tv/[redacted]
[cli][info] Found matching plugin twitch for URL https://twitch.tv/[redacted]
[plugins.twitch][info] Using cached client-integrity token
Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 720p60, 1080p60 (best)

Any idea if the caching will work fine when running multiple instances at the same time?

@glubsy
Copy link

glubsy commented Jul 8, 2023

Works on Debian 11 (headless)

> streamlink --loglevel=debug --twitch-force-client-integrity --twitch-purge-client-integrity --webbrowser-executable "/usr/bin/chromium" https://www.twitch.tv/nacho_dayo          
[cli][debug] OS:         Linux-5.10.0-22-amd64-x86_64-with-glibc2.31
[cli][debug] Python:     3.9.2
[cli][debug] Streamlink: 5.1.1+312.g8300a795
[cli][debug] Dependencies:
[cli][debug]  certifi: 2023.5.7
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 4.9.3
[cli][debug]  pycountry: 22.3.5
[cli][debug]  pycryptodome: 3.18.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.31.0
[cli][debug]  trio: 0.22.1
[cli][debug]  trio-websocket: 0.10.3
[cli][debug]  typing-extensions: 4.7.1
[cli][debug]  urllib3: 2.0.3
[cli][debug]  websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug]  url=https://www.twitch.tv/nacho_dayo
[cli][debug]  --loglevel=debug
[cli][debug]  --webbrowser-executable=/usr/bin/chromium
[cli][debug]  --twitch-purge-client-integrity=True
[cli][debug]  --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL https://www.twitch.tv/nacho_dayo
[plugins.twitch][debug] Getting live HLS streams for nacho_dayo
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/chromium
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.twitch][debug] Client-Integrity token: {"client_id":"[redacted]","client_ip":"[redacted]","device_id":"[redacted]","exp":"2023-07-09T08:30:24Z","iat":"2023-07-08T16:30:24Z","is_bad_bot":"false","iss":"Twitch Client Integrity","nbf":"2023-07-08T16:30:24Z","user_id":""}
[plugins.twitch][info] Is bad bot? False
[plugins.twitch][debug] {'adblock': False, 'geoblock_reason': '', 'hide_ads': False, 'server_ads': True, 'show_ads': True}
[utils.l10n][debug] Language code: en_US
Available streams: audio_only, 160p (worst), 360p, 480p, 720p60, 1080p60 (best)

@ckorn
Copy link

ckorn commented Jul 9, 2023 •
edited
Loading

Does not work here with vivaldi on EndeavourOS:

$ streamlink --loglevel=debug --twitch-force-client-integrity --twitch-purge-client-integrity --webbrowser-executable "/usr/bin/vivaldi-stable" https://www.twitch.tv/the_whitey
[cli][debug] OS: Linux-6.4.2-arch1-1-x86_64-with-glibc2.37
[cli][debug] Python: 3.11.3
[cli][debug] Streamlink: 5.1.1+312.g8300a795
[cli][debug] Dependencies:
[cli][debug] certifi: 2023.5.7
[cli][debug] isodate: 0.6.1
[cli][debug] lxml: 4.9.3
[cli][debug] pycountry: 22.3.5
[cli][debug] pycryptodome: 3.18.0
[cli][debug] PySocks: 1.7.1
[cli][debug] requests: 2.31.0
[cli][debug] trio: 0.22.1
[cli][debug] trio-websocket: 0.10.3
[cli][debug] typing-extensions: 4.7.1
[cli][debug] urllib3: 2.0.3
[cli][debug] websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug] url=https://www.twitch.tv/the_whitey
[cli][debug] --loglevel=debug
[cli][debug] --webbrowser-executable=/usr/bin/vivaldi-stable
[cli][debug] --twitch-purge-client-integrity=True
[cli][debug] --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL https://www.twitch.tv/the_whitey
[plugins.twitch][debug] Getting live HLS streams for the_whitey
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/vivaldi-stable
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.twitch][error] CDPError: Sending CDP message and receiving its response timed out
[plugins.twitch][warning] No client-integrity token acquired
[plugins.twitch][debug] {'adblock': False, 'geoblock_reason': '', 'hide_ads': False, 'server_ads': True, 'show_ads': True}
[utils.l10n][debug] Language code: de_DE
Available streams: audio_only, 160p (worst), 360p, 480p, 720p, 1080p (best)

But maybe I did something wrong? Tried to follow the instructions.

@bastimeyer
Copy link
Member Author

bastimeyer commented Jul 9, 2023 •
edited
Loading

vivaldi

Looks like Vivaldi doesn't return a response when sending the Page.enable() CDP command.
https://chromedevtools.github.io/devtools-protocol/tot/Page/#method-enable

No idea. It's a proprietary browser and it's possible that it's a bug on their end or they simply don't support it. It worked in my Windows VM though, just their headless implementation didn't work properly.
#5380 (comment)

Surprisingly, the Fetch.enable(...) command worked.
https://chromedevtools.github.io/devtools-protocol/tot/Fetch/#method-enable

$ streamlink twitch.tv/CHANNEL -l all --twitch-force-client-integrity --twitch-purge-client-integrity --webbrowser-executable /usr/bin/vivaldi-stable
...
[12:44:13.720761][webbrowser.webbrowser][info] Launching web browser: /usr/bin/vivaldi-stable
[12:44:13.974808][webbrowser.cdp.connection][all] Sending message: {"id":0,"method":"Target.createTarget","params":{"url":""}}
[12:44:13.978437][webbrowser.cdp.connection][all] Received message: {"id":0,"result":{"targetId":"B2B3005323C5B8C397DBD34FC3025804"}}
[12:44:13.978665][webbrowser.cdp.connection][all] Sending message: {"id":1,"method":"Target.attachToTarget","params":{"flatten":true,"targetId":"B2B3005323C5B8C397DBD34FC3025804"}}
[12:44:13.979141][webbrowser.cdp.connection][all] Received message: {"method":"Target.attachedToTarget","params":{"sessionId":"8FF6EC14C479AB9A49C8B5202877904B","targetInfo":{"targetId":"B2B3005323C5B8C397DBD34FC3025804","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"623D634E940300092945AE5CDECF2B0A"},"waitingForDebugger":false}}
[12:44:13.979223][webbrowser.cdp.connection][all] Received event: AttachedToTarget(session_id=SessionID('8FF6EC14C479AB9A49C8B5202877904B'), target_info=TargetInfo(target_id=TargetID('B2B3005323C5B8C397DBD34FC3025804'), type_='page', title='', url='', attached=True, can_access_opener=False, opener_id=None, opener_frame_id=None, browser_context_id=BrowserContextID('623D634E940300092945AE5CDECF2B0A'), subtype=None), waiting_for_debugger=False)
[12:44:13.979354][webbrowser.cdp.connection][all] Received message: {"id":1,"result":{"sessionId":"8FF6EC14C479AB9A49C8B5202877904B"}}
[12:44:13.979694][webbrowser.cdp.connection][all] Sending message: {"id":0,"method":"Fetch.enable","params":{"handleAuthRequests":true,"patterns":[{"requestStage":"Request","urlPattern":"https://www.twitch.tv/CHANNEL"}]},"sessionId":"8FF6EC14C479AB9A49C8B5202877904B"}
[12:44:13.980059][webbrowser.cdp.connection][all] Received message: {"id":0,"result":{},"sessionId":"8FF6EC14C479AB9A49C8B5202877904B"}
[12:44:13.980185][webbrowser.cdp.connection][all] Sending message: {"id":1,"method":"Page.enable","sessionId":"8FF6EC14C479AB9A49C8B5202877904B"}
[12:44:15.984049][webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[12:44:16.012908][plugins.twitch][error] CDPError: Sending CDP message and receiving its response timed out
...

@ckorn
Copy link

ckorn commented Jul 9, 2023 •
edited
Loading

It works however with brave on EndeavourOS:

$ streamlink --loglevel=debug --twitch-force-client-integrity --twitch-purge-client-integrity --webbrowser-executable "/usr/bin/brave" https://www.twitch.tv/kiaraalexiel[cli][debug] OS: Linux-6.4.1-arch2-1-x86_64-with-glibc2.37
[cli][debug] Python: 3.11.3
[cli][debug] Streamlink: 5.1.1+312.g8300a795
[cli][debug] Dependencies:
[cli][debug] certifi: 2023.5.7
[cli][debug] isodate: 0.6.1
[cli][debug] lxml: 4.9.3
[cli][debug] pycountry: 22.3.5
[cli][debug] pycryptodome: 3.18.0
[cli][debug] PySocks: 1.7.1
[cli][debug] requests: 2.31.0
[cli][debug] trio: 0.22.1
[cli][debug] trio-websocket: 0.10.3
[cli][debug] typing-extensions: 4.7.1
[cli][debug] urllib3: 2.0.3
[cli][debug] websocket-client: 1.6.1
[cli][debug] Arguments:
[cli][debug] url=https://www.twitch.tv/kiaraalexiel
[cli][debug] --loglevel=debug
[cli][debug] --webbrowser-executable=/usr/bin/brave
[cli][debug] --twitch-purge-client-integrity=True
[cli][debug] --twitch-force-client-integrity=True
[cli][info] Found matching plugin twitch for URL https://www.twitch.tv/kiaraalexiel
[plugins.twitch][debug] Getting live HLS streams for kiaraalexiel
[plugins.twitch][info] Removing cached client-integrity token...
[plugins.twitch][info] Acquiring new client-integrity token...
[webbrowser.webbrowser][info] Launching web browser: /usr/bin/brave
[webbrowser.webbrowser][debug] Waiting for web browser process to terminate
[plugins.twitch][debug] Client-Integrity token: {"client_id":"x","client_ip":"x","device_id":"x","exp":"2023-07-10T03:55:02Z","iat":"2023-07-09T11:55:02Z","is_bad_bot":"false","iss":"Twitch Client Integrity","nbf":"2023-07-09T11:55:02Z","user_id":""}
[plugins.twitch][info] Is bad bot? False
[plugins.twitch][debug] {'adblock': False, 'geoblock_reason': '', 'hide_ads': False, 'server_ads': True, 'show_ads': True}
[utils.l10n][debug] Language code: de_DE
Available streams: audio_only, 160p (worst), 360p, 480p, 720p60, 1080p60 (best)

@bastimeyer bastimeyer force-pushed the plugins/twitch/client-integrity branch from 8300a79 to c5fda93 Compare July 11, 2023 15:50
@bastimeyer bastimeyer marked this pull request as ready for review July 11, 2023 15:51
@bastimeyer
plugins.twitch: webbrowser-based client-integrity
6323d7e 
Implement client-integrity-token acquirement using the webbrowser API:

1. Generate a new device ID
2. Open www.twitch.tv/CHANNEL and intercept its request with a dummy
   response that returns an empty HTML document
3. Execute JS code which acquires the client-integrity token, similar
   to the setup of Twitch's web player
4. Parse result and check the `is_bad_bot` value
5. Cache result and device-ID on success and set the GQL API endpoint
   headers for the access token

Also:

- Add `--twitch-purge-client-integrity`
- Forward any `--twitch-api-header` values to the web browser when
  acquiring a new client-integrity token (e.g. auth data)
- Rewrite access token schema result logic:
  - Don't rely on `TypeError`s for checking whether a channel is offline
  - Propagate any `PluginError`s when parsing the API response
  - Always attempt to get an access token without a client-integrity
    token first in order to avoid having to launch the web browser
    when Twitch doesn't require a client-integrity token
- Update tests
@bastimeyer bastimeyer force-pushed the plugins/twitch/client-integrity branch from c5fda93 to 6323d7e Compare July 11, 2023 15:54
@bastimeyer
Copy link
Member Author

--twitch-force-client-integrity removed again... going to merge now.
As long as Twitch keeps the restriction disabled, there won't be any changes in behavior anyway...

@bastimeyer bastimeyer merged commit e9500df into streamlink:master Jul 11, 2023
@bastimeyer bastimeyer deleted the plugins/twitch/client-integrity branch July 11, 2023 16:01
@samfundev samfundev mentioned this pull request Jan 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
API: webbrowser plugin issue A Plugin does not work correctly
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

plugins.twitch: no playable streams on this url 2023-05-31
8 participants
@bastimeyer @Hakkin @sw2719 @Chiitoo @hauk92 @hanacchi @glubsy @ckorn