Skip to content

plugins.vtvgo: acquire AWS WAF token #6102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 2, 2024
Merged

plugins.vtvgo: acquire AWS WAF token #6102

merged 2 commits into from
Aug 2, 2024

Conversation

bastimeyer
Copy link
Member

Resolves #6099

  1. This adds the plugin.api.aws_waf module which uses Streamlink's webbrowser API in order to get an AWS Web Application Firewall token
  2. Fixes the vtvgo plugin's bot-detection issue by acquiring an AWS WAF token

The token is cached for four days once successfully acquired. Chromium's headless mode unfortunately does not work because AWS detects it, so it's disabled (--webbrowser-headless has no effect).

$ ./script/test-plugin-urls.py vtvgo
:: https://vtvgo.vn/xem-truc-tuyen-kenh-k%C3%AAnh-vtv5-t%C3%A2y-nguy%C3%AAn-163.html
:::: Getting new AWS WAF token (bot-detection)
:::: Launching web browser: /usr/bin/chromium
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv1-1.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv2-2.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv3-3.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv4-4.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv5-5.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv5-t%C3%A2y-nam-b%E1%BB%99-7.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv6-6.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv7-27.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv8-36.html
::  480p, 720p, worst, best
:: https://vtvgo.vn/xem-truc-tuyen-kenh-vtv9-39.html
::  480p, 720p, worst, best

@HuynhNhan0104 please test and report back with a debug log, so I know it's working correctly.
https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#pull-request-feedback

@bastimeyer bastimeyer added plugin issue A Plugin does not work correctly API: plugin labels Jul 31, 2024
gravyboat
gravyboat approved these changes Aug 1, 2024
View reviewed changes
Copy link
Member

@gravyboat gravyboat left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure a single plugin is worth the extra complexity here but I guess we'll see if it causes a headache. I have not tested this, just reviewing the code portion.

@HuynhNhan0104
Copy link

Thank you so much, this is exactly what I needed. Everything working well.

@bastimeyer bastimeyer force-pushed the plugins/vtvgo/aws-waf-token branch from 389f8fe to e2c9671 Compare August 2, 2024 10:08
@bastimeyer
Copy link
Member Author

I'm not going to merge this without at least another person having actually tested this. This is not a trivial plugin fix, it's adding a Chromium web browser requirement, which has so far only been added to the Twitch plugin last year as a fallback mechanic, so it's never been used as a full requirement in a plugin yet. I explicitly asked for proof via a full debug log output, so that I know that this doesn't just work on my system and network.

@HuynhNhan0104
Copy link

It seems everything is working well in my case, there is my logs for this command line.
streamlink "https://vtvgo.vn/xem-truc-tuyen-kenh-vtv3-3.html" --loglevel=debug
[session][debug] Loading plugin: vtvgo
[cli][debug] OS: Windows 10
[cli][debug] Python: 3.11.5
[cli][debug] OpenSSL: OpenSSL 3.0.9 30 May 2023
[cli][debug] Streamlink: 6.8.3+15.g389f8fe1
[cli][debug] Dependencies:
[cli][debug] certifi: 2024.2.2
[cli][debug] isodate: 0.6.1
[cli][debug] lxml: 5.1.0
[cli][debug] pycountry: 23.12.11
[cli][debug] pycryptodome: 3.20.0
[cli][debug] PySocks: 1.7.1
[cli][debug] requests: 2.31.0
[cli][debug] trio: 0.24.0
[cli][debug] trio-websocket: 0.11.1
[cli][debug] typing-extensions: 4.9.0
[cli][debug] urllib3: 2.2.1
[cli][debug] websocket-client: 1.7.0
[cli][debug] Arguments:
[cli][debug] url=https://vtvgo.vn/xem-truc-tuyen-kenh-vtv3-3.html
[cli][debug] --loglevel=debug
[plugins.vtvgo][debug] Restored cookies: aws-waf-token
[cli][info] Found matching plugin vtvgo for URL https://vtvgo.vn/xem-truc-tuyen-kenh-vtv3-3.html
[utils.l10n][debug] Language code: en_US
Available streams: 360p (worst), 480p, 720p (best)

@bastimeyer
plugin.api: add aws_waf module
276a249 
New module for acquiring an AWS Web Application Firewall (WAF)
challenge-token via Streamlink's webbrowser API.
@bastimeyer
plugins.vtvgo: acquire AWS WAF token
57cef9e
@bastimeyer bastimeyer force-pushed the plugins/vtvgo/aws-waf-token branch from e2c9671 to 57cef9e Compare August 2, 2024 16:25
@bastimeyer bastimeyer merged commit ac9bab3 into streamlink:master Aug 2, 2024
16 checks passed
@bastimeyer bastimeyer deleted the plugins/vtvgo/aws-waf-token branch August 2, 2024 16:28
@Hakkin Hakkin mentioned this pull request Aug 4, 2024
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gravyboat gravyboat gravyboat approved these changes

Assignees
No one assigned
Labels
API: plugin plugin issue A Plugin does not work correctly
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

plugins.vtvgo: "error: Unable to validate response text: ValidationError(Callable): iselement(None) is not true"
3 participants
@bastimeyer @HuynhNhan0104 @gravyboat