Skip to content

plugins.bloomberg: update/fix headers to avoid bot detection #4919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 2, 2022
Merged

plugins.bloomberg: update/fix headers to avoid bot detection #4919

merged 1 commit into from
Nov 2, 2022

Conversation

mkbloke
Copy link
Member

@mkbloke mkbloke commented Nov 2, 2022 •
edited
Loading

As far as my testing goes, all the previously added headers seem to be unnecessary. Adding the Accept-Encoding header back seems to trigger bot detection, possibly because with requests it only includes gzip and deflate, but not br, which Firefox also includes. Removing the header entirely seems to work.

$ streamlink https://www.bloomberg.com/live
[cli][info] Found matching plugin bloomberg for URL https://www.bloomberg.com/live
Available streams: 272p (worst), 360p, 720p_alt, 720p, 1080p (best)

$ streamlink https://www.bloomberg.com/news/videos/2022-10-26/rishi-sunak-and-the-uk-economy-showdown
[cli][info] Found matching plugin bloomberg for URL https://www.bloomberg.com/news/videos/2022-10-26/rishi-sunak-and-the-uk-economy-showdown
Available streams: 100k (worst), 176p, 272p, 360p, 540p, 720p_alt, 720p, 1080p (best)

closes #4918

@mkbloke mkbloke added the plugin issue A Plugin does not work correctly label Nov 2, 2022
@bastimeyer bastimeyer merged commit fc19ae5 into streamlink:master Nov 2, 2022
@mkbloke mkbloke deleted the bloomberg branch November 2, 2022 20:16
@bastimeyer
Copy link
Member

possibly because with requests it only includes gzip and deflate, but not br

Brotli does only get added by urllib3 if the brotlicffi or brotli package is installed, hence why we're installing brotli for the CI test runners:

The bot detection triggers regardless whether brotli is available or not, just checked it. No idea why removing the header works... 😕

@bastimeyer bastimeyer mentioned this pull request Nov 2, 2022
Closed
4 tasks
@mkbloke
Copy link
Member Author

mkbloke commented Nov 2, 2022

Ah, I just read your comment here after writing: #4918 (comment)

Well, that's very strange indeed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
plugin issue A Plugin does not work correctly
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

plugins.bloomberg: Always get Could not Find JSON data error from today
2 participants
@mkbloke @bastimeyer