http_session: override urllib3 percent-encoding #4003
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was
linked to
issues
Sep 10, 2021
4 tasks
Billy2011
added a commit
to Billy2011/streamlink-27
that referenced
this pull request
Sep 10, 2021
|
Thank You Bastimeyer and Back-to, The vidio plugin is working again... Good Job... For your hard working off course I will donate again.. Thank You So Much! |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
According to RFC 3986, URLs are considered equal even if they differ in the case of the hexadecimal digits of percent-encoded characters, eg.
%FF==%ff. URL normalizers should always use uppercase characters, which is whyurllib3correctly transforms those lowercase digits to uppercase ones.This is a problem though for sites which compare the request URL (or parts of it) byte for byte and return different responses depending on that. See #4002 (comment)
This PR therefore adds overrides for
urllib3which make it not transform hexadecimal digits of percent-encoded characters to uppercase. Since URLs are still considered equal, this shouldn't break anything and fix the problem mentioned above. If the original behavior is required, then request URLs can still be transformed manually. HTTP redirections could be a problem, but the risk is rather low.Streamlink currently requires
urllib3>=1.21.1,<1.27, as defined byrequests>=2.26.0here:https://github.com/psf/requests/blob/v2.26.0/setup.py#L48
There are two different overrides for
urllib3>=1.25.8andurllib3>=1.25.4,<1.25.8, as noted in the code comments, becauseurllib3==1.25.8had a minor revision of the string substitution. Versions of urllib3 prior to1.25.4unfortunately are using external libs for modifying the percent-encoding stuff and I'm not sure if the problem can be fixed there.The added tests are working fine in both version ranges when the overrides are applied and will only fail in the lowercase test fixture when no override is applied, which is expected.
Should I add a test runner for
urllib3==1.25.4so that it can test both version ranges?