Skip to content

plugin.api.validate: ListSchema, re.Pattern and NoneOrAllSchema #4691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

bastimeyer
Copy link
Member

This adds new schema validation methods that are useful for commonly used patterns.

ListSchema

The list instance validation schema (eg. [int, str]) only validates a subset of the input, which means it defines any items that can match. This is often not what plugin authors have in mind when trying to validate a list where each indexed item needs to be validated.

The ListSchema (via validate.list(...)) on the other hand therefore checks for equal length of the input list and then validates each item according to the schema.

re.Pattern

Instead of having to define the validate.transform(re.compile(...).search) validation pattern, a simple re.compile(...) is now supported, which will call the search() method internally and return either None or an instance of re.Match. Other methods like findall() for example can still be called via validate.transform().

NoneOrAllSchema

This avoids having to repeatedly define the validate.any(None, validate.all(...)) validation pattern via the newly added validate.none_or_all(...) schema. This is especially useful for re.Pattern or validate.xml_xpath_string(...) validations.

I am not really happy with the naming of this though. Any better suggestions than none_or_all?

@bastimeyer
Copy link
Member Author

bastimeyer commented Jul 31, 2022

The linting failure is caused by a new major version of flake8 with unrelated linting errors. Will take a look at this later and then rebase here once fixed.

In contrast to the `list` instance validation where sequence subsets
get validated, the `ListSchema` validates that the input list has the
same length and that all list items validate successfully according to
the defined schemas.
Use the `search()` method and return `None` or a `re.Match` instance.

This avoids having to explicitly define the validation pattern
```py
validate.transform(re.compile(...).search)
```
@bastimeyer bastimeyer force-pushed the plugin/api/validate/new-schemas branch from de6b0ce to 4a4e2e8 Compare July 31, 2022 16:46
This helps avoiding the validation pattern
```py
validate.any(None, validate.all(...))
```

and is useful for `re.Pattern` and `xml_xpath_string` validations,
where the validation result can be `None` and is usually used for
gracefully returning no validation results instead of raising an error.
@bastimeyer bastimeyer force-pushed the plugin/api/validate/new-schemas branch from 4a4e2e8 to 9ddfa2f Compare July 31, 2022 17:29
@gravyboat
Copy link
Member

gravyboat commented Jul 31, 2022

These changes look really good @bastimeyer, lowering the complexity is always appreciated. Thank you!

I am not really happy with the naming of this though. Any better suggestions than none_or_all?

Based on what the code does and the description you've added doc wise, what about changing AllSchema to AllValidSchema, and then NoneOrAllSchema to NoneOrAllValidSchema? I don't know, these don't seem like major improvements over what you have, but they do clarify a bit more without having to look at the code/docs to see what is actually happening. The only other thing I can think of is for NoneOrAllSchema it could potentially be NoneIsSkippedAllValidSchema but that's a mouthful.

@bastimeyer
Copy link
Member Author

I was talking about renaming the none_or_all export as part of the public interface, not any internal class names. Class names are only relevant for the ValidationError messages, and the existing ones are all fine and shouldn't be changed. Changing none_or_all obviously requires changing the NoneOrAllSchema class name as well.

I was thinking about maybe renaming it to optional_all, but that's not as specific/clear, as it could suggest that validations could optionally fail, which they can not.

all_or_nothing sounds stupid.

@gravyboat
Copy link
Member

I was thinking about maybe renaming it to optional_all, but that's not as specific/clear, as it could suggest that validations could optionally fail, which they can not.

Yeah I don't think this is is very helpful. I don't think it's worth stressing about too much, I can't think of much better than none_or_all and you're right that all_or_nothing sounds like we're in some sort of anime showdown.

@bastimeyer
Copy link
Member Author

Let's keep the validate.none_or_all(...) name. If nobody has any further comments here, then this should be good.

There's one minor concern I have with the ListSchema, but I don't think it's important. Originally, I had implemented this as a SequenceSchema, which would also take tuples, sets and frozensets as inputs when validating, but then I figured that it didn't make much sense considering that usually JSON data is being validated, so these other data structures are useless. And there was the tuple() is tuple() issue that came up when writing tests, as the schema is supposed to create new instances of the sequences, so I changed my mind.

Btw, I've already modified, fixed and rewritten 52 plugins based on these new validation changes the last couple of days. Not sure though how I will submit these changes, as there are a couple of complete rewrites that shouldn't be included in commits of basic/simple validation pattern changes.

@gravyboat
Copy link
Member

Btw, I've already modified, fixed and rewritten 52 plugins based on these new validation changes the last couple of days. Not sure though how I will submit these changes, as there are a couple of complete rewrites that shouldn't be included in commits of basic/simple validation pattern changes.

What about only breaking out the full rewrites in to separate PRs, and all the small fixes in to one PR? Seems like a hassle to have a PR for every small fix just because it's a different plugin.

@bastimeyer
Copy link
Member Author

That's what I was planning to do.

@bastimeyer
Copy link
Member Author

Any further comments? If not, then please merge, so I can submit (some of) the plugin changes I was talking about.

@gravyboat gravyboat merged commit cb56e06 into streamlink:master Aug 4, 2022
@bastimeyer bastimeyer deleted the plugin/api/validate/new-schemas branch August 4, 2022 20:08
Billy2011 added a commit to Billy2011/streamlink-27 that referenced this pull request Aug 11, 2022
Billy2011 added a commit to Billy2011/streamlink-27 that referenced this pull request Aug 11, 2022
In contrast to the `list` instance validation where sequence subsets
get validated, the `ListSchema` validates that the input list has the
same length and that all list items validate successfully according to
the defined schemas.
Billy2011 added a commit to Billy2011/streamlink-27 that referenced this pull request Aug 11, 2022
This helps avoiding the validation pattern
```py
validate.any(None, validate.all(...))
```
Billy2011 added a commit to Billy2011/streamlink-27 that referenced this pull request Aug 13, 2022
Billy2011 added a commit to Billy2011/streamlink-27 that referenced this pull request Aug 13, 2022
In contrast to the `list` instance validation where sequence subsets
get validated, the `ListSchema` validates that the input list has the
same length and that all list items validate successfully according to
the defined schemas.
Billy2011 added a commit to Billy2011/streamlink-27 that referenced this pull request Aug 13, 2022
This helps avoiding the validation pattern
```py
validate.any(None, validate.all(...))
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants