session: implement lazy plugins loading #5822
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
plugins_lazy
keyword toStreamlink
session classutils.modules
and addget_finder()
function--plugins
output from "Loaded plugins" to "Available plugins"test_session
totest_plugins
Resolves #4741
This currently includes the commits of #5793, as explained there...
Loading plugins lazily saves a couple of MiBs of memory and makes resolving plugins a bit faster. It's not that much faster than I was hoping for, but it's an improvement nonetheless.
Instead of loading all module files of Streamlink's built-in plugins into memory at once, the pre-built plugins JSON data (#5793) is loaded and
Matchers
/Arguments
objects are created from that and used in place of the actual objects from the plugin modules when resolving URLs.The plugins data loading implementation always compares the checksum of the
_plugins.json
file (stored in the packageRECORDS
metadata) before evaluating it, so users can't easily modify it. I think this is better than applying a validation schema to the JSON data and unnecessarily slowing down the session initialization.What I don't like though is that the
match_url()
debug log message when loading the resolved plugin is written to the output before the environment/package/config debug messages. This is because of the CLI implementation which wants to load all plugin config files first, which is only possible after resolving the plugin from the input URL.Sideloading plugins still works the same way as before, where sideloaded plugins override built-in ones, even when those built-in ones aren't loaded yet.
Alternatively, the lazy plugins loading could've been implemented using Python's
pickle
module, which would probably lead to better loading times, but this would come at the cost of not being able to read the plugins data, unlike JSON.Some CPU time and max-memory comparisons between 6.5.1 and this branch using GNU Time:
6.5.1
PR