session: implement lazy plugins loading #5822
Conversation
bastimeyer
force-pushed
the
session/plugins/lazy
branch
2 times, most recently
from
8c5da03 to
5e2150a
Compare
|
Reviewed this last night but had to put some thought in to your points:
I'm fine with this.
Is this really that big a deal? I know it is a bit annoying but are there cases where we'll be somehow missing debug data that is written later or that will result in more difficult debugging because of this?
Let's avoid pickle, load times are good enough and being able to read the plugin data is useful. I won't mark as approved in case there is anything else you want the check, but the changes seem fine to me. |
bastimeyer
force-pushed
the
session/plugins/lazy
branch
2 times, most recently
from
840c433 to
cc0a276
Compare
- Add `plugins_lazy` keyword to `Streamlink` session class - Load pre-built plugins JSON data including matchers and arguments data - Fall back to loading all built-in plugins if loading JSON data fails - Iterate matcher/argument data of loaded and still unloaded plugins - Load plugin modules of unloaded plugins when matching URLs are found - Refactor `utils.modules` and add `get_finder()` function - Change `--plugins` output from "Loaded plugins" to "Available plugins" - Add tests - Move matcher priority tests from `test_session` to `test_plugins`
bastimeyer
force-pushed
the
session/plugins/lazy
branch
from
cc0a276 to
5090e1d
Compare
plugins_lazykeyword toStreamlinksession classutils.modulesand addget_finder()function--pluginsoutput from "Loaded plugins" to "Available plugins"test_sessiontotest_pluginsResolves #4741
This currently includes the commits of #5793, as explained there...
Loading plugins lazily saves a couple of MiBs of memory and makes resolving plugins a bit faster. It's not that much faster than I was hoping for, but it's an improvement nonetheless.
Instead of loading all module files of Streamlink's built-in plugins into memory at once, the pre-built plugins JSON data (#5793) is loaded and
Matchers/Argumentsobjects are created from that and used in place of the actual objects from the plugin modules when resolving URLs.The plugins data loading implementation always compares the checksum of the
_plugins.jsonfile (stored in the packageRECORDSmetadata) before evaluating it, so users can't easily modify it. I think this is better than applying a validation schema to the JSON data and unnecessarily slowing down the session initialization.What I don't like though is that the
match_url()debug log message when loading the resolved plugin is written to the output before the environment/package/config debug messages. This is because of the CLI implementation which wants to load all plugin config files first, which is only possible after resolving the plugin from the input URL.Sideloading plugins still works the same way as before, where sideloaded plugins override built-in ones, even when those built-in ones aren't loaded yet.
Alternatively, the lazy plugins loading could've been implemented using Python's
picklemodule, which would probably lead to better loading times, but this would come at the cost of not being able to read the plugins data, unlike JSON.Some CPU time and max-memory comparisons between 6.5.1 and this branch using GNU Time:
6.5.1
PR