FFmpeg version logging and validation #4861
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The py311 tests are failing because of issues with the logging module: The |
New utility class for executing a subprocess and asynchronously reading its stdout and stderr streams line by line while the process is running, with an optional timeout which kills the process. The output streams and exit code get passed to callback methods which can control the process execution and final output result. This is useful for validating unknown executables, e.g. when querying a version string output, as it avoids reading the entire stdout/stderr at once and also avoids waiting for the process to terminate on its own.
bastimeyer
force-pushed
the
ffmpeg-validation
branch
from
a589166 to
9539821
Compare
gravyboat
approved these changes
Oct 2, 2022
There was a problem hiding this comment.
Looks good @bastimeyer, the logic note I don't care much about but the log output typo could be a bit confusing. If that can be fixed prior to this being merged it would be great! Thanks for your hard work on this one, hopefully it will benefit users.
and log FFmpeg version output on the debug logging level
bastimeyer
force-pushed
the
ffmpeg-validation
branch
from
9539821 to
45c3f9e
Compare
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Recently, warning messages were added to the
FFMPEGMuxerif muxing streams is unavailable because FFmpeg could not be found on the user's system:#4781
There are a couple of things which can be further improved though:
This will make debugging issues easier, e.g. stream.ffmpegmux: stream with independent segments (audio+video) cannot be remuxed #4825
The user can set arbitrary executables as the
--ffmpeg-ffmpegvalue, and Streamlink will execute it blindly. Not only is this bad, but if a different executable gets set, or if theffmpegexecutable on the user's system doesn't work, no error messages will be printed.The FFmpeg version should be parsed from the version check output, so that it can be compared against a minimum version. This is currently not implemented, but could be added in the future. Related:
hls-multi from arte.tv has seconds of hanging video #4520 (comment)
Implementation
ProcessOutpututility class which can asynchronously read the subprocess's stdout/stderr streams line-by-line while the process is running, as opposed to waiting for process termination and reading the whole stream data into memory at once via the available high-level stdlib APIssubprocess.run(capture_output=True)orasyncio.create_subprocess_exec().communicate()Also add the
--ffmpeg-no-validationCLI argument for disabling the validation.Why asyncio?
This makes the implementation and testing way easier compared to running in separate threads and making everything thread-safe. Using asyncio was not possible until last year due to the support for py2. This has changed now, so it should be utilized. Adding it however requires adding two more dev-requirements, namely
pytest-asyncio- already a transitive dependency, so nothing will change heremock- for backwards compatibility ofunittest.mockon py37 (AsyncMock).mockwas already removed in the past (d04767f) after it wasn't required anymore. Just a quick mention, by adding it back, the annoying compatibility of theMock.call_argsargs/kwargs could also be solved, but that would require changing imports everywhere, and it's probably not worth it.In regards to the tests,
freezegunis used for simulating subprocess timeouts, and it might be possible that something will change in future freezegun releases in terms of the asyncio module integration. This will need to be kept in mind and checked, but most likely it'll be harmless.FFmpeg version output
As you can see here (also added as comments to the code)
the FFmpeg version string that gets checked is stable. One issue however could be forks which use a different name, but since we've just dropped
avconvin 5.0.0, I don't think this is going to be a problem. I'm also not aware of any actively used FFmpeg forks, and should there be any, users can disable the validation by setting the CLI argument mentioned above.Only the first stdout line will be checked during the validation. On success, everything else will be logged (with indentation). I wouldn't mind changing this if you don't agree with this and think that it's too much.
Potential issues
Executing FFmpeg introduces a small delay at the beginning while resolving and validating the executable. I don't think this is a problem though, even on weak/limited systems. FFmpeg has to be run anyway when muxing, so loading-wise, I don't see this as an issue. The validation timeout value has been set to a very generous 4 seconds.
On error, the infamous "Unexpected version check" error message (phrased differently) will be shown without any actual error stream outputs. Infamous, because the Twitch GUI also needs to validate Streamlink, and the same logic gets used there, which has caused problems for dozens of users in the past who had broken Streamlink installs or configs. For Streamlink, this shouldn't be an issue though, as it also tells which command it's been running for checking the FFmpeg output, and there are also more error and warning messages.
Example