plugin.raiplay: add vod support #5662

g-caronte · 2023-11-10T17:14:33Z

Hi,
I've added support for on-demand videos that require authentication.
The implementation of the authentication system is quite naive, but I hope it's sufficient.

bastimeyer

Thanks for the PR.

Some fixes are required before this can be merged. Please see the annotated comments.

I've checked my suggested changes using a VPN and a temp account.

Invalid auth:

[cli][debug] Arguments:
[cli][debug]  url=https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
[cli][debug]  --loglevel=debug
[cli][debug]  --player=mpv
[cli][debug]  --raiplay-email=********
[cli][debug]  --raiplay-password=********
[cli][info] Found matching plugin raiplay for URL https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
[plugins.raiplay][debug] Found video URL: authentication is required.
[plugins.raiplay][error] Combinazione email/password errata.
error: No playable streams found on this URL: https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html

Valid auth:

[cli][debug] Arguments:
[cli][debug]  url=https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
[cli][debug]  --loglevel=debug
[cli][debug]  --player=mpv
[cli][debug]  --raiplay-email=********
[cli][debug]  --raiplay-password=********
[cli][info] Found matching plugin raiplay for URL https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
[plugins.raiplay][debug] Found video URL: authentication is required.
[plugins.raiplay][debug] Found JSON URL: https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.json
[utils.l10n][debug] Language code: en_US
[stream.ffmpegmux][debug] ffmpeg version n6.0 Copyright (c) 2000-2023 the FFmpeg developers
[stream.ffmpegmux][debug]  built with gcc 13.2.1 (GCC) 20230801
[stream.ffmpegmux][debug]  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
[stream.ffmpegmux][debug]  libavutil      58.  2.100 / 58.  2.100
[stream.ffmpegmux][debug]  libavcodec     60.  3.100 / 60.  3.100
[stream.ffmpegmux][debug]  libavformat    60.  3.100 / 60.  3.100
[stream.ffmpegmux][debug]  libavdevice    60.  1.100 / 60.  1.100
[stream.ffmpegmux][debug]  libavfilter     9.  3.100 /  9.  3.100
[stream.ffmpegmux][debug]  libswscale      7.  1.100 /  7.  1.100
[stream.ffmpegmux][debug]  libswresample   4. 10.100 /  4. 10.100
[stream.ffmpegmux][debug]  libpostproc    57.  1.100 / 57.  1.100
[stream.hls][debug] Using external audio tracks for stream 576p (language=Italiano, name=Italiano)
[stream.hls][debug] Using external audio tracks for stream 720p (language=Italiano, name=Italiano)
Available streams: 576p (worst), 720p (best)

src/streamlink/plugins/raiplay.py

g-caronte · 2023-11-11T07:50:34Z

Thanks for the suggestions.

There is something else that you think i should improve and/or test?
If so if you can think about a plugin that already solved similar issue, it would be helpful.

bastimeyer · 2023-11-11T13:54:36Z

There is something else that you think i should improve and/or test?
If so if you can think about a plugin that already solved similar issue, it would be helpful.

The auth token could be cached, so the user doesn't have to set the email and pass every time.

This is done either via Plugin.save_cookies() or via the Plugin.cache object, which implements the Cache.get(key: str, default: Any = None) and Cache.set(key: str, value: Any, expires: int = one_week_in_seconds, expires_at: datetime | None = None) methods. Since the auth data is HTTP-header based, cookies should not be used for storing auth data, because it would update the HTTPSession.cookies dict, which we don't want.

Storing auth data however also means that a purge-credentials plugin argument needs to be added.

diff --git a/src/streamlink/plugins/raiplay.py b/src/streamlink/plugins/raiplay.py
index 57e54bf5..8f2e686b 100644
--- a/src/streamlink/plugins/raiplay.py
+++ b/src/streamlink/plugins/raiplay.py
@@ -9,6 +9,7 @@ import logging
 import re
 from urllib.parse import parse_qsl, urlparse, urlunparse
 
+from streamlink.exceptions import PluginError
 from streamlink.plugin import Plugin, pluginargument, pluginmatcher
 from streamlink.plugin.api import validate
 from streamlink.stream.hls import HLSStream
@@ -43,19 +44,25 @@ log = logging.getLogger(__name__)
     metavar="PASSWORD",
     help="A raiplay.it account password to use with --raiplay-email.",
 )
+@pluginargument(
+    "purge-credentials",
+    action="store_true",
+    help="Purge cached RaiPlay credentials to initiate a new session and reauthenticate.",
+)
 class RaiPlay(Plugin):
     _DEFAULT_MEDIAPOLIS_OUTPUT = "64"
     _AUTH_URL = "https://www.raiplay.it/raisso/login/domain/app/social"
     _DOMAIN_API_KEY = "arSgRtwasD324SaA"
+    _CACHE_KEY_UA_TOKEN = "ua-token"
 
     def _get_streams(self):
+        if self.options.get("purge-credentials"):
+            log.info("Removing cached user-authentication token...")
+            self.cache.set(self._CACHE_KEY_UA_TOKEN, None, 0)
+
         if self.matches["vod"]:
             log.debug("Found video URL: authentication is required.")
-            if not self.get_option("email"):
-                log.error("RaiPlay requires a login using --raiplay-email and --raiplay-password")
-                return
-            if not self.login():
-                return
+            self.auth()
 
         json_url = self.session.http.get(
             self.url,
@@ -113,7 +120,20 @@ class RaiPlay(Plugin):
 
         yield from HLSStream.parse_variant_playlist(self.session, stream_url).items()
 
+    def auth(self):
+        ua_token = self.cache.get(self._CACHE_KEY_UA_TOKEN)
+        if not ua_token:
+            ua_token = self.login()
+        else:
+            log.info("Using cached user-authentication token")
+
+        self.session.http.headers.update({"x-ua-token": ua_token})
+        self.cache.set(self._CACHE_KEY_UA_TOKEN, ua_token)
+
     def login(self):
+        if not self.get_option("email") or not self.get_option("password"):
+            raise PluginError("RaiPlay requires a login using --raiplay-email and --raiplay-password")
+
         response, data = self.session.http.post(
             self._AUTH_URL,
             data={
@@ -146,11 +166,10 @@ class RaiPlay(Plugin):
             ),
         )
         if response != "OK":
-            log.error(data or "Authentication failure")
-            return False
+            raise PluginError(data or "Authentication failure")
 
-        self.session.http.headers.update({"x-ua-token": data})
-        return True
+        log.info("Authentication successful")
+        return data
 
 
 __plugin__ = RaiPlay

g-caronte · 2023-11-12T11:09:53Z

I took the opportunity to experiment with validate extracting the token's expiration date and using it for caching purposes.

I hope that _JWT_EXP_SCHEMA is not unnecessarily too complex.

bastimeyer

I'm not sure if adding a validation schema for the expiration time is necessary, as the expiration time seems to be 24h and could just be hardcoded. It doesn't hurt though...

Please move the schemas from the global module scope to the scope of the login method. Otherwise, they will be kept in memory and can't be cleaned up by the garbage collector. Streamlink currently holds all plugin modules in memory due to how they're being loaded by the session instance.

bastimeyer · 2023-11-12T12:36:38Z

src/streamlink/plugins/raiplay.py

 from streamlink.plugin.api import validate
 from streamlink.stream.hls import HLSStream
 from streamlink.utils.url import update_qsd


 log = logging.getLogger(__name__)

+_JWT_EXP_SCHEMA = validate.Schema(
+    re.compile(r"[\w-]+\.([\w-]+)\.[\w-]+"),


The expiration schema needs to check if the input is a str. Otherwise, any JSON object could be passed to the re.Pattern.search call.

re.Pattern validations can also return None if they don't match, which will lead to an AttributeError being raised in the following validate.get() validation. This must be avoided.

Everything after the pattern therefore must be wrapped in a validate.none_or_all schema and a None return value must be handled accordingly. Alternatively, since this is already a non-optional validation schema, use validate.regex(re.compile(...)), as this makes the regex match mandatory:
https://streamlink.github.io/latest/api/validate.html#streamlink.plugin.api.validate.regex
The regex start and end anchors should be set though, or match should be chosen instead of search as the regex pattern method.

Or, to keep it simple, instead of the regex stuff, just use this:

str, validate.transform(str.split, "."), lambda l: len(l) == 3, validate.get(1),

bastimeyer · 2023-11-12T13:10:21Z

src/streamlink/plugins/raiplay.py

+_JWT_EXP_SCHEMA = validate.Schema(
+    re.compile(r"[\w-]+\.([\w-]+)\.[\w-]+"),
+    validate.get(1),
+    validate.transform(lambda s: base64.b64decode(s + "=" * (-len(s) % 4))),


f-strings are preferred over string concatenation:
f"{s}{'=' * (-len(s) % 4)}"

bastimeyer · 2023-11-12T13:10:38Z

src/streamlink/plugins/raiplay.py

+    validate.get(1),
+    validate.transform(lambda s: base64.b64decode(s + "=" * (-len(s) % 4))),
+    validate.parse_json(),
+    validate.all({"exp": int}, validate.get("exp")),


Unnecessary validate.all wrapper. validate.Schema already is an all schema.
https://streamlink.github.io/latest/api/validate.html#streamlink.plugin.api.validate.Schema

bastimeyer · 2023-11-12T13:19:48Z

src/streamlink/plugins/raiplay.py

+    validate.transform(lambda s: base64.b64decode(s + "=" * (-len(s) % 4))),
+    validate.parse_json(),
+    validate.all({"exp": int}, validate.get("exp")),
+    validate.transform(datetime.utcfromtimestamp),


datetime.utcfromtimestamp is deprecated since Python 3.12:
https://docs.python.org/3/library/datetime.html#datetime.datetime.utcfromtimestamp

Use fromtimestamp from the streamlink.utils.times module:
https://github.com/streamlink/streamlink/blob/6.3.1/src/streamlink/utils/times.py#L19-L20

bastimeyer · 2023-11-12T13:34:58Z

src/streamlink/plugins/raiplay.py

+        ),
+        validate.all(
+            {
+                "response": "KO",


If any of the error responses is not "KO", then this won't match. Hence why I set it to str in the diff I had posted...

Co-authored-by: bastimeyer <mail@bastimeyer.de>

bastimeyer · 2023-11-13T15:43:41Z

Rebased to master, squashed commits with a fixed commit message, fixed code issues, and force-pushed to the author's PR branch. Going to merge now... Thanks for the PR!

New valid VOD user-auth:

$ streamlink --raiplay-purge-credentials --raiplay-email htmfvgrijrzrdsdsgv@cwmxc.com --raiplay-password Qwerty123! https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
[cli][info] Found matching plugin raiplay for URL https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
[plugins.raiplay][info] Removing cached user-authentication token...
Available streams: 576p (worst), 720p (best)

Invalid VOD user-auth:

$ streamlink --raiplay-purge-credentials --raiplay-email htmfvgrijrzrdsdsgv@cwmxc.com --raiplay-password wrong-pass https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
[cli][info] Found matching plugin raiplay for URL https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
[plugins.raiplay][info] Removing cached user-authentication token...
error: Combinazione email/password errata.

Test URLs (without VPN and cached user-auth token):

$ ./script/test-plugin-urls.py raiplay
:: Finding streams for URL: https://raiplay.it/dirette/rai1
:::: Geo-restricted content
!! No streams found
:: Finding streams for URL: https://raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
:::: Using cached user-authentication token
:: Found streams: 576p, 720p, worst, best
:: Finding streams for URL: https://www.raiplay.it/dirette/rai1
:::: Geo-restricted content
!! No streams found
:: Finding streams for URL: https://www.raiplay.it/dirette/rai2
:::: Geo-restricted content
!! No streams found
:: Finding streams for URL: https://www.raiplay.it/dirette/rai3
:::: Geo-restricted content
!! No streams found
:: Finding streams for URL: https://www.raiplay.it/dirette/rainews24
:: Found streams: 432p, 576p, 720p, worst, best
:: Finding streams for URL: https://www.raiplay.it/video/2023/11/Un-posto-al-sole---Puntata-del-08112023-EP6313-1377bcf9-db3f-40f7-aa05-fefeb086ec68.html
:::: Using cached user-authentication token
:: Found streams: 576p, 720p, worst, best

bastimeyer reviewed Nov 10, 2023

View reviewed changes

src/streamlink/plugins/raiplay.py Outdated Show resolved Hide resolved

src/streamlink/plugins/raiplay.py Outdated Show resolved Hide resolved

bastimeyer added restriction: account needed plugin enhancement A new feature for a working Plugin restriction: geo blocked labels Nov 10, 2023

g-caronte force-pushed the plugins/raiplay/vod branch from 01253b2 to ef0f835 Compare November 12, 2023 11:21

bastimeyer requested changes Nov 12, 2023

View reviewed changes

plugin.raiplay: add VOD and authentication support

defa02f

Co-authored-by: bastimeyer <mail@bastimeyer.de>

bastimeyer force-pushed the plugins/raiplay/vod branch from ef0f835 to defa02f Compare November 13, 2023 15:38

bastimeyer merged commit 3e47211 into streamlink:master Nov 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

plugin.raiplay: add vod support #5662

plugin.raiplay: add vod support #5662

Uh oh!

g-caronte commented Nov 10, 2023

Uh oh!

bastimeyer left a comment

Uh oh!

Uh oh!

Uh oh!

g-caronte commented Nov 11, 2023

Uh oh!

bastimeyer commented Nov 11, 2023

Uh oh!

g-caronte commented Nov 12, 2023

Uh oh!

bastimeyer left a comment

Uh oh!

bastimeyer Nov 12, 2023

Uh oh!

bastimeyer Nov 12, 2023

Uh oh!

bastimeyer Nov 12, 2023

Uh oh!

bastimeyer Nov 12, 2023

Uh oh!

bastimeyer Nov 12, 2023

Uh oh!

bastimeyer commented Nov 13, 2023

Uh oh!

Uh oh!

Uh oh!

plugin.raiplay: add vod support #5662

plugin.raiplay: add vod support #5662

Uh oh!

Conversation

g-caronte commented Nov 10, 2023

Uh oh!

bastimeyer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

g-caronte commented Nov 11, 2023

Uh oh!

bastimeyer commented Nov 11, 2023

Uh oh!

g-caronte commented Nov 12, 2023

Uh oh!

bastimeyer left a comment

Choose a reason for hiding this comment

Uh oh!

bastimeyer Nov 12, 2023

Choose a reason for hiding this comment

Uh oh!

bastimeyer Nov 12, 2023

Choose a reason for hiding this comment

Uh oh!

bastimeyer Nov 12, 2023

Choose a reason for hiding this comment

Uh oh!

bastimeyer Nov 12, 2023

Choose a reason for hiding this comment

Uh oh!

bastimeyer Nov 12, 2023

Choose a reason for hiding this comment

Uh oh!

bastimeyer commented Nov 13, 2023

Uh oh!

Uh oh!