Skip to content

plugins.douyin: new plugin #6059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 3, 2024
Merged

plugins.douyin: new plugin #6059

merged 1 commit into from
Jul 3, 2024

Conversation

v2wy
Copy link
Contributor

@v2wy v2wy commented Jul 2, 2024 •
edited by mkbloke
Loading

Resolves #6062

cookies are necessary

test url : https://live.douyin.com/774321108336

python.exe -m src.streamlink_cli -j https://live.douyin.com/774321108336 --http-cookie __ac_nonce=xxx --http-cookie __ac_signature=xxx --http-cookie sessionid=xxx

{
  "plugin": "douyin",
  "metadata": {
    "id": "7386955391675960064",
    "author": "\u5927\u65d7\u672c\u4eba",
    "category": null,
    "title": "\u6296\u97f35\u5e74\u7ecf\u9a8c\u90fd\u5728\u8fd9\u4e86"
  },
  "streams": {
    "ao": {
      "type": "http",
      "method": "GET",
      "url": "http://pull-l3.douyincdn.com/third/stream-691881931476828639.flv?auth_key=1720514018-0-0-0184ef050d0c0ace125c665b29be2bb3&only_audio=1",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    },
    "md": {
      "type": "http",
      "method": "GET",
      "url": "https://pull-l3-admin.douyincdn.com/third/stream-691881931476828639_md.flv?auth_key=1720514018-0-0-0403d2638d6cc2d44470f5dbb69b1247",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    },
    "ld": {
      "type": "http",
      "method": "GET",
      "url": "http://pull-l3.douyincdn.com/third/stream-691881931476828639_ld.flv?auth_key=1720514018-0-0-f28a74c5d70f3775165067d3d035f1b9",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    },
    "sd": {
      "type": "http",
      "method": "GET",
      "url": "http://pull-l3.douyincdn.com/third/stream-691881931476828639_sd.flv?auth_key=1720514018-0-0-b76528578f6f60266d7bb7c5c74129bb",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    },
    "hd": {
      "type": "http",
      "method": "GET",
      "url": "http://pull-l3.douyincdn.com/third/stream-691881931476828639_hd.flv?auth_key=1720514018-0-0-ea229e278c01ba1c5209a15a62ac032d",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    },
    "uhd": {
      "type": "http",
      "method": "GET",
      "url": "http://pull-l3.douyincdn.com/third/stream-691881931476828639_uhd.flv?auth_key=1720514018-0-0-7f77a745f895f8f3a8a3d6538884dbde",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    },
    "origin": {
      "type": "http",
      "method": "GET",
      "url": "http://pull-l3.douyincdn.com/third/stream-691881931476828639_or4.flv?auth_key=1720514018-0-0-2aa15b22e37440546b15bd606d393703",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    },
    "worst": {
      "type": "http",
      "method": "GET",
      "url": "https://pull-l3-admin.douyincdn.com/third/stream-691881931476828639_md.flv?auth_key=1720514018-0-0-0403d2638d6cc2d44470f5dbb69b1247",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    },
    "best": {
      "type": "http",
      "method": "GET",
      "url": "http://pull-l3.douyincdn.com/third/stream-691881931476828639_or4.flv?auth_key=1720514018-0-0-2aa15b22e37440546b15bd606d393703",
      "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept-Encoding": "gzip, deflate",
        "Accept": "*/*",
        "Connection": "keep-alive",
        "Origin": "https://live.douyin.com/774321108336",
        "Referer": "https://live.douyin.com/774321108336",
        "Cookie": "xxx"
      },
      "body": null
    }
  }
}

@mkbloke mkbloke changed the title support douyin plugins.douyin: new plugin Jul 2, 2024
@mkbloke
Copy link
Member

mkbloke commented Jul 2, 2024

For info: plugins should be accompanied with a test. See: https://github.com/streamlink/streamlink/tree/master/tests/plugins

bastimeyer
bastimeyer requested changes Jul 2, 2024
View reviewed changes
Copy link
Member

@bastimeyer bastimeyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New plugins will require a plugin request issue first, with all the required details:
https://github.com/streamlink/streamlink/issues/new/choose

cookies are necessary

Merging a new plugin where custom HTTP headers like cookies are required won't happen. A plugin will have to work out of the box, and if authentication is required, then this need to be implemented using plugin arguments. From a quick test in my web browser though, there's no authentication required.

@steven7851
Copy link
Contributor

steven7851 commented Jul 2, 2024 •
edited
Loading

cookies are not necessary, here is my own use for your reference

"""
$description douyin
$url live.douyin.com
$type live
"""

import json
import logging
import re
import uuid

from streamlink.plugin import Plugin, pluginmatcher
from streamlink.plugin.api import validate
from streamlink.stream.hls import HLSStream
from streamlink.stream.http import HTTPStream


log = logging.getLogger(__name__)


@pluginmatcher(re.compile(
    r"https?://live\.douyin\.com/(?P<channel>[^/]+)",
))
class Douyin(Plugin):
    def _get_streams(self):
        channel = self.match.group("channel")
        uuid21 = uuid.uuid4().hex[:21]
        self.session.http.headers.update({"Cookie": "__ac_nonce=" + uuid21})
        res = self.session.http.get(self.url)
        try:
            data = re.findall(r'self.__pace_f.push\(\[\d,("[a-z]:.+?")\]\)</script>', res.text)[-1]
        except:
            return
        data = json.loads(data)
        m = re.search(r'(\[.+\])', data)
        data = json.loads(m.groups()[0])[-1]
        try:
            video_info = data['state']['roomStore']['roomInfo'].get('room')
        except:
            return

        try:
            if video_info['status'] != 2:
                return
        except:
            return

        try:
            video_info_h264 = data['state']['streamStore']['streamData']['H264_streamData']['stream']['origin'].get('main')
            flv_url = video_info_h264['flv']
            hls_url = video_info_h264['hls']
        except:
            flv_url = video_info['stream_url']['flv_pull_url']['FULL_HD1']
            hls_url = video_info['stream_url']['hls_pull_url_map']['FULL_HD1']

        yield "live", HTTPStream(self.session, flv_url)

        s = HLSStream.parse_variant_playlist(self.session, hls_url)
        if not s:
            yield "live", HLSStream(self.session, hls_url)
        elif len(s) == 1:
            yield "live", next(iter(s.values()))
        else:
            yield from s.items()


__plugin__ = Douyin

@v2wy
Copy link
Contributor Author

v2wy commented Jul 2, 2024

cookies are not required now

@v2wy
Copy link
Contributor Author

v2wy commented Jul 2, 2024

#6062

@v2wy
Copy link
Contributor Author

v2wy commented Jul 3, 2024

cookies are not necessary, here is my own use for your reference

"""
$description douyin
$url live.douyin.com
$type live
"""

import json
import logging
import re
import uuid

from streamlink.plugin import Plugin, pluginmatcher
from streamlink.plugin.api import validate
from streamlink.stream.hls import HLSStream
from streamlink.stream.http import HTTPStream


log = logging.getLogger(__name__)


@pluginmatcher(re.compile(
    r"https?://live\.douyin\.com/(?P<channel>[^/]+)",
))
class Douyin(Plugin):
    def _get_streams(self):
        channel = self.match.group("channel")
        uuid21 = uuid.uuid4().hex[:21]
        self.session.http.headers.update({"Cookie": "__ac_nonce=" + uuid21})
        res = self.session.http.get(self.url)
        try:
            data = re.findall(r'self.__pace_f.push\(\[\d,("[a-z]:.+?")\]\)</script>', res.text)[-1]
        except:
            return
        data = json.loads(data)
        m = re.search(r'(\[.+\])', data)
        data = json.loads(m.groups()[0])[-1]
        try:
            video_info = data['state']['roomStore']['roomInfo'].get('room')
        except:
            return

        try:
            if video_info['status'] != 2:
                return
        except:
            return

        try:
            video_info_h264 = data['state']['streamStore']['streamData']['H264_streamData']['stream']['origin'].get('main')
            flv_url = video_info_h264['flv']
            hls_url = video_info_h264['hls']
        except:
            flv_url = video_info['stream_url']['flv_pull_url']['FULL_HD1']
            hls_url = video_info['stream_url']['hls_pull_url_map']['FULL_HD1']

        yield "live", HTTPStream(self.session, flv_url)

        s = HLSStream.parse_variant_playlist(self.session, hls_url)
        if not s:
            yield "live", HLSStream(self.session, hls_url)
        elif len(s) == 1:
            yield "live", next(iter(s.values()))
        else:
            yield from s.items()


__plugin__ = Douyin

Thank you very much. I successfully ran it using your method. I have refactored my code using your method.

@bastimeyer
Copy link
Member

@v2wy, instead of reviewing and annotating changes, I decided to fix any left over code issues myself and force-push onto your PR branch. The commit author data is left intact. Hope that's okay...

  • Updated plugin description
  • Fixed linting issues
  • Fixed validation schema
    • Removed try-except block
    • Improved data extraction
    • Split into sub-schemas (channel/room data and stream data)
    • Fixed schemas for offline/invalid channels
  • Forced HTTPS stream URLs

As commented in the plugin code, HLS streams are available, but using those would cause an unnecessary delay when fetching the streams, because there are multivariant playlists for each quality which consist of only a single media playlist, so each multivariant playlist would need to be queried. That's not worth it, despite progressive HTTP streams being pretty bad for live streaming.

Live channel:

$ streamlink -l debug https://live.douyin.com/99457681876 best
[cli][debug] OS:         Linux-6.9.7-1-git-x86_64-with-glibc2.39
[cli][debug] Python:     3.12.4
[cli][debug] OpenSSL:    OpenSSL 3.3.1 4 Jun 2024
[cli][debug] Streamlink: 6.8.1+9.gd19b93d5
[cli][debug] Dependencies:
[cli][debug]  certifi: 2024.6.2
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 5.2.2
[cli][debug]  pycountry: 24.6.1
[cli][debug]  pycryptodome: 3.20.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.32.3
[cli][debug]  trio: 0.25.1
[cli][debug]  trio-websocket: 0.11.1
[cli][debug]  typing-extensions: 4.12.2
[cli][debug]  urllib3: 2.2.2
[cli][debug]  websocket-client: 1.8.0
[cli][debug] Arguments:
[cli][debug]  url=https://live.douyin.com/99457681876
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --player=/usr/bin/mpv
[cli][info] Found matching plugin douyin for URL https://live.douyin.com/99457681876
[plugins.douyin][debug] self.QUALITY_WEIGHTS={'md': 800000, 'ao': 0, 'origin': 12248000, 'sd': 2000000, 'ld': 1000000, 'uhd': 6000000, 'hd': 4000000}
[cli][info] Available streams: ao, md (worst), ld, sd, hd, uhd, origin (best)
[cli][info] Opening stream: origin (http)
[cli][info] Starting player: /usr/bin/mpv
[cli][debug] Pre-buffering 8192 bytes
[cli.output][debug] Opening subprocess: ['/usr/bin/mpv', '--force-media-title=https://live.douyin.com/99457681876', '-']
[cli][debug] Writing stream to output
[cli][info] Player closed
[cli][info] Stream ended
[cli][info] Closing currently open stream...

$ streamlink -j https://live.douyin.com/99457681876 best | jq .metadata
{
  "id": "7387370035625937701",
  "author": "CSGO奕剑",
  "category": null,
  "title": "打完PL看看完美巅峰赛强度"
}

Offline channel:

$ streamlink https://live.douyin.com/830806814749
[cli][info] Found matching plugin douyin for URL https://live.douyin.com/830806814749
[plugins.douyin][info] The channel is currently offline
error: No playable streams found on this URL: https://live.douyin.com/830806814749

Non-existing channel:

$ streamlink https://live.douyin.com/drytfcyuvghbujnkftcuygvhujb
[cli][info] Found matching plugin douyin for URL https://live.douyin.com/drytfcyuvghbujnkftcuygvhujb
error: No playable streams found on this URL: https://live.douyin.com/drytfcyuvghbujnkftcuygvhujb

@v2wy @bastimeyer
plugins.douyin: new plugin
cfa44c5 
Co-Authored-By: bastimeyer <mail@bastimeyer.de>
@bastimeyer bastimeyer merged commit a362d28 into streamlink:master Jul 3, 2024
16 checks passed
@v2wy
Copy link
Contributor Author

v2wy commented Jul 3, 2024

@v2wy, instead of reviewing and annotating changes, I decided to fix any left over code issues myself and force-push onto your PR branch. The commit author data is left intact. Hope that's okay...

  • Updated plugin description

  • Fixed linting issues

  • Fixed validation schema

    • Removed try-except block
    • Improved data extraction
    • Split into sub-schemas (channel/room data and stream data)
    • Fixed schemas for offline/invalid channels

  • Forced HTTPS stream URLs

As commented in the plugin code, HLS streams are available, but using those would cause an unnecessary delay when fetching the streams, because there are multivariant playlists for each quality which consist of only a single media playlist, so each multivariant playlist would need to be queried. That's not worth it, despite progressive HTTP streams being pretty bad for live streaming.

Live channel:

$ streamlink -l debug https://live.douyin.com/99457681876 best
[cli][debug] OS:         Linux-6.9.7-1-git-x86_64-with-glibc2.39
[cli][debug] Python:     3.12.4
[cli][debug] OpenSSL:    OpenSSL 3.3.1 4 Jun 2024
[cli][debug] Streamlink: 6.8.1+9.gd19b93d5
[cli][debug] Dependencies:
[cli][debug]  certifi: 2024.6.2
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 5.2.2
[cli][debug]  pycountry: 24.6.1
[cli][debug]  pycryptodome: 3.20.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.32.3
[cli][debug]  trio: 0.25.1
[cli][debug]  trio-websocket: 0.11.1
[cli][debug]  typing-extensions: 4.12.2
[cli][debug]  urllib3: 2.2.2
[cli][debug]  websocket-client: 1.8.0
[cli][debug] Arguments:
[cli][debug]  url=https://live.douyin.com/99457681876
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --player=/usr/bin/mpv
[cli][info] Found matching plugin douyin for URL https://live.douyin.com/99457681876
[plugins.douyin][debug] self.QUALITY_WEIGHTS={'md': 800000, 'ao': 0, 'origin': 12248000, 'sd': 2000000, 'ld': 1000000, 'uhd': 6000000, 'hd': 4000000}
[cli][info] Available streams: ao, md (worst), ld, sd, hd, uhd, origin (best)
[cli][info] Opening stream: origin (http)
[cli][info] Starting player: /usr/bin/mpv
[cli][debug] Pre-buffering 8192 bytes
[cli.output][debug] Opening subprocess: ['/usr/bin/mpv', '--force-media-title=https://live.douyin.com/99457681876', '-']
[cli][debug] Writing stream to output
[cli][info] Player closed
[cli][info] Stream ended
[cli][info] Closing currently open stream...

$ streamlink -j https://live.douyin.com/99457681876 best | jq .metadata
{
  "id": "7387370035625937701",
  "author": "CSGO奕剑",
  "category": null,
  "title": "打完PL看看完美巅峰赛强度"
}

Offline channel:

$ streamlink https://live.douyin.com/830806814749
[cli][info] Found matching plugin douyin for URL https://live.douyin.com/830806814749
[plugins.douyin][info] The channel is currently offline
error: No playable streams found on this URL: https://live.douyin.com/830806814749

Non-existing channel:

$ streamlink https://live.douyin.com/drytfcyuvghbujnkftcuygvhujb
[cli][info] Found matching plugin douyin for URL https://live.douyin.com/drytfcyuvghbujnkftcuygvhujb
error: No playable streams found on this URL: https://live.douyin.com/drytfcyuvghbujnkftcuygvhujb

Thank you very much for your review and modifications. I have learned a lot from your comments. The code written earlier was somewhat casual. I will submit a PR following the guidelines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bastimeyer bastimeyer bastimeyer requested changes

Assignees
No one assigned
Labels
new plugin
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

douyin support
4 participants
@v2wy @mkbloke @steven7851 @bastimeyer