2023年5月3日 星期三

yt-dlp 開發筆記 @ Python3, macOS 13.2.1

yt-dlp 是知名的網路影音網站研究的專案,內含不少各大網站的爬蟲技術。若某個服務失效想要貢獻的話,可這樣操作,邊改 code 變測試:

% git clone https://github.com/yt-dlp/yt-dlp.git
% cd yt-dlp
% python3 yt_dlp/__main__.py 'URL' --skip-download -j

以 2023/08/31 即將關站的 Xuite 為例,從 github.com/yt-dlp/yt-dlp/tree/master/yt_dlp/extractor/xuite.py 找一則 unit test 來跑:

% python3 yt_dlp/__main__.py 'http://vlog.xuite.net/embed/cE1xbENoLTI3NDQ3MzM2LmZsdg==?ar=0&as=0' -f mp4 --skip-download -j | jq '' | tail -n 25
  "format_id": "720",
  "height": 720,
  "protocol": "http",
  "resolution": "720p",
  "dynamic_range": "SDR",
  "aspect_ratio": null,
  "http_headers": {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.50 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-us,en;q=0.5",
    "Sec-Fetch-Mode": "navigate"
  },
  "video_ext": "mp4",
  "audio_ext": "none",
  "format": "720 - 720p",
  "_filename": "男女平權只是口號?專家解釋約會時男生是否該幫女生付錢 (中字) [27447336].mp4",
  "filename": "男女平權只是口號?專家解釋約會時男生是否該幫女生付錢 (中字) [27447336].mp4",
  "_type": "video",
  "_version": {
    "version": "2023.03.04",
    "current_git_head": "b423b6a48",
    "release_git_head": "392389b7df7b818f794b231f14dc396d4875fbad",
    "repository": "yt-dlp/yt-dlp"
  }
}

至於,若要貢獻給 yt-dlp 社群,可以仿其架構,挑一個 extractor 來模仿,過程要繼承 InfoExtractor 來做事,例如實作 _VALID_URL 和 _real_extract(self, url) 等等,可以挑幾個實作沒那麼複雜的網站研究一下流程: