Skip to content

installer: rewrite wheels config, fix pycountry #4180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 17, 2021
Merged

installer: rewrite wheels config, fix pycountry #4180

merged 1 commit into from
Nov 17, 2021

Conversation

bastimeyer
Copy link
Member

Resolves #4179

This fixes the broken pycountry import in the Windows installer's Python environment by building the pycountry wheel locally in the makeinstaller.sh script and including it instead of the source tarball:
#4179 (comment)

A 3.0.1 release needs to be published right after testing/confirming and merging this.

This fix isn't an ideal solution. All I want is to resolve this as quickly as possible. I will take a look at rewriting this properly afterwards, similar to the appimages build config, which uses a package "lockfile" (which we don't have here) with hashes. Pynsist unfortunately doesn't support this and since we're now using pip for building wheels locally anyway, we can utilize that and switch to the local_wheels config completely (via pip download $WHEEL_DEPS and pip wheel $SRC_DEPS).

$ ./script/makeinstaller.sh
[makeinstaller.sh] Setting up clean build directories
[makeinstaller.sh] Building streamlink-3.0.0+1.g132d37c package
running build
running build_py
UPDATING build/lib/streamlink/_version.py
set build/lib/streamlink/_version.py to '3.0.0+1.g132d37c'
[makeinstaller.sh] Creating empty plugin files
[makeinstaller.sh] Creating icons
[makeinstaller.sh] Locally building missing dependency wheels
Collecting pycountry==20.7.3
  Downloading pycountry-20.7.3.tar.gz (10.1 MB)
     |████████████████████████████████| 10.1 MB 1.0 MB/s            
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: pycountry
  Building wheel for pycountry (setup.py) ... done
  Created wheel for pycountry: filename=pycountry-20.7.3-py2.py3-none-any.whl size=10746883 sha256=6db926a1bbf100cf484c3120595f756546ab8aad0cd62f8a4bc4e58899c5c4d0
  Stored in directory: /tmp/pip-ephem-wheel-cache-pvl9c2j6/wheels/de/cb/0e/b40fff1168704e2498630eee7ad70a78b458fe4a902179ae2c
Successfully built pycountry
[makeinstaller.sh] Configuring installer
[makeinstaller.sh] Preparing assets
/home/basti/repos/streamlink/build/cache/ffmpeg-n4.4.1-win32-gpl-4.4.zip: OK
[makeinstaller.sh] Assembling files directory
Archive:  /home/basti/repos/streamlink/build/cache/ffmpeg-n4.4.1-win32-gpl-4.4.zip
   creating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/LICENSE.txt  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/BUILDINFO.txt  
   creating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/bin/
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/bin/ffplay.exe  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/bin/ffprobe.exe  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/bin/ffmpeg.exe  
   creating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffplay.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-protocols.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/mailing-list-faq.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/faq.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-formats.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/libavdevice.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/platform.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/libavcodec.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffprobe.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/fate.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/libavfilter.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-codecs.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/general.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/nut.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-bitstream-filters.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffplay-all.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-resampler.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/developer.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-devices.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-filters.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/git-howto.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-scaler.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffprobe-all.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-utils.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/libswresample.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/libavformat.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/ffmpeg-all.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/libswscale.html  
  inflating: /tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/doc/libavutil.html  
install: creating directory '/home/basti/repos/streamlink/build/files/ffmpeg'
'/tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/bin/ffmpeg.exe' -> '/home/basti/repos/streamlink/build/files/ffmpeg/ffmpeg.exe'
'/tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/LICENSE.txt' -> '/home/basti/repos/streamlink/build/files/ffmpeg/LICENSE.txt'
'/tmp/tmp.1nSkP7NsJn/0/ffmpeg-n4.4.1-win32-gpl-4.4/BUILDINFO.txt' -> '/home/basti/repos/streamlink/build/files/ffmpeg/BUILDINFO.txt'
[makeinstaller.sh] Building streamlink-3.0.0_1.g132d37c installer
Unpacking Python...
Copying packages into build directory...
Using cached wheel: /home/basti/.cache/pynsist/pypi/certifi/2021.5.30/certifi-2021.5.30-py2.py3-none-any.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/charset-normalizer/2.0.4/charset_normalizer-2.0.4-py3-none-any.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/idna/3.2/idna-3.2-py3-none-any.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/isodate/0.6.0/isodate-0.6.0-py2.py3-none-any.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/lxml/4.6.4/lxml-4.6.4-cp39-cp39-win32.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/pycryptodome/3.10.1/pycryptodome-3.10.1-cp35-abi3-win32.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/PySocks/1.7.1/PySocks-1.7.1-py3-none-any.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/requests/2.26.0/requests-2.26.0-py2.py3-none-any.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/six/1.16.0/six-1.16.0-py2.py3-none-any.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/urllib3/1.26.6/urllib3-1.26.6-py2.py3-none-any.whl
Using cached wheel: /home/basti/.cache/pynsist/pypi/websocket-client/1.2.1/websocket_client-1.2.1-py2.py3-none-any.whl
Collecting wheel file: pycountry-20.7.3-py2.py3-none-any.whl (from: /tmp/tmp.Se2qzPr8Y2/*.whl)
Writing NSI file to nsis/installer.nsi

~~~ Running makensis ~~~
Processing config: /etc/nsisconf.nsh
Processing script file: "nsis/installer.nsi" (UTF8)

Processed 1 file, writing output (x86-unicode):

Output: "/home/basti/repos/streamlink/dist/streamlink-3.0.0_1.g132d37c.exe"
Install: 6 pages (384 bytes), 5 sections (2 required) (163960 bytes), 3430 instructions (96040 bytes), 1663 strings (58828 bytes), 1 language table (354 bytes).
Uninstall: 1 page (128 bytes), 1 section (32792 bytes), 185 instructions (5180 bytes), 112 strings (3268 bytes), 1 language table (198 bytes).
Datablock optimizer saved 48300 bytes (~0.1%).

Using lzma compression.

EXE header size:              134144 / 102912 bytes
Install code:                  20367 / 238110 bytes
Install data:               39629435 / 120274517 bytes
Uninstall code+data:           26561 / 38745 bytes
CRC (0x080A95BA):                  4 / 4 bytes

Total size:                 39810511 / 120654288 bytes (32.9%)
Installer written to /home/basti/repos/streamlink/dist/streamlink-3.0.0_1.g132d37c.exe
[makeinstaller.sh] Success!

$ streamlink --loglevel debug
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.9.8
[cli][debug] Streamlink: 3.0.0+1.g132d37c
[cli][debug] Requests(2.26.0), Socks(1.7.1), Websocket(1.2.1)
[cli][debug] Arguments:
[cli][debug]  --loglevel=debug
[cli][debug]  --ffmpeg-ffmpeg=C:\Program Files (x86)\Streamlink\ffmpeg\ffmpeg.exe
usage: streamlink [OPTIONS] <URL> [STREAM]

Use -h/--help to see the available options or read the manual at https://streamlink.github.io

@bastimeyer bastimeyer force-pushed the installer/fix-pycountry branch from 132d37c to 8a7b779 Compare November 17, 2021 10:15
@bastimeyer
Copy link
Member Author

If there's no review within the next two hours, I'm going to merge and release 3.0.1, because 3.0.0 is broken and shouldn't stay like this.

@bastimeyer bastimeyer force-pushed the installer/fix-pycountry branch from 8a7b779 to 0b2ef88 Compare November 17, 2021 13:31
@bastimeyer bastimeyer changed the title installer: fix pycountry installer: rewrite wheels config, fix pycountry Nov 17, 2021
@bastimeyer
Copy link
Member Author

I've updated the PR with a wheel lockfile, similar to the appimage build config.

As mentioned, pynsist doesn't support wheel hash information, and since we need to build the pycountry wheel locally and include it via the local_wheels pynsist directive anyway, we can easily download the remaining wheels via pip too and set the hashes.

This by the way also updates some dependency versions.

Check the installer build log for the wheel lists:

A quick validation check from my locally built installer (built on Linux, copied into W10 VM):

$ streamlink -l debug
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.9.8
[cli][debug] Streamlink: 3.0.0+1.g0b2ef88
[cli][debug] Requests(2.26.0), Socks(1.7.1), Websocket(1.2.1)
[cli][debug] Arguments:
[cli][debug]  --loglevel=debug
[cli][debug]  --ffmpeg-ffmpeg=C:\Program Files (x86)\Streamlink\ffmpeg\ffmpeg.exe
usage: streamlink [OPTIONS] <URL> [STREAM]

Use -h/--help to see the available options or read the manual at https://streamlink.github.io

@bastimeyer bastimeyer merged commit b19458c into streamlink:master Nov 17, 2021
@bastimeyer bastimeyer deleted the installer/fix-pycountry branch November 17, 2021 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Windows installer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

3.0.0 - The 'pycountry' distribution was not found and is required by the application
1 participant
@bastimeyer