Isolate macOS wheel builds from Homebrew #8497
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,6 +23,8 @@ OPENJPEG_VERSION=2.5.2 | |
| XZ_VERSION=5.6.3 | ||
| TIFF_VERSION=4.6.0 | ||
| LCMS2_VERSION=2.16 | ||
| RAQM_VERSION=0.7.1 | ||
| FRIBIDI_VERSION=1.0.16 | ||
| if [[ -n "$IS_MACOS" ]]; then | ||
| GIFLIB_VERSION=5.2.2 | ||
| else | ||
|
|
@@ -38,7 +40,23 @@ BZIP2_VERSION=1.0.8 | |
| LIBXCB_VERSION=1.17.0 | ||
| BROTLI_VERSION=1.1.0 | ||
|
|
||
| function build_pkg_config { | ||
| if [ -e pkg-config-stamp ]; then return; fi | ||
| # This essentially duplicates the Homebrew recipe: | ||
| # https://github.com/Homebrew/homebrew-core/blob/master/Formula/p/pkg-config.rb | ||
radarhere marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ORIGINAL_CFLAGS=$CFLAGS | ||
| CFLAGS="$CFLAGS -Wno-int-conversion" | ||
| build_simple pkg-config 0.29.2 https://pkg-config.freedesktop.org/releases tar.gz \ | ||
| --disable-debug --disable-host-tool --with-internal-glib \ | ||
| --with-pc-path=$BUILD_PREFIX/share/pkgconfig:$BUILD_PREFIX/lib/pkgconfig \ | ||
| --with-system-include-path=$(xcrun --show-sdk-path --sdk macosx)/usr/include | ||
| CFLAGS=$ORIGINAL_CFLAGS | ||
| export PKG_CONFIG=$BUILD_PREFIX/bin/pkg-config | ||
| touch pkg-config-stamp | ||
| } | ||
|
|
||
| function build_brotli { | ||
| if [ -e brotli-stamp ]; then return; fi | ||
| local cmake=$(get_modern_cmake) | ||
| local out_dir=$(fetch_unpack https://github.com/google/brotli/archive/v$BROTLI_VERSION.tar.gz brotli-$BROTLI_VERSION.tar.gz) | ||
| (cd $out_dir \ | ||
|
|
@@ -48,25 +66,25 @@ function build_brotli { | |
| cp /usr/local/lib64/libbrotli* /usr/local/lib | ||
| cp /usr/local/lib64/pkgconfig/libbrotli* /usr/local/lib/pkgconfig | ||
| fi | ||
| touch brotli-stamp | ||
| } | ||
|
|
||
| function build_harfbuzz { | ||
| if [ -e harfbuzz-stamp ]; then return; fi | ||
| python -m pip install meson ninja | ||
|
|
||
| local out_dir=$(fetch_unpack https://github.com/harfbuzz/harfbuzz/releases/download/$HARFBUZZ_VERSION/$HARFBUZZ_VERSION.tar.xz harfbuzz-$HARFBUZZ_VERSION.tar.xz) | ||
| (cd $out_dir \ | ||
| && meson setup build --prefix=$BUILD_PREFIX --buildtype=release -Dfreetype=enabled -Dglib=disabled) | ||
| (cd $out_dir/build \ | ||
| && meson install) | ||
| if [[ "$MB_ML_LIBC" == "manylinux" ]]; then | ||
| cp /usr/local/lib64/libharfbuzz* /usr/local/lib | ||
| fi | ||
| touch harfbuzz-stamp | ||
| } | ||
|
|
||
| function build { | ||
| build_xz | ||
| if [ -z "$IS_ALPINE" ] && [ -z "$IS_MACOS" ]; then | ||
| yum remove -y zlib-devel | ||
|
|
@@ -77,17 +95,23 @@ function build { | |
| if [ -n "$IS_MACOS" ]; then | ||
| build_simple xorgproto 2024.1 https://www.x.org/pub/individual/proto | ||
| build_simple libXau 1.0.11 https://www.x.org/pub/individual/lib | ||
| build_simple libXdmcp 1.1.5 https://www.x.org/pub/individual/lib | ||
|
There was a problem hiding this comment.
I suspect you took out all the I'm saying that this wouldn't be part of the Pillow wheels at the moment, so if it's still not after this PR, that sounds fine to me. This function Line 834 in 5bff2f3 There was a problem hiding this comment. My recollection was that Xdmcp was needed for XCB to compile at all; but I've just removed it and done a clean build, and everything worked as expected (AFAICT). I'll remove it. |
||
| build_simple libpthread-stubs 0.5 https://xcb.freedesktop.org/dist | ||
| else | ||
| sed s/\${pc_sysrootdir\}// /usr/local/share/pkgconfig/xcb-proto.pc > /usr/local/lib/pkgconfig/xcb-proto.pc | ||
| fi | ||
| build_simple libxcb $LIBXCB_VERSION https://www.x.org/releases/individual/lib | ||
|
|
||
| build_libjpeg_turbo | ||
| if [ -n "$IS_MACOS" ]; then | ||
| # Custom tiff build to include jpeg; by default, configure won't include | ||
| # headers/libs in the custom macOS prefix | ||
| build_simple tiff $TIFF_VERSION https://download.osgeo.org/libtiff tar.gz \ | ||
| --with-jpeg-include-dir=$BUILD_PREFIX/include --with-jpeg-lib-dir=$BUILD_PREFIX/lib | ||
| else | ||
| build_tiff | ||
| fi | ||
|
|
||
| build_libpng | ||
| build_lcms2 | ||
| build_openjpeg | ||
|
|
@@ -113,33 +137,55 @@ function build { | |
| fi | ||
|
|
||
| build_harfbuzz | ||
|
|
||
| if [ -n "$IS_MACOS" ]; then | ||
| build_simple fribidi $FRIBIDI_VERSION https://github.com/fribidi/fribidi/releases/download/v$FRIBIDI_VERSION tar.xz --enable-shared | ||
| build_simple raqm $RAQM_VERSION https://github.com/Host_Oman/libraqm/releases/download/v$RAQM_VERSION tar.gz --enable-shared | ||
| fi | ||
| } | ||
|
|
||
| # Perform all dependency builds in the build subfolder. | ||
| mkdir -p build | ||
| pushd build > /dev/null | ||
|
|
||
| # Any stuff that you need to do before you start building the wheels | ||
| # Runs in the root directory of this repository. | ||
| if [[ ! -d pillow-depends-main ]]; then | ||
| if [[ ! -f pillow-depends-main.zip ]]; then | ||
| echo "Download pillow dependency sources..." | ||
| curl -fSL -o pillow-depends-main.zip https://github.com/python-pillow/pillow-depends/archive/main.zip | ||
| fi | ||
| untar pillow-depends-main.zip | ||
| fi | ||
|
|
||
| if [[ -n "$IS_MACOS" ]]; then | ||
| # Build and install into the `deps` folder. | ||
| BUILD_PREFIX=$(pwd)/deps | ||
|
|
||
| # Homebrew (or similar packaging environments) install can contain some of | ||
| # the libraries that we're going to build. However, they may be compiled | ||
| # with a MACOSX_DEPLOYMENT_TARGET that doesn't match what we want to use, | ||
| # and they may bring in other dependencies that we don't want. The same will | ||
| # be true of any other locations on the path. To avoid conflicts, strip the | ||
| # path down to the bare mimimum (which, on macOS, won't include any | ||
freakboy3742 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # development dependencies). | ||
| export PATH="$BUILD_PREFIX/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:$(dirname $(which python))" | ||
| export CMAKE_PREFIX_PATH=$BUILD_PREFIX | ||
|
|
||
| # Link the brew command into our isolated build directory. | ||
radarhere marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| mkdir -p "$BUILD_PREFIX/bin" | ||
| mkdir -p "$BUILD_PREFIX/lib" | ||
|
|
||
| # Ensure pkg-confg and cmake are available | ||
| build_pkg_config | ||
| python -m pip install cmake | ||
freakboy3742 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| fi | ||
|
|
||
| wrap_wheel_builder build | ||
|
|
||
| # Return to the project root to finish the build | ||
| popd > /dev/null | ||
|
|
||
| # Append licenses | ||
| for filename in wheels/dependency_licenses/*; do | ||
| echo -e "\n\n----\n\n$(basename $filename | cut -f 1 -d '.')\n" | cat >> LICENSE | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,6 +19,7 @@ lib64/ | |
| parts/ | ||
| sdist/ | ||
| var/ | ||
| wheelhouse/ | ||
| *.egg-info/ | ||
| .installed.cfg | ||
| *.egg | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule multibuild
updated
20 files
| +0 −91 | .appveyor.yml | |
| +6 −12 | .github/workflows/lint_python.yml | |
| +20 −11 | .github/workflows/test.yml | |
| +1 −1 | LICENSE | |
| +8 −8 | README.rst | |
| +19 −8 | common_utils.sh | |
| +12 −17 | library_builders.sh | |
| +42 −25 | osx_utils.sh | |
| +16 −12 | supported_wheels.py | |
| +1 −2 | tests/test_common_utils.sh | |
| +3 −1 | tests/test_fill_pyver.sh | |
| +1 −1 | tests/test_fill_submodule.sh | |
| +2 −0 | tests/test_library_builders.sh | |
| +0 −2 | tests/test_multibuild.sh | |
| +33 −9 | tests/test_osx_utils.sh | |
| +1 −11 | tests/test_python_install.sh | |
| +3 −3 | tests/test_supported_wheels.sh | |
| +5 −2 | travis_linux_steps.sh | |
| +2 −1 | travis_osx_steps.sh | |
| +6 −0 | travis_steps.sh |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It definitely makes sense to use the most recent available version; but pillow-depends-main doesn't include this version. What's the procedure for updating that repo?
(Related: pkg-config, libXdmcp, and an updated webp sources should probably be added to that project)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The theory is that pillow-depends-main is just a more reliable source for the dependencies, and PRs should work without it.
Once this is merged, I will likely add them there.