Removing Python as a dependency of R
·Surprisingly Python was a runtime dependency of R on conda-forge. As R doesn’t need Python to run, this was a bit weird. We got rid of this by splitting up the GLib package.
Typically people argue whether they should use R or Python to solve a problem.
In the case of conda-forge’s r-base package, on Linux, you got both when you only asked for one as Python was listed as a runtime dependency of R.
As they are both independent languages one doesn’t need the other to run, this was rather surprising.
Creating a new environment for R 4.0 using conda create -n r-test -c conda-forge r-base=4.0 lead to the following solution at the beginning of September:
The following NEW packages will be INSTALLED:
_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
_r-mutex conda-forge/noarch::_r-mutex-1.0.1-anacondar_1
binutils_impl_lin~ conda-forge/linux-64::binutils_impl_linux-64-2.35-h18a2f87_9
binutils_linux-64 conda-forge/linux-64::binutils_linux-64-2.35-heab0d09_28
bwidget conda-forge/linux-64::bwidget-1.9.14-0
bzip2 conda-forge/linux-64::bzip2-1.0.8-h516909a_3
c-ares conda-forge/linux-64::c-ares-1.16.1-h516909a_3
ca-certificates conda-forge/linux-64::ca-certificates-2020.6.20-hecda079_0
cairo conda-forge/linux-64::cairo-1.16.0-h3fc0475_1005
certifi conda-forge/linux-64::certifi-2020.6.20-py38h32f6830_0
curl conda-forge/linux-64::curl-7.71.1-he644dc0_5
fontconfig conda-forge/linux-64::fontconfig-2.13.1-h1056068_1002
freetype conda-forge/linux-64::freetype-2.10.2-he06d7ca_0
fribidi conda-forge/linux-64::fribidi-1.0.10-h516909a_0
gcc_impl_linux-64 conda-forge/linux-64::gcc_impl_linux-64-7.5.0-hdb87b24_16
gcc_linux-64 conda-forge/linux-64::gcc_linux-64-7.5.0-hf34d7eb_28
gettext conda-forge/linux-64::gettext-0.19.8.1-hc5be6a0_1002
gfortran_impl_lin~ conda-forge/linux-64::gfortran_impl_linux-64-7.5.0-h1104b78_16
gfortran_linux-64 conda-forge/linux-64::gfortran_linux-64-7.5.0-ha781d05_28
glib conda-forge/linux-64::glib-2.66.0-h0dae87d_0
graphite2 conda-forge/linux-64::graphite2-1.3.13-he1b5a44_1001
gsl conda-forge/linux-64::gsl-2.6-h294904e_0
gxx_impl_linux-64 conda-forge/linux-64::gxx_impl_linux-64-7.5.0-h1104b78_16
gxx_linux-64 conda-forge/linux-64::gxx_linux-64-7.5.0-ha781d05_28
harfbuzz conda-forge/linux-64::harfbuzz-2.7.2-hee91db6_0
icu conda-forge/linux-64::icu-67.1-he1b5a44_0
jpeg conda-forge/linux-64::jpeg-9d-h516909a_0
kernel-headers_li~ conda-forge/noarch::kernel-headers_linux-64-2.6.32-h77966d4_13
krb5 conda-forge/linux-64::krb5-1.17.1-hfafb76e_3
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.35-h769bd43_9
libblas conda-forge/linux-64::libblas-3.8.0-17_openblas
libcblas conda-forge/linux-64::libcblas-3.8.0-17_openblas
libcurl conda-forge/linux-64::libcurl-7.71.1-hcdd3856_5
libedit conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
libev conda-forge/linux-64::libev-4.33-h516909a_1
libffi conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007
libgcc-devel_linu~ conda-forge/linux-64::libgcc-devel_linux-64-7.5.0-h42c25f5_16
libgcc-ng conda-forge/linux-64::libgcc-ng-9.3.0-h24d8f2e_16
libgfortran-ng conda-forge/linux-64::libgfortran-ng-7.5.0-hdf63c60_16
libgomp conda-forge/linux-64::libgomp-9.3.0-h24d8f2e_16
libiconv conda-forge/linux-64::libiconv-1.16-h516909a_0
liblapack conda-forge/linux-64::liblapack-3.8.0-17_openblas
libnghttp2 conda-forge/linux-64::libnghttp2-1.41.0-h8cfc5f6_2
libopenblas conda-forge/linux-64::libopenblas-0.3.10-pthreads_hb3c22a3_4
libpng conda-forge/linux-64::libpng-1.6.37-hed695b0_2
libssh2 conda-forge/linux-64::libssh2-1.9.0-hab1572f_5
libstdcxx-devel_l~ conda-forge/linux-64::libstdcxx-devel_linux-64-7.5.0-h4084dd6_16
libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-9.3.0-hdf63c60_16
libtiff conda-forge/linux-64::libtiff-4.1.0-hc7e4089_6
libuuid conda-forge/linux-64::libuuid-2.32.1-h14c3975_1000
libwebp-base conda-forge/linux-64::libwebp-base-1.1.0-h516909a_3
libxcb conda-forge/linux-64::libxcb-1.13-h14c3975_1002
libxml2 conda-forge/linux-64::libxml2-2.9.10-h68273f3_2
lz4-c conda-forge/linux-64::lz4-c-1.9.2-he1b5a44_3
make conda-forge/linux-64::make-4.3-h516909a_0
ncurses conda-forge/linux-64::ncurses-6.2-he1b5a44_1
openssl conda-forge/linux-64::openssl-1.1.1g-h516909a_1
pango conda-forge/linux-64::pango-1.42.4-h7062337_4
pcre conda-forge/linux-64::pcre-8.44-he1b5a44_0
pcre2 conda-forge/linux-64::pcre2-10.35-h2f06484_0
pip conda-forge/noarch::pip-20.2.3-py_0
pixman conda-forge/linux-64::pixman-0.38.0-h516909a_1003
pthread-stubs conda-forge/linux-64::pthread-stubs-0.4-h14c3975_1001
python conda-forge/linux-64::python-3.8.5-h1103e12_7_cpython
python_abi conda-forge/linux-64::python_abi-3.8-1_cp38
r-base conda-forge/linux-64::r-base-4.0.2-he766273_1
readline conda-forge/linux-64::readline-8.0-he28a2e2_2
sed conda-forge/linux-64::sed-4.8-hbfbb72e_0
setuptools conda-forge/linux-64::setuptools-49.6.0-py38h32f6830_0
sqlite conda-forge/linux-64::sqlite-3.33.0-h4cf870e_0
sysroot_linux-64 conda-forge/noarch::sysroot_linux-64-2.12-h77966d4_13
tk conda-forge/linux-64::tk-8.6.10-hed695b0_0
tktable conda-forge/linux-64::tktable-2.10-h555a92e_3
wheel conda-forge/noarch::wheel-0.35.1-pyh9f0ad1d_0
xorg-kbproto conda-forge/linux-64::xorg-kbproto-1.0.7-h14c3975_1002
xorg-libice conda-forge/linux-64::xorg-libice-1.0.10-h516909a_0
xorg-libsm conda-forge/linux-64::xorg-libsm-1.2.3-h84519dc_1000
xorg-libx11 conda-forge/linux-64::xorg-libx11-1.6.12-h516909a_0
xorg-libxau conda-forge/linux-64::xorg-libxau-1.0.9-h14c3975_0
xorg-libxdmcp conda-forge/linux-64::xorg-libxdmcp-1.1.3-h516909a_0
xorg-libxext conda-forge/linux-64::xorg-libxext-1.3.4-h516909a_0
xorg-libxrender conda-forge/linux-64::xorg-libxrender-0.9.10-h516909a_1002
xorg-renderproto conda-forge/linux-64::xorg-renderproto-0.11.1-h14c3975_1002
xorg-xextproto conda-forge/linux-64::xorg-xextproto-7.3.0-h14c3975_1002
xorg-xproto conda-forge/linux-64::xorg-xproto-7.0.31-h14c3975_1007
xz conda-forge/linux-64::xz-5.2.5-h516909a_1
zlib conda-forge/linux-64::zlib-1.2.11-h516909a_1009
zstd conda-forge/linux-64::zstd-1.4.5-h6597ccf_2
As one can spot, among several other dependencies, python and pip were listed. To check why python was installed, we use conda-tree to find the reverse dependencies:
# conda-tree whoneeds python
['certifi', 'python_abi', 'setuptools', 'pip', 'wheel', 'glib'
Except glib, all of these packages are actually cyclic dependencies of python:
# conda-tree cycles
python_abi -> python -> pip -> setuptools -> python_abi
python_abi -> python -> pip -> setuptools -> certifi -> python_abi
certifi -> python -> pip -> setuptools -> certifi
wheel -> python -> pip -> wheel
setuptools -> python -> pip -> setuptools
python -> pip -> python
Thus the issue boils down to certain parts of glib needing python as declared in the meta.yaml.
# Note that Python is in fact a runtime dependency because this package
# provides a couple of developer support tools that are Python scripts.
requirements:
build:
- …
- python >=2.7
host:
- …
run:
- python >=2.7
- …
python is a runtime dependency of glib as some of its building tooling is based on Python scripts.
We don’t need that build tooling when we only want to run code that depends on glib, thus we create a new package called libglib which only contains the runtime dependencies and components.
The main package that is listed in the host requirements is still glib as we need these Python-depending build tools in most cases.
The main trick is that glib has a run_exports: - {{ pin_subpackage("libglib") }}, meaning that packages built with glib in host will automatically depend on libglib at runtime.
To make this work, we have rebuilt the latest version, 2.66.1 and 2.64.6, the last stable version before 2.66 and also the one that still works with 10.9 macOS SDK.
Another important aspect of 2.64.6 is that we haven’t built this patch version before.
This means that we use safely use it in a migration to rebuild all packages depending on glib so that the libglib run_exports is applied correctly.
If we were using a version of glib where we already had built a version without the package split, we wouldn’t be 100% sure in all cases that the variant with the split packages would be selected.
While the new recipes and the new pinning should ensure that the correct run_exports is set, we sadly still have glib mentioned explicitly in many of the recipes.
To ensure that on an r-base installation only libglib and not glib and python are installed, we need to remove glib from the run sections of the packages that r-base pulls in.
This is in addition to r-base also cairo, pango, harfbuzz.
Finally, we also run the glib migration for r-base: r-base#140.
Here we can already see as part of the CI builds that conda-build doesn’t install python or glib in the test environment.
Creating a new environment for R 4.0 using conda create -n r-test -c conda-forge r-base=4.0 leads to the following solution as of 18th October:
_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
_r-mutex conda-forge/noarch::_r-mutex-1.0.1-anacondar_1
binutils_impl_lin~ conda-forge/linux-64::binutils_impl_linux-64-2.35-h18a2f87_9
binutils_linux-64 conda-forge/linux-64::binutils_linux-64-2.35-hc3fd857_29
bwidget conda-forge/linux-64::bwidget-1.9.14-0
bzip2 conda-forge/linux-64::bzip2-1.0.8-h516909a_3
c-ares conda-forge/linux-64::c-ares-1.16.1-h516909a_3
ca-certificates conda-forge/linux-64::ca-certificates-2020.6.20-hecda079_0
cairo conda-forge/linux-64::cairo-1.16.0-h488836b_1006
curl conda-forge/linux-64::curl-7.71.1-he644dc0_8
fontconfig conda-forge/linux-64::fontconfig-2.13.1-h1056068_1002
freetype conda-forge/linux-64::freetype-2.10.3-he06d7ca_0
fribidi conda-forge/linux-64::fribidi-1.0.10-h516909a_0
gcc_impl_linux-64 conda-forge/linux-64::gcc_impl_linux-64-7.5.0-hda68d29_13
gcc_linux-64 conda-forge/linux-64::gcc_linux-64-7.5.0-he2a3fca_29
gettext conda-forge/linux-64::gettext-0.19.8.1-hf34092f_1003
gfortran_impl_lin~ conda-forge/linux-64::gfortran_impl_linux-64-7.5.0-hfca37b7_17
gfortran_linux-64 conda-forge/linux-64::gfortran_linux-64-7.5.0-ha081f1e_29
graphite2 conda-forge/linux-64::graphite2-1.3.13-he1b5a44_1001
gsl conda-forge/linux-64::gsl-2.6-h294904e_0
gxx_impl_linux-64 conda-forge/linux-64::gxx_impl_linux-64-7.5.0-h64c220c_13
gxx_linux-64 conda-forge/linux-64::gxx_linux-64-7.5.0-h547f3ba_29
harfbuzz conda-forge/linux-64::harfbuzz-2.7.2-hb1ce69c_1
icu conda-forge/linux-64::icu-67.1-he1b5a44_0
jpeg conda-forge/linux-64::jpeg-9d-h516909a_0
kernel-headers_li~ conda-forge/noarch::kernel-headers_linux-64-2.6.32-h77966d4_13
krb5 conda-forge/linux-64::krb5-1.17.1-hfafb76e_3
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.35-h769bd43_9
libblas conda-forge/linux-64::libblas-3.8.0-17_openblas
libcblas conda-forge/linux-64::libcblas-3.8.0-17_openblas
libcurl conda-forge/linux-64::libcurl-7.71.1-hcdd3856_8
libedit conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
libev conda-forge/linux-64::libev-4.33-h516909a_1
libffi conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007
libgcc-ng conda-forge/linux-64::libgcc-ng-9.3.0-h5dbcf3e_17
libgfortran-ng conda-forge/linux-64::libgfortran-ng-7.5.0-hae1eefd_17
libgfortran4 conda-forge/linux-64::libgfortran4-7.5.0-hae1eefd_17
libglib conda-forge/linux-64::libglib-2.66.1-h0dae87d_1
libgomp conda-forge/linux-64::libgomp-9.3.0-h5dbcf3e_17
libiconv conda-forge/linux-64::libiconv-1.16-h516909a_0
liblapack conda-forge/linux-64::liblapack-3.8.0-17_openblas
libnghttp2 conda-forge/linux-64::libnghttp2-1.41.0-h8cfc5f6_2
libopenblas pkgs/main/linux-64::libopenblas-0.3.10-h5a2b251_0
libpng conda-forge/linux-64::libpng-1.6.37-hed695b0_2
libssh2 conda-forge/linux-64::libssh2-1.9.0-hab1572f_5
libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-9.3.0-h2ae2ef3_17
libtiff conda-forge/linux-64::libtiff-4.1.0-hc7e4089_6
libuuid conda-forge/linux-64::libuuid-2.32.1-h14c3975_1000
libwebp-base conda-forge/linux-64::libwebp-base-1.1.0-h516909a_3
libxcb conda-forge/linux-64::libxcb-1.13-h14c3975_1002
libxml2 conda-forge/linux-64::libxml2-2.9.10-h68273f3_2
lz4-c conda-forge/linux-64::lz4-c-1.9.2-he1b5a44_3
make conda-forge/linux-64::make-4.3-hd18ef5c_1
ncurses conda-forge/linux-64::ncurses-6.2-he1b5a44_2
openssl conda-forge/linux-64::openssl-1.1.1h-h516909a_0
pango conda-forge/linux-64::pango-1.42.4-h80147aa_5
pcre conda-forge/linux-64::pcre-8.44-he1b5a44_0
pcre2 conda-forge/linux-64::pcre2-10.35-h2f06484_0
pixman conda-forge/linux-64::pixman-0.38.0-h516909a_1003
pthread-stubs conda-forge/linux-64::pthread-stubs-0.4-h14c3975_1001
r-base conda-forge/linux-64::r-base-4.0.3-hd272fe0_2
readline conda-forge/linux-64::readline-8.0-he28a2e2_2
sed conda-forge/linux-64::sed-4.8-hbfbb72e_0
sysroot_linux-64 conda-forge/noarch::sysroot_linux-64-2.12-h77966d4_13
tk conda-forge/linux-64::tk-8.6.10-hed695b0_1
tktable conda-forge/linux-64::tktable-2.10-h555a92e_3
xorg-kbproto conda-forge/linux-64::xorg-kbproto-1.0.7-h14c3975_1002
xorg-libice conda-forge/linux-64::xorg-libice-1.0.10-h516909a_0
xorg-libsm conda-forge/linux-64::xorg-libsm-1.2.3-h84519dc_1000
xorg-libx11 conda-forge/linux-64::xorg-libx11-1.6.12-h516909a_0
xorg-libxau conda-forge/linux-64::xorg-libxau-1.0.9-h14c3975_0
xorg-libxdmcp conda-forge/linux-64::xorg-libxdmcp-1.1.3-h516909a_0
xorg-libxext conda-forge/linux-64::xorg-libxext-1.3.4-h516909a_0
xorg-libxrender conda-forge/linux-64::xorg-libxrender-0.9.10-h516909a_1002
xorg-renderproto conda-forge/linux-64::xorg-renderproto-0.11.1-h14c3975_1002
xorg-xextproto conda-forge/linux-64::xorg-xextproto-7.3.0-h14c3975_1002
xorg-xproto conda-forge/linux-64::xorg-xproto-7.0.31-h14c3975_1007
xz conda-forge/linux-64::xz-5.2.5-h516909a_1
zlib conda-forge/linux-64::zlib-1.2.11-h516909a_1010
zstd conda-forge/linux-64::zstd-1.4.5-h6597ccf_2
Besides still being quite some dependencies, this no longer includes glib and python, thus we have achieved the goal of this task.
Title picture: Photo by boris misevic on Unsplash