Testing on multiple platforms

Sage is intended to build and run on a variety of platforms, including all major Linux distributions, as well as MacOS, and Windows (with Cygwin and WSL).

There is considerable variation among these platforms. To ensure that Sage continues to build correctly on users’ machines, it is crucial to test changes to Sage, in particular when external packages are added or upgraded, on a wide spectrum of platforms.

Sage patchbots

The Sage patchbots will automatically test your Trac ticket by attempting an incremental build of Sage and running doctests.

Sage buildbots

The Sage Release buildbot builds entire tarballs (e.g., all the development releases) on a variety of machines.

Developers’ and users’ tests on sage-release

Sage developers and users are encouraged to contribute to testing releases that are announced on Sage Release on their machines and to report test results (success and failures) by responding to the announcements.

Testing Sage on a different platform using Docker

Docker is a popular virtualization software, running Linux operating system images (“Docker images”) in containers on a shared Linux kernel. These containers can be run using a Docker client on your Linux, Mac, or Windows box, as well as on various cloud services.

To get started, you need to install a Docker client. The clients are available for Linux, Mac, and Windows. The clients for the latter are known as “Docker Desktop”.

All examples in this section were obtained using Docker Desktop for Mac; but the command-line user interface for the other platforms is identical.

All major Linux distributions provide ready-to-use Docker images, which are published via Docker Hub. For example, to run the current stable (LTS) version of Ubuntu interactively, you can use the shell command:

[mkoeppe@sage sage]$ docker run -it ubuntu:latest
root@9f3398da43c2:/#

Here ubuntu is referred to as the “image (name)” and latest as the “tag”. Other releases of Ubuntu are available under different tags, such as xenial or devel.

The above command drops you in a root shell on the container:

root@9f3398da43c2:/# uname -a
Linux 9f3398da43c2 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@9f3398da43c2:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         181G  116G   56G  68% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.7G     0  2.7G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/sda1       181G  116G   56G  68% /etc/hosts
tmpfs           2.7G     0  2.7G   0% /proc/acpi
tmpfs           2.7G     0  2.7G   0% /sys/firmware

Exiting the shell terminates the container:

root@9f3398da43c2:/# ^D
[mkoeppe@sage sage]$

Let us work with a distclean Sage source tree. If you are using git, a good way to get one (without losing a precious installation in SAGE_LOCAL) is by creating a new worktree:

[mkoeppe@sage sage] git worktree add worktree-ubuntu-latest
[mkoeppe@sage sage] cd worktree-ubuntu-latest
[mkoeppe@sage worktree-ubuntu-latest] ls
COPYING.txt ... Makefile ... configure.ac ... src tox.ini

This is not bootstrapped (configure is missing), so let’s bootstrap it:

[mkoeppe@sage worktree-ubuntu-latest] make configure
...

We can start a container again with same image, ubuntu:latest, but this time let’s mount the current directory into it:

[mkoeppe@sage worktree-ubuntu-latest]$ docker run -it --mount type=bind,source=$(pwd),target=/sage ubuntu:latest
root@39d693b2a75d:/# mount | grep sage
osxfs on /sage type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
root@39d693b2a75d:/# cd sage
root@39d693b2a75d:/sage# ls
COPYING.txt ... Makefile ... config configure configure.ac ... src tox.ini

Typical Docker images provide minimal installations of packages only:

root@39d693b2a75d:/sage# command -v python
root@39d693b2a75d:/sage# command -v gcc
root@39d693b2a75d:/sage#

As you can see above, the image ubuntu:latest has neither a Python nor a GCC installed, which are among the build prerequisites of Sage. We need to install them using the distribution’s package manager first.

Sage facilitates testing various distributions on Docker as follows.

Discovering the system’s package system

root@39d693b2a75d:/sage# build/bin/sage-guess-package-system
debian

Let’s install gcc, hoping that the Ubuntu package providing it is simply named gcc. If we forgot what the package manager on Debian-derived distributions is called, we can ask Sage for a reminder:

root@39d693b2a75d:/sage# build/bin/sage-print-system-package-command debian install gcc
apt-get install gcc

We remember that we need to fetch the current package lists from the server first:

root@39d693b2a75d:/sage# apt-get update
root@39d693b2a75d:/sage# apt-get install gcc

Using Sage’s database of distribution prerequisites

The source code of the Sage distribution contains a database of package names in various distributions’ package managers. For example, the file build/pkgs/_prereq/distros/debian.txt contains the following

# This file, build/pkgs/_prereq/distros/debian.txt, contains names
# of Debian/Ubuntu packages needed for installation of Sage from source.
#
# In addition, the files build/pkgs/SPKG/distros/debian.txt contain the names
# of packages that provide the equivalent of SPKG.
#
# Everything on a line after a # character is ignored.
binutils
make
m4
perl
# python3-minimal is not enough on debian buster, ubuntu bionic - it does not have urllib
python3    # system python for bootstrapping the build
tar
bc
gcc
# On debian buster, need C++ even to survive 'configure'. Otherwise:
# checking how to run the C++ preprocessor... /lib/cpp
# configure: error: in `/sage':
# configure: error: C++ preprocessor "/lib/cpp" fails sanity check
g++
# Needed if we download some packages from a https upstream URL
ca-certificates

From this information, we know that we can use the following command on our container to install the necessary build prerequisites:

root@39d693b2a75d:/sage# apt-get install binutils make m4 perl python3 tar bc gcc g++ ca-certificates
Reading package lists... Done
Building dependency tree
Reading state information... Done
tar is already the newest version (1.29b-2ubuntu0.1).
The following additional packages will be installed:
...
Done.

(The Sage Installation Guide also provides such command lines for some distributions; these are automatically generated from the database of package names.)

Now we can start the build:

root@39d693b2a75d:/sage# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking for root user... yes
configure: error: You cannot build Sage as root, switch to an unprivileged user.  (If building in a container, use --enable-build-as-root.)

Let’s just follow this helpful hint:

root@39d693b2a75d:/sage# ./configure --enable-build-as-root
checking for a BSD-compatible install... /usr/bin/install -c
...

Using Sage’s database of equivalent distribution packages

At the end of the ./configure run, Sage issued a message like the following:

configure: notice: the following SPKGs did not find equivalent system packages: arb boost boost_cropped bzip2 ... yasm zeromq zlib
checking for the package system in use... debian
configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source:
configure:   $ sudo apt-get install libflint-arb-dev ... yasm libzmq3-dev libz-dev
configure: After installation, re-run configure using:
configure:   $ ./config.status --recheck && ./config.status

This information comes from Sage’s database of equivalent distribution packages. For example:

root@39d693b2a75d:/sage# ls build/pkgs/arb/distros/
arch.txt      conda.txt       debian.txt      gentoo.txt
root@39d693b2a75d:/sage# cat build/pkgs/arb/distros/debian.txt
libflint-arb-dev

Note that these package equivalencies are based on a current stable or testing version of the distribution; the packages are not guaranteed to exist in every release or derivative distribution.

The Sage distribution is intended to build correctly no matter what superset of the set of packages forming the minimal build prerequisites is installed on the system. If it does not, this is a bug of the Sage distribution and should be reported and fixed on a ticket. Crucial part of a bug report is the configuration of the system, in particular a list of installed packages and their versions.

Let us install a subset of these packages:

root@39d693b2a75d:/sage# apt-get install libbz2-dev bzip2 yasm libz-dev
Reading package lists... Done
...
Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ...
root@39d693b2a75d:/sage#

Committing a container to disk

After terminating the container, we can create a new image corresponding to its current state:

root@39d693b2a75d:/sage# ^D
[mkoeppe@sage worktree-ubuntu-latest]$ docker ps -a | head -n3
CONTAINER ID        IMAGE                   COMMAND                   CREATED             STATUS
39d693b2a75d        ubuntu:latest           "/bin/bash"               8 minutes ago       Exited (0) 6 seconds ago
9f3398da43c2        ubuntu:latest           "/bin/bash"               8 minutes ago       Exited (0) 8 minutes ago
[mkoeppe@sage worktree-ubuntu-latest]$ docker commit 39d693b2a75d ubuntu-latest-minimal-17
sha256:4151c5ca4476660f6181cdb13923da8fe44082222b984c377fb4fd6cc05415c1

Here, 39d693b2a75d was the container id (which appeared in the shell prompts and in the output of docker ps), and ubuntu-latest-minimal-17 is an arbitrary symbolic name for the new image. The output of the command is the id of the new image. We can use either the symbolic name or the id to refer to the new image.

We can run the image and get a new container with the same state as the one that we terminated. Again we want to mount our worktree into it; otherwise, because we did not make a copy, the new container will have no access to the worktree:

[mkoeppe@sage worktree-ubuntu-latest]$ docker run -it \
  --mount type=bind,source=$(pwd),target=/sage ubuntu-latest-minimal-17
root@73987568712c:/# cd sage
root@73987568712c:/sage# command -v gcc
/usr/bin/gcc
root@73987568712c:/sage# command -v yasm
/usr/bin/yasm
root@73987568712c:/sage# ^D
[mkoeppe@sage worktree-ubuntu-latest]$

The image ubuntu-latest-minimal-17 can be run in as many containers as we want and can also be shared with other users or developers so that they can run it in a container on their machine. (See the Docker documentation on how to share images on Docker Hub or to save images to a tar archive.)

This facilitates collaboration on fixing portability bugs of the Sage distribution. After reproducing a portability bug on a container, several developers can work on fixing the bug using containers running on their respective machines.

Generating Dockerfiles

Sage also provides a script for generating a Dockerfile, which is a recipe for automatically building a new image:

[mkoeppe@sage sage]$ build/bin/write-dockerfile.sh debian ":standard: :optional:" > Dockerfile

(The second argument is passed to sage -package list to find packages for the listed package types.)

The Dockerfile instructs the command docker build to build a new Docker image. Let us take a quick look at the generated file; this is slightly simplified:

[mkoeppe@sage sage]$ cat Dockerfile
# Automatically generated by SAGE_ROOT/build/bin/write-dockerfile.sh
# the :comments: separate the generated file into sections
# to simplify writing scripts that customize this file
...

First, it instructs docker build to start from an existing base image…:

...
ARG BASE_IMAGE=ubuntu:latest
FROM ${BASE_IMAGE}
...

Then, to install system packages…:

...
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes binutils make m4 perl python3 ... yasm libzmq3-dev libz-dev && apt-get clean

Then, to bootstrap and configure…:

RUN mkdir -p /sage
WORKDIR /sage
ADD Makefile VERSION.txt README.md bootstrap configure.ac sage ./
ADD src/doc/bootstrap src/doc/bootstrap
ADD m4 ./m4
ADD build ./build
RUN ./bootstrap
ADD src/bin src/bin
ARG EXTRA_CONFIGURE_ARGS=""
RUN ./configure --enable-build-as-root ${EXTRA_CONFIGURE_ARGS} || (cat config.log; exit 1)

Finally, to build and test…:

ARG NUMPROC=8
ENV MAKE="make -j${NUMPROC}"
ARG USE_MAKEFLAGS="-k"
RUN make ${USE_MAKEFLAGS} base-toolchain
ARG TARGETS_PRE="sagelib-build-deps"
RUN make ${USE_MAKEFLAGS} ${TARGETS_PRE}
ADD src src
ARG TARGETS="build ptest"
RUN make ${USE_MAKEFLAGS} ${TARGETS}

You can customize the image build process by passing build arguments to the command docker build. For example:

[mkoeppe@sage sage]$ docker build . -f Dockerfile \
  --build-arg BASE_IMAGE=ubuntu:latest \
  --build-arg NUMPROC=4 \
  --build-arg EXTRA_CONFIGURE_ARGS="--with-python=/usr/bin/python3.42"

These arguments (and their default values) are defined using ARG commands in the Dockerfile.

The above command will build Sage from scratch and will therefore take quite long. Let us instead just do a partial build, consisting of one small package, by setting the arguments TARGETS_PRE and TARGETS. We use a silent build (make V=0):

[mkoeppe@sage sage]$ docker build . -f Dockerfile \
  --build-arg TARGETS_PRE=ratpoints \
  --build-arg TARGETS=ratpoints \
  --build-arg USE_MAKEFLAGS="V=0"
Sending build context to Docker daemon    285MB
Step 1/28 : ARG BASE_IMAGE=ubuntu:latest
...
Step 2/28 : FROM ${BASE_IMAGE}
 ---> 549b9b86cb8d
...
Step 25/28 : RUN make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE}
...
make[1]: Entering directory '/sage/build/make'
sage-logger -p 'sage-spkg -y -o  ratpoints-2.1.3.p5' '/sage/logs/pkgs/ratpoints-2.1.3.p5.log'
[ratpoints-2.1.3.p5] installing. Log file: /sage/logs/pkgs/ratpoints-2.1.3.p5.log
  [ratpoints-2.1.3.p5] successfully installed.
make[1]: Leaving directory '/sage/build/make'

real  0m18.886s
user  0m1.779s
sys   0m0.314s
Sage build/upgrade complete!
...
---> 2d06689d39fa
Successfully built 2d06689d39fa

We can now start a container using the image id shown in the last step:

[mkoeppe@sage sage]$ docker run -it 2d06689d39fa bash
root@fab59e09a641:/sage# ls -l logs/pkgs/
total 236
-rw-r--r-- 1 root root 231169 Mar 26 22:07 config.log
-rw-r--r-- 1 root root   6025 Mar 26 22:27 ratpoints-2.1.3.p5.log
root@fab59e09a641:/sage# ls -l local/lib/*rat*
-rw-r--r-- 1 root root 177256 Mar 26 22:27 local/lib/libratpoints.a

You can customize the image build process further by editing the Dockerfile. For example, by default, the generated Dockerfile configures, builds, and tests Sage. By deleting or commenting out the commands for the latter, you can adjust the Dockerfile to stop after the configure phase, for example.

Dockerfile is the default filename for Dockerfiles. You can change it to any other name, but it is recommended to use Dockerfile as a prefix, such as Dockerfile-debian-standard. It should be placed within the tree rooted at the current directory (.); if you want to put it elsewhere, you need to learn about details of “Docker build contexts”.

Note that in contrast to the workflow described in the above sections, the Dockerfile copies a snapshot of your Sage worktree into the build container, using ADD commands, instead of mounting the directory into it. This copying is subject to the exclusions in the .gitignore file (via a symbolic link from .dockerignore). Therefore, only the sources are copied, but not your configuration (such as the file config.status), nor the $SAGE_LOCAL tree, nor any other build artefacts.

Because of this, you can build a Docker image using the generated Dockerfile from your main Sage development tree. It does not have to be distclean to start, and the build will not write into it at all. Hence, you can continue editing and compiling your Sage development tree even while Docker builds are running.

Debugging a portability bug using Docker

Let us do another partial build. We choose a package that we suspect might not work on all platforms, surf, which was marked as “experimental” in 2017:

[mkoeppe@sage sage]$ docker build . -f Dockerfile \
  --build-arg BASE_IMAGE=ubuntu:latest \
  --build-arg NUMPROC=4 \
  --build-arg TARGETS_PRE=surf \
  --build-arg TARGETS=surf
Sending build context to Docker daemon    285MB
Step 1/28 : ARG BASE_IMAGE=ubuntu:latest
Step 2/28 : FROM ${BASE_IMAGE}
 ---> 549b9b86cb8d
...
Step 24/28 : ARG TARGETS_PRE="sagelib-build-deps"
 ---> Running in 17d0ddb5ad7b
Removing intermediate container 17d0ddb5ad7b
 ---> 7b51411520c3
Step 25/28 : RUN make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE}
 ---> Running in 61833bea6a6d
make -j4 build/make/Makefile --stop
...
[surf-1.0.6-gcc6] Attempting to download package surf-1.0.6-gcc6.tar.gz from mirrors
...
[surf-1.0.6-gcc6] http://mirrors.mit.edu/sage/spkg/upstream/surf/surf-1.0.6-gcc6.tar.gz
...
[surf-1.0.6-gcc6] Setting up build directory for surf-1.0.6-gcc6
...
[surf-1.0.6-gcc6] /usr/bin/ld: cannot find -lfl
[surf-1.0.6-gcc6] collect2: error: ld returned 1 exit status
[surf-1.0.6-gcc6] Makefile:504: recipe for target 'surf' failed
[surf-1.0.6-gcc6] make[3]: *** [surf] Error 1
...
[surf-1.0.6-gcc6] Error installing package surf-1.0.6-gcc6
...
Makefile:2088: recipe for target '/sage/local/var/lib/sage/installed/surf-1.0.6-gcc6' failed
make[1]: *** [/sage/local/var/lib/sage/installed/surf-1.0.6-gcc6] Error 1
make[1]: Target 'surf' not remade because of errors.
make[1]: Leaving directory '/sage/build/make'
...
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make surf'):

* package:         surf-1.0.6-gcc6
  last build time: Mar 26 22:07
  log file:        /sage/logs/pkgs/surf-1.0.6-gcc6.log
  build directory: /sage/local/var/tmp/sage/build/surf-1.0.6-gcc6

...
Makefile:31: recipe for target 'surf' failed
make: *** [surf] Error 1
The command '/bin/sh -c make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE}' returned a non-zero code: 2

Note that no image id is shown at the end; the build failed, and no image is created. However, the container in which the last step of the build was attempted exists:

[mkoeppe@sage sage]$ docker ps -a |head -n3
CONTAINER ID        IMAGE                      COMMAND                   CREATED             STATUS
61833bea6a6d        7b51411520c3               "/bin/sh -c 'make SA…"    9 minutes ago       Exited (2) 1 minute ago
73987568712c        ubuntu-latest-minimal-17   "/bin/bash"               24 hours ago        Exited (0) 23 hours ago

We can copy the build directory from the container for inspection:

[mkoeppe@sage sage]$ docker cp 61833bea6a6d:/sage/local/var/tmp/sage/build ubuntu-build
[mkoeppe@sage sage]$ ls ubuntu-build/surf*/src
AUTHORS         TODO            curve           misc
COPYING         acinclude.m4    debug           missing
ChangeLog       aclocal.m4      dither          mkinstalldirs
INSTALL         background.pic  docs            mt
Makefile        config.guess    draw            src
Makefile.am     config.log      drawfunc        surf.1
Makefile.global config.status   examples        surf.xpm
Makefile.in     config.sub      gtkgui          yaccsrc
NEWS            configure       image-formats
README          configure.in    install-sh

Alternatively, we can use docker commit as explained earlier to create an image from the container:

[mkoeppe@sage sage]$ docker commit 61833bea6a6d
sha256:003fbd511016fe305bd8494bb1747f0fbf4cb2c788b4e755e9099d9f2014a60d
[mkoeppe@sage sage]$ docker run -it 003fbd511 bash
root@2d9ac65f4572:/sage# (cd /sage/local/var/tmp/sage/build/surf* && /sage/sage --buildsh)

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.
...
Note: SAGE_ROOT=/sage
(sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ ls /usr/lib/libfl*
/usr/lib/libflint-2.5.2.so  /usr/lib/libflint-2.5.2.so.13.5.2  /usr/lib/libflint.a  /usr/lib/libflint.so
(sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-get update && apt-get install apt-file
(sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-file update
(sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-file search "/usr/lib/libfl.a"
flex-old: /usr/lib/libfl.a
freebsd-buildutils: /usr/lib/libfl.a
(sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-get install flex-old
(sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ ./spkg-install
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
...
  /usr/bin/install -c  surf /sage/local/bin/surf
 /usr/bin/install -c -m 644 ./surf.1 /sage/local/share/man/man1/surf.1
...
make[1]: Leaving directory '/sage/local/var/tmp/sage/build/surf-1.0.6-gcc6/src'
(sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ exit
root@2d9ac65f4572:/sage# exit
[mkoeppe@sage sage]$

A standard case of bitrot.

Automatic Docker-based build testing using tox

tox is a Python package that is widely used for automating tests of Python projects.

Install tox for use with your system Python, for example using:

[mkoeppe@sage sage]$ pip install --user tox

A tox “environment” is identified by a symbolic name composed of several Tox “factors”, which are defined in the file $SAGE_ROOT/tox.ini.

The technology factor describes how the environment is run:

  • docker builds a Docker image as described above.

  • local runs testing on the host OS instead. We explain this technology in a later section.

The next two factors determine the host system configuration: The system factor describes a base operating system image.

  • Examples are ubuntu-focal, debian-buster, archlinux-latest, fedora-30, slackware-14.2, centos-7-i386, and ubuntu-bionic-arm64.

  • See $SAGE_ROOT/tox.ini for a complete list, and to which images on Docker hub they correspond.

The packages factor describes a list of system packages to be installed on the system before building Sage:

  • minimal installs the system packages known to Sage to provide minimal prerequisites for bootstrapping and building the Sage distribution. This corresponds to the packages _bootstrap and _prereq.

  • standard additionally installs all known system packages that are equivalent to standard packages of the Sage distribution, for which the mechanism spkg-configure.m4 is implemented. This corresponds to the packages listed by:

    [mkoeppe@sage sage]$ sage --package list --has-file=spkg-configure.m4 :standard:

  • maximal does the same for all standard and optional packages. This corresponds to the packages listed by:

    [mkoeppe@sage sage]$ sage --package list :standard: :optional:

The factors are connected by a hyphen to name a system configuration, such as debian-buster-standard and centos-7-i386-minimal.

Finally, the configuration factor (which is allowed to be empty) controls how the configure script is run.

The factors are connected by a hyphen to name a tox environment. (The order of the factors does not matter; however, for consistency and because the ordered name is used for caching purposes, we recommend to use the factors in the listed order.)

To run an environment:

[mkoeppe@sage sage]$ tox -e docker-slackware-14.2-minimal
[mkoeppe@sage sage]$ tox -e docker-ubuntu-bionic-standard

Arbitrary extra arguments to docker build can be supplied through the environment variable EXTRA_DOCKER_BUILD_ARGS. For example, for a non-silent build (make V=1), use:

[mkoeppe@sage sage]$ EXTRA_DOCKER_BUILD_ARGS="--build-arg USE_MAKEFLAGS=\"V=1\"" \
  tox -e docker-ubuntu-bionic-standard

By default, tox uses TARGETS_PRE=sagelib-build-deps and TARGETS=build, leading to a complete build of Sage without the documentation. If you pass positional arguments to tox (separated from tox options by --), then both TARGETS_PRE and TARGETS are set to these arguments. In this way, you can build some specific packages instead of all of Sage, for example:

[mkoeppe@sage sage]$ tox -e docker-centos-8-standard -- ratpoints

If the build succeeds, this will create a new image named sage-docker-centos-8-standard-with-targets:9.1.beta9-431-gca4b5b2f33-dirty, where

  • the image name is derived from the tox environment name and the suffix with-targets expresses that the make targets given in TARGETS have been built;

  • the tag name describes the git revision of the source tree as per git describe --dirty.

You can ask for tox to create named intermediate images as well. For example, to create the images corresponding to the state of the OS after installing all system packages (with-system-packages) and the one just after running the configure script (configured):

[mkoeppe@sage sage]$ DOCKER_TARGETS="with-system-packages configured with-targets" \
  tox -e docker-centos-8-standard -- ratpoints
...
Sending build context to Docker daemon ...
Step 1/109 : ARG BASE_IMAGE=fedora:latest
Step 2/109 : FROM ${BASE_IMAGE} as with-system-packages
...
Step 109/109 : RUN yum install -y zlib-devel || echo "(ignoring error)"
...
Successfully built 4bb14c3d5646
Successfully tagged sage-docker-centos-8-standard-with-system-packages:9.1.beta9-435-g861ba33bbc-dirty
Sending build context to Docker daemon ...
...
Successfully tagged sage-docker-centos-8-standard-configured:9.1.beta9-435-g861ba33bbc-dirty
...
Sending build context to Docker daemon ...
...
Successfully tagged sage-docker-centos-8-standard-with-targets:9.1.beta9-435-g861ba33bbc-dirty

Let’s verify that the images are available:

(base) egret:~/s/sage/sage-rebasing/worktree-algebraic-2018-spring (mkoeppe *$%>)$ docker images | head
REPOSITORY                                                TAG                               IMAGE ID
sage-docker-centos-8-standard-with-targets                9.1.beta9-435-g861ba33bbc-dirty   7ecfa86fceab
sage-docker-centos-8-standard-configured                  9.1.beta9-435-g861ba33bbc-dirty   4314929e2b4c
sage-docker-centos-8-standard-with-system-packages        9.1.beta9-435-g861ba33bbc-dirty   4bb14c3d5646
...

Automatic build testing on the host OS using tox -e local-direct

The local technology runs testing on the host OS instead.

In contrast to the docker technology, it does not make a copy of the source tree. It is most straightforward to run it from a separate, distclean git worktree.

Let us try a first variant of the local technology, the tox environment called local-direct. Because all builds with tox begin by bootstrapping the source tree, you will need autotools and other prerequisites installed in your system. See build/pkgs/_bootstrap/distros/*.txt for a list of system packages that provide these prerequisites.

We start by creating a fresh (distclean) git worktree.

[mkoeppe@sage sage] git worktree add worktree-local [mkoeppe@sage sage] cd worktree-local [mkoeppe@sage worktree-local] ls COPYING.txt … Makefile … configure.ac … src tox.ini

Again we build only a small package. Build targets can be passed as positional arguments (separated from tox options by --):

[mkoeppe@sage worktree-local] tox -e local-direct -- ratpoints
local-direct create: /Users/mkoeppe/.../worktree-local/.tox/local-direct
local-direct run-test-pre: PYTHONHASHSEED='2211987514'
...
src/doc/bootstrap:48: installing src/doc/en/installation/debian.txt...
bootstrap:69: installing 'config/config.rpath'
configure.ac:328: installing 'config/compile'
configure.ac:113: installing 'config/config.guess'
...
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
...
sage-logger -p 'sage-spkg -y -o  ratpoints-2.1.3.p5' '.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log'
[ratpoints-2.1.3.p5] installing. Log file: .../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log
  [ratpoints-2.1.3.p5] successfully installed.
...
  local-direct: commands succeeded
  congratulations :)

Let’s investigate what happened here:

[mkoeppe@sage worktree-local]$ ls -la
total 2576
drwxr-xr-x  35 mkoeppe  staff    1120 Mar 26 22:20 .
drwxr-xr-x  63 mkoeppe  staff    2016 Mar 27 09:35 ..
...
lrwxr-xr-x   1 mkoeppe  staff      10 Mar 26 20:34 .dockerignore -> .gitignore
-rw-r--r--   1 mkoeppe  staff      74 Mar 26 20:34 .git
...
-rw-r--r--   1 mkoeppe  staff    1212 Mar 26 20:41 .gitignore
...
drwxr-xr-x   7 mkoeppe  staff     224 Mar 26 22:11 .tox
...
-rw-r--r--   1 mkoeppe  staff    7542 Mar 26 20:41 Makefile
...
lrwxr-xr-x   1 mkoeppe  staff     114 Mar 26 20:45 config.log -> .tox/local-direct/log/config.log
-rwxr-xr-x   1 mkoeppe  staff   90411 Mar 26 20:46 config.status
-rwxr-xr-x   1 mkoeppe  staff  887180 Mar 26 20:45 configure
-rw-r--r--   1 mkoeppe  staff   17070 Mar 26 20:41 configure.ac
...
lrwxr-xr-x   1 mkoeppe  staff     103 Mar 26 20:45 logs -> .tox/local-direct/log
drwxr-xr-x  24 mkoeppe  staff     768 Mar 26 20:45 m4
lrwxr-xr-x   1 mkoeppe  staff     105 Mar 26 20:45 prefix -> .tox/local-direct/local
-rwxr-xr-x   1 mkoeppe  staff    4868 Mar 26 20:34 sage
drwxr-xr-x  16 mkoeppe  staff     512 Mar 26 20:46 src
-rw-r--r--   1 mkoeppe  staff   13478 Mar 26 20:41 tox.ini
drwxr-xr-x   4 mkoeppe  staff     128 Mar 26 20:46 upstream

There is no local subdirectory. This is part of a strategy to keep the source tree clean to the extent possible. In particular:

  • tox configured the build to use a separate $SAGE_LOCAL hierarchy in a directory under the tox environment directory .tox/local-direct. It created a symbolic link prefix that points there, for convenience:

    [mkoeppe@sage worktree-local]$ ls -l prefix/lib/*rat*
-rw-r--r--  1 mkoeppe  staff  165968 Mar 26 20:46 prefix/lib/libratpoints.a

  • Likewise, it created a separate logs directory, again under the tox environment directory, and a symbolic link.

This makes it possible for advanced users to test several local tox environments (such as local-direct) out of one worktree. However, because a build still writes configuration scripts and build artefacts (such as config.status) into the worktree, only one local build can run at a time in a given worktree.

The tox environment directory will be reused for the next tox run, which will therefore do an incremental build. To start a fresh build, you can use the -r option.

Automatic build testing on the host OS with best-effort isolation using tox -e local

tox -e local (without -direct) attempts a best-effort isolation from the user’s environment as follows:

  • All environment variables are set to standard values; with the exception of MAKE and EXTRA_CONFIGURE_ARGS. In particular, PATH is set to just /usr/bin:/bin:/usr/sbin:/sbin; it does not include /usr/local/bin.

Note, however, that various packages have build scripts that use /usr/local or other popular file system locations such as /opt/sfw/. Therefore, the isolation is not complete. Using /usr/local is considered standard behavior. On the other hand, we consider a package build script that inspects other file system locations to be a bug of the Sage distribution, which should be reported and fixed on a ticket.

Automatic build testing on macOS with a best-effort isolated installation of Homebrew

XCode on macOS does not provide the prerequisites for bootstrapping the Sage distribution. A good way to install them is using the Homebrew package manager.

In fact, Sage provides a tox environment that automatically installs an isolated copy of Homebrew with all prerequisites for bootstrapping:

[mkoeppe@sage worktree-local]$ tox -e local-homebrew-macos-minimal -- lrslib
local-homebrew-macos-minimal create: .../worktree-local/.tox/local-homebrew-macos-minimal
local-homebrew-macos-minimal run-test-pre: PYTHONHASHSEED='4246149402'
...
Initialized empty Git repository in .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/.git/
...
Tapped 2 commands and 4942 formulae (5,205 files, 310.7MB).
==> Downloading https://ftp.gnu.org/gnu/gettext/gettext-0.20.1.tar.xz
...
==> Pouring autoconf-2.69.catalina.bottle.4.tar.gz
...
==> Pouring pkg-config-0.29.2.catalina.bottle.1.tar.gz
  .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/Cellar/pkg-config/0.29.2: 11 files, 623.4KB
==> Caveats
==> gettext
gettext is keg-only, which means it was not symlinked into .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew,
because macOS provides the BSD gettext library & some software gets confused if both are in the library path.

If you need to have gettext first in your PATH run:
  echo 'export PATH=".../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/bin:$PATH"' >> ~/.bash_profile

For compilers to find gettext you may need to set:
  export LDFLAGS="-L.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/lib"
  export CPPFLAGS="-I.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/include"
...
local-homebrew-macos-minimal run-test: commands[0] | bash -c 'export PATH=.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin && . .homebrew-build-env && ./bootstrap && ./configure --prefix=.../worktree-local/.tox/local-homebrew-macos-minimal/local    && make -k V=0 ... lrslib'
...
bootstrap:69: installing 'config/config.rpath'
...
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
...
configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer ... tachyon xz yasm zeromq
checking for the package system in use... homebrew
configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source:
configure:   $ brew install cmake gcc gsl mpfi ninja openblas gpatch r readline xz yasm zeromq
...
sage-logger -p 'sage-spkg -y -o  lrslib-062+autotools-2017-03-03.p1' '.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log'
[lrslib-062+autotools-2017-03-03.p1] installing. Log file: .../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log
  [lrslib-062+autotools-2017-03-03.p1] successfully installed.
...
  local-homebrew-macos-minimal: commands succeeded
  congratulations :)

The tox environment uses the subdirectory homebrew of the environment directory .tox/local-homebrew-macos-minimal as the Homebrew prefix. This installation does not interact in any way with a Homebrew installation in /usr/local that you may have.

The test script sets the PATH to the bin directory of the Homebrew prefix, followed by /usr/bin:/bin:/usr/sbin:/sbin. It then uses the script $SAGE_ROOT/.homebrew-build-env to set environment variables so that Sage’s build scripts will find “keg-only” packages such as gettext.

The local-homebrew-macos-minimal environment does not install Homebrew’s python3 package. It uses XCode’s /usr/bin/python3 as system python. However, because various packages are missing that Sage considers as dependencies, Sage builds its own copy of these packages and of python3.

The local-homebrew-macos-standard environment additionally installs (in its separate isolated copy of Homebrew) all Homebrew packages known to Sage for which the spkg-configure.m4 mechanism is implemented; this is similar to the docker-standard tox environments described earlier. In particular it installs and uses Homebrew’s python3 package.

By using configuration factors, more variants can be tested. The local-homebrew-macos-standard-python3_xcode environment installs the same packages, but uses XCode’s /usr/bin/python3.

The local-homebrew-macos-standard-python3_pythonorg expects an installation of Python 3.7 in /Library/Frameworks/Python.framework; this is where the binary packages provided by python.org install themselves.

Automatic build testing with a best-effort isolated installation of Conda

Sage provides environments local-conda-forge-standard and local-conda-forge-minimal that create isolated installations of Miniconda in the subdirectory conda of the environment directory. They do not interact in any way with other installations of Anaconda or Miniconda that you may have on your system.

The environments use the conda-forge channel and use the python package and the compilers from this channel.

Options for build testing with the local technology

The environments using the local technology can be customized by setting environment variables.

  • If SKIP_SYSTEM_PKG_INSTALL is set to 1 (or yes), then all steps of installing system packages are skipped in this run. When reusing a previously created tox environment, this option can save time and also give developers more control for experiments with system packages.

  • If SKIP_BOOTSTRAP is set to 1 (or yes), then the bootstrapping phase is skipped. When reusing a previously created tox environment, this option can save time.

  • If SKIP_CONFIGURE is set to 1 (or yes), then the configure script is not run explicitly. When reusing a previously created tox environment, this option can save time. (The Makefile may still rerun configuration using config.status --recheck.)

The local technology also defines a special target bash: Instead of building anything with make, it just starts an interactive shell. For example, in combination with the above options:

[mkoeppe@sage worktree-local]$ SKIP_SYSTEM_PKG_INSTALL=yes SKIP_BOOTSTRAP=1 SKIP_CONFIGURE=1 tox -e local-homebrew-macos-minimal -- bash

Automatic parallel tox runs on GitHub Actions

The Sage source tree includes a default configuration for GitHub Actions that runs tox on a multitude of platforms on every pull request and on every push of a tag (but not of a branch) to a repository for which GitHub Actions are enabled.

This is defined in the file $SAGE_ROOT/.github/workflows/tox.yml.

An additional GitHub Actions workflow for testing on Cygwin, not based on tox, is defined in the file $SAGE_ROOT/.github/workflows/ci-cygwin.yml.

GitHub Actions runs these build jobs on 2-core machines with 7 GB of RAM memory and 14 GB of SSD disk space, cf. here, and has a time limit of 6h per job. This is just barely enough for a typical minimal build followed by make ptest to succeed; and plenty of time for a typical standard build to succeed.

Build logs become available as “artifacts” when all jobs of the workflow have finished. Each job generates one tarball. “Annotations” highlight certain top-level errors or warnings issued during the build.

The following procedure triggers a run of tests with the default set of system configurations.

  • Push your changes to trac.

  • Go to the Actions page on the GitHub mirror and select the workflow you would like to run.

  • Click on “Run workflow” above the list of workflow runs and select the branch where the workflow will run.

For more information, see the GitHub documentation.

Alternatively, you can create and push a custom tag in order to trigger a run of tests as follows. Let’s assume that github is the name of the remote corresponding to your GitHub fork of the Sage repository:

$ git remote -v | grep /my-github
my-github      https://github.com/mkoeppe/sage.git (fetch)
my-github      https://github.com/mkoeppe/sage.git (push)

  • Create a (“lightweight”, not “annotated”) tag with an arbitrary name, say ci (for “Continuous Integration”):

    git tag -f ci

  • Then push the tag to your GitHub repository:

    git push -f my-github ci

(In both commands, the “force” option (-f) allows overwriting a previous tag of that name.)

For testing branches against a custom set of system configurations during development, the following procedure seems to work well. It avoids changing the CI configuration on your development branch:

  • Create a branch from a recent beta release that contains the default GitHub Actions configuration; name it TESTER, say.

  • Edit $SAGE_ROOT/.github/workflows/tox.yml to include the system config you wish to test.

  • Commit and push the branch to your GitHub fork of sage.

  • Push your development branch to your GitHub repository and create a pull request against the TESTER branch. This will trigger the GitHub Actions workflow.

You will find a workflow status page in the “Actions” tab of your repository.

Here is how to read it. Each of the items in the left pane represents a full build of Sage on a particular system configuration. A test item in the left pane is marked with a green checkmark in the left pane if make build doc-html finished without error. (It also runs package testsuites and the Sage doctests but failures in these are not in reflected in the left pane; see below.)

The right pane (“Artifacts”) offers archives of the logs for download.

Scrolling down in the right pane shows “Annotations”:

  • Red “check failure” annotations appear for each log file that contains a build error. For example, you might see:

    docker (fedora-28, standard)
artifacts/logs-commit-8ca1c2df8f1fb4c6d54b44b34b4d8320ebecb164-tox-docker-fedora-28-standard/logs/pkgs/sagetex-3.4.log#L1
==== ERROR IN LOG FILE artifacts/logs-commit-8ca1c2df8f1fb4c6d54b44b34b4d8320ebecb164-tox-docker-fedora-28-standard/logs/pkgs/sagetex-3.4.log ====

  • Yellow “check warning” annotations. There are 2 types of these:

    1. Package testsuite or Sage doctest failures, like the following:

      docker (fedora-30, standard)
artifacts/logs-commit-8ca1c2df8f1fb4c6d54b44b34b4d8320ebecb164-tox-docker-fedora-30-standard/logs/ptest.log#L1
==== TESTSUITE FAILURE IN LOG FILE artifacts/logs-commit-8ca1c2df8f1fb4c6d54b44b34b4d8320ebecb164-tox-docker-fedora-30-standard/logs/ptest.log ====

    2. Notices from ./configure about not finding equivalent system packages, like the following:

      docker (fedora-31, standard)
artifacts/logs-commit-8ca1c2df8f1fb4c6d54b44b34b4d8320ebecb164-tox-docker-fedora-31-standard/config.log#L1
configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cddlib cmake eclib ecm fflas_ffpack flint flintqs fplll givaro gp

Clicking on the annotations does not take you to a very useful place. To view details, click on one of the items in the pane. This changes the right pane to a log viewer.

The docker workflows automatically push images to docker.pkg.github.com. You find them in the Packages tab of your GitHub repository.

In order to pull them for use on your computer, you need to first generate a Personal Access Token providing the read:packages scope as follows. Visit https://github.com/settings/tokens/new (this may prompt you for your GitHub password). As “Note”, type “Access docker.pkg.github.com”; then in “Select scopes”, select the checkbox for read:packages. Finally, push the “Generate token” button at the bottom. This will lead to a page showing your token, such as de1ec7ab1ec0ffee5ca1dedbaff1ed0ddba11. Copy this token and paste it to the command line:

$ echo de1ec7ab1ec0ffee5ca1dedbaff1ed0ddba11 | docker login docker.pkg.github.com --username YOUR-GITHUB-USERNAME

where you replace the token by your token, of course, and YOUR-GITHUB-USERNAME by your GitHub username.

Now you can pull the image and run it:

$ docker pull docker.pkg.github.com/YOUR-GITHUB-USERNAME/sage/sage-docker-fedora-31-standard-configured:f4bd671
$ docker run -it docker.pkg.github.com/YOUR-GITHUB-USERNAME/sage/sage-docker-fedora-31-standard-configured:f4bd671 bash