Building Bundled Executables

NOTE: Weave is now removed from PyCBC, so this documentation page may need removing. However, instructions for bundling may still be useful

For most applications, PyCBC executables can be built and run from a standard Python install following the instructions in Installing PyCBC. The standard installation requires that the install directory is available at runtime so that Python can find the libraries. It also requires that the build and runtime environments are compatible (e.g. compatible versions of glibc, gcc, and Python). Some PyCBC executables (e.g. pycbc_inspiral) require runtime compilation of code using weave, and so the execution environment must have a full installation of gcc and the Python development libraries.

When running on, e.g. the Open Science Grid, these requirements may not be satisfied. Although CVMFS can provide access to the PyCBC libraries at runtime on the OSG, many OSG execute machines do not have the required environment to weave-compile code at runtime. Running PyCBC using Einstein@Home places even stricter limitations on the runtime environment of the code, as execute machines do not have access to CVMFS and may be running a variety of operating systems.

For both OSG and Einstein@Home, a bundled executable must be built using PyInstaller. This bundle contains all of the Python and user-space C libraries required, as well as a Python interperter to run the code. This bundle must also contain pre-compiled objects for all of the weave code that is needed at runtime. The script pycbc_build_eah.sh can be used to build a self-contained PyInstaller bundle for pycbc_inspiral so that it can be run on OSG and Einstein@Home.

Note that the PyInstaller bundles are not completely static and still require a dynamically linked version of glibc at runtime. Since Linux systems are backwards, but not forwards compatible the bundle must be built on the lowest-common denominator operating system for the execution platform. For OSG this is RHEL6, as the OSG executale machines are typically RHEL6 or RHEL7. RHEL6 executables also work on Debian Wheezy and Jessie. Einstein@Home requires an even older build platforms, typically Debian Etch. Since these older build platforms may not have the required software installed (e.g. FFTW, Python, etc.) the pycbc_build_eah.sh downloads and builds a complete installation environment of all of the required software. It can therefore be used to build PyCBC on a machine without the standard LIGO Data Grid software installed.

Using the build script

Note

The build script creates a new virtual environment to build the bundled executables, and so it should not be run from within an existing virtual environment. Run the deactivate command if you are currently in a virtual environment. You may also need to specify the full path to the pycbc_build_eah.sh script once you leave your virtual environment.

The command-line arguments for the pycbc_build_eah.sh build script are:

$ ../tools/einsteinathome/pycbc_build_eah.sh --force-debian4 --help
>> [Wed Jun  8 00:27:47 UTC 2022] Start ../tools/einsteinathome/pycbc_build_eah.sh --force-debian4 --help


>> [Wed Jun  8 00:27:47 UTC 2022] Using Debian 4.0 (etch) settings
Options:

    --force-debian4                 force Debian 4.0 build, must be first option in command-line

    --print-env                     dump environment at beginning, must be first option in command-line

    --help                          print this messge and exit

    --clean                         perform a clean build (takes quite some time); delete ~/.cache and
                                    tarballs containing precompiled libraries (lalsuite, scipy etc.)

    --clean-lalsuite                checkout and build lalsuite from scratch

    --clean-sundays                 perform a clean-lalsuite build on sundays

    --clean-pycbc                   check out pycbc git repo from scratch

    --lalsuite-commit=<commit>      specify a commit (or tag or branch) of lalsuite to build from

    --blessed-lalsuite              get the lalsuite commit to build from
                                    https://github.com/ligo-cbc/pycbc/releases/latest

    --pycbc-commit=<commit>         specify a commit or tag of pycbc to build from (specifying a
                                    branch will only work reliably in conjunction with --clean-pycbc)

    --pycbc-remote=<username>       add pycbc repository github.com/username as remote

    --pycbc-branch=<branch>         checkout branch before building

    --no-pycbc-update               don't update local pycbc repo

    --no-lalsuite-update            don't update local lalsuite repo

    --bema-testing                  use einsteinathome_testing branch of bema-ligo/pycbc repo

    --no-cleanup                    keep build directories after successful build for later inspection

    --download-url=<url>            download test frame, template bank, and roms from <url>

    --with-extra-libs=<url>         add extra files from a tar file at <url> to the bundles

    --with-extra-bank=<file>        run pycbc_inspiral again with an extra template bank

    --with-extra-approximant=<file> run pycbc_inspiral again with an extra approximant

    --with-lal-data-path=<path>     run test job using ROM data from <path>

    --processing-scheme=<scheme>    run test job using processing scheme <scheme>

    --verbose-python                run PyInstalled Python in verbose mode, showing imports

    --no-analysis                   for testing, don't run analysis

    --silent-build                  do not brint build messages unless there is an error

    --pycbc-fetch-ref               fetch and use a specific reference for pycbc

Note

Command line parsing is performed with shell syntax matching, not Python argument parsing, so command line arguments must be given as --option=argument with the option name spelled out in full and including the = between the option and the argument.

The build script creates the directories pycbc-sources and pycbc-build in the directory that it is run from. The build script creates the bundles in the directory pycbc-build/environment/dist/ relative to the directory from which it run. The pycbc_inspiral executable will be named pycbc_inspiral_osg or pycbc_inspiral_osg_vX.Y.Z if the git commit matches a git tag of the form vX.Y.Z (where X, Y, and Z are integers).

Note

The build script checks that it is being run on one of the lowest-common denominator platforms that it knows about. To run the script on another platform, pass the command-line argument --force-debian4 to the script as the first argument in the list of arguments.

The minimal set of command line options required to build the pycbc_inspiral bundle is typically the hash of the versions of LALSuite and PyCBC:

pycbc_build_eah.sh --lalsuite-commit=a2a5a476d33f169b8749e2840c306a48df63c936 --pycbc-commit=b68832784969a47fe2658abffb3888ee06cd1be4

To include extra run-time libraries in the bundle, e.g. to add the Intel MKL libraries, specify them with the command-line argument:

--with-extra-libs=file://opt/intel/composer_xe_2015.0.090.tar.gz

The script executes pycbc_inspiral as part of the build process. This may require LAL data at build time. The LAL data can be given with the command line argument:

--with-lal-data-path=/cvmfs/oasis.opensciencegrid.org/ligo/sw/pycbc/lalsuite-extra/current/share/lalsimulation

The default command line arguments clone PyCBC from the standard GitHub repository. If you would like to build a bundle using code from your own GitHub repository or branch you can use the arguments mentioned below. In this case you would not need to specify a --pycbc-commit:

--pycbc-remote=soumide1102 --pycbc-branch=comp_wave_in_search

You may also tell the script to run pycbc_inspiral with additional waveform approximants to ensure that all of the necessary weave code is compiled into the executable with the arguments. The argument --with-extra-approximant can be specified multiple times to weave-compile and bundle different waveform approximants, for example:

--with-extra-approximant='SPAtmplt:mtotal<4' --with-extra-approximant='SEOBNRv4_ROM:else'

To test with compressed waveform banks, you can provide the following option after all the other --with-extra-approximant arguments:

--with-extra-approximant=--use-compressed-waveforms

The weave-compilation step can also be run with additional template banks by passing the argument:

--with-extra-bank=/home/soumi.de/projects/cbc/SEOBNRROM-proj/testbank_TF2v4ROM.hdf

Building Releases for CVMFS

To build a release of pycbc_inspiral for installation in CVMFS, run the script with the arguments:

pycbc_build_eah.sh --lalsuite-commit=a3a5a476d33f169b8749e2840c306a48df63c936 --pycbc-commit=b68832784969a47fe2658abffb3888ee06cd1be4 --with-extra-libs=file:///home/pycbc/build/composer_xe_2015.0.090.tar.gz --with-lal-data-path=/cvmfs/oasis.opensciencegrid.org/ligo/sw/pycbc/lalsuite-extra/current/share/lalsimulation

changing the --lalsuite-commit, --pycbc-commit, and --with-lal-data-path options to the values for the release.