Introduction to Singularity
In this section we're going to be working with Singularity Community Edition (CE)
Wait, what is "Apptainer", and what is the difference between SingularityCE and Apptainer?
The Singularity project was split into multiple projects managed by different organizations since it was created in 2017.
In a nutshell:
- Greg Kurtzer founded the Singularity project while at the Lawrence Berkeley National Laboratory
- Kurtzer created Sylabs, a private company, around Singularity
- Kurtzer left Sylabs to focus on CIQ, another private company, and moved Singularity to HPCng (a Community Org)
- Sylabs forked Singularity for control and professionially licensed support creating Singularity Community Edition"
- HPCng gave the official project to Linux Foundation and renamed it "Apptainer"
- Apptainer is being marketed by CIQ
At the present time, Apptainer and Singularity CE have highly similar syntax and will run Singularity .sif images interoperably
Docker vs SingularityCE & Apptainer
Apptainer and SingularityCE are 100% compatible with Docker but they do have some distinct differences
Docker
Docker containers run as root
- This privilege is almost never supported by administrators of High Performance Computing (HPC) centers. Meaning Docker is not, and will likely never be, installed natively on your HPC cluster.
uses compressed layers to create one image
SingularityCE & Apptainer:
Same user and group identity inside as outside the container
User only has root privileges if elevated with sudo when the container is run
Can run and modify any existing Docker image
- These key differences allow Singularity to be installed on most HPC centers. Because you can run virtually all Docker containers in Singularity, you can effectively run Docker on an HPC.
SingularityCE Installation¶
Sylabs Singularity Community Edition (CE) homepage: https://www.sylabs.io/docs/
Apptainer Linux Foundation homepage: https://apptainer.org/
Conda¶
SingularityCE or Apptainer can both be installed from Conda:
Install Locally¶
To Install Singularity follow the instructions for your specific OS:
Module loading on HPC¶
If you are interested in working with SingularityCE on HPC, you may need to contact your systems administrator and request they install SingularityCE. Because SingularityCE ideally needs setuid, your admins may have some qualms about giving SingularityCE this privilege. If that is the case, you might consider forwarding this letter to your admins.
Most HPC systems are running Environment Modules with the simple command module.
You can check to see what is available:
If Singularity is listed as being installed, load a specific version, e.g.:
Install in CodeSpaces¶
Let's use Conda (or optionally, Mamba)
Singularity CLI¶
Singularity’s command line interface allows you to build and interact with containers transparently. You can run programs inside a container as if they were running on your host system. You can easily redirect IO, use pipes, pass arguments, and access files, sockets, and ports on the host system from within a container.
help¶
The help command gives an overview of Singularity options and
subcommands as follows:
$ singularity help pull
Pull an image from a URI
Usage:
singularity pull [pull options...] [output file] <URI>
Description:
The 'pull' command allows you to download or build a container from a given
URI. Supported URIs include:
library: Pull an image from the currently configured library
library://user/collection/container[:tag]
docker: Pull a Docker/OCI image from Docker Hub, or another OCI registry.
docker://user/image:tag
shub: Pull an image from Singularity Hub
shub://user/image:tag
oras: Pull a SIF image from an OCI registry that supports ORAS.
oras://registry/namespace/image:tag
http, https: Pull an image using the http(s?) protocol
https://library.sylabs.io/v1/imagefile/library/default/alpine:latest
Options:
--arch string architecture to pull from library (default "amd64")
--dir string download images to the specific directory
--disable-cache dont use cached images/blobs and dont create them
--docker-login login to a Docker Repository interactively
-F, --force overwrite an image file if it exists
-h, --help help for pull
--library string download images from the provided library
--no-cleanup do NOT clean up bundle after failed build, can be
helpful for debugging
--no-https use http instead of https for docker:// oras://
and library://<hostname>/... URIs
Examples:
From Sylabs cloud library
$ singularity pull alpine.sif library://alpine:latest
From Docker
$ singularity pull tensorflow.sif docker://tensorflow/tensorflow:latest
From Shub
$ singularity pull singularity-images.sif shub://vsoch/singularity-images
From supporting OCI registry (e.g. Azure Container Registry)
$ singularity pull image.sif oras://<username>.azurecr.io/namespace/image:tag
For additional help or support, please visit https://www.sylabs.io/docs/
search¶
Just like with Docker, you can search the Singularity container registries for images.
pull¶
The easiest way to use a Singularity is to pull an existing container
from one of the Registries.
Downloading pre-built images¶
You can use the pull command to download pre-built images from a
number of Container Registries, here we'll be focusing on the
Singularity-Hub or
DockerHub.
Container Registries:
library://- images hosted on Sylabs Clouddocker://- images hosted on Docker Hublocalimage://- images saved on your machineyum://- yum based systems such as CentOS and Scientific Linuxdebootstrap://- apt based systems such as Debian and Ubuntuarch://- Arch Linuxbusybox://- BusyBox-
zypper://- zypper based systems such as Suse and OpenSuse -
shub://- (archived) images hosted on Singularity Hub, no longer maintained
Pulling an image from Singularity Hub¶
Similar to previous example, in this example I am pulling a base Ubuntu container from Singularity-Hub:
$ singularity pull shub://singularityhub/ubuntu
WARNING: Authentication token file not found : Only pulls of public images will succeed
88.58 MiB / 88.58 MiB [===============================================================================================] 100.00% 31.86 MiB/s 2s
You can rename the container using the --name flag:
$ singularity pull --name ubuntu_test.simg shub://singularityhub/ubuntu
WARNING: Authentication token file not found : Only pulls of public images will succeed
88.58 MiB / 88.58 MiB [===============================================================================================] 100.00% 35.12 MiB/s 2s
The above command will save the alpine image from the Container Library
as alpine.sif
Pulling an image from Docker Hub¶
This example pulls an ubuntu:16.04 image from DockerHub and saves it
to the working directory.
$ singularity pull docker://ubuntu:20.04
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob 8f6b7df711c8 done
Copying blob 0703c52b8763 done
Copying blob 07304348ce1b done
Copying blob 4795dceb8869 done
Copying config 05ac933964 done
Writing manifest to image destination
Storing signatures
2020/03/09 16:14:12 info unpack layer: sha256:8f6b7df711c8a4733138390ff2aba1bfeb755bf4736c39c6e4858076c40fb5eb
2020/03/09 16:14:13 info unpack layer: sha256:0703c52b8763604318dcbb1730c82ad276a487335ecabde2f43f69a6222e8090
2020/03/09 16:14:13 info unpack layer: sha256:07304348ce1b6d24f136a3c4ebaa800297b804937a6942ce9e9fe0dac0b0ca74
2020/03/09 16:14:13 info unpack layer: sha256:4795dceb8869bdfa64f3742e1df492e6f31baf9cfc36f1a042a8f981607e99a2
INFO: Creating SIF file...
INFO: Build complete: ubuntu_20.04.sif
Warning
Pulling Docker images reduces reproducibility. If you were to pull a Docker image today and then wait six months and pull again, you are not guaranteed to get the same image. If any of the source layers has changed the image will be altered. If reproducibility is a priority for you, try building your images from the Container Library.
Pulling an image from Sylabs cloud library¶
Let’s use an easy example of alpine.sif image from the container
library
Tip
You can use singularity search <name> command to locate groups,
collections, and containers of interest on the Container Library
Interact with images¶
You can interact with images in several ways such as shell, exec and
run.
For these examples we will use a cowsay_latest.sif image that can be
pulled from the Docker Hub.
$ singularity pull docker://tswetnam/cowsay
INFO: Downloading library image
67.00 MiB / 67.00 MiB [=====================================================================================================] 100.00% 5.45 MiB/s 12s
WARNING: unable to verify container: cowsay_latest.sif
WARNING: Skipping container verification
$ sudo singularity run cowsay_latest.sif
________________________________________
/ Expect a letter from a friend who will \
\ ask a favor of you. /
----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
shell¶
The shell command allows you to spawn a new shell within your
container and interact with it as though it were a small virtual
machine.
The change in prompt indicates that you have entered the container (though you should not rely on that to determine whether you are in container or not).
Once inside of a Singularity container, you are the same user as you are on the host system.
Warning
shell also works with the library://, docker://, and shub:// URIs.
This creates an ephemeral container that disappears when the shell is
exited.
exec¶
The exec command allows you to execute a custom command within a
container by specifying the image file. For instance, to execute the
cowsay program within the cowsay_latest.sif container:
$ singularity exec cowsay_latest.sif cowsay container camp rocks
______________________
< container camp rocks >
----------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
exec also works with the library://, docker://, and shub:// URIs.
This creates an ephemeral container that executes a command and disappears.
run¶
Singularity containers contain
runscripts.
These are user defined scripts that define the actions a container
should perform when someone runs it. The runscript can be triggered with
the run command, or simply by calling the container as though it were
an executable.
singularity run lolcow_latest.sif
_________________________________________
/ You will remember, Watson, how the \
| dreadful business of the Abernetty |
| family was first brought to my notice |
| by the depth which the parsley had sunk |
| into the butter upon a hot day. |
| |
\ -- Sherlock Holmes /
-----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
inspect¶
The inspect command will provide information about labels, metadata, and environmental variables.
build¶
See Next Section for details of build