Deploying a Linux-based CI

This document contains instructions to build and deploy a new bare-OS CI for Linux. Instructions for deployment assume a freshly installed machine, with Idiap’s latest Debian distribution running. Our builds use Docker images. We also configure docker-in-docker to enable to run docker builds (and other tests) within docker images.

Warning

Idiap has throttling rules that are typically applied to all machines in the lab network. To avoid issues for newly installed CI nodes, ensure you request throttling to be disabled for new CI machines.

Docker and Gitlab-runner setup

Base docker installation: https://secure.idiap.ch/intranet/system/software/docker

Ensure to add/configure for auto-loading the overlay kernel module in /etc/modules. Then update/create /etc/docker/daemon.json to contain the entry "storage-driver": "overlay2".

To ensure that you can control memory and CPU usage for launched docker containers, you’ll need to enable “cgroups” on your machine. In essence, change /etc/default/grub to contain the line GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1". Then, re-run update-grub after such change.

To install docker at Idiap, you also need to follow the security guidelines.

If you do not follow these guidelines, the machine will not be accessible from outside via the login gateway, as the default docker installation conflicts with Idiap’s internal setup. You may also find other network connectivity issues.

Also, you want to place /var/lib/docker on a fast disk. Normally, we have a scratch partition for this. Follow the instructions at https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux for this step:

  $ mkdir /scratch/docker
  $ chmod g-rw,o-rw /scratch/docker
  $ service docker stop
  $ rsync -aqxP /var/lib/docker/ /scratch/docker
  $ rm -rf /var/lib/docker
  $ vim /etc/docker/daemon.json  # add data-root -> /scratch/docker
  $ service docker start


Reboot the machine to ensure everything works fine.

Hosts section

We re-direct calls to www.idiap.ch to our internal server, for speed. Just add this to /etc/hosts:

$ echo "" >> /etc/hosts
$ echo "#We fake www.idiap.ch to keep things internal" >> /etc/hosts
$ echo "What is the internal server IPv4 address?"
$ read ipv4add
$ echo "${ipv4add} www.idiap.ch" >> /etc/hosts
$ echo "What is the internal server IPv6 address?"
$ read ipv6add
$ echo "${ipv6add} www.idiap.ch" >> /etc/hosts

Note

You should obtain the values of the internal IPv4 and IPv6 addresses from inside the Idiap network. We cannot replicate them in this manual for security reasons.

Gitlab runner configuration

Once that is setup, install gitlab-runner from https://docs.gitlab.com/runner/install/linux-repository.html, and then register it https://docs.gitlab.com/runner/register/.

We are currently using this kind of configuration (notice you need to replace the values of <internal.ipv4.address> and <token> on the template below):

concurrent = 20
check_interval = 10

[session_server]
  session_timeout = 1800

[[runners]]
  name = "<machine-name>"
  output_limit = 102400
  url = "https://gitlab.idiap.ch/"
  token = "<token>"
  executor = "shell"
  shell = "bash"
  builds_dir = "/scratch/builds"
  cache_dir = "/scratch/cache"

[[runners]]
  name = "bp-srv01"
  output_limit = 102400
  url = "https://gitlab.idiap.ch/"
  token = "<token>"
  executor = "docker"
  builds_dir = "/scratch/builds"
  cache_dir = "/scratch/cache"
  [runners.docker]
    tls_verify = false
    image = "continuumio/conda-concourse-ci"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/scratch/cache"]
    shm_size = 0
    extra_hosts = ["www.idiap.ch:<internal.ipv4.address>"]
  [runners.cache]
    Insecure = false

Note

You must make both /scratch/builds and /scratch/cache owned by the user running the gitlab-runner process. Typically, it is gitlab-runner. These commands, in this case, are in order to complete the setup:

.. code-block:: sh

$ mkdir /scratch/builds $ chown gitlab-runner:gitlab-runner /scratch/builds $ mkdir /scratch/cache $ chown gitlab-runner:gitlab-runner /scratch/cache

Once the configuration is done, add the gitlab-runner user to the docker group so it can do tasks related to Docker (images pulling, python client call, etc.):

.. code-block:: sh

$ usermod -a -G docker gitlab-runner

Access to Idiap’s docker registry

If you want the Idiap docker registry (docker.idiap.ch) to be accessible from the shell executors, you must also ensure Idiap registry certificates are available on the host. You may copy the contents of docker.idiap.ch directory in this documentation set for that purpose, to the directory /etc/docker/certs.d. Then, ensure to use something like: docker login -u gitlab-ci-token -p $CI_JOB_TOKEN docker.idiap.ch on the (global) before_script phase in jobs requiring access to the registry.

Repository cloning from CI jobs

If you’d like to allow the (shell-based) runner to clone repositories other than the one being built, you need to ensure the following is configured at ~/.ssh/config of the user running the gitlab-runner process (typically gitlab-runner):

Host gitlab.idiap.ch
  ForwardX11 no
  ForwardX11Trusted no
  ForwardAgent yes
  StrictHostKeyChecking no
  ControlMaster auto
  ControlPath /tmp/%r@%h-%p
  ControlPersist 600
  Compression yes

Make sure to use an “https” git-clone strategy in your recipes.

Git

The version of git (2.11) shipped with Debian Stretch (9.x) is broken. The git-clean command does not honour the --exclude passed via the command-line. I advise you install the most recent version from debian backports by enabling this repository or configuring it with instructions from https://backports.debian.org. To install the newest git version, after an apt update, just run the following command as root:

$ apt-get -t stretch-backports install "git" "git-lfs"

X11

Some utilities such as dot (graphviz) require X11 support. If you intend to make use of the shell builder and graphviz, you must install basic X11 support. Just run the following command as root to fix this:

$ apt install libxrender1 libxext6

Crontabs

# crontab -l
MAILTO=""
0 12 * * SUN /usr/share/gitlab-runner/clear-docker-cache

Conda and shared builds

To avoid problems with conda and using shared builders, consider creating the directory ~gitlab-runner/.conda and touching the file environments.txt in that directory, setting a mode of 444 (i.e., make it read-only).

Extra packages

List of extra packages to ensure are installed on the shell environment:

  • rsync

  • libgl1

libgl1 is required to run the beat/beat.editor> tests. While the offscreen plugin is used and therefor no X11 server is required, the libraries still needs the OpenGL symbols.

Locale

Ensure to set the default locale as C.UTF-8 by re-running dpkg-reconfigure locales. The click (python) package requires it.