Deploying a macOS-based CI¶
This document contains instructions to build and deploy a new bare-OS CI for macOS. Instructions for deployment assume a freshly installed machine.
For sanity, don’t use an OS with lower version number than the macOS SDK code that will be installed (currently 10.9). There may be undesired consequences. You may use the latest OS version in case of doubt, but by default we recommend the one before the last stable version, for stability. So, if the current version is 10.14, a good base install would use 10.13.
Idiap has throttling rules that are typically applied to all machines in the lab network. To avoid issues for newly installed CI nodes, ensure you request throttling to be disabled for new CI machines.
Building the reference setup¶
Make sure the computer name is correctly set or execute the following on the command-line, as an admin user:
$ sudo scutil --get LocalHostName ... $ sudo scutil --get HostName ... $ sudo scutil --get ComputerName ... # if applicable, run the following commands $ sudo scutil --set LocalHostName "<hostname-without-domain-name>" $ sudo scutil --set HostName "<fully-qualified-domain-name>" $ sudo scutil --set ComputerName "<fully-qualified-domain-name>"
Disable all energy saving features. Go to “System Preferences” then “Energy Saver”:
Enable “Prevent computer from sleeping…”
Disable “Put hard disks to sleep when possible”
Leave “Wake for network access” enabled
You may leave the display on sleep to 10 minutes
To be able to send e-mails from the command-line (e.g., when completing cronjobs), via the Idiap SMTP, you will need to modify the postfix configuration:
Edit the file
/etc/postfix/main.cfto add a line stating
relayhost = [smtp.lab.idiap.ch](all e-mails should be routed by this SMTP host)
Edit the file
/etc/postfix/genericto add a line stating
email@example.com firstname.lastname@example.org(all e-mails leaving the lab infrastruture need to have
postmap /etc/postfix/genericas root (required to update the internal postfix aliases)
Create a new user (without administrative priviledges) called
gitlab. Choose a password to protect access to this user. In “Login Options”, select this user to auto-login, type its password to confirm
Enable SSH access to the machine by going on
Sharingand then selecting
Remote Login(for ssh) and
Screen Sharing(for remote desktop connections). Make sure only users on the
Administratorsgroup can access the machine.
Create as many
Administratorusers as required to manage the machine
Login as administrator of the machine (so, not on the gitlab account). As that user, run the
admin-install.shscript (after copying this repo from https://gitlab.idiap.ch/bob/bob.devtools via a zip file download):
$ cd $ curl -o bob.devtools-master.zip https://gitlab.idiap.ch/bob/bob.devtools/-/archive/master/bob.devtools-master.zip $ unzip ~/Downloads/bob.devtools-master.zip $ cd bob.devtools-master/doc/macos-ci-install $ sudo ./admin-install.sh 10.9 gitlab
Check that script for details on what is installed and the order. You may execute pieces of the script by hand if something fails. In that case, please investigate why it fails and properly fix the scripts so the next install runs more smoothly.
Check the maximum number of files that can be opened on a shell session with the command
launchctl limit maxfiles. If smaller than 4096, set the maximum number of open files to 4096 by creating the file
/Library/LaunchDaemons/limit.maxfiles.plistwith the following contents:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>limit.maxfiles</string> <key>ProgramArguments</key> <array> <string>launchctl</string> <string>limit</string> <string>maxfiles</string> <string>4096</string> <string>unlimited</string> </array> <key>RunAtLoad</key> <true/> <key>ServiceIPC</key> <false/> </dict> </plist>
At this occasion, verify if the kernel limits are not lower than this value using:
$ sysctl kern.maxfilesperproc 10240 #example output $ sysctl kern.maxfiles 12288 #example output
If that is the case (i.e., the values are lower than 4096), set those values so they are slightly higher than that new limit with
sudo sysctl -w kern.maxfilesperproc=10240and
sudo sysctl -w kern.maxfiles=12288respectively, for example.
Install oh-my-zsh for both the admin and gitlab users. Set ZSH theme “ys”. Add the following bits to
.zshrcto ensure completions work:
# Enables homebrew auto-completions for zsh (add: right at the top!) if type brew &>/dev/null; then FPATH=$(brew --prefix)/share/zsh/site-functions:$FPATH ZSH_DISABLE_COMPFIX="true" fi ... # plugins (add: just before sourcing oh-my-zsh) plugins=() plugins+=(docker) plugins+=(git) plugins+=(gitfast) plugins+=(python) plugins+=(themes) plugins+=(z) plugins+=(zsh-syntax-highlighting) plugins+=(history-substring-search)
Enter as gitlab user and install/configure the gitlab runner:
Configure the runner for shell executor, with local caching. As
gitlabuser, execute on the command-line:
# notice that running `brew services gitlab-runner start or restart` will # break the configuration of the service once more. Execute the following # to correct for it: $ /bin/bash <(curl -s https://gitlab.idiap.ch/bob/bob.devtools/raw/master/doc/macos-ci-install/reconfig-gitlab-runner.sh)
Once that is set, your runner configuration (
~/.gitlab-runner/config.toml) should look like this (remove comments if gitlab does not like them):
concurrent = 8 # set this to the number of cores available check_interval = 10 # do **not** leave this to zero [[runners]] name = "<runner-name>" # use a suggestive name output_limit = 102400 # this value is in kb, so we mean 100 mb url = "https://gitlab.idiap.ch" # this is our gitlab service token = "abcdefabcdefabcdefabcdefabcdef" # this is specific to the conn. executor = "shell" # select this builds_dir = "/Users/gitlab/builds" # set this or bugs occur cache_dir = "/Users/gitlab/caches" # this is optional, but desirable shell = "bash"
So conda works properly on a shared builder, as the
gitlabuser, make sure to create an empty, read-only file named
~/.conda/environments.txt. Failure to create this file and make it read-only to the gitlab user, will create a concurrence issue on the shared builder, w.r.t. to conda.
While at the gitlab user, install Docker for Mac. Ensure to set it up to start at login. In “Preferences > Filesystem Sharing”, ensure that /var/folders is included in the list (that is the default location for temporary files in macOS).
Once installed, go the the settings and in “General” uncheck the option “Use gRPC FUSE for file sharing”. At the time of writing (04.10.2021), the gRPC Fuse system does not work well with beat/beat.core> testing.
Import Idiap’s self-signed root certificate:
$ curl -o cert.crt -s https://pki.idiap.ch/download/Idiap_2016_Root-cacert.crt $ sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain cert.crt $ rm -f cert.crt
Reboot the machine. At this point, the gitlab user should be auto-logged and the runner process should be executing. Congratulations, you’re done!
Running regular updates¶
We recommend that the CI machine to have homebrew and installed pip packages
updated regularly (once a week). To do so, setup a cronjob like the following,
SHELL=/bin/bash MAILTO="email@example.com" 00 12 * * 0 /bin/bash <(curl -s https://gitlab.idiap.ch/bob/bob.devtools/raw/master/doc/macos-ci-install/update-ci.sh)
And one line the following for the
gitlab user, about 30 minutes later, to
give time for the machine updating to be performed. The second cronjob will
re-spawn the gitlab-runner, which may be necessary if it was updated on the
SHELL=/bin/bash MAILTO="firstname.lastname@example.org" 30 12 * * 0 /bin/bash <(curl -s https://gitlab.idiap.ch/bob/bob.devtools/raw/master/doc/macos-ci-install/reconfig-gitlab-runner.sh)