Philip's coding lair

XMake: Combining the power of GIT submodules and CMake for library development

WARNING: This page is a work-in-progress

The problem

Imagine that you are developing a complex library (or application) using some other libraries (which also relies on external libraries themselves). Take the following image as an example:

A "complex" application dependency tree

How do you setup your repository? There's 2 main ways:

  1. the repository contains only the source code of your library/application, with a documentation stating how to get and compile the external libraries
  2. the repository contains the source code of your library/application and the source code of the external libraries

The first way is quite common, but isn't user-friendly at all: configuring, compiling (and most of the time installing) all the needed libraries manually can take a lot of time, especially if more than one computer are used for development.

The second way is better: everything is in the repository, and there's usually a way to compile all the stuff easily (like a script, a hand-crafted Visual Studio solution, ...). But it's often not trivial to maintain those compilation scripts.

Due to the complexity of some of my projects, and the need to be able to compile them easily on several platforms, I developed a personal standard to setup my projects, using a combination of GIT submodules and CMake.

With an incredible lack of inspiration, I called this system XMake, for eXtended cMake. It takes the form of a CMake file containing helper functions and a strict way to organize my GIT repositories.

Simple case: a standalone library

Let's first have a look at the simplest case: we develop a standalone library (say libA in the picture above). libA doesn't rely on any other library. The build system needs to be able to:

  1. Compile libA as a shared library (libA.so, libA.dll, libA.dylib) or as a framework (on MacOS X)
  2. If the license permits it, compile libA as a static library (libA.a, libA.lib)
  3. Eventually, compile and run the tests of libA
  4. Eventually, compile and run the examples and tools coming with libA
  5. Install libA system-wide, in a user-supplied location, or by default in the appropriate platform-specific location (/usr/local/, C:\Windows\System32\, ...)

Nothing special here. Now If you look at it from the perspective of the implementor of Application (still in the picture above), you may want to:

  1. Specify the folder into which the compiled library is written (for instance, to have application.exe and libA.dll in the same folder)
  2. Override the installation steps of libA

Our build system needs to take those informations into account. Pretty simple, no big deal here.

For the rest of the article, assume that the libA repository contains the following files:

Source tree of the 'libA' library

Second case: a library using another library

Take libB now: it relies on both libC and libD (whose structure is similar to libA). This is where GIT submodules are used: libC and libD are both submodules of libB. In order to configure and compile them alongside libB, our build system only needs to set their options to appropriate values, and include their CMake-related files. From that point, it looks like if the three libraries were in the same repository from the start!

Source tree of the 'libB' library

The two submodules are highlighted in green, and located in a folder named dependencies.

But wait! There is a problem: libE also relies on libC. Does it means that the source code of libC will be included (and worse: compiled) twice when compiling Application? Of course not.

First, to deal with several occurrences of a specific library (libC) in the final project, we'll follow this rule:

Rule #1: Only the submodules of the top-most project are checked out of their repository

This means that Application also has libC and libD as submodules (event if it doesn't use them itself). Those will be the only instances of these libraries in the whole Application folder.

Source tree of the whole application

The submodules are highlighted in green. You can see that (for instance) the libB submodule still has an empty libC folder (highlighted in blue). This is because it still reference libC as one of its submodule. But this particular instance of libC will never be checked out. We'll use the one in Application/dependencies/libC instead.

This leads us to our second rule:

Rule #2: Each project must allow to override the path to each of its submodules.

In our case: by default, libB will look in dependencies/libC for the headers of libC. Its CMake files will also compile it. But when instructed to use another path for libC, it will only look at this path to find the headers. It will not attempt to compile libC in any way. This is the job of Application in our case, since it is the one that really has a working copy of libC in its dependencies folder.

Compilation of the project

This system allows us to easily compile any project using the same sequence of commands, whatever the complexity of the library dependencies. Here they are, assuming the use of the command-line (Linux/MacOS X):

$ git clone ssh://path/to/the/git/repository/of/the/project/Application.git

(The project is retrieved from its repository)

$ cd Application
Application$ git submodule init
Application$ git submodule update

(The submodules are retrieved from their repositories)

Application$ mkdir build
Application$ cd build
build$ ccmake ..

(Configuration of the compilation)

build$ make