Dependency flag soup needs some fiber

Description of why usage requirements and how flag soup is not good enough in the context of modules.

On "single installation prefix" (or "few installation prefix") systems such as Linux, BSD, or other Unix-like platforms where all dependencies tend to live in a few places, toolchains tend to search these places by default and simple flags such as -llibname to load the code (and the headers are already found implicitly).

In this case, the name of the libraries to load is all that is necessary and -l can be prefixed to the required library names and given to the toolchain to search for.

On Windows there is a mechanism to request a library be linked from the source code. This allows for dependencies to be used through just an include and library search path addition. The syntax for this is:

During linking, this will cause it to act as if libname.lib were specified as an additional library to link to. However, the search path must be provided by other mechanisms.

Some projects ended up writing their own "compiler" which wraps the base toolchain and adds the required flags as necessary while mediating between the consumer and the base toolchain.

The biggest issue with this approach is that it does not compose. These do not, generally, have a mechanism to stack them arbitrarily and instead expect to chain to the "lower" compiler by looking for it through the typical compiler names such as the CC environment variable or searching for "common" names for known compilers such as cc, gcc, or clang.

If the wrapper is a compiled tool, using it in a cross-compilation context can be difficult as it tends to be provided by the project itself. Even if not compiled, it may not understand how to incorporate flags such as --sysroot to re-root any paths it wants to pass to the underlying compiler.

Projects such as freetype and libpng provide executables which provide the flags necessary to use the library. These can work well, but have real questions on how to support cross-compilation if provided in a compiled form (most tend to be shell scripts in practice).

These tools essentially manually implement .pc file resolution manually (and, in fact, are sometimes implemented in terms of pkgconf today). However, there is a lack of consistency at times, sometimes various features are missing, and there’s no coordination with other ways of collecting dependency information.

The pkg-config tool and its associated .pc files are widely used to provide these flags. These files are widely available and used on Unix-like platforms. Some projects provide them on Windows as well, but they are not widely available enough to be relied upon.

There is some affordance for getting dependencies and making sure that there is logic for the different linking requirements for static versus shared libraries, but the query language is a bit coarse and requesting the static version of library A while getting the shared version of library B is not easy to guarantee if both libraries have both variants available. There is also no way to provide different compilation flags between shared and static (e.g., to control the export macros on platforms such as Windows or AIX).

.pc files also tend to prefer -L and -l pairs which can confound each other. This seems to be done typically to support static and shared usage with a single set of flags.

I’ll adopt some of CMake’s terminology for the rest of this paper to discuss more concrete usage and limitations of the model.

CMake has a number of usage requirements today, but this is certainly not an exhaustive list of what makes sense. In fact, I feel that projects should be able to define their own. Actually using them requires some level of support in tools, but the ability to express the information is, I feel, a good starting place for representing the information in the first place.

Further usage requirements I have wished for in the past:

Usage requirements can be used to represent all kinds of information. It is my feeling that projects should be able to declare and define their own usage requirements. Support for consuming particular usage requirements may differ as needed, though I would hope that compelling use cases become influential for supporting more usage requirements in more tools over time.

There may be checks for consistency across targets in a usage subgraph. For example, all code in a single program image must agree on the usage of the -fPIC flag when compiling its constituent objects.

Usage requirements allow querying the consuming environment in order to modify the set of provided information. However, this is not an unlimited capability. Of note, nothing that can be queried may be modified through a usage requirement. If this were possible, statements akin to "this statement is false" are possible. More concretely, imagine a target (eve) that is able to set some property on the consuming target (e.g., "set property use-alice to 0 on the consuming target"). Another target charlie uses alice if it has and use-alice set to a non-0 value. alice then has a usage requirement that consumers use eve. Now that the usage requirements of eve apply, it sets the use-alice property on charlie to 0. This is now a paradox that cannot be resolved because alice should now not be used based on the property settings.

Therefore, usage requirements may only rely on properties of the context that they may not affect. This includes support for "intersectional usage requirements" such as "if X and Y are used, add usage requirement A".

When a layer of "visibility" is layered on top of usage requirements, they may also be used to represent the properties of the target for itself as well as for consumers. CMake gives these names:

This additionally gives a classification of what flags are important for consuming modules.

Modules have a narrower compatibility contract than the ABI of libraries and (see [P2581R2]). This means that there may be more than one BMI per build graph so that different consumers can import a module. This is because BMIs may involve internal data structures that are sensitive to flags such as -std= where the selection of the TU importing affects what BMIs may actually be loaded safely. There are many such flags that affect this compatibility in practice.

This narrow compatibility means that when consuming module interface units from an external project cannot rely on any BMI artifacts to be provided by that project and must be prepared to build BMIs for these interface units as part of its own build graph.

In terms of usage requirements, this manifests itself as variants of the target provided by the project for each unique usage pattern which gain some usage requirements from the direct consumer. An optimization may be made to share BMI builds if it is found that different importers can share BMIs while constructing the build graph.

In order to do this effectively, it is necessary to know what flags that end up being given to the compiler can be skipped when compiling these BMIs (the "safe" option is to give all flags and every potential importer gets its own BMI as necessary).

Sanitizers are accessed via flags such as -fsanitize=…. These flags have both compiler and linker effects, namely generating the calls to extra checks in the code itself and adding the library providing the implementation of these extra checks to the linker.

In addition, for some santizers, if the code is loaded by an executable not using the sanitizer itself (e.g., a sanitized Python module loaded by a pre-existing interpreter), LD_PRELOAD is necessary to ensure that the sanitizer can install hooks early enough to provide their functionality.

While Boost is largely a header-only project, some of its libraries have additional compilation and linking requirements. For example, Boost.Archive can optionally use various compression libraries which must be provided if used. There is no mechanism for such optional dependencies when just providing flags a priori.

Boost also uses the autolinking mechanism to a note to the linker about libraries that are required based on the included headers. Projects which use Boost through more explicit means typically want to disable autolinking they prefer to provide unambiguous.

MPI implementations have historically provided "compiler wrappers" named mpicc and mpic++. These are intended to be used instead of using the platform toolchain directly. These wrappers essentially just pass the required compiler and linker flags as necessary.

Python is among one of the more complicated libraries to use due to the various ways in which it can be used and deployment strategies used for its distributions. Some consumers are creating modules, others are embedding the interpreter. Some module-using consumers additionally wish to limit their symbol usage using its "stable ABI" mechanism. These different consumption patterns are not easily expressible via a simple .pc file as the tools do not generally support a way to provide this kind of context (particularly for the "use the stable ABI as of version 3.x" case).

The expected usage pattern for Python is that the interpreter always provides the libpython ABI at runtime. In order to do this, libraries which expect to be used via an interpreter should not link to the libpython library. Modern linkers prefer to error if used symbols are not provided by a library given, so special flags (e.g., -Wl,--unresolved-symbols=ignore-in-shared-libs) are required to inform the toolchain to not care about missing symbols.

Things get more complicated when using these modules via further plugins. For example, a Python-using library may reasonably expect to be loaded by a Python interpreter (and so not link to its own libpython) while also wanting to be used from a plugin of another project that does not use Python itself. Here, the plugin must be told "you need to guarantee that libpython is provided" and so it must link to libpython even if it does not directly use any of its symbols itself. This can also require special flags (e.g., -Wl,--no-as-needed) to force the plugin to actually ensure that libpython is loaded.

On the whole, Qt is largely a normal dependency as far as direct usage is concerned. There are some extra tools Qt provides for various subsystems such as moc, rcc, and uic, but these are essentially additional toolchains which result in code generation that need to be used. Projects tend to handle these things explicitly and these use cases are not intended to be solved here.

Qt does get more complicated in two specific situations: use from executables on Windows and when using a static Qt. On Windows, Qt requires that its qtmain library is linked to, but only for executables in order to assist in creating a cross-platform main function. However, sometimes this is not wanted, so there should be a way to skip it (e.g., if some other framework is also assisting with main portability).