Format for describing dependencies of source files

Added:

The following changes have been made in response to feedback in SG15:

The following changes have been made in response to feedback in SG16:

Description of the format and its semantics.

Historically, dependency information of a build rule has been handled by Makefile snippets. An example of this is:

This states that the artifact of the build rule is output and files input_a, input_b, and input_c were read during its creation. This allows the build system to know that if any of the listed input_* files changes, the rule for output needs to be brought up-to-date as well.

This works decently well for the kinds of dependencies that have occurred in C++ to date, namely header includes. This is because these dependencies can be discovered while executing the rule associated with output.

The issue that arises with the Makefile design is that modules are a new kind of dependency that cannot be represented in declarative Makefile syntax. For example, GCC outputs variable modifications (CXX_MODULES+=…) into these snippets which is commonly not supported by the consuming tools. In addition, because these dependencies must be discovered before the compilation rule is executed, there would need to be one rule that writes dependency information for another.

As an example of the restrictions placed on these Makefile snippets, the ninja [ninja] build tool requires that output be the same for the rule which wrote out the dependency snippet and that no other outputs are mentioned. No other Makefile syntax is supported (variables, adding rules, special variables, macro expansions, etc.). This is because ninja is reading these for just the dependency information.

For the information provided by the format, the following JSON Schema [JSON-Schema] may be used.

This format uses UTF-8 as a communication channel between a dependency scanning tool and a build tool, but filepath encodings are specific to the platform which means considerations for paths containing non-UTF-8 sequences must be made. However, the most common uses of paths and filenames are either valid UTF-8 sequences or may be unambiguously represented using UTF-8 (e.g., a platform using UTF-16 for its path APIs has a valid UTF-8 encoding), so requiring excessive obfuscation in all cases is unnecessary.

In order to store a non-UTF-8 sequence losslessly, there must be a way to encode the non-UTF-8 sequence into this format. There have been multiple ways utilized in the past for storing binary data into JSON including Base64 (as well as other related encodings such as Base85 or Base91), integer arrays, and going so far as to convert the entire file format over to binary (e.g., [BSON], [UBJSON], etc.). These encodings do not handle sequences of 16-bit data well since endianness information is not stored in them. These solutions are over-pessimistic about the common case of valid UTF-8 paths used so this encoding scheme uses UTF-8 wherever possible while dropping down to a less efficient encoding only when necessary.

Note that some JSON encoders and decoders will normalize Unicode sequences. Due to the presence of platforms where non-normalized sequences are valid paths, any such normalization logic should be disabled when interacting with this format.

The most general format for storing data is to use an array of integers tagged with the size of the values in memory. This is done by using an object with two required properties: code-units storing the unsigned integers representing the raw data and format describing the bit size of the integers in memory. Supported formats are raw8 and raw16. Other formats are ill-formed. There is an optional readable-name property which contains a string for communicating the contents in a human-readable format using UTF-8. The value of the readable-name property is purely information and does not have any normative meaning to the interpretation of the format.

Requirements for passing data to the platform’s APIs such as a terminating ASCII NUL byte or endianness are not included in the format. Using integer values outside of the range specified for the format is ill-formed.

When a path can be communicated as a series of UTF-8 codepoints, it should be done, but it is not required. That is, all fields which may contain binary data in the format are allowed to be unconditionally encoded using the most general format.

Filepaths may either be relative or absolute. To this end, the dependency scanning tool must output its working directory in the work-directory property at the root of the document. The build tool may then construct the absolute paths as necessary.

For concrete examples where absolute paths may not be suitable:

The rules array allows for the dependency information of multiple rules to be specified in a single file.

The only restriction on the contents of the collective set of rules objects is that the set of all outputs in each object and future-compile object must be unique. This is because if they are not unique, there are outputs which have multiple rules that write to them, which is undefined behavior.

Each rule represented in the rules array is a JSON object which has a single required property, inputs. The value of this property is an array of datablock entries representing a set of filepaths that are read directly by the execution of this rule. A rule must have at least one input because otherwise the rule is idempotent and never needs dependency information to be discovered. Two optional properties exist to indicate the dependencies of the execution of the dependency scanning tool itself: the outputs array and the depends array. The depends value is an array in which each element is a filepath for files that affect the results of the run. For C++, this will generally be paths due to #include, but other mechanisms may be in effect.

The core of this specification is the future-compile property on a rules object. future-compile objects have three optional properties, outputs, provides, and requires.

The outputs array contains filepaths which will be written to when the source is compiled. The provides and requires arrays contain descriptions of modules that. The provides array is for modules that the inputs will produce and the requires array is for modules that the inputs require. Each item of these arrays is a JSON object with one required property, logical-name, and two optional properties: compiled-module-path and source-path. All of these property’s values are filepaths. The logical-name value is what build tools must use to discover the ordering among translation unit compilations. In C++, this will generally be the name of the module (including its partition, if any) as included in the source. The compiled-module-path should be provided only if the location of the module’s artifact is known when the dependency information is discovered. The source-path is the path to the main source of the module. This is intended to be used to communicate the location of a header for a header-unit import or a module interface unit for a C++20 module when it is known.

Vendor extensions may be added to the format using properties prefixed with an underscore (_) followed by a vendor-specific string followed by another underscore. None of these may be used to store semantically relevant information required to execute a correct build. Consumers must be able to ignore all _-prefixed properties and not suffer any loss of essential functionality.