Format for describing dependencies of source files

The following changes have been made in response to feedback in SG15:

The following changes have been made in response to feedback in SG16:

Description of the format and its semantics.

For the information provided by the format, the following JSON Schema [JSON-Schema] may be used.

This format uses UTF-8 as a communication channel between a dependency scanning tool and a build tool, but filepath encodings are specific to the platform in use. Therefore, considerations for paths containing non-UTF-8 sequences must be made. However, the most common uses of paths and filenames are either valid UTF-8 sequences or may be unambiguously represented using UTF-8 (e.g., a platform using UTF-16 for its path APIs has a valid UTF-8 encoding), so requiring excessive obfuscation in all cases is unnecessary.

In order to store a non-UTF-8 sequence losslessly, there must be a way to encode the non-UTF-8 sequence into this format. There have been multiple ways utilized in the past for storing binary data into JSON including Base64 (as well as other related encodings such as Base85 or Base91), integer arrays, and going so far as to convert the entire file format over to binary (e.g., [BSON], [UBJSON], etc.). These encodings do not handle sequences of 16-bit data well either since endianness information is not stored in them. These solutions are over-pessimistic about the common case of valid UTF-8 paths used in this format so this encoding scheme uses UTF-8 wherever possible while dropping down to a less efficient encoding only when necessary.

Note that some JSON encoders and decoders will normalize Unicode sequences. Due to the presence of platforms where non-normalized sequences are valid paths, any such normalization logic should be disabled when interacting with this format.

The most general format for storing data is to use an array of integers tagged with the size of the values in memory. This is done by using an object with two required keys: code-units storing the integers representing the raw data and format describing the size of the integers in memory. Supported formats are raw8 and raw16. Other formats are ill-formed. There is an optional readable-name key which contains a string for communicating the contents in a human-readable format using UTF-8. The value of the readable-name key is purely information and does not have any normative meaning to the interpretation of the format.

Requirements for passing data to the platform’s APIs such as a terminating ASCII NUL byte or endianness are not included in the format. Using integer values outside of the range specified for the format is ill-formed.

When a path can be communicated as a series of UTF-8 codepoints, it should be done, but it is not required. That is, all fields which may contain binary data in the format are allowed to be unconditionally encoded using the most general format.

Filepaths may either be relative or absolute. It is preferred to use relative paths because the compilation may occur in a different working directory than the scanning tool uses. However, any paths which are not dependent on the working directory of the tool must be output using an absolute path. To this end, the dependency scanning tool must output its working directory in the work-directory key at the root of the document. The build tool may then construct the absolute paths as necessary.

For concrete examples where absolute paths may not be suitable:

The sources array allows for the dependency information of multiple files to be specified in a single file. The only restriction placed on this is that the input field across all sources entries be unique after decoding it as a filepath.

Each source represented in the sources array is a JSON object which only requires a single key, input. Its value is a datablock representing a filepath. Two optional keys exist to indicate the dependencies of the execution of the dependency scanning tool: the outputs array and the depends array. The outputs array in which each element is a filepath for files written by the dependency scanning tool due to the specified input file. The depends array in which each element is a filepath for files which affect the results of the run. For C++, this will generally paths be due to #include, but other mechanisms may be in effect.

The core of this specification is the future-compile key on a sources object. They both use the same specification for their values, but contain the information for different phases of source compilation. These JSON objects have three optional keys, outputs, provides, and requires.

The outputs array contains filepaths which will be written to when the source is compiled. Only filepaths which are known to the dependency scanning tool that will be created at compile time should be included here.

The provides and requires arrays contain descriptions of modules that will be provided or required at compile time. Each item of these arrays is a JSON object with one required key, logical-name, and two optional keys: compiled-module-path and source-path. All of these key’s values are filepaths. The logical-name value is what build tools should use to discover the ordering among translation unit compilations. In C++, this will generally be the name of the module (including its partition, if any) as included in the source. The compiled-module-path should be provided only if the location of the module’s future on-disk representation is known when the dependency information is discovered. The source-path is the path to the main source of the module. This is intended to be used to communicate the on-disk header for a header-unit import when it is known.

Extensions may be added to the format using keys prefixed with an underscore (_) followed by a vendor-specific string followed by another underscore. None of these may be used to store semantically relevant information required to execute a correct build. Essentially, consumers of the format may ignore all _-prefixed keys and not suffer any loss of essential functionality.