Format for describing dependencies of source files

Description of the format and its semantics.

For the information provided by the format, the following JSON Schema [JSON-Schema] may be used.

This format uses UTF-8 as a communication channel between a dependency scanning tool and a build tool, but filepath encodings are specific to the platform in use. Therefore, considerations for paths containing non-UTF-8 sequences must be made. However, the most common uses of paths and filenames are either valid UTF-8 sequences or may be unambiguously represented using UTF-8 (e.g., a platform using UTF-16 for its path APIs has a valid UTF-8 encoding), so requiring excessive obfuscation in all cases is unnecessary.

In order to store a non-UTF-8 sequence losslessly, there must be a way to encode the non-UTF-8 sequence into this format. There have been multiple ways utilized in the past for storing binary data into JSON including Base64 (as well as other related encodings such as Base85 or Base91), integer arrays, and going so far as to convert the entire file format over to binary (e.g., [BSON], [UBJSON], etc.). These encodings do not handle sequences of 16-bit data well either since endianness information is not stored in them. These solutions are over-pessimistic about the common case of valid UTF-8 paths used in this format so this encoding scheme uses UTF-8 wherever possible while dropping down to a less efficient encoding only when necessary.

The most general format for storing data is to use an array of integers tagged with the size of the values in memory. This is done by using an object with two required keys: data storing the integers representing the raw data and format describing the size of the integers in memory. Supported formats are raw8 and raw16. Other formats are ill-formed. There is an optional readable key which contains a string for communicating the contents in a human-readable format using UTF-8. The value of the readable key is purely information and does not have any normative meaning to the interpretation of the format.

Requirements for passing data to the platform’s APIs such as a terminating ASCII NUL byte or endianness are not included in the format. Using integer values outside of the range specified for the format is ill-formed.

If data are mostly UTF-8, to avoid encoding the entire path with the integer array, the percent encoding specified in [RFC3986]§2.1 may be used for these bytes to avoid encoding the entire path as an integer array. For example, a filepath with the 0xf5 byte in it may be encoded as "file/path/with/raw-%f5-byte". Due to this support, any literal % (ASCII 0x25) bytes must be encoded using percent encoding as well (%25).

When a path can be communicated as a series of UTF-8 codepoints, it should be done, but it is not required. That is, all fields which may contain binary data in the format are allowed to be unconditionally encoded using the most general format.

Filepaths may either be relative or absolute. It is preferred to use relative paths because the compilation may occur in a different working directory than the scanning tool uses. However, any paths which are not dependent on the working directory of the tool must be output using an absolute path. To this end, the dependency scanning tool must output its working directory in the work-directory key at the root of the document. The build tool may then construct the absolute paths as necessary.

For concrete examples where absolute paths may not be suitable:

The sources array allows for the dependency information of multiple files to be specified in a single file. The only restriction placed on this is that the input field across all sources entries be unique after decoding it as a filepath.

Each source represented in the sources array is a JSON object which only requires a single key, input. Its value is a datablock representing a filepath. Two optional keys exist to indicate the dependencies of the execution of the dependency scanning tool: the outputs array and the depends array. The outputs array in which each element is a filepath for files written by the dependency scanning tool due to the specified input file. The depends array in which each element is a filepath for files which affect the results of the run. For C++, this will generally paths be due to #include, but other mechanisms may be in effect.

The core of this specification is the future-compile and future-link keys on a sources object. They both use the same specification for their values, but contain the information for different phases of source compilation. These JSON objects have three optional keys, outputs, provides, and requires.

The outputs array contains filepaths which will be written to when the source is compiled. Only filepaths which are known to the dependency scanning tool that will be created at compile time should be included here.

The provides and requires arrays contain descriptions of modules that will be provided or required at compile time. Each item of these arrays is a JSON object with one required key, logical, and one optional key, filepath. Both of these key’s values are filepaths. The logical value is what build tools should use to discover the ordering among translation unit compilations. In C++, this will generally be the name of the module (including its partition, if any). The filepath should be provided only if the location of the module’s future on-disk representation is known when the dependency information is discovered.

Extensions may be added to the format using keys prefixed with an underscore (_). In addition, each source entry has an extensions key. Neither of these may be used to store semantically relevant information required to execute a correct build. Essentially, consumers of the format may ignore both _-prefixed keys and the extensions field and not suffer any loss of essential functionality.