Structured Response Files

Document #:	ISO/IEC/JTC1/SC22/WG21/P3051R2
Date:	2024-06-29
Audience:	EWG, LEWG
Authors:	René Ferdinand Rivera Morell
Reply-to:	grafikrobot@gmail.com
Copyright:	Copyright 2024 René Ferdinand Rivera Morell, Creative Commons Attribution 4.0 International License (CC BY 4.0)

1. Abstract

This aims to define a standard structured response file format that can become the best way to communicate compiling C++.

2. Revision History

2.1. Revision 2 (June 2024)

Only allow one of arguments or options fields at a time.
Define the concept of "processing" arguments and options.
Rename the wording section from "Structured Options" to "Structured Parameters" to avoid conflagration withe options.
Change options to a a JSON object moving to a unique structured option model.
Add design considerations section to track design choice rationale.
Remove flag options as they are not needed as they can be expressed more naturally as a boolean value.

2.2. Revision 1 (May 2024)

Specify that options use abstract features instead of literal arguments, per SG15 feedback. Addition of wording.

2.3. Revision 0 (December 2023)

Initial text.

3. Motivation

A key aspect of inter-operation between tools in the ecosystem is having a common language to express tool commands, i.e. in compiler drivers, that can be understood and/or translated between different tools and platforms.

Currently tools use differing, but related, ways for users (and other tools) to specify the set of options to "toolsets" (compiler drivers, linkers, etc). While there are some commonalities in how those options are specified as "configuration response files" containing bare options, there are sufficient differences to hinder general inter-operation.

4. Scope

This proposal aims to specify a method for tools to specify arguments to other tools in a consistent and flexible manner. As such what it does and doesn’t aim to accomplish:

It does not aim to remove current arguments handling. It does allow for incrementally adoption of an alternative that facilitates common tool arguments.
It does not specify any particular options to replace existing options (except the ones to indicate the new response file). It does aim to specify an additional alternative option style that reduces the parsing complexity, and perhaps ambiguities, in tools.

5. Current Response Files

Current response files commonly contain "unstructured" sequence of command line arguments. Some also allow recursive inclusion and expansion of additional response files. Below are a summary of the syntax, capabilities, and restrictions of some of the compiler drivers.

5.1. Clang

Use of @filename argument.
Use of --config=filename argument.
References to other response files allowed.

Example response file:

# Several options on line
-c --target=x86_64-unknown-linux-gnu

# Long option split between lines
-I/usr/lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/C++/5.4.0

# other config files may be included
@linux.options

Source: Clang Compiler User’s Manual ^[1]

5.1.1. GNU Compiler Collection, GCC

Use of @filename argument.
References to other response files allowed.

Example response file:

-o "hello" -Wl,--start-group "bin/hello.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -fPIC -g

Source: GCC Documentation ^[2]

5.2. Intel® oneAPI DPC++/C++

Use of @filename argument.
References to other response files disallowed.
Platform specific option syntax.
# prefixed line comments.

Example response file for "Linux":

# compile with these options
  -O0
# end of response file

Example response file for "Windows":

# compile with these options
  /0d
# end of response file

Source: Intel® oneAPI DPC++/C++ Compiler Developer Guide and Reference ^[3]

5.3. NVIDIA CUDA Compiler Driver NVCC

Use of --options-file filename,…, or -optf filename,….

-O0

Source: NVIDIA CUDA Compiler Driver NVCC Documentation ^[4]

5.4. Microsoft Visual C++

Use of @filename argument.
References to other response files disallowed.

Example response file:

"hello.cpp" -c -Fo"bin\hello.obj" -TP /wd4675 /EHs /GR /Zc:throwingNew /Z7 /Od /Ob0 /W3 /MDd /Zc:forScope /Zc:wchar_t /Zc:inline /favor:blend

Source: Microsoft C++, C, and Assembler documentation ^[5]

5.5. Other

Edison Design Group C++ Front End (edgcpfe): Does not support response configuration files. ^[6]
Embarcadero C++ Builder: Supports at least @filename option, with bare arguments syntax. ^[7]
IBM Open XL C/C++ for AIX 17.1.0: Supports the Clang ^[1] --config option.
IBM Open XL C/C++ for Linux on Power 17.1.1: Has migrated to using the Clang ^[1] toolchain and supports the same options.
IBM Open XL C/C++ and XL C/C++ for z/OS: Supports the Clang ^[1] --config option.
HPE Cray Programming Environment (CPE): Support depends on the platform complier environment.
NVIDIA HPC C++ (NVC++): Does not support response configuration files. ^[8]
Oracle® Developer Studio 12.6: Supports a single global options configuration file, with bare arguments syntax.
Python argparse module: Supports arbitrarily character prefixed response files. Where the files, by default, contain an argument per line. ^[9]

6. Design

Abstractly response files are files in the operating system storage that contain arguments and options in addition to the ones given directly in the tool invocation. For this design we are going to refer to two different types of ways to pass the information to the tools:

Arguments: Arguments use the syntax that one would specify directly in the command line as a user. This would be things like the -O1 optimization flag argument.
Options: Options are the conceptual flag option that the tool understands that does not necessarily follow the same syntax as the flags specify in command line arguments.

Using those two distinct definitions allows us to specify them differently in the response file. Using arguments we will follow the same existing command line syntax. Keeping a form of compatibility with existing tools. But we would then be restricted to that syntax. Using options we can use a definition that fits best with a structured data definition.

Last consideration is the choice of structured data format we will have in the response files. Keeping with previous work and practice we will use JSON text as that format. ^[10] ^[11]

With that context, here are two example structured response files:

Simple Arguments:

{
  "arguments": [
    "-fPIC",
    "-O0",
    "-fno-inline",
    "-Wall",
    "-Werror",
    "-g",
    "-I\"util/include\"",
    "-c"
  ]
}

Structured Options:

{
  "options": {
    "position-independent-code": true,
    "optimization": { "level": "off" },
    "inline": false,
    "warnings": { "level": "all", "as-errors": true },
    "debug-symbols": "on",
    "include": [ "util/include" ],
    "action": "compile"
  }
}

ℹ	The names and values in the `options` example is for exposition only. It may not reflect the actual names and values as proposed, now or in the future.

The simple arguments example shows specifying an "arguments" key with an array of values corresponding to the regular command line arguments. This mirrors what one would see in a JSON compilation database ^[11]. This style has some advantages:

There is a direct correlation with the JSON compilation database format which some tools support. Which would mean that they already have code to deal with the JSON loading and understanding of it.
There’s a direct mapping for regular command line arguments. Hence it will be less effort to support this style for tools.

And there are some disadvantages:

The parsing of the arguments still has to happen to get at the option and value.
It is subject to the same limitations of regular command line arguments. Like complexity and ambiguities of command line syntax.

The structured options example shows specifying an "options" key with a dictionary of option names and option values. Where the option objects contain an option name and structured value. Some advantages of this style of structured data are:

The option names do not indicate a particular option prefix (i.e. -, --, /, etc) making it possible to use tool agnostic common names.
The ability to use arrays, or possibly objects, for the option values allows for logical groupings avoiding extra tracking of such as is present in command line parsing of options.
The names can be more descriptive, and hence easier for people to understand.

Some possible disadvantages:

Tools will need an alternative path to understand the new options. Although hopefully this is balanced by the more direct availability of the values.
If this style is also to be supported in the JSON compile database format it means more work to accomplish that. But again, the hope is that there is an easier mapping from internal structures to this format.

The design we are proposing has the following key points:

The format of the file is well formed JSON text.
The top level of that is a JSON object with one of arguments or options fields.
The addition of one command line option to specify the structured response file.
The arguments field has a single array value with string values.
The arguments values are single strings containing the same options as would be specified in the command line.
The arguments values can contain the option for other structure response files. Which will be recursively inserted at the location of the option.
The options field has a dictionaty of kesy and values for structured options.
The options values will follow an option specific schema.
The arguments are not specified, and as such are implementation defined.
The options are specified with both a schema and semantics for each.

6.1. Command Line

We propose to add a single new command line option as a requirement to implementing this capability:

$ tool --std-param=file

Or:

$ tool -std-param:file

The std-param command line option, which can be repeated, will read the indicated file and parse the JSON text contents to configure the tool as needed.

6.2. File Format

The response file is a valid JSON text file with a single JSON object as the root object. There are two mutually exclusive fields to specify the command information: arguments or options.

There are two additional, optional, fields: $schema and version. The $schema field points at the released JSON Schema. ^[12] The version field indicated the response format of the file. The version number follows those specified in the capability introspection version numbering and semantics.

The arguments field specifies a single array value of strings. Each string array entry is a command line argument to be used directly by the tool. The specific syntax of the arguments is up to the specific tool. For example a compile invocation for GCC, and compatible compiler front-ends, would look like:

{
  "$schema": "https://isocpp.org/release/schema/std_param-1.0.0.json",
  "version": "1",
  "arguments": ["-fPIC", "-O0", "-fno-inline", "-Wall", "-Werror", "-g", "-I\"util/include\"", "-c" ]
}

You can also include --std-param=file options in the list of arguments to include the arguments that are referenced in another response file, and so on. For example, given a common.json response file as such:

{
  "$schema": "https://isocpp.org/release/schema/std_param-1.0.0.json",
  "version": "1",
  "arguments": ["-fPIC", "-O0", "-fno-inline", "-Wall", "-Werror", "-g", "-I\"util/include\"", "-c" ]
}

One can refer to it in a main response file that compiles a C++ source file:

{
  "$schema": "https://isocpp.org/release/schema/std_param-1.0.0.json",
  "version": "1",
  "arguments": [ "--std-param=common.json", "main.cpp", "-o", "main.o" ]
}

The effect is that the options in common.json are inserted in the arguments array at the location of the --std-param=common.json argument.

The options field specifies a dictionary value of option keys and values. An option value can be any value that follows the schema for the collection of specified options.

When compared to having the arguments field there are more constraints to the syntax of the options:

The option names do not contain prefix characters (i.e. the --, -, /, etc) and without the value separator (i.e. =, :, etc).

Additional response files can be inherited by specifying a { "std.param": { "pre": [ "file.json" ] } } option. Or they can be appended by specifying a { "std.param": { "post": [ "file.json" ] } } option.

Do note though that even though the option names have a specific naming format they are still defined by the tool. The goal of having the restrictions on the option names is to make it possible in the future to specify tool agnostic options to facilitate general interop. But that is a subject for future proposals.

‼	Tools need to support being able to either refer to an `options` style response file from an `arguments` style response file, and conversely.

6.3. Flags or Names

There is a question as to wether it’s better to use command line flags (for example W, o, I, etc) or non-command line names (for example warning, output, include, etc) in the options field. We will call the former the "Flags" choice, and the latter the "Names" choice. Each would mean:

Flags: The keys would be direct correlation to the specific tool, i.e. compiler, command line options. For example a I field name would match the -I.
Names: The keys would be symbolic names correlating to a concept that may map to one or more current command line options. For example an include field name would map to one or more -I command line options. But it would also map to one or more new --include command line options.

Given those definitions we can consider the pros and cons of each method.

	Flags	Names
Pros	Small effort for compilers to implement. No additional CLI parsing to implement.	Common names and meanings across different tools. Names could be used in other formats, like CPS.
Cons	Continues the increased implementation burden on build systems, etc, to manage different options per compiler, etc.	Possibly complex implementation to map from names to internal data.

Flags

Names

Pros

Small effort for compilers to implement.
No additional CLI parsing to implement.

Common names and meanings across different tools.
Names could be used in other formats, like CPS.

Cons

Continues the increased implementation burden on build systems, etc, to manage different options per compiler, etc.

Possibly complex implementation to map from names to internal data.

ℹ	One key consideration is that this proposal does not prevent choosing either flags or names as the `options` fields. Specifying the fields can be accomplished in a further proposal that specifies either flags or names as common specified syntax.

7. Considerations

Specify options fields as "Flags" or "Names"?

The discussions in WG21/SG15 concluded that it’s preferable to use "names" for the keys. This allows, as outlined in previous sections, a dual operation. The "arguments" for some form of backward compatibility. And the "options" for the modern structure tool invocation.

Specify the options in this proposal or a separate proposal?

We decided early on that it would be best to separate out specifying additional options in further proposals. A key advantage of the separation is that groups of such option definitions can indicate a capability. And hence other tools can introspect which option groups are supported.

Structured options as an array.

In previous revisions of this feature we used a JSON array for the options. While that approach facilitated translation ability to traditional command parameters it did it at the expense of increased complexity for the handling of the ordinality of the structured options.

Structured options need merge rules.

We recognize that having structured options as a dictionary means that when multiple structured arguments files are present we need to define how multiple specifications for the options need to operate. But we also recognize that such rules can´t be defined in a vacuum. Future papers that define options will define such rules as they are needed. And while categories of such rules will be shared across different specific options the effect they have will be option specific.

Why use a new option (--std-param=file) instead of existing response file methods like @file?

Implementing support in tools for using the same response file options would:

Mean that the option to add the structured response files would vary from tool to tool as some use different styles for specifying the file.
It would be harder to implement as it would require inspecting the file content to determine the parsing method needed.

Why have an arguments field instead of just using the options field?

Having an arguments field has a couple of benefits:

It makes it easier for tools to immediately support use of this format as they can directly inject the arguments into their existing command line argument parsing.
Makes it easier for tools that already support compile_commands.json to produce or consume structured response files as they are the same format for the arguments field in both.

8. Polls

8.1. SG15: P3051R1 (2024-06-23)

Structured options should be an unordered dictionary instead of an ordered list.

SF	F	N	A	SA
3	5	0	0	0

std.param should look like: "param": {"pre": [], "post": []}

Unanimous Consent

Forward the initial set of Ecosystem IS papers (P2656, P3342, P3051, and P2717), with the agreed changes, to EWG with the request to create a work item.

SF	F	N	A	SA
8	0	0	0	0

8.2. SG15: P3051R0 (2024-03-19)

SG15 wants to pursue defining a common response file format starting with supporting utf8 simple command lines, accepting the limitations for non-unicode arguments for the first version.

SF	F	N	A	SA
6	6	0	0	0

Result: Consensus

SG15 wants to persue defining common structured tool configuration arguments as part of the response file format.

SF	F	N	A	SA
6	3	3	0	0

Result: Consensus

9. Wording

Wording is relative to ecosystem-is/56aeadb. ^[13]

9.1. Specification: Structured Parameters

Insert clause after Introspection [intspct].

9.1.1. Structured Parameters [strctparam]

9.1.1.1. Preamble [strctparam.pre]

This clause describes options, output, and formats that control the behavior of applications through the specification of arguments and options from structured parameters.

This clause specifies the std.strctparam capability ([intspct.cap]) version 1.0.0.

An application can implement this capability.

An application that implements the std.strctparam capability shall include the std.strctparam field and version value in the introspection JSON text output ([intspct.schema.cap]).

9.1.1.2. Overview [strctparam.overview]

application [ std-strctparam-input file ]

9.1.1.3. Input Option [strctparam.input]

std-strctparam-input: The pathname of a file to read the structured parameters from. The option is specified as --std-param=file or -std-param:file. If file is ‘-’, the standard input shall be used.

9.1.1.4. Files [strctparam.file]

An application shall read a valid JSON text file that conforms to the structured parameters schema (strctparam.schema).

An application shall interpret the information in the file as if the options and arguments in the file occur in the same position as the std-strctparam-input parameter of the list of parameters given to the application. Given either directly as part of the application command line or as part of the arguments field (strctparam.schema.args).

An application processes arguments and options by applying the semantics of the arguments and options to change the state of the application that is relevant to the arguments and options.

An application shall process arguments in the file in the order that they appear.

An application shall process options in the file as if the a new modified state replaces the current state.

9.1.1.5. Schema [strctparam.schema]

A structured parameters JSON text file shall contain one structured parameters JSON object (strctparam.schema.param).

9.1.1.6. Structured Parameters Object [strctparam.schema.param]

The structured parameters object is the root JSON object of the structured parameters JSON text.

A structured parameters object can have the following fields.

A structured parameters object shall have only one of arguments and options fields.

9.1.1.7. JSON Schema Field [strctparam.schema.schema]

Name: $schema
Type: string
Value: The value shall be a reference to a JSON Schema specification.
Description: A structured parameters object can contain this field. If a structured parameters object does not contain this field the value shall be a reference to the JSON Schema corresponding to the current edition of this standard (strctparamjschm).

9.1.1.8. Version Field [strctparam.schema.ver]

Name: version
Type: string
Value: 1 or 1.0 or 1.0.0
Description: The version field indicates the version of the structured parameters represented in the contents of the JSON text. If a structured parameters object does not contain this field the value shall be '1.0.0'.

9.1.1.9. Arguments Field [strctparam.schema.args]

Name: arguments
Type: array
Value: The value shall be a JSON array. The items in the array shall be of JSON string types.
Description: The arguments field specifies items to be interpreted directly as if they occur in the command line of the program.

The application shall process the items as if they replace the std-strctparam-input argument.

9.1.1.10. Options Field [strctparam.schema.opts]

Name: options
Type: object
Value: The value shall be a JSON object.
Description: A structured parameters object can contain this structured options field. The keys of items shall be name (strctparam.schema.names) values.

9.1.1.11. Names [strctparam.schema.names]

name:: one or more of:
U+0061 .. U+007A LATIN SMALL LETTER A .. Z
U+0030 .. u+0039 DIGIT ZERO .. NINE
U+005F LOW LINE U+002D HYPHEN-MINUS
scope:: name scope-designator scope_opt
scope-designator:: U+002E FULL STOP

The name std is reserved for structured options defined in this standard.

Applications can specify vendor designated name parts outside of this standard.

9.1.1.12. Structured Option `std.param` [strctparam.schema.opt.param]

The std.param structured option defines an option to refer to additional structured parameters (strctparam) to process.

An application shall implement this option.

The std.param option shall have the following fields.

Name: pre
Type: string or array
Value: (for string) A pathname to a file containing structured parameters (strctparam).
Value: (for array) A list of pathname string items to files containing structured parameters (strctparam)
Description: One or more references to files that include additional structured parameters (strctparam).

Name: post
Type: string or array
Value: (for string) A pathname to a file containing structured parameters (strctparam).
Value: (for array) A list of pathname string items to files containing structured parameters (strctparam)
Description: One or more references to files that include additional structured parameters (strctparam).

A string value in the pre or post field shall be as if the value was given as an array with the string value as the only value in the array.

An application shall process the structured parameters in the pre field before processing the structured options where the std.param option is specified.

An application shall process the structured parameters in the post field after processing the structured options where the std.param option is specified.

An application shall process the structured parameters in the pre and post field in the order given in the value applying semantics as specified in the parameters or options of the structured parameters.

9.2. JSON Schema

Insert clause after Annex A.

9.2.1. Annex B (informative) Structured Parameters JSON Schema [strctparamjschm]

9.2.1.1. General [strctparamjschm.general]

This Annex defines the structured parameters capability schema (strctparam.schema) in terms of a JSON Schema. A JSON Schema refers to the IETF RFC draft "JSON Schema: A Media Type for Describing JSON Documents" as specified in https://json-schema.org/draft/2020-12/json-schema-core.html.

This JSON Schema can be referenced as the $schema field with URI value of "https://isocpp.org/release/schema/std_param-1.0.0.json".

9.2.1.2. JSON Schema Specification [strctparamjschm.spec]

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://isocpp.org/release/schema/std_param-1.0.0.json",
  "title": "Structured Parameters Version 1.0.0 JSON Schema",
  "type": "object",
  "properties": {
    "$schema": {
      "description": "JSON Schema URI for the version of the structured parameters format.",
      "type": "string",
      "format": "uri"
    },
    "version": {
      "description": "The Structured Parameters format version.",
      "type": "string",
      "$ref": "#/$defs/Version"
    },
    "arguments": {
      "description": "Application direct arguments.",
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "options": {
      "description": "Application structured options.",
      "type": "object",
      "allOf": [
        {
          "$ref": "#/$opt/Std.Param"
        }
      ],
      "propertyName": {
        "$ref": "#/$defs/Name"
      }
    }
  },
  "oneOf": [
    {
      "required": [
        "arguments"
      ]
    },
    {
      "required": [
        "options"
      ]
    }
  ],
  "$defs": {
    "Version": {
      "type": "string",
      "pattern": "^[0-9]+([.][0-9]+){0,2}$"
    },
    "Name": {
      "type": "string",
      "pattern": "^([a-z0-9_-]+[.])*([a-z0-9_-]+)$"
    },
    "StringOrArray": {
      "type": [
        "string",
        "array"
      ],
      "items": {
        "type": "string"
      }
    }
  },
  "$opt": {
    "Std.Param": {
      "properties": {
        "std.param": {
          "description": "Recursive reference to one or more structured parameters files.",
          "type": "object",
          "properties": {
            "pre": {
              "$ref": "#/$defs/StringOrArray"
            },
            "post": {
              "$ref": "#/$defs/StringOrArray"
            }
          },
          "additionalProperties": false
        }
      },
      "param": {
        "$ref": "#/$opt/Std.Param/properties/std.param"
      }
    }
  }
}

10. License

This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

1. Clang Compiler User’s Manual (https://clang.llvm.org/docs/UsersManual.html#configuration-files)

2. GCC Documentation (https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Overall-Options.html)

3. Intel® oneAPI DPC++/C++ Compiler Developer Guide and Reference (https://www.intel.com/content/dam/develop/external/us/en/documents/oneapi_dpcpp_cpp_compiler_2022.pdf)

4. NVIDIA CUDA Compiler Driver NVCC Documentation (https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)

5. Microsoft C++, C, and Assembler documentation (https://learn.microsoft.com/en-us/cpp/build/reference/at-specify-a-compiler-response-file)

6. Edison Design Group C++ documentation. https://www.edg.com/docs/edg_cpp.pdf

7. B2 Embarcadero C++ Builder @ file implementation. (https://github.com/bfgroup/b2/blob/4.10.1/src/tools/borland.jam#L282)

8. NVIDIA HPC C++ Reference (https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-ref-guide/index.html)

9. Python argparse module (https://docs.python.org/3/library/argparse.html#fromfile-prefix-chars)

10. P2717 Tool Introspection (https://wg21.link/P2717)

11. JSON Compilation Database Format Specification (https://clang.llvm.org/docs/JSONCompilationDatabase.html)

12. JSON Schema: A Media Type for Describing JSON Documents (http://json-schema.org/latest/json-schema-core.html)