Document #:

ISO/IEC/JTC1/SC22/WG21/P3051R1
Date:

2024-05-20
Audience:

SG15
Authors:

René Ferdinand Rivera Morell
Reply-to:

grafikrobot@gmail.com
Copyright:

Copyright 2024 René Ferdinand Rivera Morell, Creative Commons Attribution 4.0 International License (CC BY 4.0)

1. Abstract

This aims to define a standard structured response file format that can become the best way to communicate compiling C++.

2. Revision History

2.1. Revision 1 (May 2024)

Specify that options use abstract features instead of literal arguments, per SG15 feedback. Addition of wording.

2.2. Revision 0 (December 2023)

Initial text.

3. Motivation

A key aspect of inter-operation between tools in the ecosystem is having a common language to express tool commands, i.e. in compiler drivers, that can be understood and/or translated between different tools and platforms.

Currently tools use differing, but related, ways for users (and other tools) to specify the set of options to "toolsets" (compiler drivers, linkers, etc). While there are some commonalities in how those options are specified as "configuration response files" containing bare options, there are sufficient differences to hinder general inter-operation.

4. Scope

This proposal aims to specify a method for tools to specify arguments to other tools in a consistent and flexible manner. As such what it does and doesn’t aim to accomplish:

  • It does not aim to remove current arguments handling. It does allow for incrementally adoption of an alternative that facilitates common tool arguments.

  • It does not specify any particular options to replace existing options (except the ones to indicate the new response file). It does aim to specify an additional alternative option style that reduces the parsing complexity, and perhaps ambiguities, in tools.

5. Current Response Files

Current response files commonly contain "unstructured" sequence of command line arguments. Some also allow recursive inclusion and expansion of additional response files. Below are a summary of the syntax, capabilities, and restrictions of some of the compiler drivers.

5.1. Clang

  • Use of @filename argument.

  • Use of --config=filename argument.

  • References to other response files allowed.

Example response file:

# Several options on line
-c --target=x86_64-unknown-linux-gnu

# Long option split between lines
-I/usr/lib/gcc/x86_64-linux-gnu/5.4.0/../../../../include/C++/5.4.0

# other config files may be included
@linux.options

Source: Clang Compiler User’s Manual [1]

5.1.1. GNU Compiler Collection, GCC

  • Use of @filename argument.

  • References to other response files allowed.

Example response file:

-o "hello" -Wl,--start-group "bin/hello.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -fPIC -g

Source: GCC Documentation [2]

5.2. Intel® oneAPI DPC++/C++

  • Use of @filename argument.

  • References to other response files disallowed.

  • Platform specific option syntax.

  • # prefixed line comments.

Example response file for "Linux":

# compile with these options
  -O0
# end of response file

Example response file for "Windows":

# compile with these options
  /0d
# end of response file

Source: Intel® oneAPI DPC++/C++ Compiler Developer Guide and Reference [3]

5.3. NVIDIA CUDA Compiler Driver NVCC

  • Use of --options-file filename,…​, or -optf filename,…​.

-O0

Source: NVIDIA CUDA Compiler Driver NVCC Documentation [4]

5.4. Microsoft Visual C++

  • Use of @filename argument.

  • References to other response files disallowed.

Example response file:

"hello.cpp" -c -Fo"bin\hello.obj" -TP /wd4675 /EHs /GR /Zc:throwingNew /Z7 /Od /Ob0 /W3 /MDd /Zc:forScope /Zc:wchar_t /Zc:inline /favor:blend

Source: Microsoft C++, C, and Assembler documentation [5]

5.5. Other

Edison Design Group C++ Front End (edgcpfe)

Does not support response configuration files. [6]

Embarcadero C++ Builder

Supports at least @filename option, with bare arguments syntax. [7]

IBM Open XL C/C++ for AIX 17.1.0

Supports the Clang [1] --config option.

IBM Open XL C/C++ for Linux on Power 17.1.1

Has migrated to using the Clang [1] toolchain and supports the same options.

IBM Open XL C/C++ and XL C/C++ for z/OS

Supports the Clang [1] --config option.

HPE Cray Programming Environment (CPE)

Support depends on the platform complier environment.

NVIDIA HPC C++ (NVC++)

Does not support response configuration files. [8]

Oracle® Developer Studio 12.6

Supports a single global options configuration file, with bare arguments syntax.

Python argparse module

Supports arbitrarily character prefixed response files. Where the files, by default, contain an argument per line. [9]

6. Design

Abstractly response files are files in the operating system storage that contain arguments and options in addition to the ones given directly in the tool invocation. For this design we are going to refer to two different types of ways to pass the information to the tools:

Arguments

Arguments use the syntax that one would specify directly in the command line as a user. This would be things like the -O1 optimization flag argument.

Options

Options are the conceptual flag option that the tool understands that does not necessarily follow the same syntax as the flags specify in command line arguments.

Using those two distinct definitions allows us to specify them differently in the response file. Using arguments we will follow the same existing command line syntax. Keeping a form of compatibility with existing tools. But we would then be restricted to that syntax. Using options we can use a definition that fits best with a structured data definition.

Last consideration is the choice of structured data format we will have in the response files. Keeping with previous work and practice we will use JSON text as that format. [10] [11]

With that context, here are two example structured response files:

Simple Arguments:

{
  "arguments": [
    "-fPIC",
    "-O0",
    "-fno-inline",
    "-Wall",
    "-Werror",
    "-g",
    "-I\"util/include\"",
    "-c"
  ]
}

Structured Options:

{
  "options": [
    "position-independent-code",
    { "name": "optimization", "level": "off" },
    "no-inline",
    { "name": "warnings", "level": "all" },
    "warnings-as-errors",
    { "name": "debug-symbols", "value": "on" },
    { "name": "include", "dirs": [ "util/include" ] },
    "compile"
  ]
}
ℹ
 The names and values in the options example is for exposition only. It may not reflect the actual names and values as proposed, now or in the future.

The simple arguments example shows specifying an "arguments" key with an array of values corresponding to the regular command line arguments. This mirrors what one would see in a JSON compilation database [11]. This style has some advantages:

  • There is a direct correlation with the JSON compilation database format which some tools support. Which would mean that they already have code to deal with the JSON loading and understanding of it.

  • There’s a direct mapping for regular command line arguments. Hence it will be less effort to support this style for tools.

And there are some disadvantages:

  • The parsing of the arguments still has to happen to get at the option and value.

  • It is subject to the same limitations of regular command line arguments. Like complexity and ambiguities of command line syntax.

The structured options example shows specifying an "options" key with an array of option names or option objects. Where the option objects contain an option name and structured value. Some advantages of this style of structured data are:

  • The option names do not indicate a particular option prefix (i.e. -, --, /, etc) making it possible to use tool agnostic common names.

  • The ability to use arrays, or possibly objects, for the option values allows for logical groupings avoiding extra tracking of such as is present in command line parsing of options.

  • The names can be more descriptive, and hence easier for people to understand.

Some possible disadvantages:

  • Tools will need an alternative path to understand the new options. Although hopefully this is balanced by the more direct availability of the values.

  • If this style is also to be supported in the JSON compile database format it means more work to accomplish that. But again, the hope is that there is an easier mapping from internal structures to this format.

The design we are proposing has the following key points:

  • The format of the file is well formed JSON text.

  • The top level of that is a JSON object with one of, or both, arguments and options fields.

  • The addition of one command line option to specify the structured response file.

  • The arguments field has a single array value with string values.

  • The arguments values are single strings containing the same options as would be specified in the command line.

  • The arguments values can contain the option for other structure response files. Which will be recursively inserted at the location of the option.

  • The options field has an array of values for structured options.

  • The options values will follow an option specific schema.

  • The arguments are not specified, and as such are implementation defined.

  • The options are specified with both a schema and semantics for each.

6.1. Command Line

We propose to add a single new command line option as a requirement to implementing this capability:

$ tool --std-opt=file

Or:

$ tool -std-opt:file

The std-opt command line option, which can be repeated, will read the indicated file and parse the JSON text contents to configure the tool as needed.

6.2. File Format

The response file is a valid JSON text file with a single JSON object as the root object. There are two mutually exclusive fields to specify the command information: arguments or options.

There are two additional, optional, fields: $schema and version. The $schema field points at the released JSON Schema. [12] The version field indicated the response format of the file. The version number follows those specified in the capability introspection version numbering and semantics.

The arguments field specifies a single array value of strings. Each string array entry is a command line argument to be used directly by the tool. The specific syntax of the arguments is up to the specific tool. For example a compile invocation for GCC, and compatible compiler front-ends, would look like:

{
  "$schema": "https://raw.githubusercontent.com/cplusplus/ecosystem-is/release/schema/std_opt-1.0.0.json",
  "version": "1",
  "arguments": ["-fPIC", "-O0", "-fno-inline", "-Wall", "-Werror", "-g", "-I\"util/include\"", "-c" ]
}

You can also include --std-opt=file options in the list of arguments to include the arguments that are referenced in another response file, and so on. For example, given a common.json response file as such:

{
  "$schema": "https://raw.githubusercontent.com/cplusplus/ecosystem-is/release/schema/std_opt-1.0.0.json",
  "version": "1",
  "arguments": ["-fPIC", "-O0", "-fno-inline", "-Wall", "-Werror", "-g", "-I\"util/include\"", "-c" ]
}

One can refer to it in a main response file that compiles a C++ source file:

{
  "$schema": "https://raw.githubusercontent.com/cplusplus/ecosystem-is/release/schema/std_opt-1.0.0.json",
  "version": "1",
  "arguments": [ "--std-opt=common.json", "main.cpp", "-o", "main.o" ]
}

The effect is that the options in common.json are inserted in the arguments array at the location of the --std-opt=common.json argument.

The options field specifies a single array value of options values. An option value can be any value that follows the schema for the collection of specified options.

When compared to having the arguments field there are more constraints to the syntax of the options:

  • The option names do not contain prefix characters (i.e. the --, -, /, etc) and without the value separator (i.e. =, :, etc).

Like the arguments field, additional response files can be inserted at the indicated location by specifying a { "std.opt": "file.json" } option.

Do note though that even though the option names have a specific naming format they are still defined by the tool. The goal of having the restrictions on the option names is to make it possible in the future to specify tool agnostic options to facilitate general interop. But that is a subject for future proposals.

‼
 Tools need to support being able to either refer to an options style response file from an arguments style response file, and conversely.

6.3. Flags or Names

There is a question as to wether it’s better to use command line flags (for example W, o, I, etc) or non-command line names (for example warning, output, include, etc) in the options field. We will call the former the "Flags" choice, and the latter the "Names" choice. Each would mean:

Flags

The keys would be direct correlation to the specific tool, i.e. compiler, command line options. For example a I field name would match the -I.

Names

The keys would be symbolic names correlating to a concept that may map to one or more current command line options. For example an include field name would map to one or more -I command line options. But it would also map to one or more new --include command line options.

Given those definitions we can consider the pros and cons of each method.

Flags Names

Pros

  • Small effort for compilers to implement.

  • No additional CLI parsing to implement.

  • Common names and meanings across different tools.

  • Names could be used in other formats, like CPS.

Cons

  • Continues the increased implementation burden on build systems, etc, to manage different options per compiler, etc.

  • Possibly complex implementation to map from names to internal data.
ℹ
 One key consideration is that this proposal does not prevent choosing either flags or names as the options fields. Specifying the fields can be accomplished in a further proposal that specifies either flags or names as common specified syntax.

There are a set of possible design questions for this proposal that come to mind:

  1. Specify options fields as "Flags" or "Names"?

  2. Specify the options in this proposal or a separate proposal?

The discussions in WG21/SG15 concluded that it’s preferable to use "names" for the keys. This allows, as outlined in previous sections, a dual operation. The "arguments" for some form of backward compatibility. And the "options" for the modern structure tool invocation.

7. Questions

Why use a new option (--std-opt=file) instead of existing response file methods like @file?

Implementing support in tools for using the same response file options would:

  • Mean that the option to add the structured response files would vary from tool to tool as some use different styles for specifying the file.

  • It would be harder to implement as it would require inspecting the file content to determine the parsing method needed.

Why have an arguments field instead of just using the options field?

Having an arguments field has a couple of benefits:

  • It makes it easier for tools to immediately support use of this format as they can directly inject the arguments into their existing command line argument parsing.

  • Makes it easier for tools that already support compile_commands.json to produce or consume structured response files as they are the same format for the arguments field in both.

8. Polls

8.1. SG15: P3051R0 (2024-03-19)

SG15 wants to pursue defining a common response file format starting with supporting utf8 simple command lines, accepting the limitations for non-unicode arguments for the first version.

SF F N A SA

6

6

0

0

0

Result: Consensus

SG15 wants to persue defining common structured tool configuration arguments as part of the response file format.

SF F N A SA

6

3

3

0

0

Result: Consensus

9. Wording

Wording is relative to ecosystem-is/56aeadb. [13]

9.1. Specification: Structured Options

Insert clause after Introspection [intspct].

9.1.1. Structured Options [strctopt]

9.1.1.1. Preamble [strctopt.pre]

This clause describes options, output, and formats that control the behavior of applications through the specification of arguments and options from structured options.

This clause specifies the std.strctopt capability ([intspct.cap]) version 1.0.0.

An application can implement this capability.

An application that implements the std.strctopt capability shall include the std.strctopt field and version value in the introspection JSON text output ([intspct.schema.cap]).

9.1.2. Overview [strctopt.overview]

application [ std-strctopt-input-opt file ]

9.1.2.1. Input Option [strctopt.input]
std-strctopt-input-opt

The pathname of a file to read the structured options from. The option is specified as --std-opt=file or -std-opt:file. If file is ‘-’, the standard input shall be used.

9.1.2.2. Files [strctopt.file]

An application shall read a valid JSON text file that conforms to the structured options schema (strctopt.schema).

An application shall interpret the information in the file as if the options and arguments in the file occur in the same position as the std-strctopt-input-opt argument of the list of arguments given to the application. Given either directly as part of the application command line or as part of the arguments field (strctopt.schema.args).

9.1.2.3. Schema [strctopt.schema]

A structured options JSON text file shall contain one structured options JSON object (strctopt.schema.obj).

9.1.2.3.1. Structured Options Object [strctopt.schema.obj]

The structured options object is the root JSON object of the structured options JSON text.

A structure options object can have the following fields.

9.1.2.3.2. JSON Schema Field [strctopt.schema.schema]

Name: $schema
Type: string
Value: The value shall be a reference to a JSON Schema specification.
Description: A structure options object can contain this field. If a structure options object does not contain this field the value shall be a reference to the JSON Schema corresponding to the current edition of this standard (strctoptjschm).

9.1.2.3.3. Version Field [strctopt.schema.ver]

Name: version
Type: string
Value: 1 or 1.0 or 1.0.0
Description: The version field indicates the version of the structured options represented in the contents of the JSON text.

9.1.2.3.4. Arguments Field [strctopt.schema.args]

Name: arguments
Type: array
Value: The value shall be a JSON array. The items in the array shall be of JSON string types.
Description: The arguments field specifies items to be interpreted directly as if they occur in the command line of the program.

The application shall process the items as if they replace the std-strctopt-input-opt argument.

9.1.2.3.5. Options Field [strctopt.schema.opts]

Name: options
Type: array
Value: The value shall be a JSON array. The items in the array shall be of either JSON string or object types.
Items: (for string) The item shall be a single flag option.
Items: (for object) The item shall be a valid structured option.
Description: A structured options object can contain this field.

9.1.2.3.6. Names [[strctopt-strctopt-names]]
name:

one or more of:
U+0061 .. U+007A LATIN SMALL LETTER A .. Z
U+0030 .. u+0039 DIGIT ZERO .. NINE
U+005F LOW LINE U+002D HYPHEN-MINUS

scope:

name scope-designator scopeopt

scope-designator:

U+002E FULL STOP

The name std is reserved for flag options and structured options defined in this standard.

Applications can specify vendor designated name parts outside of this standard.

9.1.2.3.7. Flag Option [strctopt.schema.flag]
flag-option:

one of: yes-flag no-flag

yes-flag:

one of: yes-flag-name scoped-yes-flag-name

yes-flag-name:

name

scoped-yes-flag-name:

scopeopt name

no-flag:

one of: no-flag-name scoped-no-flag-name

no-flag-name:

no-flag-prefix name

scoped-no-flag-name:

scopeopt no-flag-name

no-flag-prefix:

U+006E LATIN SMALL LETTER N U+006F LATIN SMALL LETTER O
U+002D HYPHEN-MINUS

A flag option is an option that allows or prevents the specified semantics of the option.

A flag option shall be either a yes-flag or no-flag.

A yes-flag allows the semantics in the application as specified by effects of that behavior.

A no-flag prevents the semantics in the application as specified by effects of that behavior.

A flag option without a scope shall be equivalent the same flag option with a std. scope.

9.1.2.3.8. Structured Option [strctopt.schema.struct]

A structured option JSON object shall contain a structured-option-name field.

A structured option JSON object can contain additional fields as specified by the option specification.

Name: name
Type: string
Value: A valid structured-option-name.
Description: The name of the structured option.

structured-option-name:

scopeopt name

A structured option without a scope shall be equivalent to the same structured option with a std. scope.

9.1.2.3.9. Option std.opt [strctopt.schema.opt.opt]

The std.opt option defines an option to refer to additional structured options (strctopt) to process.

An application shall implement this option.

The std.opt option shall have the following fields.

Name: name
Type: string
Value: std.opt
Description: The name of the option.

Name: files
Type: string or array
Value: (for string) A pathname to a file containing structured options (strctopt).
Value: (for array) A list of pathname string items to files containing structured options (strctopt)
Description: One or more references to files that include additional structured options (strctopt) that are processed as if they occur at the location of the std.opt option.

9.2. JSON Schema

Insert clause after Annex A.

9.2.1. Annex B (informative) Structured Options JSON Schema [strctoptjschm]

9.2.1.1. General [strctoptjschm.general]

This Annex defines the structured options capability schema (strctopt.schema) in terms of a JSON Schema. A JSON Schema refers to the IETF RFC draft "JSON Schema: A Media Type for Describing JSON Documents" as specified in https://json-schema.org/draft/2020-12/json-schema-core.html.

This JSON Schema can be referenced as the $schema field with URI value of "https://raw.githubusercontent.com/cplusplus/ecosystem-is/release/schema/std_opt-1.0.0.json".

9.2.1.2. JSON Schema Specification [strctoptjschm.spec]
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://raw.githubusercontent.com/cplusplus/ecosystem-is/release/schema/std_opt-1.0.0.json",
  "title": "Structured Options Version 1.0.0 JSON Schema",
  "$defs": {
    "Version": {
      "type": "string",
      "pattern": "^[0-9]+([.][0-9]+){0,2}$"
    },
    "Name": {
      "type": "string",
      "pattern": "^[a-z0-9_\\-.]*[a-z0-9_-]+$"
    },
    "StringOrArray": {
      "type": [
        "string",
        "array"
      ],
      "items": {
        "type": "string"
      }
    }
  },
  "type": "object",
  "properties": {
    "$schema": {
      "description": "JSON Schema URI for the version of the structured options format.",
      "type": "string",
      "format": "uri"
    },
    "version": {
      "description": "The Structured Options format version.",
      "type": "string",
      "$ref": "#/$defs/Version"
    },
    "arguments": {
      "description": "Application direct arguments.",
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "options": {
      "description": "Application structured options.",
      "type": "array",
      "items": {
        "oneOf": [
          {
            "$ref": "#/$opts/Std.Opt"
          },
          {
            "$ref": "#/$opts/Any"
          }
        ]
      }
    }
  },
  "$opts": {
    "Std.Opt": {
      "description": "Recursive reference to one or more structured option files.",
      "type": "object",
      "properties": {
        "name": {
          "enum": [
            "std.opt",
            "opt"
          ]
        },
        "files": {
          "$ref": "#/$defs/StringOrArray"
        },
        "required": [
          "name",
          "files"
        ]
      },
      "additionalProperties": false
    },
    "Any": {
      "description": "Any other structured option.",
      "oneOf": [
        {
          "type": "object",
          "properties": {
            "name": {
              "$ref": "#/$defs/Name"
            },
            "required": [
              "name"
            ]
          },
          "additionalProperties": true
        },
        {
          "$ref": "#/$defs/Name"
        }
      ]
    }
  }
}

10. License

This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

1. Clang Compiler User’s Manual (https://clang.llvm.org/docs/UsersManual.html#configuration-files)
2. GCC Documentation (https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Overall-Options.html)
3. Intel® oneAPI DPC++/C++ Compiler Developer Guide and Reference (https://www.intel.com/content/dam/develop/external/us/en/documents/oneapi_dpcpp_cpp_compiler_2022.pdf)
4. NVIDIA CUDA Compiler Driver NVCC Documentation (https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html)
5. Microsoft C++, C, and Assembler documentation (https://learn.microsoft.com/en-us/cpp/build/reference/at-specify-a-compiler-response-file)
6. Edison Design Group C++ documentation. https://www.edg.com/docs/edg_cpp.pdf
7. B2 Embarcadero C++ Builder @ file implementation. (https://github.com/bfgroup/b2/blob/4.10.1/src/tools/borland.jam#L282)
8. NVIDIA HPC C++ Reference (https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-ref-guide/index.html)
9. Python argparse module (https://docs.python.org/3/library/argparse.html#fromfile-prefix-chars)
10. P2717 Tool Introspection (https://wg21.link/P2717)
11. JSON Compilation Database Format Specification (https://clang.llvm.org/docs/JSONCompilationDatabase.html)
12. JSON Schema: A Media Type for Describing JSON Documents (http://json-schema.org/latest/json-schema-core.html)
13. Working Draft, C++ Ecosystem International Standard 2023-11-10 (https://github.com/cplusplus/ecosystem-is/tree/56aeadb86d52a15aa4ec948a1eb8f247a8b703d8)