- Document number:
-
ISO/IEC/JTC1/SC22/WG21/P2656R2
- Date:
-
2023-02-14
- Audience:
-
SG15
- Reply-to:
-
René Ferdinand Rivera Morell - grafikrobot at gmail dot com - B2 Maintainer
Ben Craig - ben dot craig at gmail dot com
1. Abstract
We propose to publish an International Standard that specifies formats, processes, definitions, and so on, that facilitates the interoperation of the tools and systems that implement, and interface with, the C++ International Standard (ISO/IEC 14882).
2. Revision History
2.1. Revision 2 (February 2023)
Narrows the set of goals for the initial edition of the IS per SG15 polls. And adds explanations of those goals.
3. Motivation
Interoperability is a challenge in today’s C++ ecosystem. At a time when the C++ language is advancing, the community is struggling to manage the challenges of the complexity and variability of the tools, technologies, and systems that make C++ possible. In the view of users the C++ ecosystem is fractured in ways that hinder its successful advancement.
The continued success of C++ is tied not solely to the language, but to the C++ ecosystem. The interoperability within that ecosystem is key to surmounting the challenges of further growth of the language for the benefit of users. It is therefore critical that we expand the specifications that WG21 produces to bring coherence to the C++ ecosystem.
Users should be able to mix and match their preferred build systems, compilers, linkers, package managers, static analyzers, runtime analyzers, debuggers, profilers, etc. without needing the tools to have vendor specific knowledge of each other. Vendors should be able to focus on direct tool improvements, rather than figuring out how to interact with yet another proprietary interface.
4. Scope
This new standard aims to clarify practice in a common way. It can contain various aspects of the C++ Ecosystem:
-
Definitions : We will need a common language to refer to the many components and aspects of the ecosystem. With a shared understanding of components like: compilers, linkers, analyzers, debuggers, package managers, preprocessors, source files, object files, library files, shared library files, executables, and more, we can subsequently formulate specifications for them.
-
Format Specifications : The tools that make up the ecosystem work by consuming and producing a variety of data in a variety of formats. We will need to specify those formats such that the tools and components can operate effectively.
-
Operation Specifications : It’s not enough to know what the information the ecosystem contains, we also need to specify how that data is consumed and produced to aid in inter-operation of the variety of use cases in the C++ ecosystem.
This new standard will not:
-
Mandate any single vendor tools : It is not a goal to seek single "blessed" tools in the ecosystem. We have a wide variety of good tools in the ecosystem. And we look forward to those tools cooperating with each other.
-
Prohibit vendor extensions : It is not a goal to prevent vendor innovation in what the ecosystem tools can achieve. As such we welcome extensions and look towards the advancement that such extensions bring.
-
Modify the C++ Language Standard : It is not a goal to alter, in any way, the C++ Language Standard. It is important that how we define the tools of the C++ ecosystem evolves independent of the language.
5. Goals
Like the C++ Language Standard this one will never be complete or finished. And it will take time and effort to provide coverage of the specifications needed to put together a good basic picture of the ecosystem. While the scope above defines an ideal completion, the goals for a first edition of this standard include:
-
Definitions.
-
Build System ⇐⇒ Package Manager Interoperation.
-
Minimum set of recognized file extensions.
-
Tool introspection.
-
Portable diagnostics format via SARIF. [1]
-
Command line portability.
|
ℹ
|
This is not a closed set of goals. It is what we think is achievable with what we know now. We welcome additional goals if people come with complete proposals. |
5.1. Definitions
We will need some basic definitions as needed to circumscribe the specifications included in this first standard.
5.2. Build System ⇐⇒ Package Manager Interoperation
Specification of formats and operation of interoperability between build systems and package managers. Current work:
And previous work on this:
-
The CppCon 2022 presentation "The Case For a Standardized Package Description Format", [5] prompted ongoing work to specify standard communication format between package managers and build systems.
-
P2577 C++ Modules Discovery in Prebuilt Library Releases [6]
-
P2536 Distributing C++ Module Libraries with dependencies json files. [7]
-
P2473 Distributing C++ Module Libraries. [8]
-
P1767 Packaging C++ Modules. [9]
-
libman, A Dependency Manager ➔ Build System Bridge [10] -
P1313 Let’s Talk About Package Specification. [11]
-
P1177 Package Ecosystem Plan. [12]
5.3. File Name Extensions
Specification of a minimal set of file name extension understood, and for what they are understood, by the various tools in the ecosystem. Current work is forthcoming.
Previous work on this:
5.4. Introspection
Specification of format and command options to provide implementation information of the IS.
-
P2717 Tool Introspection [14]
5.5. Portable Diagnostics Format
C++ tools spend a lot of their time reading the output of other tools and processing it to either do more work or to present it to users. Unfortunately much of that information is not in a structured form. But instead is in plain stream output, i.e. log and error text which takes considerable effort and is specific to each tool generating it. Recently some tools have implemented the common structured output as SARIF format.[1] The format is designed for presenting results of static analysis. But is finding alternate uses. We aim to incorporate the SARIF format.
5.6. Command Line Portability
A key aspect of interoperation between tools in the ecosystem is having a common language to express tool commands, i.e. in compiler drivers, that can be understood and/or translated between different tools and platforms. This aims to define a standard structured response file format that can become the best way to communicate compiling C++.
6. Timeline
We believe that improving the interoperability in the C++ ecosystem is an urgent problem to solve.
-
We can’t solve all the challenges of the ecosystem interoperation at once; there are just too many of them.
-
We need solutions sooner to show that vendors can count on a stable future for them to build their tools on.
-
We need implementations sooner to show users the value of the IS.
-
We recognize that the IS will have errors that need to be addressed quickly.
Hence we aim to publish a standard quickly and provide updates to it as quickly. The goal is to publish this new IS on a two (2) year cycle starting in 2023. This means publishing the first edition in 2025. Subsequent versions would then publish in 2027, 2029, and so on. Because we plan on a small initial standard document we will follow the 24 month standards development track (SDT 24). [15]
The timeline that follows lists milestones for relevant WG21 meetings.
6.1. 2023.2: Plan
- Goal
-
Finalize the plan for the development of the IS.
With the intent of keeping the first edition of the IS limited we expect to have a rough idea of what will go into the IS by this time. SG15 will poll the plan by the end of this meeting. From this point we will have one year (12 months) to hone proposals to merge into the IS.
6.2. 2023.3: Pre-Draft
- Goal
-
Approve skeleton draft of the IS.
We will have a minimal skeleton draft of the IS prepared. This draft will have one or more papers merged into it, and will have outlines for the rest of the content, as possible. We will ask EWG approval on this content to checkpoint the work so far and the work going forward.
6.3. 2024.2: Proposal
- Goal
-
Submit formal proposal to create work item for the publication of the new IS.
The proposal will include an initial, mostly complete, draft of the intended content of the IS. Submitting at this meeting allows following the SDT 24 track of development [15] with a target publication in Q3 2025. The goal being to avoid the rush of the preparations for the C++ 26 IS. As the work will be completed by Q1 2025.
|
ℹ
|
Provide for an 8 week ballot period on proposal acceptance. |
6.4. 2025.1: Committee Draft (CD) Finalized
- Goal
-
Approve Committee Draft for National Body comments.
From submitting an initial draft in 2024.1 we will have completed incorporating any detail changes that the draft text will be ready to get voted on. This will mark, approximately, 1.5 years since the beginning of work on the new IS. The goal at this WG21 meeting will be to address any urgent issues that could prevent NB balloting of the IS draft.
|
ℹ
|
Provide for an 8 week ballot period on proposal acceptance. And 2 (4?) weeks of comment compilation time. |
6.5. 2025.2: Draft International Standard (DIS) Finalized
- Goal
-
Resolve collected NB comments and approve the final draft of the IS.
Consider and resolve NB comments compiled during the CD polling. With the first IS on its way to publishing approval we can start discussions on what the process and content will be going forward.
From here we can start the ongoing two (2) year cycles of releasing updates to the IS. In comparison to the C++ IS that would look like:
7. Process
We expect the development of the IS to use two processes that mesh into the existing processes of WG21:
- Bootstrap
-
Initial development and review in Tooling Study Group (SG15), followed by review and approvals in Evolution Working Group (EWG) or Library Evolution Working Group (LEWG). And from there continuing to the regular review and approval of wording process.
- Parallel
-
Development and review can originate in any existing study group depending as appropriate. Followed for review and approval by a new Tooling Working Group (TWG). TWG would include consideration of wording of the IS itself. And, hence, produce polls for WG21 plenary votes.
7.1. Bootstrap
We would use the Bootstrap process for the first edition of the IS. Following this process for the first edition has some advantages:
-
It’s a process we know. Which means it reduces initial overhead.
-
The reduced overhead allows us to concentrate more time on the IS development itself.
-
Gives us time to recruit people for the subsequent editions of the IS.
-
Contributors build up knowledge on the process to prepare for the next IS edition.
But it has some drawbacks:
-
It places higher burden on time for the EWG and Core groups to review the work.
-
The EWG and Core groups usual experts might not all be familiar with the tooling and ecosystem domain.
Those are significant drawbacks although we think are ameliorated by: First, scheduling the IS to complete a full year before the C++26 time frame. And, second, limiting the scope of the IS early in the time line resulting in more time spent in SG15 on draft wording details.
7.2. Parallel
After the first edition of the IS we would switch to the Parallel process. Visually this process would alter the regular flow of WG21 in minor ways resulting in:
Here the drawbacks from the Bootstrap process are addressed. And we maintain the advantages as we will have people ready and able to develop and process further editions of the IS.
This process structure will, clearly, change over time as the IS grows and experts fill similar roles to what we now have in WG21. Hence we expect to eventually need a wording group and narrower domain specific study groups.
|
ℹ
|
SG15 would cease to operate. As TWG would assume the same responsibilities. |
8. Polls
8.1. SG15: P2656R1 (2023-02-10)
SG15 thinks that the initial Ecosystem IS should include recommended / recognized file extensions.
| SF | F | N | A | SA |
|---|---|---|---|---|
3 |
4 |
3 |
0 |
0 |
SG15 is interested in a structured diagnostics format in the initial Ecosystem IS.
| SF | F | N | A | SA |
|---|---|---|---|---|
6 |
3 |
1 |
0 |
0 |
8.2. SG15: P2656R1 (2022-12-16)
SG15 recommends a two year timeline for the tooling IS as described in D2656R1.
| SF | F | N | A | SA |
|---|---|---|---|---|
1 |
5 |
4 |
0 |
0 |
Author: SF
SG15 recommends a three year timeline for the tooling IS offset from the C++ Language IS.
| SF | F | N | A | SA |
|---|---|---|---|---|
0 |
5 |
4 |
0 |
0 |
Author: F