"Привет, κόσμος!"
― anonymous user
1. Introduction
A new I/O-agnostic text formatting library was introduced in C++20 ([FORMAT]). This paper proposes integrating it with standard I/O facilities via a simple and intuitive API achieving the following goals:
-
Usability
-
Unicode support
-
Good performance
-
Small binary footprint
2. Revision history
Changes since R12:
-
Changed section numbers to permalinks in the wording.
-
Replaced "string literal encoding" with "the ordinary literal encoding ([lex.charset]) in [print.fun].2.
-
Added missing template parameter to
in [print.fun].3.forward -
Replaced
with "The function initializes an automatic variable viaLet out = vformat ( fmt , args )
in [print.fun].6 and [ostream.formatted.print].3.string out = vformat ( fmt , args ); -
Replaced "As specified in [format.err.report] or" with "Any exception thrown by the call to
([format.err.report])." in [print.fun].7 and [print.fun].11.vformat -
Removed "�" from [print.fun].8 and [ostream.formatted.print].5.
-
Clarified that writing to an output stream is done via an unformatted output function in [ostream.formatted.print].3 and [ostream.formatted.print].6.
-
Added a missing definition of the
overload ofostream
.println -
Moved "The Unicode® Standard Version 14.0 – Core Specification" from Normative references to Bibliography per LWG request and verified that ISO 10646:2020 doesn’t include an equivalent of "U+FFFD Substitution of Maximal Subparts" section from the Unicode standard and therefore can’t be referenced instead.
-
Rephrase Throws clauses per LWG feedback and clarify that exceptions from the underlying unformatted functions ([ostream.unformatted]) are propagated in [ostream.formatted.print].4 and [ostream.formatted.print].7.
Changes since R11:
-
Corrected header names in the feature test macro definition replacing "also in <format>" with "also in <print> and <ostream>".
-
Replaced
withstream
in [ostream.formatted.print].2.os
Changes since R10:
-
Moved "If invoking the (native Unicode) API requires transcoding, implementations should substitute invalid code units with U+FFFD � REPLACEMENT CHARACTER per The Unicode Standard Version 13.0 – Core Specification, Chapter 3.9." to the Recommended practice clause in the definition of
per LEWG feedback.vprint_unicode -
Made the definition of the
overload ofostream
consistent with thevprint_unicode
overload.FILE * -
Namespace-qualified the calls to
.forward -
Removed a call to
in [print.fun].1.make_format_args -
Removed extra
in the Effects clauses of)
andvprint_unicode
.vprint_nonunicode -
Rendered
in italics instr
.fmt . str -
Defined the
overload that doesn’t take a stream in terms of the one that does.println -
Add missing
to. str
in thefmt
overload ofostream
.print -
Updated The Unicode® Standard Version to 14.
Changes since R9:
-
Rebased the wording onto the latest draft, most importantly adding support for types that are neither const-iterable nor copyable introduced by [P2418].
Changes since R8:
-
Added new SG16 poll results.
-
Improved wording for [print.syn].6 (previously [print.syn].31) to remove ambiguities per SG16 feedback.
-
Fixed paragraph numbering in the wording.
-
Clarified the difference from the Python’s
in § 10 API and naming.print
Changes since R7:
-
Added a reference to LLVM’s
that implements similar mojibake prevention mechanism to § 15 Implementation.raw_ostream
Changes since R6:
-
Added new SG16 poll results.
-
Rebased the wording onto the latest draft, most importantly adding compile-time checks introduced by [P2216].
-
Added "If
contains invalid code units, the behavior is undefined and implementations are encouraged to diagnose it." to the definition ofout
per SG16 feedback.vprint_unicode -
Replaced "invalid code points are substituted with U+FFFD � REPLACEMENT CHARACTER" with " implementations should substitute invalid code units with U+FFFD � REPLACEMENT CHARACTER per The Unicode® Standard Version 13.0 – Core Specification, Chapter 3.9" in § 16 Wording per SG16 feedback.
-
Added "The Unicode® Standard Version 13.0 – Core Specification" to Normative references in § 16 Wording.
-
Clarified the behavior when mixing encodings in § 11 Unicode.
Changes since R5:
-
Added new LEWG poll results.
-
Added new SG16 poll results.
-
Replaced
with< io >
per LEWG feedback.< print > -
Clarified the choice of U+FFFD � REPLACEMENT CHARACTER for transcoding errors in § 11 Unicode.
-
Clarified the choice of literal encoding in § 11 Unicode.
-
Clarified that ANSI escape codes for specifying coding systems are not considered a native system API that supports Unicode in § 11 Unicode.
-
Added a reference to Rust’s standard output facility that implements the same mojibake prevention mechanism to § 15 Implementation
Changes since R4:
-
Added SG16 Unicode poll results.
-
Added a list of candidate headers formatted output functions can be added to.
-
Moved the non-
overloads of formatted output functions to a separate header for cleaner separation of formatting and I/O facilities.ostream
Changes since R3:
-
Replaced
with_isatty ( _fileno ( stream ))
in a note in [format.functions] because the former may return 1 for streams not referring to a terminal.GetConsoleMode ( _get_osfhandle ( _fileno ( stream )), ...)
Changes since R2:
-
Added better compatibility with other formatted I/O facilities as another advantage of using
in § 10 API and naming per SG16 feedback.stdout -
Clarified that [P1885] can be used for literal encoding detection per SG16 feedback.
-
Added comparison of Unicode handling in various languages in Appendix A: Unicode tests and a summary in § 11 Unicode per SG16 request.
-
Removed incorrect "exposition-only" in § 11 Unicode.
-
Replaced "both source and literal encodings are UTF-8, which is enabled by the
/utf-8
compiler flag" with "the literal (execution) encoding is UTF-8, which is enabled by the/execution-charset:utf-8
compiler flag" since the source encoding is irrelevant there. -
Rephrased Effects of
for clarity.vprint_unicode
Changes since R1:
-
Added missing
overloads that takeprintln
andFILE *
.ostream & -
Moved the print functions that take
toostream &
since< ostream >
shouldn’t depend on< format >
.< ostream > -
Clarified why it is useful to provide
functions in § 13 Binary code.vprint * -
Rebased the wording onto the latest working draft, N4861, in particular updating the Throws clauses to match existing wording.
-
Replaced
withstd :: system_error
in the wording.system_error -
Added paragraph numbers to the wording.
Changes since R0:
-
Clarified that adding
overloads will be addressed in a separate paper per the UK C++ panel feedback.wchar_t
3. SG16 polls (R8)
Poll: Use of UTF-8 as the literal encoding is sufficient for
facilities to establish encoding expectations.
SF | F | N | A | SA |
---|---|---|---|---|
2 | 3 | 2 | 1 | 0 |
Consensus: Consensus in favour.
Poll: Correct the P2093R8 wording for [print.syn].31 to remove ambiguities, and forward P2093 as revised to LEWG with a recommended ship vehicle of C++23.
SF | F | N | A | SA |
---|---|---|---|---|
1 | 4 | 2 | 0 | 0 |
Consensus: Consensus in favour.
4. SG16 polls (R6)
Poll: When
facilities must transcode their formatting results for
display on a device, and during that process invalidly-encoded text is
encountered,
should replace the erroneously-encoded code units with
.
SF | F | N | A | SA |
---|---|---|---|---|
3 | 3 | 1 | 2 | 0 |
Outcome: Consensus for the position
Poll: When
facilities need not transcode their formatting results
for display on a device, and during that process invalidly-encoded text is
encountered,
should nevertheless replace the erroneously-encoded code
units with
.
SF | F | N | A | SA |
---|---|---|---|---|
1 | 0 | 2 | 2 | 3 |
Outcome: Consensus against the direction.
Poll:
and
facilities should have consistent behavior
with respect to encoding expectations for the format string.
No objection to unanimous consent.
Poll: formatters should not be sensitive to whether they are being used with a
or
facility.
No objection to unanimous consent.
Poll: Regardless of format string encoding assumptions,
facilities
may be used to format binary data.
SF | F | N | A | SA |
---|---|---|---|---|
5 | 1 | 1 | 0 | 0 |
Consensus: Strong consensus in favor.
Poll: Regardless of format string encoding assumptions,
facilities may
be used to format binary data.
SF | F | N | A | SA |
---|---|---|---|---|
2 | 1 | 3 | 1 | 0 |
Consensus: Weak consensus in favor.
Poll:
facilities exhibit undefined behavior when an encoding
expectation is present and a format string or formatter output does not match
those expectations.
SF | F | N | A | SA |
---|---|---|---|---|
2 | 4 | 0 | 0 | 1 |
Consensus: Strong consensus in favor.
Poll:
facilities exhibit undefined behavior when an encoding
expectation is present and a format string or formatter output does not match
those expectations and output is directed to a device that has encoding
expectations.
SF | F | N | A | SA |
---|---|---|---|---|
6 | 0 | 1 | 0 | 0 |
Consensus: Stronger consensus in favor relative to previous poll.
Poll:
facility implementors are encouraged to provide a run-time
means for diagnosing format strings and formatter output that is not well-formed
according to the expected encoding.
SF | F | N | A | SA |
---|---|---|---|---|
4 | 0 | 2 | 1 | 0 |
Consensus: Consensus in favor.
Poll:
and
facilities should have consistent behavior
with respect to encoding expectations for the output of formatters.
SF | F | N | A | SA |
---|---|---|---|---|
0 | 1 | 1 | 5 | 1 |
Consensus: Strong consensus against.
Poll:
facility implementors are encouraged to substitute
replacement characters following Unicode guidance when output is directed to a
device and transcoding is necessary.
SF | F | N | A | SA |
---|---|---|---|---|
2 | 5 | 0 | 0 | 1 |
Consensus: Consensus in favor.
Poll:
facilities must provide an explicit program-controlled error
handling mechanism for violations of encoding expectations.
SF | F | N | A | SA |
---|---|---|---|---|
0 | 0 | 3 | 3 | 3 |
Poll: Use of UTF-8 as the literal encoding is sufficient for
facilities to establish encoding expectations.
SF | F | N | A | SA |
---|---|---|---|---|
3 | 1 | 3 | 2 | 0 |
Consensus: Very weak consensus.
5. LEWG polls (R5)
Poll: Block P2093 until we have a proposal for a lower level facility that can query tty/console and perform direct output to the console
SF | F | N | A | SA |
---|---|---|---|---|
2 | 1 | 5 | 5 | 6 |
Outcome: Consensus Against
Poll: We want
et al. non
overloads to go to:
Header | Approve | Disapprove |
---|---|---|
| 3 | 7 |
| 6 | 5 |
| 10 | 3 |
| 12 | 1 |
| 1 | 13 |
Outcome: both
and
have consensus, author can choose one for the
next revision.
6. SG16 polls (R3)
Poll: Forward P2093R3 to LEWG.
SF | F | N | A | SA |
---|---|---|---|---|
4 | 2 | 2 | 0 | 1 |
Consensus? Yes
7. LEWG polls (R2)
Poll: We want P2093R2 to revert moving the
overloads to the
header.
SF | F | N | A | SA |
---|---|---|---|---|
0 | 0 | 6 | 6 | 1 |
Outcome: Keep the paper as is
8. LEWG polls (R1)
Poll: We prefer
as the default output target.
SF | F | N | A | SA |
---|---|---|---|---|
3 | 2 | 6 | 6 | 5 |
Poll: Add a member function on
instead of a
free function overload.
SF | F | N | A | SA |
---|---|---|---|---|
0 | 2 | 5 | 17 | 3 |
Consensus against.
Poll: Remove
from the paper?
SF | F | N | A | SA |
---|---|---|---|---|
1 | 10 | 7 | 4 | 4 |
No consensus for change.
Poll: We are happy with the design with regards to UTF-8 output.
Unanimous consent.
Attendance: 35
9. Motivating examples
Consider a common task of printing formatted text to
:
C++20 | Proposed |
---|---|
|
|
The proposed
function improves usability, avoids allocating a
temporary
object and calling
which performs formatted
I/O on text that is already formatted. The number of function calls is reduced
to one which, together with
-like type erasure, results in much
smaller binary code (see § 13 Binary code).
Existing alternatives in C++20:
Code | Comments |
---|---|
| Requires even more formatted I/O function calls; message is interleaved with parameters; can result in interleaved output. |
| Only works if is a null-terminated character string.
|
| Constructs a temporary string; requires a call to and a separate
I/O function call, although potentially cheaper than .
|
Another problem is formatting of Unicode text:
If the source and execution encoding is UTF-8 this will produce the expected output on most GNU/Linux and macOS systems. Unfortunately on Windows it is almost guaranteed to produce mojibake despite the fact that the system is fully capable of printing Unicode, for examplestd :: cout << "Привет, κόσμος!" ;
Приветeven when compiled with, κόσμος!
/utf-8
using Visual C++
([MSVC-UTF8]). This happens because the terminal assumes code page 437 in this
case independently of the execution encoding.
With the proposed paper
will printstd :: ( "Привет, κόσμος!" );
"Привет, κόσμος!"
as expected allowing programmers to write Unicode
text portably using standard facilities. This will bring C++ on par with other
languages where such functionality has been available for a long time.
For comparison this just works in Python 3.8 on Windows with the same active
code page and console settings:
>>> ( "Привет, κόσμος!" ) Привет, κόσμος!
This problem is independent of formatting
strings but the same
solution applies there. Adding
and
overloads will be
explored in a separate paper in a more general context.
10. API and naming
Many programming languages provide functions for printing text to standard output, often combined with formatting:
Language | Function(s) |
---|---|
C | [N2176]
|
C#/.NET | [DOTNET-WRITE]
|
COBOL | statement [N0147]
|
Fortran | and statements [N2162]
|
Go | [GO-FMT]
|
Java | , , [JAVA-PRINT]
|
JavaScript | [WHATWG-CONSOLE]
|
Perl | [PERL-PRINTF]
|
PHP | [PHP-PRINTF]
|
Python | statement or function [PY-FUNC]
|
R | [R-PRINT]
|
Ruby | and [RUBY-PRINT]
|
Rust | [RUST-PRINT]
|
Swift | [SWIFT-PRINT]
|
Variations of
appear to be the most popular naming choice for this
functionality. It is either provided as a free function (most common) or a
member function (less common) together with a global object representing
standard output stream. Notable exceptions are COBOL, Fortran, and Python 2
which have dedicated language statements and Rust where
is a
function-like macro.
We propose adding a free function called
with overloads for writing to
the standard output (the default) and an explicitly passed output stream object.
The default output stream can be either
or
. We propose
using
for the following reasons:
-
is considerably faster on at least two major implementations (see § 12 Performance).stdout -
Better compatibility with other formatted I/O facilities compared to
and its associatedstd :: cout
that suffer from private buffering, localization and conversion services that must be synchronized at a lower level.std :: streambuf -
won’t use any formatted output functionality ofprint
.ostream
In some languages like Python
only provides the default formatting
although some formatting control may be achieved by other means such as named
arguments and/or doing formatting manually via
or string
interpolation. The current paper doesn’t propose such a default formatting
facility. A search in a large Python codebase revealed that over 70% of
calls either take a string literal or use interpolation and this doesn’t even
account for string variables. These use cases are covered by the current
proposal with better usability and potentially better performance because there
are no separate formatting function calls.
Since
doesn’t have an associated
locale we propose using the current global locale for locale-specific formatting
which is consistent with
. With
or another explicitly passed
stream, the stream’s locale will be used. In all cases the default formatting is
locale-independent.
Another option is to make
a member function of
. This
would make usage somewhat more awkward:
A free function can also be overloaded to takestd :: cout . ( "Hello, {}!" , name );
FILE *
to simplify migration
(possibly automated) of code from printf
to the new facility.
There are multiple approaches to appending a trailing newline:
-
Don’t append a newline automatically:
in C and other languages.printf -
Append a newline but don’t format arguments:
in C (inconsistent withputs
).fputs -
Have two formatting functions/macros, one that appends newline and another that doesn’t:
/print
in Java,println
/print !
in Rust,println !
/Printf
in Go,Println
/Write
in C#/.NET.WriteLine -
Let the user choose a terminating string defaulting to
and do limited formatting:" \n "
in Python and Swift.print
We propose not appending a newline automatically for consistency with
and iostreams:
std :: ( "Hello, {}!" , name ); // doesn’t print a newline std :: ( "Hello, {}! \n " , name ); // prints a newline
Additionally we can provide a function that appends a newline:
std :: println ( "Hello, {}!" , name ); // prints a newline
Although
doesn’t provide much usability improvement compared to
with explicit
, it has been an occasionally requested feature in
the fmt library ([FMT]).
Another question is which header non-
overloads of formatted output
functions should go to. Possible options:
-
< io > -
< print > -
< format > -
< ostream > -
< utility >
Earlier versions of the paper proposed
analogous to
so that
the future I/O facilities that don’t depend on ostream could be added there.
This was changed to a more narrow-focused
but
can be added in
the future once a symmetric input facility becomes available. Using
is undesirable because this header and its transitive dependencies are very big
(42 thousand lines preprocessed on libc++):
% echo '#include < ostream > '| clang ++ - E - x c ++ - | wc - l 42491
It also pulls in a lot of unrelated symbols such as
insertion
operators, global
,
,
variables and their
counterparts.
overloads are added to the
header.
11. Unicode
We can prevent mojibake in the Unicode example by detecting if the string literal encoding is UTF-8 and dispatching to a different function that correctly handles Unicode, for example:
where theconstexpr bool is_utf8 () { const unsigned char micro [] = " \u00B5 " ; return sizeof ( micro ) == 3 && micro [ 0 ] == 0xC2 && micro [ 1 ] == 0xB5 ; } template < typename ... Args > void ( string_view fmt , const Args & ... args ) { if ( is_utf8 ()) vprint_unicode ( fmt , make_format_args ( args ...)); else vprint_nonunicode ( fmt , make_format_args ( args ...)); }
vprint_unicode
function formats and prints text in UTF-8 using the
native system API that supports Unicode and vprint_nonunicode
does the same
for other encodings. The latter ensures that interoperability with code using
legacy encodings is preserved even though print
is a new API and it is not
strictly necessary. If calling the system API requires transcoding we propose
substituting invalid code units with U+FFFD � REPLACEMENT CHARACTER
which is consistent with the treatment of malformed UTF-8 in UTF-8-native
terminals. For example
prints#include <stdio.h>int main () { puts ( " \xc3\x28 " ); // Invalid 2 Octet Sequence }
�(
in iTerm2 and ? (
in macOS Terminal. So whether transcoding is done
or not in the UTF-8 case, you will normally get similar observed behavior.
In Visual C++
will return true
if the literal (execution) encoding
is UTF-8, which is enabled by the /execution-charset:utf-8
compiler flags or other means, and false
otherwise. Literal encoding detection
can be implemented in a more elegant way using [P1885].
Note that ANSI escape codes for specifying coding systems ([ISO2022]) are not considered a native system API that supports Unicode for the purposes of this proposal.
We propose using the literal encoding for the following reasons:
-
Consistency with the design of
which is locale-independent by default ([P0645]) and disallows implicitly mixing encodings e.g. passing a narrow string into a widestd :: format
is ill-formed.std :: format -
Consistency with the encoding used for width estimation ([P1868]). The standard wording doesn’t mention the literal encoding explicitly but the fact that the format strings are either literals or other compile-time strings ([P2216]) makes it the only conformant option.
-
Safety: the result of
does not depend on the global locale by default and a buffer allocated with this size can be passed safely toformatted_size
even if the locale has been changed in the meantime, possibly from another thread.format_to -
Implementation and usage experience.
-
In the vast majority of cases format strings are literals. For example, analyzing a sample of 100
calls from [CODESEARCH] showed that 98 of them are string literals and 2 are string literals wrapped in theprintf
gettext macro._ -
The active code page and the terminal encoding being unrelated on popular Windows localizations such as Russian where the former is CP1251 while the latter is CP866. Instead of assuming one encoding regardless of the string origin which would often result in mojibake, an explicit encoding indication can be done via the standard extension API, e.g. (exposition only)
This is already possible to implement by providing appropriateprint ( "Привет, {}!" , locale_enc ( string_in_locale_encoding ));
specializations.std :: formatter
This approach has been implemented in the fmt library ([FMT]), successfully tested and used on a variety of platforms.
Users can sometimes restrict the set of used characters to the common subset among multiple encodings (often ASCII) in which case encoding becomes mostly irrelevant. Such "polyglot" strings are fully supported for legacy encodings and partially supported for UTF-8 by the current proposal even though mixing encodings in such a way is a clearly bad practice from a general software engineering point of view.
Here’s an example output on Windows:
At the same time interoperability with legacy code is preserved when literal
encoding is not UTF-8. In particular, in case of EBCDIC, Shift JIS or a
non-Unicode Windows code page,
will perform no transcoding and the text
will be printed as is.
The following table summarizes the behavior of formatted output facilities in different programming languages:
Linux | macOS | Windows | ||||
---|---|---|---|---|---|---|
Language | Terminal | Redirect | Terminal | Redirect | Terminal | Redirect |
C | Correct | UTF-8 | Correct | UTF-8 | Wrong | UTF-8 |
Go | Correct | UTF-8 | Correct | UTF-8 | Correct | UTF-8 |
Java | Correct | UTF-8* | Correct | UTF-8* | Wrong | CP1251 (lossy) |
JavaScript | Correct | UTF-8* | Correct | UTF-8* | Correct | UTF-8* |
Python | Correct | UTF-8* | Correct | UTF-8* | Correct | Error |
Rust | Correct | UTF-8 | Correct | UTF-8 | Correct | UTF-8 |
* - the output is transcoded from a different UTF representation.
Correct means that the test message "Привет, κόσμος!" was fully readable in the
terminal output. None of the tested language facilities were able to produce
readable output when piped through the standard
command on Windows.
Java gave the worst results producing both mojibake and replacement characters
in this case: "╧ЁштхЄ, ??????!". Most other languages produced valid UTF-8
when the output of
was redirected to a file.
The current paper proposes following C, Go, JavaScript and Rust and preserving
the original encoding (modulo UTF conversion). The only difference compared to
is that we fix the console output on Windows. Java’s approach is
problematic for the following reasons:
-
There is a silent data loss for valid Unicode code points when the output is redirected to a file.
-
It is more expensive because of transcoding.
-
It may give an unusable result when piped through standard Windows commands like
.findstr -
It transcodes into legacy encodings that are rarely used in practice nowadays. For example, usage of CP1251 dropped from 4.3% to 0.6% in the last 12+ years ([ENCODING-TRENDS]), including a 0.4% drop while the current paper was in review.
The full listings of test programs are given in Appendix A: Unicode tests.
12. Performance
All the performance benefits of
([FORMAT]) automatically carry
over to this proposal. In particular, locale-independence by default reduces
global state and makes formatting more efficient compared to stdio and
iostreams. There are fewer function calls (see § 13 Binary code) and no shared
formatting state compared to iostreams.
The following benchmark compares the reference implementation of
with
and
. This benchmark formats a simple message and prints it to
the output stream redirected to
. It uses the Google Benchmark
library [GOOGLE-BENCH] to measure timings:
#include <cstdio>#include <iostream>#include <benchmark/benchmark.h>#include <fmt/ostream.h>void printf ( benchmark :: State & s ) { while ( s . KeepRunning ()) std :: printf ( "The answer is %d. \n " , 42 ); } BENCHMARK ( printf ); void ostream ( benchmark :: State & s ) { std :: ios :: sync_with_stdio ( false); while ( s . KeepRunning ()) std :: cout << "The answer is " << 42 << ". \n " ; } BENCHMARK ( ostream ); void ( benchmark :: State & s ) { while ( s . KeepRunning ()) fmt :: ( "The answer is {}. \n " , 42 ); } BENCHMARK ( ); void print_cout ( benchmark :: State & s ) { std :: ios :: sync_with_stdio ( false); while ( s . KeepRunning ()) fmt :: ( std :: cout , "The answer is {}. \n " , 42 ); } BENCHMARK ( print_cout ); void print_cout_sync ( benchmark :: State & s ) { std :: ios :: sync_with_stdio ( true); while ( s . KeepRunning ()) fmt :: ( std :: cout , "The answer is {}. \n " , 42 ); } BENCHMARK ( print_cout_sync ); BENCHMARK_MAIN ();
The benchmark was compiled with Apple clang version 11.0.0 (clang-1100.0.33.17)
with
and run on macOS 10.15.4. Below are the results:
Run on (8 X 2800 MHz CPU s) CPU Caches: L1 Data 32K (x4) L1 Instruction 32K (x4) L2 Unified 262K (x4) L3 Unified 8388K (x1) Load Average: 1.83, 1.88, 1.82 ---------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------- printf 87.0 ns 86.9 ns 7834009 ostream 255 ns 255 ns 2746434 print 78.4 ns 78.3 ns 9095989 print_cout 89.4 ns 89.4 ns 7702973 print_cout_sync 91.5 ns 91.4 ns 7903889
Both
and
are ~3 times faster than
even with
synchronization to the standard C streams turned off.
is 14% faster when
printing to
than to
. For this reason and because
doesn’t
use formatting facilities of
we propose using
as the default
output stream and providing an overload for writing to
.
On Windows 10 with Visual C++ 2019 the results are similar although the
difference between
writing to
and
is smaller with
being 7% faster:
Run on (1 X 2808 MHz CPU ) CPU Caches: L1 Data 32K (x1) L1 Instruction 32K (x1) L2 Unified 262K (x1) L3 Unified 8388K (x1) ---------------------------------------------------------- Benchmark Time CPU Iterations ---------------------------------------------------------- printf 835 ns 816 ns 746667 ostream 2410 ns 2400 ns 280000 print 580 ns 572 ns 1120000 print_cout 623 ns 614 ns 1120000 print_cout_sync 615 ns 614 ns 1120000
13. Binary code
We propose minimizing per-call binary code size by applying the type erasure
mechanism from [P0645]. In this approach all the formatting and printing logic
is implemented in a non-variadic function
. Inline variadic
function only constructs a
object, representing an array of
type-erased argument references, and passes it to
. Here is a
simplified example:
void vprint ( string_view fmt , format_args args ); template < class ... Args > inline void ( string_view fmt , const Args & ... args ) { return vprint ( fmt , make_format_args ( args ...)); }
We provide
overloads so that users can apply the same technique to
their own code. For example:
void vlog ( log_level level , string_view fmt , format_args args ) { // Print the log level and use vprint* overloads to format and print the // message. } template < class ... Args > inline void log ( log_level level , string_view fmt , const Args & ... args ) { return vlog ( level , fmt , make_format_args ( args ...)); }
Here
that implements the logging logic is not parameterized on formatting
argument types resulting in less code bloat compared to a naive templated
version. As a real-world example, this technique has been applied in the Folly
Logger ([FOLLY]) bringing ~5x binary size reduction per logging function call.
Below we compare the reference implementation of
to standard
formatting facilities. All the code snippets are compiled with clang (Apple
clang version 11.0.0 clang-1100.0.33.17) with -O3 -DNDEBUG -c -std=c++17
and the resulting binaries are disassembled
with objdump -S
:
void printf_test ( const char * name ) { printf ( "Hello, %s!" , name ); }
__Z11printf_testPKc: 0: 55 pushq %rbp 1: 48 89 e5 movq %rsp, %rbp 4: 48 89 fe movq %rdi, %rsi 7: 48 8d 3d 08 00 00 00 leaq 8(%rip), %rdi e: 31 c0 xorl %eax, %eax 10: 5d popq %rbp 11: e9 00 00 00 00 jmp 0 <__Z11printf_testPKc+0x16>
void ostream_test ( const char * name ) { std :: cout << "Hello, " << name << "!" ; }
__Z12ostream_testPKc: 0: 55 pushq %rbp 1: 48 89 e5 movq %rsp, %rbp 4: 41 56 pushq %r14 6: 53 pushq %rbx 7: 48 89 fb movq %rdi, %rbx a: 48 8b 3d 00 00 00 00 movq (%rip), %rdi 11: 48 8d 35 6c 03 00 00 leaq 876(%rip), %rsi 18: ba 07 00 00 00 movl $7, %edx 1d: e8 00 00 00 00 callq 0 <__Z12ostream_testPKc+0x22> 22: 49 89 c6 movq %rax, %r14 25: 48 89 df movq %rbx, %rdi 28: e8 00 00 00 00 callq 0 <__Z12ostream_testPKc+0x2d> 2d: 4c 89 f7 movq %r14, %rdi 30: 48 89 de movq %rbx, %rsi 33: 48 89 c2 movq %rax, %rdx 36: e8 00 00 00 00 callq 0 <__Z12ostream_testPKc+0x3b> 3b: 48 8d 35 4a 03 00 00 leaq 842(%rip), %rsi 42: ba 01 00 00 00 movl $1, %edx 47: 48 89 c7 movq %rax, %rdi 4a: 5b popq %rbx 4b: 41 5e popq %r14 4d: 5d popq %rbp 4e: e9 00 00 00 00 jmp 0 <__Z12ostream_testPKc+0x53> 53: 66 2e 0f 1f 84 00 00 00 00 00 nopw %cs:(%rax,%rax) 5d: 0f 1f 00 nopl (%rax)
void print_test ( const char * name ) { ( "Hello, {}!" , name ); }
__Z10print_testPKc: 0: 55 pushq %rbp 1: 48 89 e5 movq %rsp, %rbp 4: 48 83 ec 10 subq $16, %rsp 8: 48 89 7d f0 movq %rdi, -16(%rbp) c: 48 8d 3d 19 00 00 00 leaq 25(%rip), %rdi 13: 48 8d 4d f0 leaq -16(%rbp), %rcx 17: be 0a 00 00 00 movl $10, %esi 1c: ba 0d 00 00 00 movl $13, %edx 21: e8 00 00 00 00 callq 0 <__Z10print_testPKc+0x26> 26: 48 83 c4 10 addq $16, %rsp 2a: 5d popq %rbp 2b: c3 retq
The code generated for the
function that uses the reference
implementation of
described in this proposal is more than 2x smaller
than the ostream code and has one function call instead of three. The
code is further 2x smaller but doesn’t have any error handling. Adding error
handling would make its code size closer to that of
.
The following factors contribute to the difference in binary code size between
and
:
-
Passing format string as
instead ofstring_view
.const char * -
Capturing and passing argument type information.
-
Preparing the array of formatting arguments.
14. Impact on existing code
The current proposal adds new functions to the headers
and
and should have no impact on existing code.
15. Implementation
The proposed
function has been implemented in the open-source fmt
library [FMT] and has been in use for about 6 years.
Rust’s standard output facility uses essentially the same approach for preventing mojibake when printing to console on Windows ([RUST-STDIO]). The main difference is that invalid code units are reported as errors in Rust.
LLVM’s
[LLVM-OSTREAM] also implements this approach when writing
to console on Windows. The main difference is that in case of invalid UTF-8 it
falls back on writing raw (not transcoded) data.
16. Wording
Add an entry for
to section "Header
synopsis [version.syn]",
in a place that respects the table’s current alphabetic order:
#define __cpp_lib_print 202112L **placeholder** // also in <print> and <ostream>
Add the header
to the "C++ library headers" table in [headers], in a
place that respects the table’s current alphabetic order.
Add after subsection "Header
< print >
synopsis [print.syn]
namespace std { template < class ... Args > void ( format - string < Args ... > fmt , Args && ... args ); template < class ... Args > void ( FILE * stream , format - string < Args ... > fmt , Args && ... args ); template < class ... Args > void println ( format - string < Args ... > fmt , Args && ... args ); template < class ... Args > void println ( FILE * stream , format - string < Args ... > fmt , Args && ... args ); void vprint_unicode ( string_view fmt , format_args args ); void vprint_unicode ( FILE * stream , string_view fmt , format_args args ); void vprint_nonunicode ( string_view fmt , format_args args ); void vprint_nonunicode ( FILE * stream , string_view fmt , format_args args ); }
Modify subsection "Header
synopsis [ostream.syn]":
... template < class charT , class traits , class T > basic_ostream < charT , traits >& operator << ( basic_ostream < charT , traits >&& os , const T & x ); template < class ... Args > void ( ostream & os , format - string < Args ... > fmt , Args && ... args ); template < class ... Args > void println ( ostream & os , format - string < Args ... > fmt , Args && ... args ); void vprint_unicode ( ostream & os , string_view fmt , format_args args ); void vprint_nonunicode ( ostream & os , string_view fmt , format_args args );
Add a new subsection to "Formatting and manipulators [iostream.format]":
?.?.? Print functions [print.fun]template < class ... Args > void ( format - string < Args ... > fmt , Args && ... args );
1 Effects: Equivalent to:
( stdout , fmt , std :: forward < Args > ( args )...);
template < class ... Args > void ( FILE * stream , format - string < Args ... > fmt , Args && ... args );
2 Effects: If the ordinary literal encoding ([lex.charset]) is UTF-8, equivalent to:
Otherwise, equivalent to:vprint_unicode ( stream , fmt . str , make_format_args ( std :: forward < Args > ( args )...));
vprint_nonunicode ( stream , fmt . str , make_format_args ( std :: forward < Args > ( args )...));
template < class ... Args > void println ( format - string < Args ... > fmt , Args && ... args );
3 Effects: Equivalent to:
println ( stdout , fmt , std :: forward < Args > ( args )...);
template < class ... Args > void println ( FILE * stream , format - string < Args ... > fmt , Args && ... args );
4 Effects: Equivalent to:
( stream , "{} \n " , format ( fmt , std :: forward < Args > ( args )...));
void vprint_unicode ( string_view fmt , format_args args );
5 Effects: Equivalent to:
vprint_unicode ( stdout , fmt , args );
6 Effects: The function initializes an automatic variable viavoid vprint_unicode ( FILE * stream , string_view fmt , format_args args );
Ifstring out = vformat ( fmt , args );
stream
refers to a terminal capable of displaying Unicode, writes out
to
the terminal using the native Unicode API; if out
contains invalid code units,
the behavior is undefined and implementations are encouraged to diagnose it.
Otherwise writes out
to stream
unchanged.
[ Note: On POSIX and Windows,
referring to a terminal means that,
respectively,
and
return nonzero.
— end note ]
[ Note: On Windows, the native Unicode API is
. — end note ]
7 Throws: Any exception thrown by the call to
([format.err.report]).
if a call by the implementation to an operating system or
other underlying API results in an error that prevents the function from meeting
its specifications.
8 Recommended practice: If invoking the native Unicode API requires transcoding, implementations should substitute invalid code units with U+FFFD REPLACEMENT CHARACTER per The Unicode Standard Version 14.0 – Core Specification, Chapter 3.9.
void vprint_nonunicode ( string_view fmt , format_args args );
9 Effects: Equivalent to:
vprint_nonunicode ( stdout , fmt , args );
10 Effects: Writes the result ofvoid vprint_nonunicode ( FILE * stream , string_view fmt , format_args args );
vformat ( fmt , args )
to stream
.
11 Throws: Any exception thrown by the call to
([format.err.report]).
if a call by the implementation to an operating system or
other underlying API results in an error that prevents the function from meeting
its specifications.
Add subsection "Print [ostream.formatted.print]" to "Formatted output functions [ostream.formatted]":
template < class ... Args > void ( ostream & os , format - string < Args ... > fmt , Args && ... args );
1 Effects: If the ordinary literal encoding ([lex.charset]) is UTF-8, equivalent to:
Otherwise, equivalent to:vprint_unicode ( os , fmt . str , make_format_args ( std :: forward < Args > ( args )...));
vprint_nonunicode ( os , fmt . str , make_format_args ( std :: forward < Args > ( args )...));
template < class ... Args > void println ( ostream & os , format - string < Args ... > fmt , Args && ... args );
2 Effects: Equivalent to:
( os , "{} \n " , format ( fmt , std :: forward < Args > ( args )...));
3 Effects: The function initializes an automatic variable viavoid vprint_unicode ( ostream & os , string_view fmt , format_args args );
Ifstring out = vformat ( os . getloc (), fmt , args );
os
is a stream that refers to a terminal capable of displaying Unicode
which is determined in an implementation-defined manner, writes out
to
the terminal using the native Unicode API; if out
contains invalid code units,
the behavior is undefined and implementations are encouraged to diagnose it.
Otherwise writes out
to os
unchanged using an unformatted output function.
4 Throws: Any exception thrown by the call to
([format.err.report])
or the unformatted output function ([ostream.unformatted]).
if a call by the native Unicode API results in an error that
prevents the function from meeting its specifications.
5 Recommended practice: If invoking the native Unicode API requires transcoding, implementations should substitute invalid code units with U+FFFD REPLACEMENT CHARACTER per The Unicode Standard Version 14.0 – Core Specification, Chapter 3.9.
6 Effects: Writes the result ofvoid vprint_nonunicode ( ostream & os , string_view fmt , format_args args );
vformat ( os . getloc (), fmt , args )
to os
using an unformatted output function.
7 Throws: Any exception thrown by the call to
([format.err.report])
or the unformatted output function ([ostream.unformatted]).
Add to Bibliography:
– The Unicode® Standard Version 14.0 – Core SpecificationAppendix A: Unicode tests
This appendix gives full listings of programs for testing Unicode handling in various formatting facilities as well as test commands and their output on different platforms. The code contains additional sanity checks to ensure that the strings are encoded in some form of UTF as opposed to a legacy encoding.
C (
):
#include <stdio.h>#include <stdlib.h>int main () { const char * message = "Привет, κόσμος! \n " ; if (( unsigned char ) message [ 0 ] != 0xD0 && ( unsigned char ) message [ 1 ] != 0x9F ) abort (); printf ( message ); }
Go (
):
package mainimport "fmt" import "log" func main() { var message= "Привет, κόσμος!" if message[ 0 ] != 0xD0 && message[ 1 ] != 0x9F { log. Fatal( "wrong encoding" ) } fmt. Println( message) }
Java (
):
class Test { public static void main ( String [] args ) { String message = "Привет, κόσμος!\n" ; if ( message . charAt ( 0 ) != 0x41F ) throw new RuntimeException (); System . out . ( message ); } }
JavaScript / Node.js (
):
message= "Привет, κόσμος!" ; if ( message. charCodeAt( 0 ) != 0x41F ) throw "wrong encoding" ; console. log( message);
Python (
):
message = "Привет, κόσμος!" if ord( message [ 0 ]) != 0x41F : raise Exception () print( message )
Rust (
):
fn main () { if "Привет, κόσμος!" . chars (). nth ( 0 ). unwrap () as u32 != 0x41F { panic!(); } println!( "Привет, κόσμος!" ); }
Linux:
$ cc test . c - o c - test $ . / c - test Привет, κόσμος! $ . / c - test > out - c - linux . txt $ go build - o go - test test . go $ . / go - test Привет, κόσμος! $ . / go - test > out - go - linux . txt $ java Test Привет, κόσμος! $ java Test > out - java - linux . txt $ node test . js Привет, κόσμος! $ node test . js > out - js - linux . txt $ python3 test . py Привет, κόσμος! $ python3 test . py > out - py - linux . txt $ rustc test . rs - o rust - test $ . / rust - test Привет, κόσμος! $ . / rust - test > out - rust - linux . txt
All output files are in UTF-8:
Linux configuration:
-
Ubuntu Focal 20.04 with the ru_RU.UTF-8 locale
-
cc: gcc 9.3.0
-
go: go1.13.8
-
java: openjdk 11.0.9.1
-
node: v14.5.0
-
python3: 3.7.5
-
rustc: 1.47.0
macOS:
% cc test . c - o c - test % . / c - test Привет, κόσμος! % . / c - test > out - c - macos . txt % go build - o test - go test . go % . / test - go Привет, κόσμος! % . / test - go > out - go - macos . txt % java Test Привет, κόσμος! % java Test > out - java - macos . txt % node test . js Привет, κόσμος! % node test . js > out - js - macos . txt % python3 test . py Привет, κόσμος! % python3 test . py > out - py - macos . txt % rustc test . rs - o rust - test % . / rust - test Привет, κόσμος! % . / rust - test > out - rust - macos . txt
All output files are in UTF-8:
macOS configuration:
-
macOS Catalina 10.15.7 with the ru_RU.UTF-8 locale which is the default for Russian
-
cc: Apple clang version 12.0.0 (clang-1200.0.32.27)
-
go: go1.15.5
-
java: openjdk 14.0.1
-
node: v14.5.0
-
python3: 3.7.5
-
rustc: 1.47.0
Windows:
> cl / Fe : c - test . exe test . c ... > c - test ╨Я╤А╨╕╨▓╨╡╤В, ╬║╧М╧Г╬╝╬┐╧В! > c - test > out - c - windows . txt > c - test | findstr , ╨Я╤А╨╕╨▓╨╡╤В, ╬║╧М╧Г╬╝╬┐╧В! > go build - o go - test . exe test . go > go - test Привет, κόσμος! > go - test > out - go - windows . txt > go - test | findstr , ╨Я╤А╨╕╨▓╨╡╤В, ╬║╧М╧Г╬╝╬┐╧В! > java Test Привет, ??????! > java Test > out - java - windows . txt > java Test | findstr , ╧ЁштхЄ, ??????! > node test . js Привет, κόσμος! > node test . js > out - js - windows . txt > node test . js | findstr , ╨Я╤А╨╕╨▓╨╡╤В, ╬║╧М╧Г╬╝╬┐╧В! > python test . py Привет, κόσμος! > python test . py > out - py - windows . txt Traceback ( most recent call last ) : File "... \t est.py" , line 4 , in < module > ( message ) File "...\Python39\lib\encodings\cp1251.py" , line 19 , in encode return codecs . charmap_encode ( input , self . errors , encoding_table )[ 0 ] UnicodeEncodeError : 'charmap 'codec can ’t encode characters in position 8-13 : character maps to < undefined > > python test . py | findstr , Traceback ( most recent call last ) : File "... \t est.py" , line 4 , in < module > ( message ) File "...\Python39\lib\encodings\cp1251.py" , line 19 , in encode return codecs . charmap_encode ( input , self . errors , encoding_table )[ 0 ] UnicodeEncodeError : 'charmap 'codec can ’t encode characters in position 8-13 : character maps to < undefined > > rustc test . rs - o rust - test . exe > rust - test Привет, κόσμος! > rust - test > out - rust - windows . txt > rust - test | findstr , ╨Я╤А╨╕╨▓╨╡╤В, ╬║╧М╧Г╬╝╬┐╧В!
C, JavaScript (node.js), Rust and Go produced valid UTF-8 when the output was
redirected to files. Java produced a file in the legacy CP1251 encoding with
for non-representable code points. Python failed on transcoding to CP1251.
Output files:
Windows configuration:
-
Windows 10 10.0.19041 with Russian Region and Language settings
-
cl: Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29335
-
go: go1.15.6
-
java: Java HotSpot(TM) 64-Bit Server VM (build 15.0.1+9-18)
-
node: v14.15.3
-
python: 3.9.1
-
rustc: v14.15.3
17. Acknowledgements
Thanks to Corentin Jabot for his work on text encodings in C++ and in particular [P1885] that will simplify implementation of the current proposal.
Thanks to Roger Orr, Peter Brett, Hubert Tong, the BSI C++ panel and Tom Honermann for their feedback, support, constructive criticism and contributions to the proposal.