Use cases include the increasing number of text-based interchange formats such as JSON or XML that do not require internationalization support, but do require high throughput when produced by a server.
There are a lot of existing functions in C++ to perform such conversions, but none offers a high-performance solution. At a minimum, an implementation by an ordinary user of the language using an elementary textbook algorithm should not be able to outperform a quality standard library implementation. The requirements are thus:
For floating-point numbers, there should be a facility to output a floating-point number with a minimum number of decimal digits where input from the digits is guaranteed to reproduce the original floating-point value.
The deliberations in the Kona LEWG sessions resulted in the following comments:char.C++ already provides at least the facilities in the following table, each with shortcomings highlighted in the second column.
| facility | shortcomings |
|---|---|
| sprintf | format string, locale, buffer overrun |
| snprintf | format string, locale |
| sscanf | format string, locale |
| atol | locale, does not signal errors |
| strtol | locale, ignores whitespace and 0x prefix |
| strstream | locale, ignores whitespace |
| stringstream | locale, ignores whitespace, memory allocation |
| num_put / num_get facets | locale, virtual function |
| to_string | locale, memory allocation |
| stoi etc. | locale, memory allocation, ignores whitespace and 0x prefix, exception on error |
As a rough performance comparison, the following simple numeric formatting task was implemented: Output the integer numbers 0 ... 1 million, separated by a single space character, into a contiguous array buffer of 10 MB. This task was executed 10 times. The execution environment was gcc 4.9 on Intel Core i5 M450.
| strstream | 864 ms | uses std::strstream with application-provided buffer |
| streambuf | 540 ms | uses simple custom streambuf with std::num_put<> facet |
| direct | 285 ms | open-coded "divide by 10" algorithm, using the interface described below |
| fixed-point | 125 ms | fixed-point algorithm found in an older AMD optimization guide, using the interface described below |
There are various approaches for even more efficient algorithms; see, for example, https://gist.github.com/anonymous/7700052 .
The following discussion assumes that a common interface style should
be established that covers (built-in) integer and floating-point
types. The type T designates such an arithmetic type.
Note that given these restrictions, output of T to a string has a
small maximum length in all cases. The styles for input vs. output
will differ due to the differing functionality.
The fundamental interface for a string is that it is caller-allocated,
contiguous in memory, and not necessarily 0-terminated. That means,
it can be represented by a range [first,last)
where first and last are of type char
*.
Given this framework, the following subsections discuss various specific interface styles for both output and input. In each case, the signature of an integer output or input function is shown. Criteria for comparison include impact on compiler optimizations, indication of output buffer overflow, and composability (as a measure of ease-of-use).
This subsection discusses various specific interface styles for output. In each case, the signature of an integer output function is shown. There is one failure mode for output: overflow of the provided output buffer. Criteria for comparison include impact on compiler optimizations, indication of output buffer overflow, and composability (as a measure of ease-of-use). For exposition of the latter, consecutive output of two numbers is shown, without any separator.
Conceptually, an output function has four parameters and two
results. The parameters are the first and
last pointers of the buffer, the value, and the desired
base. The results are the updated first pointer and an
overflow indication.
| base 2...36 | overload provided |
| uppercase for base > 10 | not supported |
The following table lists the format specifiers of fprintf relevant to floating-point in C11 and the disposition in the context of the functionality proposed in this paper.
| field width | not supported | |
| precision (number of digits after the decimal-point) | overload provided | |
| + | mandatory sign | not supported |
| space | prefix | not supported |
| # | mandatory decimal point | not supported |
| 0 | pad with zeroes | not supported |
| L | long double argument | overload provided |
| f | fixed-precision lowercase conversion | overload provided |
| F | fixed-precision uppercase conversion | not provided |
| e | scientific lowercase conversion | overload provided |
| E | scientific uppercase conversion | not provided |
| g | switch between f and e | overload provided |
| G | switch between F and E | not provided |
| a | hexadecimal lowercase conversion | overload provided |
| A | hexadecimal uppercase conversion | not provided |
char * to_chars(char * first, char * last, T value, int base = 10);
This interface style returns the updated first pointer.
That is, the resulting string is in [first,
return-value) and [return-value, last)
is unused space in the string. Such an interface style is used for
many standard library algorithms, e.g. find [alg.find].
All parameters are passed by value which helps the optimizer.
Overflow is indicated by return-value == last.
The situation that the output exactly fits into the provided buffer
cannot be distinguished from overflow. Two consecutive outputs can be
produced trivially using:
p = to_chars(p, last, value1); p = to_chars(p, last, value2);
void to_chars(char *& first, char * last, T value, int base = 10);
This interface style updates the first pointer in place.
That is, the resulting string is in [old-first,
first) and [first,last) is
unused space in the string. Aliasing rules allow that updates to
first change the data where first points. To avoid
redundant updates, the implementation can copy first to a
local variable. Overflow is indicated by first reaching
last. The situation that the output exactly fits into the
provided buffer cannot be distinguished from overflow. Two
consecutive outputs can be produced trivially using:
to_chars(p, last, value1); to_chars(p, last, value2);
void to_chars(std::string_view& s, T value, int base = 10);This interface style groups the
first and
last pointers into a string_view which is
updated in-place. Comments on "iterator with in-situ update" apply
analogously.
bool to_chars(char *& first, char * last, T value, int base = 10);Comments on "iterator with in-situ update" apply analogously, except that the return value indicates whether overflow occurred.
int to_chars(char * first, char * last, T value, int base = 10);This interface style always returns the number of characters required to output T, regardless of whether sufficient space was provided. That is, an overflow occurred if the return value is larger than
last-first, otherwise the resulting string is
in [first, first + return-value).
Such an interface style is used for snprintf, except that
the proposed function never 0-terminates the output. All parameters
are passed by value which helps the optimizer. Overflow is indicated
by a return value strictly larger than the distance between
first and last. Computing the amount
of overflow is helpful to allocate a larger buffer, but, in general,
requires switching from the fast path, because no further characters
may be stored. The elementary functions discussed in this paper all
have (statically computable) limited maximum output size, so the
benefit of returning the exact size is small. Two consecutive
outputs require attention at the caller site to avoid buffer overflow:
int n = 0; n += to_chars(first, last, value1); n += to_chars(first + std::min(n, last-first), last, value2);
struct to_chars_result {
char* ptr;
bool overflow;
operator tuple<char *, bool>() const;
};
char* get<0>(const to_chars_result&); // for tie()
bool get<1>(const to_chars_result&);
to_chars_result to_chars(char* first, char* last, T value, int base = 10);
This interface style returns a named pair with the updated
first pointer. All parameters are passed by value which
helps the optimizer. Overflow is indicated by a separate overflow
indicator in the return value.
Two consecutive outputs can be produced easily using:
to_chars_result result = to_chars(p, last, value1); result = to_chars(result.ptr, last, value2);
An input function conceptually operates in two steps: First, it
consumes characters from the input string matching a pattern until the
first non-matching character or the last of the string is encountered.
Second, the matched characters are translated into a value of type
T. There are two failure modes: no characters match, or
the pattern translates to a value that is not in the range
representable by T.
Conceptually, an input function has three parameters and three
results. The parameters are the first and
last pointers of the string and the desired base. The
results are the updated first pointer, a
std::error_code and the parsed value.
This subsection discusses various specific interface styles for
input. Failure is indicated by std::error_code with the
appropriate value. In each case, the signature of an integer input
function is shown. Criteria for comparison include impact on compiler
optimizations and composability (as a measure of ease-of-use). For
exposition of the latter, parsing of two consecutive values is shown,
without skipping of any separator.
const char * from_chars(const char * first, const char * last, T& value, std::error_code& ec, int base = 10);This interface style returns the updated
first pointer.
That is, the returned pointer points to the first character not
matching the pattern. Such an interface style is used for many
standard library algorithms. Two consecutive inputs can be performed
like this:
T value1, value2;
std::error_code ec;
p = from_chars(p, last, value1, ec);
if (ec)
/* parse error */;
p = from_chars(p, last, value2, ec);
if (ec)
/* parse error */;
void from_chars(const char *& first, const char * last, T& value, std::error_code& ec, int base = 10);This interface style updates the
first pointer in place.
Two consecutive inputs can be performed like this:
T value1, value2;
std::error_code ec;
from_chars(p, last, value1, ec);
if (ec)
/* parse error */;
from_chars(p, last, value2, ec);
if (ec)
/* parse error */;
std::error_code from_chars(const char *& first, const char * last, T& value, int base = 10);Returning the error code allows for more compact code at the call site:
T value1, value2;
if (std::error_code ec = from_chars(p, last, value1))
/* parse error */;
if (std::error_code ec = from_chars(p, last, value2))
/* parse error */;
std::map shows that the naming of the
parts (first and second) carries no semantic
meaning which would help reading the resulting code. If the result
value moves to the return value, its type T needs to be
passed explicitly (e.g. as a template parameter). The composition
example would be:
std::pair<T, std::error_code> res;
res = from_chars<T>(p, last);
if (res.second)
/* parse error */;
T value1 = res.first;
res = from_chars<T>(p, last);
if (res.second)
/* parse error */;
T value2 = res.second;
struct from_chars_result {
const char* ptr;
error_code ec;
};
const char * get<0>(const from_chars_result&); // for tie()
error_code get<1>(const from_chars_result&);
from_chars_result from_chars(const char* first, const char* last, T& value, int base = 10);
This interface style returns the updated first pointer
and an error code. All parameters, except for the parsed value, are
passed by value, which helps the optimizer. Two consecutive inputs
can be performed like this:
T value1, value2;
from_chars_result result = from_chars(p, last, value1);
if (result.ec)
/* parse error */
result = from_chars(result.ptr, last, value2);
if (result.ec)
/* parse error */
The LEWG deliberations in Kona expressed the following naming preferences (sorted by number of votes).
| to_text | 9 |
| to_chars | 9 |
| to_digits | 7 |
| to_characters | 7 |
| to_printable | 6 |
| to_ascii | 3 |
| to_string | 3 |
| to_output | 1 |
| [de]serialize | 1 |
| [un]marshal | 1 |
| [de]stringify | 1 |
Given the tie in the first place and the author's personal preference
for to_chars, this paper proposes to_chars
for the output function and from_chars for the input
(parse) function.
to_chars_result to_chars(char* first, char* last, T value, int base = 10);vs.
to_chars_result to_chars(char* first, char* last, T value); to_chars_result to_chars(char* first, char* last, T value, int base);The difference is almost a quality-of-implementation issue, except that the standard gives appropriate liberty only for member functions, not for non-member functions (17.6.5.5 [member.functions]). The former can be implemented like this:
inline to_chars_result to_chars(char* first, char* last, T value, int base = 10)
{
if (base == 10)
return to_chars2(first, last, value);
else
return to_chars2(first, last, value, base);
}
Other than a slightly increased burden on the inlining and constant propagation capabilities of the compiler, the two signatures are thus identical in performance. I have analyzed similar cases in the past and can confirm that the inline function essentially vanishes for optimized compiles. Personally, I would prefer to give an implementation latitude to switch between the two interface styles as it sees fit, but that is a question that should be discussed in a wider context, independent of the present paper. A similar argument applies to the question of overhead for base = 16, where a very efficient implementation using SIMD vector instructions is possible.
namespace std {
struct to_chars_result {
char* ptr;
bool overflow;
};
to_chars_result to_chars(char* first, char* last, see below value, int base = 10);
to_chars_result to_chars(char* first, char* last, float value, bool hex = false);
to_chars_result to_chars(char* first, char* last, double value, bool hex = false);
to_chars_result to_chars(char* first, char* last, long double value, bool hex = false);
enum class chars_format {
scientific = unspecified,
fixed = unspecified,
hex = unspecified,
general = fixed | scientific
};
to_chars_result to_chars(char* first, char* last, float value, chars_format fmt, int precision = 6);
to_chars_result to_chars(char* first, char* last, double value, chars_format fmt, int precision = 6);
to_chars_result to_chars(char* first, char* last, long double value, chars_format fmt, int precision = 6);
struct from_chars_result {
const char* ptr;
error_code ec;
};
from_chars_result from_chars(const char* first, const char* last, see below& value, int base = 10);
from_chars_result from_chars(const char* first, const char* last, float& value, chars_format fmt = chars_format::general);
from_chars_result from_chars(const char* first, const char* last, double& value, chars_format fmt = chars_format::general);
from_chars_result from_chars(const char* first, const char* last, long double& value, chars_format fmt = chars_format::general);
}
The type chars_format is a bitmask type (17.5.2.1.3 [bitmask.types])
with elements scientific, fixed,
and hex.
For the to_chars function taking a
parameter base, the implementation shall provide
overloads for all signed and unsigned integer types
and char as the type of the parameter value.
For the from_chars function taking a
parameter base, the implementation shall provide
overloads for all signed and unsigned integer types
and char as the referenced type of the
parameter value.
All functions named to_chars convert value
into a character string by successively filling the range
[first, last). If the member
overflow of the return value is false, the
conversion was successful and the member ptr is the
one-past-the-end pointer of the characters written. Otherwise, the
member ptr has the value last and the
contents of the range [first, last) are
unspecified.
to_chars_result to_chars(char* first, char* last, T value, int base = 10);Requires:
base has a value between 2 and 36 (inclusive).
Effects: The value of value is converted to a
string of digits in the given base (with no redundant leading zeroes).
Digits in the range 10..35 (inclusive) are represented as lowercase
characters a..z. If value is less than zero, the
representation starts with a minus sign.
Throws: Nothing.
to_chars_result to_chars(char* first, char* last, float value, bool hex = false); to_chars_result to_chars(char* first, char* last, double value, bool hex = false); to_chars_result to_chars(char* first, char* last, long double value, bool hex = false);Effects:
value is converted to a string as-if by
printf in the "C" locale (see ISO C 7.19.6.1). If hex is
true, the conversion specifier is a (without leading
"0x"); otherwise the conversion specifier is f or
e. In either case, the representation is such that there
is at least one digit before the radix point (if present) and the
representation requires a minimal number of characters, yet parsing
the representation using the corresponding from_chars
function recovers value exactly. [ Note:
This guarantee applies only if to_chars
and from_chars are executed on the same
implementation. ]
Throws: Nothing.
to_chars_result to_chars(char* first, char* last, float value, chars_format fmt, int precision = 6); to_chars_result to_chars(char* first, char* last, double value, chars_format fmt, int precision = 6); to_chars_result to_chars(char* first, char* last, long double value, chars_format fmt, int precision = 6);Requires:
fmt has the value of one of the
enumerators of chars_format.
Effects: value is converted to a string as-if by
printf in the "C" locale with the given precision
(see ISO C 7.19.6.1). The conversion specifier is
f if fmt is
chars_format::fixed, e if fmt
is chars_format::scientific, a (without
leading "0x" in the result) if fmt is
chars_format::hex, and g if fmt
is chars_format::general.
Throws: Nothing.
from_chars analyze the string
[first,last) for a pattern. If no characters match the pattern,
value is unmodified, the member ptr of the
return value is first and the member ec is
equal to errc::invalid_argument.
Otherwise, the characters matching the pattern are interpreted as a
representation of a value of type T. The member ptr of
the return value points to the first character not matching the
pattern, or has the value last if all characters match.
If the parsed value is not in the range representable by the type of
value, value is unmodified and the member
ec of the return value is equal
to errc::result_out_of_range. Otherwise,
value is set to the parsed value and the member
ec is set such that the conversion to bool
yields false.
from_chars_result from_chars(const char* first, const char* last, T& value, int base = 10);Requires:
base has a value between 2 and 36 (inclusive).
Effects: The pattern is the expected form of the subject
sequence for the given non-zero base, as described for
strtol in ISO C 7.20.1.4, except that no "0x" or "0X"
prefix shall appear if the value of base is 16, and except
that a minus sign is the only sign that may appear, and only if
T is a signed type.
Throws: Nothing.
from_chars_result from_chars(const char* first, const char* last, float& value, chars_format fmt); from_chars_result from_chars(const char* first, const char* last, double& value, chars_format fmt); from_chars_result from_chars(const char* first, const char* last, long double& value, chars_format fmt);Effects: The pattern is the expected form of the subject sequence in the "C" locale, as described for
strtod in ISO C 7.20.1.3, except that
fmt has
chars_format::scientific set but not
chars_format::fixed, the otherwise optional exponent part
shall appear;fmt has chars_format::fixed
set but not chars_format::scientific, the optional
exponent part shall not appear; andfmt is chars_format::hex, the
prefix "0x" or "0X" shall not appear, but is assumed.value is one of at most two
floating-point values closest to the value of the string matching the
pattern.
Throws: Nothing.