"We are stuck with technology when what we really want is just stuff that works." ― Douglas Adams
1. Introduction
[P2845] made it possible to format and print
with
correct handling of Unicode. Unfortunately, some common path accessors still
exhibit broken behavior, which results in mojibake and data loss. This paper
proposes fixing these issues, making the path API more reliable, user-friendly
and consistent with other standard facilities.
2. Problem
Consider the following example:
std :: filesystem :: path p ( L"Выявы" ); // Выявы is Images in Belarusian. std :: cout << p << std :: endl ; std :: cout << p . string () << std :: endl ;
Even if all code pages and localization settings are set to Belarusian and both the source and literal encodings are UTF-8, this still results in mojibake on Windows:
"┬√ т√" ┬√ т√
Unfortunately, we cannot change the behavior of iostreams but at least the new
facilities such as
and
correctly handle Unicode in
paths. For example:
std :: filesystem :: path p ( L"Выявы" ); std :: ( "{} \n " , p );
prints
Выявы
However, the
accessor still exhibits the broken behavior, e.g.
std :: filesystem :: path p ( L"Выявы" ); std :: ( "{} \n " , p . string ());
prints
�����
The reason for this is that
transcodes the
path into the native encoding
([fs.path.type.cvt]) defined as:
The native encoding of an ordinary character string is the operating system dependent current encoding for pathnames ([fs.class.path]).
It is neither the literal encoding nor a locale encoding, and transcoding is usually lossy, which makes it almost never what you want. For example:
std :: filesystem :: path p ( L"Obrázky" ); std :: string s = p . string ();
throws
with the message "unknown error" on the same system
which is a terrible user experience.
The string can be passed to system-specific APIs that accept paths provided that
the system encoding hasn’t changed in the meantime. But even this use case is
limited because the transcoding is lossy, and it’s better to use an equivalent
API or
instead.
On Windows, the native encoding is effectively the Active Code Page (ACP), which is separate from the console code page. This is why paths often cannot be correctly displayed. Even Windows documentation ([CODE-PAGES]) cautions against using code pages:
New Windows applications should use Unicode to avoid the inconsistencies of varied code pages and for ease of localization.
Encoding bugs are even present in standard library implementations, see e.g. [LWG4087], where a path in the "native" encoding is incorrectly combined with text in literal and potentially other encodings when constructing an exception message.
Moreover, the result of
is affected by a runtime setting and may
work in a test environment but easily break after deployment. This is similar
to one of the problems with
but worse because in this case C++
doesn’t even provide a way to set or query the encoding. It disproportionately
affects non-English C++ users making the language not as attractive for
writing internationalized and localized software.
3. Proposal
To summarize,
has the following problems:
-
It uses encoding that is generally incompatible with nearly all standard text processing and I/O facilities including iostreams,
andstd :: format
.std :: print -
It is extremely error-prone, causing easy to miss transcoding issues that may arise after the program is deployed in a different environment or after a runtime configuration change.
-
It makes writing portable code hard because the issues may not be obvious on POSIX platforms where
is just an inefficient equivalent ofstring ()
with extra memory allocation and copy.native ()
The crux of the matter is that different use cases require different encodings, and the "native" encoding is almost always the wrong choice, making it yet another example of a wrong default.
The current paper proposes deprecating the
overload for
, replacing it with alternatives that make the target
encoding clear:
-
returningsystem_string ()
in the operating system dependent current encoding for pathnames (native ordinary encoding). Similarly to the currentstd :: string
, it is lossy and only useful for passing to legacy system APIs.string () -
returningdisplay_string ()
in the literal encoding suitable for display, e.g. formatting withstd :: string
and printing withstd :: format
. It is lossless if the literal encoding is UTF-8 and the path is valid Unicode which is almost all paths on Windows.std :: print
We use "system" instead of "native" because the latter is ambiguous: it can either refer to encoding or format (path separators, etc.).
Similarly,
, which has the same problems, is split into
and
.
This will solve common issues with
by requiring the caller to
specify which encoding they want. In particular, the original example will
work with
:
std :: filesystem :: path p ( L"Выявы" ); std :: ( "{} \n " , p . display_string ());
prints
Выявы
This also makes conversion to the system encoding for use with legacy APIs more explicit:
std :: filesystem :: path p ( L"Выявы" ); std :: remove ( p . system_string ()); // Legacy API, won't work if ACP is not CP1251.
There is usually a better way to accomplish the same task with non-legacy
APIs, e.g. using the lossless
that takes a path object
instead of
:
std :: filesystem :: remove ( p ); // Lossless, portable and more efficient.
Ideally,
should be deprecated but this is out of scope of the
current paper.
4. Wording
Modify [https://eel.is/c++draft/fs.class.path.general]:
... // [fs.path.native.obs], native format observers const string_type & native () const noexcept ; const value_type * c_str () const noexcept ; operator string_type () const ; template < class EcharT , class traits = char_traits < EcharT > , class Allocator = allocator < EcharT >> basic_string < EcharT , traits , Allocator > string ( const Allocator & a = Allocator ()) const ; std :: string string () const ; std :: string display_string () const ; std :: string system_string () const ; std :: wstring wstring () const ; std :: u8string u8string () const ; std :: u16string u16string () const ; std :: u32string u32string () const ; // [fs.path.generic.obs], generic format observers template < class EcharT , class traits = char_traits < EcharT > , class Allocator = allocator < EcharT >> basic_string < EcharT , traits , Allocator > generic_string ( const Allocator & a = Allocator ()) const ; std :: string generic_string () const ; std :: string generic_display_string () const ; std :: string generic_system_string () const ; std :: wstring generic_wstring () const ; std :: u8string generic_u8string () const ; std :: u16string generic_u16string () const ; std :: u32string generic_u32string () const ; ...
Modify [fs.path.native.obs]:
...
std :: string string () const ; std :: string system_string () const ; std :: wstring wstring () const ; std :: u8string u8string () const ; std :: u16string u16string () const ; std :: u32string u32string () const ;
Returns: native().
Remarks: Conversion, if any, is performed as specified by [fs.path.cvt].
Returns:std :: string display_string () const ;
format ( "{}" , * this )
.
[Note: The returned string is suitable for use with formatting ([format.functions]) and print functions ([print.fun]). — end note]
Modify [fs.path.generic.obs]:
...
std :: string generic_string () const ; std :: string generic_system_string () const ; std :: wstring generic_wstring () const ; std :: u8string generic_u8string () const ; std :: u16string generic_u16string () const ; std :: u32string generic_u32string () const ;
Returns: The pathname in the generic format.
Remarks: Conversion, if any, is specified by [fs.path.cvt].
Returns:std :: string generic_display_string () const ;
format ( "{:g}" , * this )
.
[Note: The returned string is suitable for use with formatting ([format.functions]) and print functions ([print.fun]). — end note]
Add a new subclause in Annex D:
Deprecated filesystem path format observers [depr.fs.path.obs]
The following
member is defined in addition to those
specified in [fs.path.native.obs]:
Returns: native().std :: string string () const ;
Remarks: Conversion, if any, is performed as specified by [fs.path.cvt].
The following
member is defined in addition to those
specified in [fs.path.generic.obs]:
Returns: The pathname in the generic format.std :: string generic_string () const ;
Remarks: Conversion, if any, is specified by [fs.path.cvt].
5. Implementation
The proposed accessors have been implemented in the {fmt} library ([FMT]).