Document Number: N2444
Submitter: Florian Weimer
Submission Date: 2019-10-09
More optionally per-thread state for the library implementation

Summary

In various places, the standard gives implementations permission not to avoid data races on shared state internal to the implementation. This lets implementations avoid the overhead of using locking or atomics. It does not allow implementations to use objects of thread storage duration to avoid data races, though. There are two reasons why the standard does not permit thread-safe implementations:

There is a strong incentive to implementations to make additional library state thread-specific because a lot of library code uses functions such as strtok or localtime in ways that assume that their effects are restricted to the current thread. (This is less of a problem for a single application, where the application developer generally knows whether a program is single-threaded or not.) In many cases, these issues are the only source of thread safety issues. Several C implementations already have made changes to better support such legacy libraries, but these implementations are currently non-conforming. For example, Solaris documents such behavior for several functions:

The strtok() function is safe to use in multithreaded applications because it saves its internal state in a thread-specific data area. [Source]
The asctime(), ctime(), gmtime(), and localtime() functions are safe to use in multithread applications because they employ thread-specific data. [Source]

Similarly, the Bionic C library for Android uses a buffer of thread storage duration for the strerror result.

For some of the affected functions (such as localtime), POSIX suggests that thread-safe implementations are possible, but does not go into details

This proposal subsumes the following previous submissions:
N2225Thread-local state for getenv, strtok, and set_constraint_handler
N2226Thread-local state for the program locale (setlocale)
N2227Thread-local state for library functions with internal state (rand, srand, c16rtomb, c32rtomb, mbrlen, mbrtoc16, mbrtoc32, mbrtowc, mbsrtowcs, mbtowc, wcrtomb, wcsrtombs, wctomb)
N2228Thread-local state for library functions returning pointers to internal state (strerror, asctime, ctime, gmtime, localtime)

Enabling additional thread-specific state

As discussed in the summary, there are two problems with objects with thread storage duration which make their introduction observable to programs: lack of sharing across threads, and deallocation on thread exit rather than on program exit. This proposal adopts the following solutions:
  1. For strtok, the wording is tailored to this function in particular. It is based on a suggestion Philipp Klaus Krause posted to the reflector on 2018-04-19 (SC22WG14.15088).
  2. For other functions that have internal state that is not directly accessible, the previous approach is used: give implementations permission to give the state thread storage duration. Mandating thread storage duration was raised on the reflector, but was met with an objection and no support.
  3. For functions which return pointers to objects, implementations are given permission to deallocate that object on thread exit. For strerror and setlocale, it is made clear that a previously returned string may be deallocated by a subsequent call and subsequent access is therefore undefined.

Proposed wording changes

Section numbers refer to the 2011 edition of the standard.
  1. In 7.11.1.1 (The setlocale function), change:
    The pointer to string returned by the setlocale function is such that a subsequent call with that string value and its associated category will restore that part of the program’s locale. The string pointed to shall not be modified by the program, but may be overwritten by a subsequent call to the setlocale function. The behavior is undefined if the returned value is used after a subsequent call to the setlocale function, or after the thread which called the function to obtain the returned value has exited.

    In J.2 (Undefined behavior), add:

    — A pointer returned by the setlocale function is used after a subsequent call to the function, or after the calling thread has exited (7.11.1.1).
  2. In 7.22.7 (Multibyte/wide character conversion functions), change:
    For a state-dependent encoding, each function is placed into its initial conversion state at program startupprior to the first call to the function and can be returned to that state by a call with a null pointer as its character pointer argument, s. Subsequent calls with s as other than a null pointer cause the internal conversion state of the function to be altered as necessary. It is implementation-defined whether internal conversion state has thread storage duration, and whether a newly created thread has the same state as the current thread at the time of creation, or the initial conversion state.
    In J.3.12 (Library functions), add:
    — Whether the internal state of multibyte/wide character conversion functions has thread-storage duration, and its initial value in newly created threads (7.22.7).
  3. In 7.24.5.8 (The strtok function), change:

    A sequence of calls to the strtok function breaks the string pointed to by s1 into a sequence of tokens, each of which is delimited by a character from the string pointed to by s2. The first call in the sequence has a non-null first argument; subsequent calls in the sequence have a null first argument. If any of the subsequent calls in the sequence is made by a different thread than the first, the behavior is undefined. The separator string pointed to by s2 may be different from call to call.

    In J.2 (Undefined behavior), add:

    — A sequence of calls of the strtok function is made from different threads (7.24.5.8).
  4. In 7.24.6.2 (The strerror function), change:
    The strerror function returns a pointer to the string, the contents of which are locale- specific. The array pointed to shall not be modified by the program, but may be overwritten by a subsequent call to the strerror function. The behavior is undefined if the returned value is used after a subsequent call to the strerror function, or after the thread which called the function to obtain the returned value has exited.
    In J.2 (Undefined behavior), add:
    — A pointer returned by the strerror function is used after a subsequent call to the function, or after the calling thread has exited (7.24.6.2).
  5. In 7.27.3 (Time conversion functions), change:
    Except for the strftime function, these functions each return a pointer to one of two types of static objects: a broken-down time structure or an array of char. Execution of any of the functions that return a pointer to one of these object types may overwrite the information in any object of the same type pointed to by the value returned from any previous call to any of them and the functions are not required to avoid data races with each other. 322) Accessing the returned pointer after the thread that called the function that returned it has exited results in undefined behavior. The implementation shall behave as if no other library functions call these functions.
    In J.2 (Undefined behavior), add:
    — An attempt is made to access the pointer returned by the time conversion functions after the thread that originally called the function to obtain it has exited (7.27.3).
  6. In 7.28.1 (Restartable multibyte/wide character conversion functions), change:
    If ps is a null pointer, each function uses its own internal mbstate_t object instead, which is initialized at program startupprior to the first call to the function to the initial conversion state; the functions are not required to avoid data races with other calls to the same function in this case. It is implementation-defined whether the internal mbstate_t object has thread storage duration; if it has thread storage duration, it is initialized to the initial conversion state prior to the first call to the function on the new thread. The implementation behaves as if no library function calls these functions with a null pointer for ps.
    In 7.29.6.4 (Restartable multibyte/wide string conversion functions), change:
    If ps is a null pointer, each function uses its own internal mbstate_t object instead, which is initialized at program startupprior to the first call to the function to the initial conversion state; the functions are not required to avoid data races with other calls to the same function in this case. It is implementation-defined whether the internal mbstate_t object has thread storage duration; if it has thread storage duration, it is initialized to the initial conversion state prior to the first call to the function on the new thread. The implementation behaves as if no library function calls these functions with a null pointer for ps.
    In J.3.12 (Library functions), add:
    — Whether internal mbstate_t objects have thread storage duration (7.28.1, 7.29.6.4).

Changes to the previous submissions