Document number: p0544R0
Date: 2017-02-01
Audience: Library Evolution Working Group
Reply to: Titus Winters <titus@google.com>, Geoff Romer <gromer@google.com>

User Injection of Filesystems

  1. Rationale
  2. Design Choices
  3. Proposed Wording

Rationale

The existing proposals in the Filesystem TS (now adopted for inclusion with C++17) effectively specify the C++ filesystem API as a set of C++ wrappers and types (like directory_iterator) directly on top of whatever is provided by the OS for filesystem interaction. This is easy to specify and implement, but inflexible and likely does not provide the functionality needed for a modern C++ codebase — for example, it is impossible to fake filesystem errors in testing, or to use the same interfaces for in-memory filesystems (avoiding the essential OS and I/O overhead). Our belief, based on extensive experience in the Google codebase and production systems, is that most robust C++ installations will wind up wrapping the proposed interface in order to provide opportunities to change the backing store, simulate errors for testing, etc. To that end, we propose the definition of a filesystem type, and injection mechanisms to allow user-customization of the filesystem backend used by the free functions provided by the Filesystem TS.

This proposal is pure-library, OS and compiler agnostic, and is backwards compatible with any existing code written against the Filesystem TS. It has not yet been implemented in this form. However, the filesystems at use in Google production work in roughly this way with small differences. In the Google design, multiple filesystems are present at once, they are all pinned into the filesystem at some root (/memfile, /virtualfs, etc) with the OS local filesystem as the generic fallback. This works for us because we have no symlinks. Once symlinks are in the equation, the design has to shift to something like what we are describing.

Design Choices

The biggest point of discussion in the design for filesystem injection is how to specify where injection takes place. Is it a new filesystem root? If so, how is it specified (especially on POSIX systems)? If you can inject at any point in a filesystem, does that injection respect symlinks? If so, that requires one (or more) OS calls on every filesystem call to resolve symlinks in order to determine if the specified path(s) fall inside the injection point. Since we cannot determine one policy that will satisfy all users for the above questions, we instead choose to present a more general injection mechanism: replace the filesystem entirely. In conjunction with the ability to get the “default” filesystem, a user can then build a solution that forwards to the underlying filesystem in a fashion congruent with any of the above designs.

Under this design, the expected cost is bounded by following an atomic pointer plus a vtable dereference. In comparison to the overhead of performing I/O through the OS, this is expected to be negligible.

Proposed Wording

Changes are relative to N4582.

Add the following to the <filesystem> synopsis in [fs.filesystem.syn]:

class filesystem_error;
class directory_entry;

class filesystem;
class default_filesystem_impl;  // exposition only
class directory_traverser;

class directory_iterator;

// range access for directory_iterator
directory_iterator begin(directory_iterator iter) noexcept;
directory_iterator end(const directory_iterator&) noexcept;

bool operator==(const directory_iterator& lhs, const directory_iterator& rhs);
bool operator!=(const directory_iterator& lhs, const directory_iterator& rhs);

class recursive_directory_iterator;
typedef chrono::time_point<trivial-clock> file_time_type;

// filesystem injection
extern atomic<filesystem*> current_filesystem;  // exposition only
void inject_filesystem(filesystem& fs);
filesystem& default_filesystem();

// filesystem operation functions

Add a new paragraph to the end of [fs.err.report] as follows:

When a function that does not take an error_code parameter is specified to call a function that takes an error_code parameter, the outer function is understood to construct a local error_code variable ec to pass to the function call. If !ec after the call completes, unless otherwise specified the function throws a filesystem_error exception containing ec, an implementation-defined message, and any paths that were passed to the function.

Insert new sections above [class.directory_iterator]:

27.10.? Class filesystem [class.filesystem]

Class filesystem defines the base class for the types of objects that provide access to the fundamental operations of a specific file system. A special filesystem object ([class.default_filesystem_impl]) provides access to the native file system provided by the operating system, and other filesystem objects can implement virtual file systems for purposes such as testing.

Classes derived from filesystem shall satisfy all of the requirements of filesystem.

class filesystem {
public:
  filesystem() = default;
  virtual ~filesystem();
  filesystem(const filesystem&) = delete;
  filesystem& operator=(const filesystem&) = delete;

  virtual bool copy_file(const path& from, const path& to,
                         copy_options options, error_code& ec) noexcept = 0;
  virtual bool create_directory(const path& p, error_code& ec) noexcept = 0;
  virtual bool create_directory(const path& p, const path& existing_p,
                                error_code& ec) noexcept = 0;
  virtual void create_directory_symlink(
      const path& to, const path& new_symlink, error_code& ec) noexcept = 0;
  virtual void create_hard_link(const path& to, const path& new_hard_link,
                                error_code& ec) noexcept = 0;
  virtual void create_symlink(const path& to, const path& new_symlink,
                              error_code& ec) noexcept = 0;
  virtual path current_path(error_code& ec) const = 0;
  virtual void current_path(const path& p, error_code& ec) noexcept = 0;
  virtual bool equivalent(const path& p1, const path& p2, error_code& ec) const noexcept = 0;
  virtual uintmax_t file_size(const path& p, error_code& ec) const noexcept = 0;
  virtual uintmax_t hard_link_count(const path& p, error_code& ec) const noexcept = 0;
  virtual file_time_type last_write_time(const path& p, error_code& ec) const noexcept = 0;
  virtual void last_write_time(const path& p, file_time_type new_time,
                               error_code& ec) const noexcept = 0;
  virtual void permissions(const path& p, perms prms, error_code& ec) const noexcept = 0;
  virtual path read_symlink(const path& p, error_code& ec) const = 0;
  virtual bool remove(const path& p, error_code& ec) noexcept = 0;
  virtual void rename(const path& old_p, const path& new_p, error_code& ec) noexcept = 0;
  virtual void resize_file(const path& p, uintmax_t new_size, error_code& ec) noexcept = 0;
  virtual space_info space(const path& p, error_code& ec) const noexcept = 0;
  virtual file_status status(const path& p, error_code& ec) const noexcept = 0;
  virtual file_status symlink_status(const path& p, error_code& ec) const noexcept = 0;
  virtual path system_complete(const path& p, error_code& ec) const = 0;
  virtual path temp_directory_path(error_code& ec) = 0;

  virtual unique_ptr<directory_traverser> directory_traverser(
      const path& p, directory_options options, error_code& ec) const = 0;
};

27.10.?.1 filesystem members [filesystem.members]

27.10.?.1.1 Copy file [filesystem.copy_file]

virtual bool copy_file(const path& from, const path& to,
                       copy_options options, error_code& ec) noexcept = 0;

Copy [fs.op.copy_file] paragraphs 3 to 9 here, with edits as specified:

27.10.?.1.2 Create directory [filesystem.create_directory]

virtual bool create_directory(const path& p, error_code& ec) noexcept = 0;

Copy [fs.op.create_directory] paragraphs 1 to 4 here, with edits as specified:

virtual bool create_directory(const path& p, const path& existing_p,
                              error_code& ec) noexcept = 0;

Copy [fs.op.create_directory] paragraphs 5 to 8 here, with edits as specified:

27.10.?.1.3 Create directory symlink [filesystem.create_directory_symlink]

virtual void create_directory_symlink(
    const path& to, const path& new_symlink, error_code& ec) noexcept = 0;

Copy [fs.op.create_dir_symlk] paragraphs 1 to 5 here, with edits as specified:

27.10.?.1.4 Create hard link [filesystem.create_hard_link]

virtual void create_hard_link(const path& to, const path& new_hard_link,
                              error_code& ec) noexcept = 0;

Copy [fs.op.create_hard_lk] paragraphs 1 to 4 here, with edits as specified:

27.10.?.1.5 Create symlink [filesystem.create_symlink]

virtual void create_symlink(const path& to, const path& new_symlink,
                            error_code& ec) noexcept = 0;

Copy [fs.op.create_symlink] paragraphs 1 to 4 here, with edits as specified:

27.10.?.1.6 Current path [filesystem.current_path]

virtual path current_path(error_code& ec) const = 0;

Copy [fs.op.current_path] paragraphs 1 to 5 here, with edits as specified:

virtual void current_path(const path& p, error_code& ec) noexcept = 0;

Copy [fs.op.current_path] paragraphs 6 to 9 here, with edits as specified:

27.10.?.1.7 Equivalent [filesystem.equivalent]

virtual bool equivalent(const path& p1, const path& p2, error_code& ec) const noexcept = 0

Copy [fs.op.equivalent] paragraphs 1 to 4 here, with edits as specified:

27.10.?.1.8 File size [filesystem.file_size]

virtual uintmax_t file_size(const path& p, error_code& ec) const noexcept = 0;

Copy [fs.op.file_size] paragraphs 1 to 2 here, with edits as specified:

27.10.?.1.9 Hard link count [filesystem.hard_link_count]

virtual uintmax_t hard_link_count(const path& p, error_code& ec) const noexcept = 0;

Copy [fs.op.hard_lk_ct] paragraphs 1 to 2 here, with edits as specified:

27.10.?.1.10 Last write time [filesystem.last_write_time]

virtual file_time_type last_write_time(const path& p, error_code& ec) const noexcept = 0;

Copy [fs.op.last_write_time] paragraphs 1 to 2 here, with edits as specified:

virtual void last_write_time(const path& p, file_time_type new_time,
                             error_code& ec) const noexcept = 0;

Copy [fs.op.last_write_time] paragraphs 3 to 5 here, with edits as specified:

27.10.?.1.11 Permissions [filesystem.permissions]

virtual void permissions(const path& p, perms prms, error_code& ec) const noexcept = 0;

Copy [fs.op.permissions] paragraphs 1 to 4 here, with edits as specified:

27.10.?.1.12 Read symlink [filesystem.read_symlink]

virtual path read_symlink(const path& p, error_code& ec) const noexcept = 0;

Copy [fs.op.read_symlink] paragraphs 1 to 2 here, with edits as specified:

27.10.?.1.13 Remove [filesystem.remove]

virtual bool remove(const path& p, error_code& ec) noexcept = 0;

Copy [fs.op.remove] paragraphs 1 to 5 here, with edits as specified:

27.10.?.1.14 Rename [filesystem.rename]

virtual void rename(const path& old_p, const path& new_p, error_code& ec) noexcept = 0;

Copy [fs.op.rename] paragraphs 1 to 3 here, with edits as specified:

27.10.?.1.15 Resize file [filesystem.resize_file]

virtual void resize_file(const path& p, uintmax_t new_size, error_code& ec) noexcept = 0;

Copy [fs.op.resize_file] paragraphs 1 to 3 here, with edits as specified:

27.10.?.1.16 Space [filesystem.space]

virtual space_info space(const path& p, error_code& ec) const noexcept = 0;

Copy [fs.op.space] paragraphs 1 to 3 here, with edits as specified:

27.10.?.1.17 Status [filesystem.status]

virtual file_status status(const path& p, error_code& ec) const noexcept = 0;

Copy [fs.op.status] paragraphs 4 to 9 here, with edits as specified:

27.10.?.1.18 Symlink status [filesystem.symlink_status]

virtual file_status symlink_status(const path& p, error_code& ec) const noexcept = 0;

Copy [fs.op.symlink_status] paragraphs 1 to 4 here, with edits as specified:

27.10.?.1.19 System complete [filesystem.system_complete]

virtual path system_complete(const path& p error_code& ec) const = 0;

Copy [fs.op.system_complete] paragraphs 1 to 6 here, with edits as specified:

27.10.?.1.20 Temp directory path [filesystem.temp_directory_path]

virtual path temp_directory_path(error_code& ec) const = 0;

Copy [fs.op.temp_dir_path] paragraphs 1 to 4 here, with edits as specified:

27.10.?.1.21 Directory traverser [filesystem.directory_traverser]

virtual unique_ptr<directory_traverser> directory_traverser(
    const path& p, directory_options options, error_code& ec) const = 0;

Copy [directory_iterator.members] paragraphs 2 to 4 here, with edits as specified:

27.10.? Class default_filesystem_impl [class.default_filesystem_impl]

Class default_filesystem_impl is for exposition only. An implementation is permitted to provide equivalent functionality without providing a class with this name. This class provides a default implementation of the filesystem interface, which is implemented by delegating to the underlying operating system.

Descriptions are provided only where this class differs from the base class filesystem.

class default_filesystem_impl : public filesystem {  // exposition only
public:
  virtual bool copy_file(const path& from, const path& to,
                         copy_options options, error_code& ec) noexcept;
  virtual bool create_directory(const path& p, error_code& ec) noexcept;
  virtual bool create_directory(const path& p, const path& existing_p,
                                error_code& ec) noexcept;
  virtual void create_directory_symlink(
      const path& to, const path& new_symlink, error_code& ec) noexcept;
  virtual void create_hard_link(const path& to, const path& new_hard_link,
                                error_code& ec) noexcept;
  virtual void create_symlink(const path& to, const path& new_symlink,
                              error_code& ec) noexcept;
  virtual path current_path(error_code& ec);
  virtual void current_path(const path& p, error_code& ec) noexcept;
  virtual bool equivalent(const path& p1, const path& p2, error_code& ec) noexcept;
  virtual uintmax_t file_size(const path& p, error_code& ec) noexcept;
  virtual uintmax_t hard_link_count(const path& p, error_code& ec) noexcept;
  virtual file_time_type last_write_time(const path& p, error_code& ec) noexcept;
  virtual void last_write_time(const path& p, file_time_type new_time,
                               error_code& ec) noexcept;
  virtual void permissions(const path& p, perms prms, error_code& ec) noexcept;
  virtual path read_symlink(const path& p, error_code& ec);
  virtual bool remove(const path& p, error_code& ec) noexcept;
  virtual void rename(const path& old_p, const path& new_p, erorr_code& ec) noexcept;
  virtual void resize_file(const path& p, uintmax_t new_size, error_code& ec) noexcept;
  virtual space_info space(const path& p, error_code& ec) noexcept;
  virtual file_status status(const path& p, error_code& ec) noexcept;
  virtual file_status symlink_status(const path& p, error_code& ec) noexcept;
  virtual path system_complete(const path& p, error_code& ec);
  virtual path temp_directory_path(error_code& ec);

  virtual unique_ptr<directory_traverser> directory_traverser(
      const path& p, directory_options options, error_code& ec);
};

27.10.?.1 filesystem members [default_filesystem_impl.members]

27.10.?.1.1 Create directory [default_filesystem_impl.create_directory]

virtual bool create_directory(const path& p, error_code& ec) noexcept;

27.10.?.1.2 Create directory symlink [default_filesystem_impl.create_directory_symlink]

virtual void create_directory_symlink(
    const path& to, const path& new_symlink, error_code& ec) noexcept;

27.10.?.1.3 Create hard link [default_filesystem.create_hard_link]

virtual void create_hard_link(const path& to, const path& new_hard_link,
                              error_code& ec) noexcept;

27.10.?.1.4 Create symlink [default_filesystem.create_symlink]

virtual void create_symlink(const path& to, const path& new_symlink,
                            error_code& ec) noexcept;

27.10.?.1.5 Current path [default_filesystem.current_path]

virtual path current_path(error_code& ec);
virtual void current_path(const path& p, error_code& ec) noexcept;

27.10.?.1.6 Equivalent [default_filesystem.equivalent]

virtual bool equivalent(const path& p1, const path& p2, error_code& ec) noexcept;

27.10.?.1.7 File size [default_filesystem.file_size]

virtual uintmax_t file_size(const path& p, error_code& ec) noexcept;

27.10.?.1.8 Last write time[default_filesystem.last_write_time]

virtual file_time_type last_write_time(const path& p, error_code& ec) noexcept;
virtual void last_write_time(const path& p, file_time_type new_time,
                             error_code& ec) noexcept;

27.10.?.1.9 Permissions [default_filesystem.permissions]

virtual void permissions(const path& p, perms prms, error_code& ec) noexcept;

27.10.?.1.10 Remove [default_filesystem.remove]

virtual bool remove(const path& p, error_code& ec) noexcept;

27.10.?.1.11 Rename [default_filesystem.rename]

virtual void rename(const path& old_p, const path& new_p, error_code& ec) noexcept;

27.10.?.1.12 Resize file [default_filesystem.resize_file]

virtual void resize_file(const path& p, uintmax_t new_size, error_code& ec) noexcept;

27.10.?.1.13 Space [default_filesystem.space]

virtual space_info space(const path& p, error_code& ec) noexcept;

27.10.?.1.14 Status [default_filesystem.status]

virtual file_status status(const path& p, error_code& ec) noexcept;

27.10.?.1.15 Symlink status [default_filesystem.symlink_status]

virtual file_status symlink_status(const path& p, error_code& ec) noexcept;

27.10.? Class directory_traverser [class.directory_traverser]

class directory_traverser {
public:
  directory_traverser() = default;
  virtual ~directory_traverser() = default;

  virtual const directory_entry& dereference() const = 0;
  virtual void increment(error_code& ec) noexcept = 0;
  virtual bool equal(const directory_traverser& rhs) const = 0;
  virtual bool is_end() const noexcept = 0;
  virtual unique_ptr<directory_traverser> clone() const = 0;
};

The directory_traverser class defines the base class for the types of objects that represent how a directory in a particular filesystem implementation is traversed. An instance of such a type is either a past-the-end value, or points to an element of a sequence of directory_entry objects representing the contents of the directory.

Classes derived from directory_traverser shall satisfy all of the requirements of directory_traverser.

[Note: If a file is removed from or added to a directory after the construction of a directory_traverser for the directory, it is unspecified whether or not subsequently incrementing the directory_traverser will ever result in it pointing to the removed or added directory entry. See POSIX readdir_r. — end note]

27.10.?.1 directory_traverser members [directory_traverser.members]

virtual const directory_entry& dereference() const = 0;
virtual void increment(error_code& ec) = 0;
virtual bool equal(const directory_traverser& rhs) const = 0;
virtual bool is_end() const = 0;
virtual unique_ptr<directory_traverser> clone() const = 0;

Revise the directory_iterator class synopsis as follows:

namespace std::filesystem {
  class directory_iterator {
  public:
    typedef directory_entry        value_type;
    typedef ptrdiff_t              difference_type;
    typedef const directory_entry* pointer;
    typedef const directory_entry& reference;
    typedef input_iterator_tag     iterator_category;

    // member functions
    directory_iterator() noexcept;
    explicit directory_iterator(const path& p);
    directory_iterator(const path& p, directory_options options);
    directory_iterator(const path& p, error_code& ec) noexcept;
    directory_iterator(const path& p, directory_options options,
                       error_code& ec) noexcept;
    directory_iterator(const directory_iterator& rhs);
    directory_iterator(directory_iterator&& rhs) noexcept;
   ~directory_iterator();

    directory_iterator& operator=(const directory_iterator& rhs);
    directory_iterator& operator=(directory_iterator&& rhs) noexcept;

    const directory_entry& operator*() const;
    const directory_entry* operator->() const;
    directory_iterator&    operator++();
    directory_iterator&    increment(error_code& ec) noexcept;

    // other members as required by 24.2.3, input iterators

  private:
    unique_ptr<directory_traverser> traverser;   // exposition only
  };
}

Revise [directory_iterator.members] as follows:

27.10.13.1 directory_iterator members [directory_iterator.members]

directory_iterator() noexcept;
explicit directory_iterator(const path& p);
directory_iterator(const path& p, directory_options options);
directory_iterator(const path& p, error_code& ec) noexcept;
directory_iterator(const path& p, directory_options options, error_code& ec) noexcept;
directory_iterator(const directory_iterator& rhs);
directory_iterator(directory_iterator&& rhs);
directory_iterator& operator=(const directory_iterator& rhs);
directory_iterator& operator=(directory_iterator&& rhs) noexcept;
const directory_entry& operator*() const;
const directory_entry* operator->() const;
directory_iterator& operator++();
directory_iterator& increment(error_code& ec) noexcept;

Add the following to the end of [directory_iterator.nonmembers]:

bool operator==(const directory_iterator& lhs, const directory_iterator& rhs);
bool operator!=(const directory_iterator& lhs, const directory_iterator& rhs);

Insert a new section above [fs.op.funcs]:

27.10.? Filesystem injection [fs.inject]

27.10.?.1 Current filesystem [fs.inject.current]

extern atomic<filesystem*> current_filesystem;

The current_filesystem variable is for exposition only. An implementation is permitted to provide equivalent functionality without providing a variable with this name.

current_filesystem will be initialized to &default_filesystem() before the body of main() begins execution. [Note Implementations are encouraged to initialize before main() begins execution. --end note]

27.10.?.2 Inject filesystem [fs.inject.inject]

void inject_filesystem(filesystem& fs);

27.10.?.3 Default filesystem [fs.inject.default]

filesystem& default_filesystem();

Revise [fs.op.copy] paragraph 6 as follows:

Otherwise if is_regular_file(f), then:

Revise [fs.op.copy_file] as follows:

bool copy_file(const path& from, const path& to);
bool copy_file(const path& from, const path& to, error_code& ec) noexcept;
bool copy_file(const path& from, const path& to, copy_options options);
bool copy_file(const path& from, const path& to, copy_options options,
               error_code& ec) noexcept;

Revise [fs.op.create_directory] as follows:

bool create_directory(const path& p);
bool create_directory(const path& p, error_code& ec) noexcept;
bool create_directory(const path& p, const path& existing_p);
bool create_directory(const path& p, const path& existing_p, error_code& ec) noexcept;

Revise [fs.op.create_dir_symlk] as follows:

void create_directory_symlink(const path& to, const path& new_symlink);
void create_directory_symlink(const path& to, const path& new_symlink,
                              error_code& ec) noexcept;

Revise [fs.op.create_hard_lk] as follows:

void create_hard_link(const path& to, const path& new_hard_link);
void create_hard_link(const path& to, const path& new_hard_link,
                      error_code& ec) noexcept;

Revise [fs.op.create_symlink] as follows:

void create_symlink(const path& to, const path& new_symlink);
void create_symlink(const path& to, const path& new_symlink,
                    error_code& ec) noexcept;

Revise [fs.op.current_path] as follows:

path current_path();
path current_path(error_code& ec);
void current_path(const path& p);
void current_path(const path& p, error_code& ec) noexcept;

Revise [fs.op.equivalent] as follows:

bool equivalent(const path& p1, const path& p2);
bool equivalent(const path& p1, const path& p2, error_code& ec) noexcept;

Revise [fs.op.file_size] as follows:

uintmax_t file_size(const path& p);
uintmax_t file_size(const path& p, error_code& ec) noexcept;

Revise [fs.op.hard_lk_ct] as follows:

uintmax_t hard_link_count(const path& p);
uintmax_t hard_link_count(const path& p, error_code& ec) noexcept;

Revise [fs.op.last_write_time] as follows:

file_time_type last_write_time(const path& p);
file_time_type last_write_time(const path& p, error_code& ec) noexcept;
void last_write_time(const path& p, file_time_type new_time);
void last_write_time(const path& p, file_time_type new_time,
                     error_code& ec) noexcept;

Revise [fs.op.permissions] as follows:

void permissions(const path& p, perms prms);
void permissions(const path& p, perms prms, error_code& ec) noexcept;

Revise [fs.op.read_symlink] as follows:

path read_symlink(const path& p);
path read_symlink(const path& p, error_code& ec);

Revise [fs.op.remove] as follows:

bool remove(const path& p);
bool remove(const path& p, error_code& ec) noexcept;

Revise [fs.op.rename] as follows:

void rename(const path& old_p, const path& new_p);
void rename(const path& old_p, const path& new_p, error_code& ec) noexcept;

Revise [fs.op.resize_file] as follows:

void resize_file(const path& p, uintmax_t new_size);
void resize_file(const path& p, uintmax_t new_size, error_code& ec) noexcept;

Revise [fs.op.space] as follows:

space_info space(const path& p);
space_info space(const path& p, error_code& ec) noexcept;

Revise [fs.op.status] as follows:

file_status status(const path& p);
file_status status(const path& p, error_code& ec) noexcept;

Revise [fs.op.symlink_status] as follows:

file_status symlink_status(const path& p);
file_status symlink_status(const path& p, error_code& ec) noexcept;

Revise [fs.op.system_complete] as follows:

path system_complete(const path& p);
path system_complete(const path& p, error_code& ec);

Revise [fs.op.temp_dir_path] as follows:

path temp_directory_path();
path temp_directory_path(error_code& ec);