A Proposal to add Classes and Functions Required for Dynamic Library Load

I. Introduction and Motivation

Adding a specific features to an existing software applications at runtime could be useful in many cases. Such extensions, or plugins, are usually implemented using Shared Library files (DLL, SO/DSO) loaded into the memory of program at runtime.

Current C++ Standard lacks support for dynamic library loading (DLL). This proposal attempts to fix that and provide a simple to use classes and functions for DLL.

A proof of concept implementation available at: Boost.DLL.

II. Impact on the Standard

This proposal is a pure library extension. It does not propose changes to existing headers. It relies on N4099, because it uses class experimental::path from that proposal. It does not require any changes in the core language and it could be implemented in standard C++.

III. Design Decisions

A. Usage of std::experimental::filesystem::path

To simplify library usage std::experimental::filesystem::path is used for specifying paths . All the path related overhead is minor, comparing to the time of loading shared library file into the memory of program or getting information about shared library file location.

B. Stick to the Filesystem error reporting scheme.

Provide two overloads for some functions, one that throws an exception to report system errors, and another that sets an error_code. This supports two common use cases:

• Uses where shared library file related system errors are truly exceptional and indicate a serious failure.
• Uses where shared library file related errors are routine and do not necessarily represent failure.

C. Do not take care of mangling.

C++ symbol mangling depend on compiler and platform. Existing attempts to get mangled symbol by unmangled name result in significant complexity and memory usage growth and overall slowdown of getting symbol from shared library file.

[Example:
    // Loading a shared library file with mangled symbols
    mangled_library lib("libmycpp.so");

    // Attempt to get function "foo::bar" with signature void(int)
    lib.get<void(int)>("foo::bar");

    // The problem is that we do not know what `foo::bar` is:
    // * `foo` could be a class and `bar` could be a static member function
    // * `foo` could be a struct and `bar` could be a static member function
    // * `foo` could be a namespace and `bar` could be function in that namespace
    //
    // We also do not know the calling convention for `foo` and it's noexcept specification.
    //
    // Mangling of `foo::bar` depends on all the knowledge from above, so we are either forced to
    // mangle and try to obtain all the available combinations; or we need to investigate all the
    // symbols exported by "libmycpp.so" shared library file and find a best match.
- end example]

While no good solution was found for obtaining mangled symbol by unmangled name, this proposal concentrates on obtaining symbols by exact name match.

D. Provide functions for simple usage.

Keeping a shared library file loaded in memory of program while using the symbol obtained from shared library file could be hard and error prone for some users. For the simplicity of those users, import functions were provided. Those functions return a symbol wrapped in a helper class along with an instance of shared_library. In that way while returned value in in scope, the shared library file won't be unloaded from program memory, so the user does not need to take care of shared library file lifetime in program memory.

E. Do not search libraries in system specific paths by default.

Searching paths for a specified shared library file may affect load times and make the proposed wording less efficient. It is assumed to be the most common case, that user exactly knows were the desired shared library file is located and provides either absolute or relative to current directory path. For that case searching system specific paths affects performance and increases the chance of finding wrong shared library file with the same name. Because some operating systems search system paths even if relative path is provided, the requirement to not do that is implicitly described in proposed wording.

F. Minimal modifications for the N4567 wording.

There have been a lot of shared library related proposals:

• N1400: Toward standardization of dynamic libraries
• N1418: Dynamic Libraries in C++
• N1428: Draft Proposal for Dynamic Libraries in C++
• N1496: Draft Proposal for Dynamic Libraries in C++ (Revision 1)
• N1976: Dynamic Shared Objects: Survey and Issues
• N2015: Plugins in C++
• N2407: C++ Dynamic Library Support

The proposal you're reading attempts to avoid issues of the proposals from above, minifying the changes to the N4567 wording. The proposal you're reading:

• is a pure library extension (does not modify Basic Concepts [basic] and does not add new keywords)
• does not add platform requiremnt for shared libraries suport

IV. Proposed wording relative to N4567

31 Dynamic library load support library[dll]

31.1 General [dll.general]

'Dynamic library load' is a runtime mechanism to load shared library file into the memory of current program, retrieve the addresses of functions and objects contained in the shared library file, execute those functions or access those objects, unload the shared library file from memory.

'Shared library file' is a file containing set of objects and functions available for use at program startup or at runtime.

Headers <experimental/dll> and <experimental/import> define classes and functions suitable for dynamic library load. For those headers term 'symbol' relates to a function, reference, class member or object that can be obtained from shared library file at runtime. Term 'symbol name' relates to a character identifier of a symbol, using which symbol can be obtained from shared library file. For symbols declared with extern "C" in code of shared library file, 'symbol name' is the name of the entity. 'symbol name' without extern "C" in C++ code of shared library file is the mangled name of the entity.

31.2 Error reporting [dll.errors]

Functions not having an argument of type error_code& report errors as follows, unless otherwise specified:

• When a call by the implementation to an operating system or other underlying API results in an error that prevents the function from meeting its specifications, an exception of type system_error is thrown.
• Failure to allocate storage is reported by throwing an exception as described in the C++ standard, 17.6.4.10 [res.on.exception.handling].
• Destructors throw nothing.

• If a call by the implementation to an operating system or other underlying API results in an error that prevents the function from meeting its specifications, the error_code& argument is set as appropriate for the specific error. Otherwise, clear() is called on the error_code& argument.
• Failure to allocate storage is reported by throwing an exception as described in the C++ standard, 17.6.4.10 [res.on.exception.handling].

31.3 Header <experimental/dll> [dll.dll]

namespace std {
  namespace experimental {
  inline namespace dll_v1 {

    // shared library file load modes
    enum class dll_mode {
        default_mode = 0,
        dont_resolve_dll_references,    // DONT_RESOLVE_DLL_REFERENCES
        load_ignore_code_authz_level,   // LOAD_IGNORE_CODE_AUTHZ_LEVEL
        rtld_lazy,                      // RTLD_LAZY
        rtld_now,                       // RTLD_NOW
        rtld_global,                    // RTLD_GLOBAL
        rtld_local,                     // RTLD_LOCAL
        rtld_deepbind,                  // RTLD_DEEPBIND
        append_decorations,             // See [dll.dll_mode]
        search_system_directories       // See [dll.dll_mode]
    };

    constexpr dll_mode  operator| (dll_mode lhs, dll_mode rhs)  noexcept;
    constexpr dll_mode& operator|=(dll_mode& lhs, dll_mode rhs) noexcept;
    constexpr dll_mode  operator& (dll_mode lhs, dll_mode rhs)  noexcept;
    constexpr dll_mode& operator&=(dll_mode& lhs, dll_mode rhs) noexcept;
    constexpr dll_mode  operator^ (dll_mode lhs, dll_mode rhs)  noexcept;
    constexpr dll_mode& operator^=(dll_mode& lhs, dll_mode rhs) noexcept;
    constexpr dll_mode& operator~ (dll_mode& lhs) noexcept;

    // class to work with shared library files
    class shared_library;

    bool operator==(const shared_library& lhs, const shared_library& rhs) noexcept;
    bool operator!=(const shared_library& lhs, const shared_library& rhs) noexcept;
    bool operator< (const shared_library& lhs, const shared_library& rhs) noexcept;
    bool operator> (const shared_library& lhs, const shared_library& rhs) noexcept;
    bool operator<=(const shared_library& lhs, const shared_library& rhs) noexcept;
    bool operator>=(const shared_library& lhs, const shared_library& rhs) noexcept;

    // free functions
    template<class T>
    filesystem::path symbol_location(const T& symbol, error_code& ec);
    template<class T>
    boost::filesystem::path symbol_location(const T& symbol);

    filesystem::path this_line_location(error_code& ec);
    filesystem::path this_line_location();

    filesystem::path program_location(error_code& ec);
    filesystem::path program_location();

  }
  }

  // support
  template <class T> struct hash;
  template <> struct hash<experimental::shared_library>;
  template <> struct hash<experimental::dll_mode>;
}

The value of each enum dll_mode constant shall be the same as the value of the macro shown in the above synopsis if that macro is available on the platform or 0 otherwise.

31.3.1 Enum dll_mode [dll.dll_mode]

Enum dll_mode provides modes for searching the shared library file and platform specific modes for loading the shared library file in memory of program. [Note: library users may extend available modes by casting the required platform specific mode to dll_mode: [ Example: static_cast<dll_mode>(RTLD_NODELETE); — end example ] — end note ]

Each of system family provides own modes, flags not supported by a particular platform must be set to 0. Special modes are listened below:

append_decorations

        // Tries to open
        //      `./my_plugins/plugin1.dll` and `./my_plugins/libplugin1.dll` on Windows
        //      `./my_plugins/libplugin1.so` on Linux
        //      `./my_plugins/libplugin1.dylib` and `./my_plugins/libplugin1.so` on MacOS.
        // If that fails, loads `./my_plugins/plugin1`
        shared_library lib("./my_plugins/plugin1", dll_mode::append_decorations);
- end example]

default_mode

search_system_directories

31.3.1.1 dll_mode operators [dll.dll_mode.operators]

constexpr dll_mode  operator| (dll_mode lhs, dll_mode rhs)  noexcept;
constexpr dll_mode& operator|=(dll_mode& lhs, dll_mode rhs) noexcept;
constexpr dll_mode  operator& (dll_mode lhs, dll_mode rhs)  noexcept;
constexpr dll_mode& operator&=(dll_mode& lhs, dll_mode rhs) noexcept;
constexpr dll_mode  operator^ (dll_mode lhs, dll_mode rhs)  noexcept;
constexpr dll_mode& operator^=(dll_mode& lhs, dll_mode rhs) noexcept;
constexpr dll_mode& operator~ (dll_mode& lhs) noexcept;

31.3.1.2 dll_mode hash support[dll.dll_mode.hash]

template <> struct hash<experimental::dll_mode>;

The template specialization shall meet the requirements of class template hash (20.9.13).

31.3.2 Class shared_library [dll.shared_library]

The class shared_library provides means to load shared library file into the memory of program, check that shared library file exports symbol with specified symbol name and obtain that symbol.

shared_library instances share reference count to an actual shared library file loaded in memory of current program, so it is safe and memory efficient to have multiple instances of shared_library referencing the same shared library file even if those instances were loaded in memory using different paths (relative and absolute) referencing the same shared library file.

It must be safe to concurrently load shared library files into memory of program, unload and get symbols from any shared library files using different shared_library instances. If current platform does not guarantee safe concurrent work with shared library files, calls to shared_library functions must be serialized by shared_library implementation.

[ Note: Constructors, comparisons and reset() functions that accept nullptr_t are not provided because that may cause confusion: some of the platforms provide interfaces for shared library files that accept nullptr_t to get a handler to the current program. — end note ]

namespace std {
  namespace experimental {
  inline namespace dll_v1 {

    class shared_library {
    public:
      typedef platform_specific native_handle_type;

      // construct/copy/destruct
      shared_library() noexcept;
      shared_library(const shared_library& lib);
      shared_library(const shared_library& lib, error_code& ec);
      shared_library(shared_library&& lib) noexcept;
      explicit shared_library(const filesystem::path& library_path, dll_mode mode = dll_mode::default_mode);
      shared_library(const filesystem::path& library_path, error_code& ec, dll_mode mode = dll_mode::default_mode);
      shared_library(const filesystem::path& library_path, dll_mode mode, error_code& ec);
      ~shared_library();

      // public member functions
      shared_library& assign(const shared_library& lib, error_code& ec);
      shared_library& assign(const shared_library& lib);
      
      shared_library& operator=(const shared_library& lib);
      shared_library& operator=(shared_library&& lib) noexcept;

      void load(const filesystem::path& library_path, dll_mode mode = dll_mode::default_mode);
      void load(const filesystem::path& library_path, error_code& ec, dll_mode mode = dll_mode::default_mode);
      void load(const filesystem::path& library_path, dll_mode mode, error_code& ec);

      void reset() noexcept;
      explicit operator bool() const noexcept;

      bool has(const char* symbol_name) const noexcept;
      bool has(const string& symbol_name) const noexcept;

      template <typename SymbolT>
      auto get(const char* symbol_name) const
        -> conditional_t<is_member_pointer_v<SymbolT>, SymbolT, add_lvalue_reference_t<SymbolT>>;

      template <typename SymbolT>
      auto get(const string& symbol_name) const
        -> conditional_t<is_member_pointer_v<SymbolT>, SymbolT, add_lvalue_reference_t<SymbolT>>;

      native_handle_type native_handle() const noexcept;

      filesystem::path location() const;
      filesystem::path location(error_code& ec) const;

      // public static member functions
      static constexpr bool platform_supports_dll() noexcept;
      static constexpr bool platform_supports_dll_of_program() noexcept;
    };

  }
  }
}

31.3.3.1 shared_library constructors [dll.shared_library.constr]

shared_library() noexcept;

shared_library(const shared_library& lib);
shared_library(const shared_library& lib, error_code& ec);

shared_library(shared_library&& lib) noexcept;

explicit shared_library(const filesystem::path& library_path, dll_mode mode = dll_mode::default_mode);
shared_library(const filesystem::path& library_path, error_code& ec, dll_mode mode = dll_mode::default_mode);
shared_library(const filesystem::path& library_path, dll_mode mode, error_code& ec);

31.3.3.2 shared_library destructor [dll.shared_library.destr]

~shared_library();

31.3.3.3 shared_library members[dll.shared_library.member]

shared_library& assign(const shared_library& lib, error_code& ec);
shared_library& assign(const shared_library& lib);

31.3.3.4 shared_library assignment [dll.shared_library.assign]

shared_library& operator=(const shared_library& lib);

shared_library& operator=(shared_library&& lib) noexcept;

void load(const filesystem::path& library_path, dll_mode mode = dll_mode::default_mode);
void load(const filesystem::path& library_path, error_code& ec, dll_mode mode = dll_mode::default_mode);
void load(const filesystem::path& library_path, dll_mode mode, error_code& ec);

void reset() noexcept;

explicit operator bool() const noexcept;

bool has(const char* symbol_name) const noexcept;
bool has(const string& symbol_name) const noexcept;

template <typename SymbolT>
auto get(const char* symbol_name) const
    -> conditional_t<is_member_pointer_v<SymbolT>, SymbolT, add_lvalue_reference_t<SymbolT>>;

template <typename SymbolT>
auto get(const string& symbol_name) const
    -> conditional_t<is_member_pointer_v<SymbolT>, SymbolT, add_lvalue_reference_t<SymbolT>>;

native_handle_type native_handle() const noexcept;

filesystem::path location() const;
filesystem::path location(error_code& ec) const;

31.3.3.5 shared_library static members[dll.shared_library.static]

static constexpr bool platform_supports_dll() noexcept;

static constexpr bool platform_supports_dll_of_program() noexcept

31.3.3.6 shared_library free operators[dll.shared_library.operators]

shared_library provides fast comparison operators that compare the referenced shared libraries file.

bool operator==(const shared_library& lhs, const shared_library& rhs) noexcept;

bool operator!=(const shared_library& lhs, const shared_library& rhs) noexcept;

bool operator<(const shared_library& lhs, const shared_library& rhs) noexcept;

bool operator>(const shared_library& lhs, const shared_library& rhs) noexcept;

bool operator<=(const shared_library& lhs, const shared_library& rhs) noexcept;

bool operator>=(const shared_library& lhs, const shared_library& rhs) noexcept;

31.3.3.7 shared_library hash support[dll.shared_library.hash]

template <> struct hash<experimental::shared_library>;

31.3.4 Runtime path functions[dll.location]

template<class T>
filesystem::path symbol_location(const T& symbol, error_code& ec);
template<class T>
boost::filesystem::path symbol_location(const T& symbol);

filesystem::path this_line_location(error_code& ec);
filesystem::path this_line_location();

filesystem::path program_location(error_code& ec);
filesystem::path program_location();

31.4 Header <experimental/import> [dll.import.header]

#include <experimental/dll>
#include <type_traits>
#include <memory>
#include <functional>

namespace std {
  namespace experimental {
  inline namespace dll_v1 {

    template <typename SymbolT>
    using imported_t = conditional_t<is_object_v<SymbolT>, shared_ptr<SymbolT>, function<SymbolT>>;

    // functions for importing a symbol
    template <typename SymbolT>
    imported_t<SymbolT> import(const filesystem::path& library_path, const char* symbol_name, dll_mode mode = dll_mode::default_mode);

    template <typename SymbolT>
    imported_t<SymbolT> import(const filesystem::path& library_path, const string& symbol_name, dll_mode mode = dll_mode::default_mode);

    template <typename SymbolT>
    imported_t<SymbolT> import(const shared_library& library, const char* symbol_name);

    template <typename SymbolT>
    imported_t<SymbolT> import(const shared_library& library, const string& symbol_name);

    template <typename SymbolT>
    imported_t<SymbolT> import(shared_library&& library, const char* symbol_name);

    template <typename SymbolT>
    imported_t<SymbolT> import(shared_library&& library, const string& symbol_name);

  }
  }
}

import functions are meant to simplify dynamic library loads of symbols by keeping shared library file loaded in program memory while imported symbol is in scope:

[Example:
    // Code of "/plugin_directory/libplugin.so" contains extern "C" void foo_function(string&&)

    auto foo_function = import<void(string&&)>("/plugin_directory/libplugin.so", "foo_function");
    foo_function("Test");
- end example]

31.4.1 import functions [dll.import.func]

template <typename SymbolT>
imported_t<SymbolT> import(const filesystem::path& library_path, const char* symbol_name, dll_mode mode = dll_mode::default_mode);

template <typename SymbolT>
imported_t<SymbolT> import(const filesystem::path& library_path, const string& symbol_name, dll_mode mode = dll_mode::default_mode);

template <typename SymbolT>
imported_t<SymbolT> import(const shared_library& library, const char* symbol_name);

template <typename SymbolT>
imported_t<SymbolT> import(const shared_library& library, const string& symbol_name);

template <typename SymbolT>
imported_t<SymbolT> import(shared_library&& library, const char* symbol_name);

template <typename SymbolT>
imported_t<SymbolT> import(shared_library&& library, const string& symbol_name);

V. Feature-testing macro

For the purposes of SG10 it is sufficient to check for header <experimental/dll> or <experimental/import> using __has_include.

V. Revision History

Revision 0:

• Initial proposal

VI. References

[Boost.DLL] Boost DLL library. Available online at https://github.com/boostorg/dll

[N4567] Working Draft, Standard for Programming Language C++. Available online at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4567.pdf

[N4099] Working Draft, Technical Specification — File System. Available online at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4099.html

[N1400] Toward standardization of dynamic libraries. Available online at http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2002/n1400.html

[N1418] Dynamic Libraries in C++. Available online at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2002/n1418.html

[N1428] Draft Proposal for Dynamic Libraries in C++. Available online at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2003/n1428.html

[N1496] Draft Proposal for Dynamic Libraries in C++ (Revision 1). Available online at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2003/n1496.html

[N1976] Dynamic Shared Objects: Survey and Issues. Available online at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n1976.html

[N2015] Plugins in C++. Available online at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2015.pdf

[N2407] C++ Dynamic Library Support. Available online at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2407.html

VII. Acknowledgements

Klemens Morgenstern highlighted some of the missing functionality in Boost.DLL and provided implementation of mangled symbols load, that showed complexities of such approach. Renato Tegon Forti started the work on Boost.DLL and provided a lot of code, help and documentation for the Boost.DLL.