Toward standardization of dynamic libraries

Matt Austern <austern@apple.com>
25 Sep 2002
N1400=02-0058

Motivation

Essentially all modern operating systems support dynamic shared libraries. Programmers rely on this feature. The C++ Standard has nothing to say about it, and OS vendors who have implemented dynamic libraries tend to have little to say about C++. There is wide variation between systems.

Vendors tend to justify this variation by saying that any program that uses dynamic libraries is outside the scope of the C++ Standard. Unfortunately,this appears to be true. The Standard says what a program is (section 3.5) and how it's put together (section 2.1). These definitions are inapplicable to a modern environment that includes dynamic linking. We run the risk of having a standard that doesn't apply to any of the programs people write.

My goal is to make it possible for users to write at least some portable programs that use dynamic libraries. We don't have to solve all problems, and we can't: there's too much variation between OSs. What we can hope to do is standardize a large enough subset of dynamic library semantics so that it's possible to write programs that are both interesting and portable, and to describe which aspects may vary from system to system. I do not intend to address component models (e.g. CORBA) or cross-language issues.

This is an informal document. The next step is an issues list, and the step after that, if we can come to an agreement on basic directions, is formal standardese.

Terminology

I'd like to avoid the word "program", since, in the context of dynamic libraries, it can mean several things. (None of which is quite what it means in the C++ standard.) Instead, some new terms:

A load unit is a series of translation units linked together by a static linker. A load unit may be an executable or a dynamic library. A load unit may contain unresolved symbols: it may depend on those symbols being defined in some other load unit.

A load unit's direct dependencies are the load units that the static linker sees when the load unit is being built, and that it explicitly depends on. A load unit's set of dependencies is the transitive closure of the direct dependencies. It can be defined recursively as follows: for a load unit with no direct dependencies its set of dependencies is the empty set, and otherwise its indirect dependencies are the union of the its direct dependencies and each of their dependencies. A load unit's indirect dependencies are its dependencies that are not direct dependencies.

A load set is an executable, and zero or more dynamic libraries, linked together. The load units that make up a load set are linked together by a dynamic linker, typically at, or even after, program start time. The dynamic linker loads a load set by starting with an executable and then automatically finding all of the executable's indirect dependencies.

A loadable library is a dynamic library that is part of a load set but that is not an indirect dependency of the executable. It is loaded into the load set by an explicit function call under the control of the programmer. These kinds of libraries are often called "plugins" or "bundles". Some OSs, but not all, enforce a rigourous distinction between loadable libraries and libraries used as load unit's dependencies.

Usage models

We can identify several different ways in which dynamic libraries may be used:

General principles:

It won't always be possible to satisfy these goals simultaneously.

Issues

Linkage

On all systems I know of that support dynamic libraries, a load unit can serve as an extra layer of scope, intermediate between a translation unit and the load set as a whole. So, for example, a symbol can be external in that it's shared between all of the translation units in a load unit, but private in that it can't be seen from other load units. I believe the Standard must take this feature into account. First, it's universally available. Second, it's a major reason that programmers organize their code into dynamic libraries.

The first issue is one of vocabulary: what words do we use to make this distinction? The three most obvious choices, public, external, and exported, are already used for other purposes! I suggest that we call a symbol global if it is intended to be visible to other load units than the one where it is defined. A symbol can only be global if it has external linkage.

(One reason "global" is a poor choice is that in some systems there's an asymmetry between making a symbol available for use in other load units, and using a symbol from a different load unit. Names like "import" and "export" reflect that asymmetry better.)

Second: how do programmers control whether or not symbols are global. The answers on existing systems are language extensions (pragmas, attributes, or new keywords), extralinguistic mechanisms (linker flags, export files), or both. For the Standard, my opinion is that the only sensible answer is a new keyword that can be used in some kinds of definitions. I propose global. I propose that symbols are always non-global by default.

This is a somewhat controversial issue, since existing practice varies widely. Either way, we need a source construct to control this on a symbol-by-symbol level. I suggest that non-globals be the default for two reasons. First, a load unit's set of global symbols is part of its interface and its set of non-global symbols is part of its implementation, and implementations are usually larger than interfaces. Second, there are existing implementations with source constructs to make symbols global selectively.

I propose that a definition must use the global keyword to allow the entity that's being defined to be accessed from other load units, but that there's no special syntax to use an entity defined in another load unit.

Third: how do programers specify that a load unit should expect to find a symbol's definition in a different load unit. There are two general classes of answers: either by leaving the symbol undefined, or by using a source construct along the same lines of extern.

Fourth, granularity of control: what kinds of definitions can be defined as global? I suggest the following:

Fifth, what exactly does this mean for things like class or template definitions, which typically show up in headers? Do we require textual differences in the header depending on which load unit it appears in, perhaps by macro hackery, or do we require that the definition use the global keyword in all load units? I propose the later. (Rationale: it's already the case that a class definition may appear in multiple translation units, so allowing it to appear in multiple load units doesn't seem like much of a stretch. I'm uncomforable with the ODR implications of saying that a class definition must be textually different in two different contexts.)

Sixth: how the global/nonglobal distinction interacts with the type system. My proposed answer: it doesn't. There's no distinction between a pointer to a global object and a pointer to a nonglobal object, it's impossible to overload on global/nonglobal, etc. (Rationale: this reflects the reality of existing practice.)

Seventh: how the global/nonglobal distinction interacts with other, similar facilities that we have in the language already. The three that I can think of are linkage, namespaces, and exported templates. Linkage: I think what we're doing is introducing a new category of linkage, in addition to the three that we have now. (no linkage, internal linkage, and external linkage) Within a single load unit, something with global linkage looks just like something that has non-global external linkage. Namespaces: I suggest that the two concepts should be orthogonal. There's nothing wrong with having a namespace that's opened in multiple load units. (Rationale: this is pretty much unavoidable, since we can expect that most load units will use namespace std. It also reflects existing practice.) Export: I don't know.

Symbol resolution

This is closely related to the issue of linkage, but not quite identical. By symbol resolution, I mean how a symbol that is undefined in one load unit can be bound to a global symbol in another load unit. By definition, this is something that happens in the dynamic linker. Issues dealing with symbol resolution include:

I regard symbol resolution rules as our nastiest problem.

Definitions in the Standard

Are any modifications needed to the following parts of the Standard?

Some possible choices of symbol resolution rules will require ODR changes. We might reasonably end up with a rule where a global symbol x means different things in different load units. That may be unavoidable; I suggest, however, that we should at least make sure that a symbol always means the same thing within a single load unit.

Loadable libraries

What is the standard interface for loading a library by name after a load set has already started running? What happens after the library has been loaded? How do its global symbols participate in the load set's symbol resolution? How can one access a global symbol from that library?

Some incomplete suggestions:

Type equivalence

If a type with the name X is defined in multiple load units, is it considered to be the same type? (Imagine a header file that's #included into multiple source files, for example.) Some specific considerations:

Object identity

Some objects may be defined in multiple places, and the Standard says the implementation is responsible for making sure that only one object appears in the final executable. What happens if an object is defined in multiple load units that get linked into the same load set? Does the answer depend on whether we're talking about direct dependencies, indirect dependencies, or loadable libraries? Examples:

A complementary question: suppose that a load unit is part of multiple load sets on the same system. Is its state (e.g. the values of static and namespace-scope variables) shared between load sets, or does each load set conceptually have a separate copy of the load unit?

Order of initialization

We need to consider order of initialization of nonlocal objects within load units, order of initialization between load units, and interaction with atexit. Some specific issues:

The Standard Library