Iterator Facade and Adaptor

Author:	David Abrahams, Jeremy Siek, Thomas Witt
Contact:	dave@boost-consulting.com, jsiek@osl.iu.edu, witt@styleadvisor.com
Organization:	Boost Consulting, Indiana University Open Systems Lab, Zephyr Associates, Inc.
Date:	2004-04-10
Number:	N1641=04-0081

copyright:	Copyright David Abrahams, Jeremy Siek, and Thomas Witt 2004. All rights reserved

abstract:	We propose a set of class templates that help programmers build standard-conforming iterators, both from scratch and by adapting other iterators. This proposal is a revision of paper n1530.

Table of Contents

Motivation
Impact on the Standard
Design
Proposed Text

Motivation

Iterators play an important role in modern C++ programming. The iterator is the central abstraction of the algorithms of the Standard Library, allowing algorithms to be re-used in in a wide variety of contexts. The C++ Standard Library contains a wide variety of useful iterators. Every one of the standard containers comes with constant and mutable iterators ², and also reverse versions of those same iterators which traverse the container in the opposite direction. The Standard also supplies istream_iterator and ostream_iterator for reading from and writing to streams, insert_iterator, front_insert_iterator and back_insert_iterator for inserting elements into containers, and raw_storage_iterator for initializing raw memory [7].

Despite the many iterators supplied by the Standard Library, obvious and useful iterators are missing, and creating new iterator types is still a common task for C++ programmers. The literature documents several of these, for example line_iterator [3] and Constant_iterator [9]. The iterator abstraction is so powerful that we expect programmers will always need to invent new iterator types.

Although it is easy to create iterators that almost conform to the standard, the iterator requirements contain subtleties which can make creating an iterator which actually conforms quite difficult. Further, the iterator interface is rich, containing many operators that are technically redundant and tedious to implement. To automate the repetitive work of constructing iterators, we propose iterator_facade, an iterator base class template which provides the rich interface of standard iterators and delegates its implementation to member functions of the derived class. In addition to reducing the amount of code necessary to create an iterator, the iterator_facade also provides compile-time error detection. Iterator implementation mistakes that often go unnoticed are turned into compile-time errors because the derived class implementation must match the expectations of the iterator_facade.

A common pattern of iterator construction is the adaptation of one iterator to form a new one. The functionality of an iterator is composed of four orthogonal aspects: traversal, indirection, equality comparison and distance measurement. Adapting an old iterator to create a new one often saves work because one can reuse one aspect of functionality while redefining the other. For example, the Standard provides reverse_iterator, which adapts any Bidirectional Iterator by inverting its direction of traversal. As with plain iterators, iterator adaptors defined outside the Standard have become commonplace in the literature:

Checked iter[13] adds bounds-checking to an existing iterator.
The iterators of the View Template Library[14], which adapts containers, are themselves adaptors over the underlying iterators.
Smart iterators [5] adapt an iterator's dereferencing behavior by applying a function object to the object being referenced and returning the result.
Custom iterators [4], in which a variety of adaptor types are enumerated.
Compound iterators [1], which access a slice out of a container of containers.
Several iterator adaptors from the MTL [12]. The MTL contains a strided iterator, where each call to operator++() moves the iterator ahead by some constant factor, and a scaled iterator, which multiplies the dereferenced value by some constant.

[1]	We use the term concept to mean a set of requirements that a type must satisfy to be used with a particular template parameter.

[2]	The term mutable iterator refers to iterators over objects that can be changed by assigning to the dereferenced iterator, while constant iterator refers to iterators over objects that cannot be modified.

To fulfill the need for constructing adaptors, we propose the iterator_adaptor class template. Instantiations of iterator_adaptor serve as a base classes for new iterators, providing the default behavior of forwarding all operations to the underlying iterator. The user can selectively replace these features in the derived iterator class. This proposal also includes a number of more specialized adaptors, such as the transform_iterator that applies some user-specified function during the dereference of the iterator.

Impact on the Standard

This proposal is purely an addition to the C++ standard library. However, note that this proposal relies on the proposal for New Iterator Concepts.

Design

Iterator Concepts

This proposal is formulated in terms of the new iterator concepts as proposed in n1550, since user-defined and especially adapted iterators suffer from the well known categorization problems that are inherent to the current iterator categories.

This proposal does not strictly depend on proposal n1550, as there is a direct mapping between new and old categories. This proposal could be reformulated using this mapping if n1550 was not accepted.

Interoperability

The question of iterator interoperability is poorly addressed in the current standard. There are currently two defect reports that are concerned with interoperability issues.

Issue 179 concerns the fact that mutable container iterator types are only required to be convertible to the corresponding constant iterator types, but objects of these types are not required to interoperate in comparison or subtraction expressions. This situation is tedious in practice and out of line with the way built in types work. This proposal implements the proposed resolution to issue 179, as most standard library implementations do nowadays. In other words, if an iterator type A has an implicit or user defined conversion to an iterator type B, the iterator types are interoperable and the usual set of operators are available.

Issue 280 concerns the current lack of interoperability between reverse iterator types. The proposed new reverse_iterator template fixes the issues raised in 280. It provides the desired interoperability without introducing unwanted overloads.

Iterator Facade

While the iterator interface is rich, there is a core subset of the interface that is necessary for all the functionality. We have identified the following core behaviors for iterators:

dereferencing
incrementing
decrementing
equality comparison
random-access motion
distance measurement

In addition to the behaviors listed above, the core interface elements include the associated types exposed through iterator traits: value_type, reference, difference_type, and iterator_category.

Iterator facade uses the Curiously Recurring Template Pattern (CRTP) [Cop95] so that the user can specify the behavior of iterator_facade in a derived class. Former designs used policy objects to specify the behavior, but that approach was discarded for several reasons:

the creation and eventual copying of the policy object may create overhead that can be avoided with the current approach.

The policy object approach does not allow for custom constructors on the created iterator types, an essential feature if iterator_facade should be used in other library implementations.

Without the use of CRTP, the standard requirement that an iterator's operator++ returns the iterator type itself would mean that all iterators built with the library would have to be specializations of iterator_facade<...>, rather than something more descriptive like indirect_iterator<T*>. Cumbersome type generator metafunctions would be needed to build new parameterized iterators, and a separate iterator_adaptor layer would be impossible.

Usage

The user of iterator_facade derives his iterator class from a specialization of iterator_facade and passes the derived iterator class as iterator_facade's first template parameter. The order of the other template parameters have been carefully chosen to take advantage of useful defaults. For example, when defining a constant lvalue iterator, the user can pass a const-qualified version of the iterator's value_type as iterator_facade's Value parameter and omit the Reference parameter which follows.

The derived iterator class must define member functions implementing the iterator's core behaviors. The following table describes expressions which are required to be valid depending on the category of the derived iterator type. These member functions are described briefly below and in more detail in the iterator facade requirements.

Expression Effects

i.dereference() Access the value referred to

i.equal(j) Compare for equality with j

i.increment() Advance by one position

i.decrement() Retreat by one position

i.advance(n) Advance by n positions

i.distance_to(j) Measure the distance to j

Expression	Effects
`i.dereference()`	Access the value referred to
`i.equal(j)`	Compare for equality with `j`
`i.increment()`	Advance by one position
`i.decrement()`	Retreat by one position
`i.advance(n)`	Advance by `n` positions
`i.distance_to(j)`	Measure the distance to `j`

In addition to implementing the core interface functions, an iterator derived from iterator_facade typically defines several constructors. To model any of the standard iterator concepts, the iterator must at least have a copy constructor. Also, if the iterator type X is meant to be automatically interoperate with another iterator type Y (as with constant and mutable iterators) then there must be an implicit conversion from X to Y or from Y to X (but not both), typically implemented as a conversion constructor. Finally, if the iterator is to model Forward Traversal Iterator or a more-refined iterator concept, a default constructor is required.

Iterator Core Access

iterator_facade and the operator implementations need to be able to access the core member functions in the derived class. Making the core member functions public would expose an implementation detail to the user. The design used here ensures that implementation details do not appear in the public interface of the derived iterator type.

Preventing direct access to the core member functions has two advantages. First, there is no possibility for the user to accidently use a member function of the iterator when a member of the value_type was intended. This has been an issue with smart pointer implementations in the past. The second and main advantage is that library implementers can freely exchange a hand-rolled iterator implementation for one based on iterator_facade without fear of breaking code that was accessing the public core member functions directly.

In a naive implementation, keeping the derived class' core member functions private would require it to grant friendship to iterator_facade and each of the seven operators. In order to reduce the burden of limiting access, iterator_core_access is provided, a class that acts as a gateway to the core member functions in the derived iterator class. The author of the derived class only needs to grant friendship to iterator_core_access to make his core member functions available to the library.

iterator_core_access will be typically implemented as an empty class containing only private static member functions which invoke the iterator core member functions. There is, however, no need to standardize the gateway protocol. Note that even if iterator_core_access used public member functions it would not open a safety loophole, as every core member function preserves the invariants of the iterator.

`operator[]`

The indexing operator for a generalized iterator presents special challenges. A random access iterator's operator[] is only required to return something convertible to its value_type. Requiring that it return an lvalue would rule out currently-legal random-access iterators which hold the referenced value in a data member (e.g. counting_iterator), because *(p+n) is a reference into the temporary iterator p+n, which is destroyed when operator[] returns.

Writable iterators built with iterator_facade implement the semantics required by the preferred resolution to issue 299 and adopted by proposal n1550: the result of p[n] is an object convertible to the iterator's value_type, and p[n] = x is equivalent to *(p + n) = x (Note: This result object may be implemented as a proxy containing a copy of p+n). This approach will work properly for any random-access iterator regardless of the other details of its implementation. A user who knows more about the implementation of her iterator is free to implement an operator[] that returns an lvalue in the derived iterator class; it will hide the one supplied by iterator_facade from clients of her iterator.

`operator->`

The reference type of a readable iterator (and today's input iterator) need not in fact be a reference, so long as it is convertible to the iterator's value_type. When the value_type is a class, however, it must still be possible to access members through operator->. Therefore, an iterator whose reference type is not in fact a reference must return a proxy containing a copy of the referenced value from its operator->.

The return types for iterator_facade's operator-> and operator[] are not explicitly specified. Instead, those types are described in terms of a set of requirements, which must be satisfied by the iterator_facade implementation.

[Cop95]

[Coplien, 1995] Coplien, J., Curiously Recurring Template Patterns, C++ Report, February 1995, pp. 24-27.

Iterator Adaptor

The iterator_adaptor class template adapts some Base ³ type to create a new iterator. Instantiations of iterator_adaptor are derived from a corresponding instantiation of iterator_facade and implement the core behaviors in terms of the Base type. In essence, iterator_adaptor merely forwards all operations to an instance of the Base type, which it stores as a member.

[3]	The term "Base" here does not refer to a base class and is not meant to imply the use of derivation. We have followed the lead of the standard library, which provides a base() function to access the underlying iterator object of a `reverse_iterator` adaptor.

The user of iterator_adaptor creates a class derived from an instantiation of iterator_adaptor and then selectively redefines some of the core member functions described in the iterator_facade core requirements table. The Base type need not meet the full requirements for an iterator; it need only support the operations used by the core interface functions of iterator_adaptor that have not been redefined in the user's derived class.

Several of the template parameters of iterator_adaptor default to use_default. This allows the user to make use of a default parameter even when she wants to specify a parameter later in the parameter list. Also, the defaults for the corresponding associated types are somewhat complicated, so metaprogramming is required to compute them, and use_default can help to simplify the implementation. Finally, the identity of the use_default type is not left unspecified because specification helps to highlight that the Reference template parameter may not always be identical to the iterator's reference type, and will keep users from making mistakes based on that assumption.

Specialized Adaptors

This proposal also contains several examples of specialized adaptors which were easily implemented using iterator_adaptor:

indirect_iterator, which iterates over iterators, pointers, or smart pointers and applies an extra level of dereferencing.
A new reverse_iterator, which inverts the direction of a Base iterator's motion, while allowing adapted constant and mutable iterators to interact in the expected ways (unlike those in most implementations of C++98).
transform_iterator, which applies a user-defined function object to the underlying values when dereferenced.
filter_iterator, which provides a view of an iterator range in which some elements of the underlying range are skipped.

counting_iterator, which adapts any incrementable type (e.g. integers, iterators) so that incrementing/decrementing the adapted iterator and dereferencing it produces successive values of the Base type.
function_output_iterator, which makes it easier to create custom output iterators.

Based on examples in the Boost library, users have generated many new adaptors, among them a permutation adaptor which applies some permutation to a random access iterator, and a strided adaptor, which adapts a random access iterator by multiplying its unit of motion by a constant factor. In addition, the Boost Graph Library (BGL) uses iterator adaptors to adapt other graph libraries, such as LEDA [10] and Stanford GraphBase [8], to the BGL interface (which requires C++ Standard compliant iterators).

Proposed Text

Header `<iterator_helper>` synopsis [lib.iterator.helper.synopsis]

struct use_default;

struct iterator_core_access { /* implementation detail */ };

template <
    class Derived
  , class Value
  , class CategoryOrTraversal
  , class Reference  = Value&
  , class Difference = ptrdiff_t
>
class iterator_facade;

template <
    class Derived
  , class Base
  , class Value      = use_default
  , class CategoryOrTraversal  = use_default
  , class Reference  = use_default
  , class Difference = use_default
>
class iterator_adaptor;

template <
    class Iterator
  , class Value = use_default
  , class CategoryOrTraversal = use_default
  , class Reference = use_default
  , class Difference = use_default
>
class indirect_iterator;

template <class Dereferenceable>
struct pointee;

template <class Dereferenceable>
struct indirect_reference;

template <class Iterator>
class reverse_iterator;

template <
    class UnaryFunction
  , class Iterator
  , class Reference = use_default
  , class Value = use_default
>
class transform_iterator;

template <class Predicate, class Iterator>
class filter_iterator;

template <
    class Incrementable
  , class CategoryOrTraversal  = use_default
  , class Difference = use_default
>
class counting_iterator;

template <class UnaryFunction>
class function_output_iterator;

Iterator facade [lib.iterator.facade]

iterator_facade is a base class template that implements the interface of standard iterators in terms of a few core functions and associated types, to be supplied by a derived iterator class.

Class template `iterator_facade`

template <
    class Derived
  , class Value
  , class CategoryOrTraversal
  , class Reference  = Value&
  , class Difference = ptrdiff_t
>
class iterator_facade {
public:
    typedef remove_const<Value>::type value_type;
    typedef Reference reference;
    typedef Value* pointer;
    typedef Difference difference_type;
    typedef /* see below */ iterator_category;

    reference operator*() const;
    /* see below */ operator->() const;
    /* see below */ operator[](difference_type n) const;
    Derived& operator++();
    Derived operator++(int);
    Derived& operator--();
    Derived operator--(int);
    Derived& operator+=(difference_type n);
    Derived& operator-=(difference_type n);
    Derived operator-(difference_type n) const;
};

// Comparison operators
template <class Dr1, class V1, class TC1, class R1, class D1,
          class Dr2, class V2, class TC2, class R2, class D2>
typename enable_if_interoperable<Dr1,Dr2,bool>::type // exposition
operator ==(iterator_facade<Dr1,V1,TC1,R1,D1> const& lhs,
            iterator_facade<Dr2,V2,TC2,R2,D2> const& rhs);

template <class Dr1, class V1, class TC1, class R1, class D1,
          class Dr2, class V2, class TC2, class R2, class D2>
typename enable_if_interoperable<Dr1,Dr2,bool>::type
operator !=(iterator_facade<Dr1,V1,TC1,R1,D1> const& lhs,
            iterator_facade<Dr2,V2,TC2,R2,D2> const& rhs);

template <class Dr1, class V1, class TC1, class R1, class D1,
          class Dr2, class V2, class TC2, class R2, class D2>
typename enable_if_interoperable<Dr1,Dr2,bool>::type
operator <(iterator_facade<Dr1,V1,TC1,R1,D1> const& lhs,
           iterator_facade<Dr2,V2,TC2,R2,D2> const& rhs);

template <class Dr1, class V1, class TC1, class R1, class D1,
          class Dr2, class V2, class TC2, class R2, class D2>
typename enable_if_interoperable<Dr1,Dr2,bool>::type
operator <=(iterator_facade<Dr1,V1,TC1,R1,D1> const& lhs,
            iterator_facade<Dr2,V2,TC2,R2,D2> const& rhs);

template <class Dr1, class V1, class TC1, class R1, class D1,
          class Dr2, class V2, class TC2, class R2, class D2>
typename enable_if_interoperable<Dr1,Dr2,bool>::type
operator >(iterator_facade<Dr1,V1,TC1,R1,D1> const& lhs,
           iterator_facade<Dr2,V2,TC2,R2,D2> const& rhs);

template <class Dr1, class V1, class TC1, class R1, class D1,
          class Dr2, class V2, class TC2, class R2, class D2>
typename enable_if_interoperable<Dr1,Dr2,bool>::type
operator >=(iterator_facade<Dr1,V1,TC1,R1,D1> const& lhs,
            iterator_facade<Dr2,V2,TC2,R2,D2> const& rhs);

// Iterator difference
template <class Dr1, class V1, class TC1, class R1, class D1,
          class Dr2, class V2, class TC2, class R2, class D2>
/* see below */
operator-(iterator_facade<Dr1,V1,TC1,R1,D1> const& lhs,
          iterator_facade<Dr2,V2,TC2,R2,D2> const& rhs);

// Iterator addition
template <class Dr, class V, class TC, class R, class D>
Derived operator+ (iterator_facade<Dr,V,TC,R,D> const&,
                   typename Derived::difference_type n);

template <class Dr, class V, class TC, class R, class D>
Derived operator+ (typename Derived::difference_type n,
                   iterator_facade<Dr,V,TC,R,D> const&);

The iterator_category member of iterator_facade is

iterator-category(CategoryOrTraversal, value_type, reference)

where iterator-category is defined as follows:

iterator-category(C,R,V) :=
   if (C is convertible to std::input_iterator_tag
       || C is convertible to std::output_iterator_tag
   )
       return C

   else if (C is not convertible to incrementable_traversal_tag)
       the program is ill-formed

   else return a type X satisfying the following two constraints:

      1. X is convertible to X1, and not to any more-derived
         type, where X1 is defined by:

           if (R is a reference type
               && C is convertible to forward_traversal_tag)
           {
               if (C is convertible to random_access_traversal_tag)
                   X1 = random_access_iterator_tag
               else if (C is convertible to bidirectional_traversal_tag)
                   X1 = bidirectional_iterator_tag
               else
                   X1 = forward_iterator_tag
           }
           else
           {
               if (C is convertible to single_pass_traversal_tag
                   && R is convertible to V)
                   X1 = input_iterator_tag
               else
                   X1 = C
           }

      2. category-to-traversal(X) is convertible to the most
         derived traversal tag type to which X is also
         convertible, and not to any more-derived traversal tag
         type.

[Note: the intention is to allow iterator_category to be one of the five original category tags when convertibility to one of the traversal tags would add no information]

The enable_if_interoperable template used above is for exposition purposes. The member operators should only be in an overload set provided the derived types Dr1 and Dr2 are interoperable, meaning that at least one of the types is convertible to the other. The enable_if_interoperable approach uses SFINAE to take the operators out of the overload set when the types are not interoperable. The operators should behave as-if enable_if_interoperable were defined to be:

template <bool, typename> enable_if_interoperable_impl
{};

template <typename T> enable_if_interoperable_impl<true,T>
{ typedef T type; };

template<typename Dr1, typename Dr2, typename T>
struct enable_if_interoperable
  : enable_if_interoperable_impl<
        is_convertible<Dr1,Dr2>::value || is_convertible<Dr2,Dr1>::value
      , T
    >
{};

`iterator_facade` Requirements

The following table describes the typical valid expressions on iterator_facade's Derived parameter, depending on the iterator concept(s) it will model. The operations in the first column must be made accessible to member functions of class iterator_core_access. In addition, static_cast<Derived*>(iterator_facade*) shall be well-formed.

In the table below, F is iterator_facade<X,V,C,R,D>, a is an object of type X, b and c are objects of type const X, n is an object of F::difference_type, y is a constant object of a single pass iterator type interoperable with X, and z is a constant object of a random access traversal iterator type interoperable with X.

iterator_facade Core Operations

Expression	Return Type	Assertion/Note	Used to implement Iterator Concept(s)
`c.dereference()`	`F::reference`		Readable Iterator, Writable Iterator
`c.equal(y)`	convertible to bool	true iff `c` and `y` refer to the same position.	Single Pass Iterator
`a.increment()`	unused		Incrementable Iterator
`a.decrement()`	unused		Bidirectional Traversal Iterator
`a.advance(n)`	unused		Random Access Traversal Iterator
`c.distance_to(z)`	convertible to `F::difference_type`	equivalent to `distance(c, X(z))`.	Random Access Traversal Iterator

`iterator_facade` operations

The operations in this section are described in terms of operations on the core interface of Derived which may be inaccessible (i.e. private). The implementation should access these operations through member functions of class iterator_core_access.

reference operator*() const;

Returns:	`static_cast<Derived const*>(this)->dereference()`

operator->() const; (see below)

Returns:

Returns:	If `reference` is a reference type, an object of type `pointer` equal to: &static_cast<Derived const>(this)->dereference() Otherwise returns an object of unspecified type such that, `(static_cast<Derived const>(this))->m` is equivalent to `(w = static_cast<Derived const>(this), w.m)` for some temporary object `w` of type `value_type`.

If reference is a reference type, an object of type pointer equal to:

&static_cast<Derived const*>(this)->dereference()

Otherwise returns an object of unspecified type such that, (*static_cast<Derived const*>(this))->m is equivalent to (w = **static_cast<Derived const*>(this), w.m) for some temporary object w of type value_type.

unspecified operator[](difference_type n) const;

Returns:	an object convertible to `value_type`. For constant objects `v` of type `value_type`, and `n` of type `difference_type`, `(this)[n] = v` is equivalent to `(this + n) = v`, and `static_cast<value_type const&>((this)[n])` is equivalent to `static_cast<value_type const&>((this + n))`

Derived& operator++();

Effects:	static_cast<Derived>(this)->increment(); return static_cast<Derived*>(this);

Derived operator++(int);

Effects:	Derived tmp(static_cast<Derived const>(this)); ++this; return tmp;

Derived& operator--();

Effects:	static_cast<Derived>(this)->decrement(); return static_cast<Derived*>(this);

Derived operator--(int);

Effects:	Derived tmp(static_cast<Derived const>(this)); --this; return tmp;

Derived& operator+=(difference_type n);

Effects:	static_cast<Derived>(this)->advance(n); return static_cast<Derived*>(this);

Derived& operator-=(difference_type n);

Effects:	static_cast<Derived>(this)->advance(-n); return static_cast<Derived*>(this);

Derived operator-(difference_type n) const;

Effects:	Derived tmp(static_cast<Derived const*>(this)); return tmp -= n;

template <class Dr, class V, class TC, class R, class D>
Derived operator+ (iterator_facade<Dr,V,TC,R,D> const&,
                   typename Derived::difference_type n);

template <class Dr, class V, class TC, class R, class D>
Derived operator+ (typename Derived::difference_type n,
                   iterator_facade<Dr,V,TC,R,D> const&);

Effects:	Derived tmp(static_cast<Derived const*>(this)); return tmp += n;

template <class Dr1, class V1, class TC1, class R1, class D1,
          class Dr2, class V2, class TC2, class R2, class D2>
typename enable_if_interoperable<Dr1,Dr2,bool>::type
operator ==(iterator_facade<Dr1,V1,TC1,R1,D1> const& lhs,
            iterator_facade<Dr2,V2,TC2,R2,D2> const& rhs);

Returns:

Returns:	if `is_convertible<Dr2,Dr1>::value` then `((Dr1 const&)lhs).equal((Dr2 const&)rhs)`. Otherwise, `((Dr2 const&)rhs).equal((Dr1 const&)lhs)`.