A URI Library for C++¶

Example Usage¶

std::network::uri uri("http://www.example.com/glynos/?key=value#frag");
assert(uri.absolute());
assert(!uri.opaque());
assert(*uri.scheme() == "http");
assert(*uri.host() == "www.example.com");
assert(*uri.path() == "/glynos/");
assert(*uri.query() == "?key=value");
assert(*uri.fragment() == "frag");

The code excerpt above shows a simple of how the proposed uri will work. The URI string is parsed during object construction and broken down into its component parts. HTTP URIs are absolute and hierarchical (i.e. not opaque).

std::network::uri uri(U"xmpp:example-node@example.com?message;subject=Hello%20World");
assert(uri.absolute());
assert(uri.opaque());
assert(*uri.scheme() == "xmpp");
assert(*uri.path() == "example-node@example.com");
assert(*uri.query() == "?message;subject=Hello%20World");

The uri in this proposal supports encoded strings and supports encoding conversion. The example above shows a uri object constructed using a std::u32string and allow the parts to be accessed as std::string objects in UTF-8 encoding.

Changes since N3484¶

Changes since N3420¶

Percent Encoding¶

URI percent encoding is described in RFC 3986, section 2.1 and RFC 3986, section 2.4.

Percent encoding is the mechanism used to encode reserved characters in a URI. According to RFC 3986, section 2.2, the set of reserved characters are:

Percent encoding is not limited to reserved characters. Any character data may be percent encoded:

Converting the Scheme / Host to Lower Case¶

The scheme and host are case-insensitive. The proposed normalization solution will convert these to lowercase.

HTTP://Example.com/ --> http://example.com/

The user info, path, query and fragment are case-sensitive and so must not be converted.

Capitalizing Characters in Escape Sequences¶

Characters in a percent-encoded triplet are case-insensitive. The proposed normalization solution will convert these to lowercase.

http://example.com/%5b%5d --> http://example.com/%5B%5D

Decoding Unreserved Characters¶

Unreserved characters that have been encoded will be decoded.

http://example.com/%7Eglynos/ --> http://example.com/~glynos/

Adding Trailing /¶

If a path refers to a directory, it should be indicated with a trailing slash.

http://example.com/glynos --> http://example.com/glynos/

But not if the path refers to a file.

http://example.com/glynos/page.html --> http://example.com/glynos/page.html

Removing dot-segments from the Path¶

The segments ”..” and ”.” can be removed according to the algorithm specified in RFC 3986, Section 5.2.4.

http://example.com/glynos/./proposals/../ --> http://example.com/glynos/

Removing the default port¶

Some schemes may have a default port (for HTTP it is 80). The default port can be removed.

http://example.com:80/ --> http://example.com/
http://example.com:/ --> http://example.com/

The Comparison Ladder¶

The Comparison Ladder is described in RFC 3986, Section 6.2. It explains that comparing URIs using normalization can be implemented in different ways according to the complexity of the method and the number of false negatives which may arise.

The final two steps in the Comparison Ladder require more information than can be provided within the limits of the proposal in order to be implemented comprehensively, and will not form part of the proposal at this stage.

Declarations¶

The <network/uri> header contains a declaration for a single uri class in the std::network namespace.

At this stage, the sub-namespace network namespace should be regarded as a placeholder for a namespace specified for network components during the standardization process (should such as sub-namespace be specified).

Factory function¶

// factory functions
template <class String>
uri make_uri(const String &u, std::error_code &e) noexcept;

This factory function is provided in order to be able to construct a uri object without throwing an exception. The error code is stored in the std::error_code object, if there is a syntax error.

Equality and Comparison Operators¶

namespace std {
namespace network {
enum uri_comparison_level { ... };
bool operator == (const uri &lhs, const uri &rhs);
bool operator != (const uri &lhs, const uri &rhs);
bool operator <  (const uri &lhs, const uri &rhs);
bool operator <= (const uri &lhs, const uri &rhs);
bool operator >  (const uri &lhs, const uri &rhs);
bool operator >= (const uri &lhs, const uri &rhs);
} // namespace network
} // namespace std

Percent Encoding and Decoding¶

namespace std {
namespace network {
template <typename InputIterator, typename OutputIterator>
OutputIterator encode_user_info(InputIterator first, InputIterator last, OutputIterator out);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename InputIterator, typename OutputIterator>
OutputIterator encode_host(InputIterator first, InputIterator last, OutputIterator out);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename InputIterator, typename OutputIterator>
OutputIterator encode_port(InputIterator first, InputIterator last, OutputIterator out);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename InputIterator, typename OutputIterator>
OutputIterator encode_path(InputIterator first, InputIterator last, OutputIterator out);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename InputIterator, typename OutputIterator>
OutputIterator encode_query(InputIterator first, InputIterator last, OutputIterator out);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename InputIterator, typename OutputIterator>
OutputIterator encode_fragment(InputIterator first, InputIterator last, OutputIterator out);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename InputIterator, typename OutputIterator>
OutputIterator decode(InputIterator first, InputIterator last, OutputIterator out);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <class String>
String decode(const String &source);
} // namespace network
} // namespace std

Stream Operators¶

namespace std {
namespace network {
std::ostream &operator <<  (std::ostream &os, const uri &u);
std::wostream &operator << (std::wostream &os, const uri &u);
} // namespace network
} // namespace std

This proposal specifies output stream operators for character and wide character streams.

namespace std {
namespace network {
std::istream &operator >>  (std::istream &is, uri &u);
std::wistream &operator >> (std::wistream &is, uri &u);
} // namespace network
} // namespace std

This proposal specifies input stream operators for character and wide character streams.

uri Requirements¶

Template parameters named InputIterator are required meet the requirements for a C++ standard library RandomIterator compliant iterator. The iterator’s value type is required to be char, wchar_t, char16_t, or char32_t.

Template parameters named Source are required to be one of:

A container with a value type of char, wchar_t, char16_t, or char32_t.

An iterator for a null terminated byte-string. The value type is required to be char, wchar_t, char16_t, or char32_t.

A C-array. The value type is required to be char, wchar_t, char16_t, or char32_t.

This is identical wording to that found in the filesystem proposal (N3365).

typedef s¶

typedef ... string_type;
typedef string_type::const_iterator iterator;
typedef string_type::const_iterator const_iterator;
typedef std::iterator_traits<iterator>::value_type value_type;

The string_type is left unspecified in this proposal and is intended to implementation defined.

Constructors and Destructors¶

uri();

template <typename InputIterator>
uri(const InputIterator &first, const InputIterator &last);

template <class Source>
explicit uri(const Source &source);

Assignment¶

uri &operator = (const uri &other);
uri &operator = (uri &&other);

Swap¶

void swap(uri &other) noexcept;

Iterators¶

const_iterator begin() const;

const_iterator end() const;

Accessors¶

std::optional<string_ref> scheme() const;

std::optional<string_ref> user_info() const;

std::optional<string_ref> host() const;

std::optional<string_ref> port() const;

std::optional<string_ref> path() const;

std::optional<string_ref> authority() const;

std::optional<string_ref> query() const;

std::optional<string_ref> fragment() const;

Query¶

bool empty() const noexcept;

bool absolute() const noexcept;

bool opaque() const noexcept;

Transformers¶

This proposal specifies three transformer functions: normalize, relativize and resolve.

uri normalize(uri_comparison_level level) const;

uri relativize(const uri &other, uri_comparison_level level) const;

std::network::uri base_uri("http://www.example.com/");
std::network::uri uri("http://www.example.com/glynos/?key=value#fragment");
std::network::uri rel_uri(base_uri.relativize(uri, uri_comparison_level::string_comparison));
assert(rel_uri.string() == "/glynos/?key=value#fragment");

uri resolve(const uri &other, uri_comparison_level level) const;

int compare(const uri &other, uri_comparison_level level) const;

String Accessors¶

string string() const;

wstring wstring() const;

u16string u16string() const;

u32string u32string() const;

Constructors¶

namespace std {
namespace network {
uri_builder::uri_builder();
uri_builder::uri_builder(const uri &base);
template <typename Source>
uri_builder::uri_builder(const Source &base);
} // namespace network
} // namespace std

Constructs a uri_builder object from a base URI, where supplied.

Builder functions¶

namespace std {
namespace network {
template <typename Source>
uri_builder &uri_builder::scheme(const Source &scheme);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &uri_builder::user_info(const Source &user_info);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &uri_builder::host(const Source &host);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &uri_builder::port(const Source &port);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &authority(const Source &authority);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &authority(const Source &user_info, const Source &host, const Source &port);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &uri_builder::path(const Source &path);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &uri_builder::append_path(const Source &path);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &uri_builder::query(const Source &query);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <class Key, class Param>
uri_builder &uri_builder::query(const Key &key, const Param &param);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri_builder &uri_builder::fragment(const Source &fragment);
} // namespace network
} // namespace std

namespace std {
namespace network {
template <typename Source>
uri uri_builder::uri() const;
} // namespace network
} // namespace std

Issue 1 - Text Encoding and Interoperability¶

The most important open issue for this proposal is how to deal with text encoding in a portable way. The main difficulty is that different platforms, libraries and applications use different encodings, Unicode or otherwise. Interoperability is therefore extremely hard.

With the advent of char16_t and char32_t, and of std::u16string and std::u32string, this problem has been acknowledged and encoding can done more explicitly, but issues with interoperability have not been addressed.

This proposal currently partially resolves interoperability issues by using a template for functions and member functions when string arguments are used. e.g. The uri constructor:

namespace std {
namespace network {
template <class Source>
uri::uri(const Source &source);
} // namespace network
} // namespace std

Internally, the uri constructor can handle strings in different encodings by using template specialization to perform the correct transformation depending on the source type.

This can help simplify the library interface but is not completely satisfactory. For example, template specialization is limited to only those types known to the standard - std::basic_string and its variants, plus character arrays. A proposal exists to add a string_ref type to the standard (N3334), which could provide better flexibility and performance. This would replace part_range in the current proposal.

This approach is limited when returning strings from functions. e.g.:

namespace std {
namespace network {
template <class Result>
Result scheme(const uri &uri_);
} // namespace network
} // namespace std

...

std::network::uri uri("http://example.com/");
auto scheme = std::network::scheme<std::u32string>(uri);

The above excerpt cannot work with character array, character pointers or string_ref since memory allocation is required.

Secondly, RFC 3987 is not well supported and often unicode text is converted using percent encoding:

std::network::uri uri(
    "http://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3%83%9A%E3%83%BC%E3%82%B8");

The encode and decode functions should take this into account.

Thirdly, the iterators returned by begin and end are not portable with the proposal in its current form. This proposal allows different internal character types, but the iterator types are completely unaware of the encoding. So the following examples are not portable:

std::network::uri uri(
    "http://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3%83%9A%E3%83%BC%E3%82%B8");
std::u32string u32;
std::copy(std::begin(uri), std::end(uri), std::back_inserter(u32));

std::network::uri uri(
    "http://ja.wikipedia.org/wiki/%E3%83%A1%E3%82%A4%E3%83%B3%E3%83%9A%E3%83%BC%E3%82%B8");
auto path_range = uri.path();
std::u32string path;
std::copy(std::begin(path_range), std::end(path_range), std::back_inserter(path));

This proposal as it stands needs better library or language support in order offer better portability. One promising solution is given in another standard proposal, N3336.

Issue 2 - The Parser Implementation¶

At this stage it is important to note that there are many compatibility issues with URIs. If this proposal were to specify full standard compliance, it may not work in practice. For example, the uri parser implementation must correctly deal with escape control characters. Furthermore, no RFC fully specifies how to parse Windows file-system paths (e.g. file:///C:\path\to\a\file.txt). Finally, it is the belief of the author of this proposal that its success depends strongly on the correct portable behavior of the parser implementation. Without wishing to run the risk of over-specifying the parser, leaving it as “implementation defined” will not be sufficient. There will be more issues that may need to specified in further detail in future revisions of this proposal.

This proposal includes internationalized URIs (RFC 3987). Since this RFC is not widely supported, and as many applications deal with unicode characters through percent encoding, the parser implementation could be simplified if appropriate by removing RFC 3987 from this proposal.

Issue 3 - Error and Exception Handling¶

There several potential sources for errors:

Issue 4 - Equivalence¶

As explained in the previous section on normalization, there is more than one way to test if two URIs are equivalent depending on different accuracy, performance and complexity trade-offs.

Providing a single operator == is sufficient to allow developers to choose different ways of testing URI equivalence, although choosing a good default comparison will be sufficient for the vast majority of cases. Future revisions of this proposal could provide functionality to allow a library to use different parts of the Comparison Ladder.

Issue 5 - Extending the URI¶

The interface can be extended to allow more flexibility for accessing parts of the URI. For example, an accessor could be provided which converts all query elements into a map:

namespace std {
namespace network {
template <class QueryMap>
QueryMap query(const uri &u);
} // namespace network
} // namespace std

...

typedef std::map<std::string, std::string> QueryMap;
QueryMap query = std::network::query<QueryMap>(uri);

Additionally, this proposal can be extended to include factory functions for common operations, such as constructing a URI from a filesystem path:

namespace std {
namespace network {
uri uri::from_path(const filesystem::path &path);
} // namespace network
} // namespace std

...

std::filesystem::path path("/usr/bin/c++");
auto uri_path = std::network::uri::from_path(path);
assert(uri_path.string() == "file:///usr/bin/c++");

Furthermore, there may be new proposals for types that represent IP addresses. If such types can be accepted into the standard, it would be possible to accommodate them in future revisions of this proposal:

namespace std {
namespace network {
uri_builder &uri_builder::host(const std::network::address_ipv4 &host);
uri_builder &uri_builder::host(const std::network::address_ipv6 &host);
} // namespace network
} // namespace std

Existing Practice¶

There exist several implementations of C++, including (but not limited to):

• cpp-netlib
• QUrl (From Qt)
• google-url

The cpp-netlib uri is the library that has motivated this proposal. It implements a generic URI, support for percent encoding, normalization and URI resolution. It also currently partially implements a URI builder.

QUrl supports much of the same functionality. It is fully internationalized, supports URI resolution and implicitly normalization. The part accessors differ from cpp-netlib, in that they return copies of the underlying parts, not references.

google-url is another existing library which is fast and robust. Its parsing strategy is such that it never fails (there is no validation), but always tries to find the best fit for the URL components. It supports normalization (but has called it canonicalization).

All of the implementations listed above do effectively the same thing - they parse a string and split it into components, either by storing separate copies of each part or by storing references to sub-strings in the original uri string.

WhatWG URL Standard¶

In addition to RFC 3986 and RFC 3987, the WhatWG URL Standard is proposal for a standard that sets out to make URLs more interoperable.

The most important features of this proposed standard seem to be:

• The obsolescence of RFC 3986 and RFC 3987 and an alignment of the URL with modern implementations
• To standardize on the term URL, instead of URI or IRI
• A detailed definition of the URL JavaScript API

It does not explicitly describe operations such as normalization, comparison or resolution.

As this proposal allows the URI parsing to be implementation defined, there is no reason why an implementor could not decide to use this standard. Apart from a few naming differences, the proposed uri interface maps closely to the WhatWG URL JavaScript API. If there is a strong interest in doing so, a future revision of this proposal could rename its class and member functions.