Copyright © 1999 W3C (MIT, INRIA, Keio ), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.
This is a W3C Working Draft for review by W3C members and other interested parties. It is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is published as part of the Internationalization Activity, see http://www.w3.org/International/Activity by the Internationalization Working Group (I18N WG), with the help of the Internationalization Interest Group (I18N IG). The I18N WG will not allow early implementation to constrain its ability to make changes to this specification prior to final release. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress". A list of current W3C working drafts can be found at http://www.w3.org/TR/.
Among else, this document addresses the requirements laid out in Requirements for String Identity and Character Indexing Definitions for the WWW. It also contains lists of topics for explicit formulation of the character model used by W3C specifications; these lists will be expanded in the next version. Comments to this Working Draft are very welcome. Comments should be sent to i18n-editor@w3.org. Public discussion of internationalization issues of the WWW takes place on www-international@w3.org.
This document defines various aspects of a character model for the WWW. It contains basic definitions and models, specifications to be used by other specifications or directly by implementations, and explanatory material. In particular, early uniform normalization, string identity matching, string indexing, and conventions for URIs are addressed.
Starting with [RFC 2070], a character model for the WWW has been emerging, with the use of the UCS (Universal Character Set, [ISO 10646]/[Unicode]) as a common reference. As long as data transfer on the WWW was primarily unidirectional (from server to browser), and the main purpose was rendering, the direct use of the UCS as a common reference posed no problems.
However, from early on, the WWW included bidirectional data transfer (forms,...). Recently, purposes other than rendering are becoming more and more important. The WWW has traditionally been seen as a collection of applications exchanging data based on protocols. However, it can also be seen as a single, very large application [Nicol]. The second view is becoming more and more important due to the following developments:
In this context, some properties of the UCS become relevant and have to be addressed. It should be noted that such properties also exist in legacy encodings, and in many cases have been inherited by the UCS in one way or another from such legacy encodings. In particular, these properties are:
This means that in order to ensure consistent behaviour on the WWW, some additional specifications, based on the UCS, are necessary. This is also taken as an occasion to clearly define the basic model and its use in various specifications and implementations.
This specification has a wide range of potential users, some of which are listed below.
The adoption, wherever appropriate, of specifications or guidelines given in this document to work outside the W3C is strongly encouraged.
Section 2 of this document is intended to give a general introduction into the treatment of characters in protocols and formats, and corresponding guidelines. Section 3 defines and explains the concept of early uniform normalization for string identity matching. Section 4 deals with string indexing. Character encodings in URIs are discussed is Section 5. A glossary gives additional explanations for some of the terms used in this document.
Words written in all capitals, such as MUST and SHOULD, are used as defined in [RFC 2119]. UCS codepoints are denoted as U+hhhh, where hhhh is a sequence of hexadecimal digits.
Where this specification contains procedural descriptions, they are understood to be a way to specify the desired external behavior. As long as observable behaviour is not affected, implementations may use some other way of achieving the same results.
This section defines a general model, e.g. in the sense of the reference processing model in [RFC 2070], and general guidelines, e.g. similar to those in [RFC 2130] and [RFC 2277]. The current version of this document only gives lists of topics to be addressed for each subsection; the WG plans to address the topics in more detail in the next version of this document.
In the next version of this document, the WG plans to address the following topics in this subsection:
Since [RFC 2070], [ISO 10646]/[Unicode] (hereafter denoted as UCS, Universal Character Set) is used as a common reference for character encoding in W3C specifications (see [HTML 4.0], [XML 1.0], and [CSS2]). This choice was motivated by the fact that the UCS:
In the next version of this document, the WG plans to address the following topic in this subsection:
In the next version of this document, the WG plans to address the following topics in this subsection:
In the next version of this document, the WG plans to address the following topics in this subsection::
Text data interchange using W3C protocols and formats is based on the principle of early normalization. This section gives a short overview of the reasons for using webwide early normalization (Section 3.1), defines the exact form to which text data has to be normalized (Section 3.2), and the cases in which normalization must be applied (Section 3.3). Section 3.4 explicitly discusses String Identity Matching, and Section 3.5 contains additional advice for compatibility equivalents and control characters.
From early on, the WWW included bidirectional data transfer (forms,...). Recently, purposes other than rendering are becoming more and more important. The WWW has traditionally been seen as a collection of applications exchanging data based on protocols. However, it can also be seen as a single, very large application [Nicol]. The second view is becoming more and more important due to the following developments:
In this context, some properties of the UCS become relevant and have to be addressed. It should be noted that such properties also exist in legacy encodings, and in many cases have been inherited by the UCS in one way or another from such legacy encodings. In particular, these properties are:
In particular, string identity matching [CharReq] is a basic operation that is carried out frequently. String identity matching is a subset of the more general problem of string matching. There are various degrees of specificity for string matching, from approximate matching such as regular expressions or phonetic matching for English, to more specific matches such as accent-insensitive or case-insensitive matching. String identity matching is concerned only with strings that contain no user-identifiable distinctions.
At various places in the WWW infrastructure, strings, and in particular identifiers, are compared for identity. If different places use different definitions of string identity matching, or if they rely on different mechanisms to test identity, the results are undesired unpredictability and unnecessary conversions. To solve the problem of string identity matching, the following issues have to be addressed:
For the following reasons, early uniform normalization was chosen:
be conservative in what you send(see [CharReq, subsection 2.8]).
Text data is in normalized form according to this specification if all of the following apply:
Note: The list of control codepoints to exclude, and of others to advise against, is still under discussion. See also Section 3.5.
Text data is also considered to be in normalized form for the purpose of this specification if all of the following apply:
Note: It is possible, in theory, that legacy encodings also exhibit the problem of duplicate encodings as described in Section 3.1. In this case, it would be appropriate if a corresponding normalization were applied. However, no such legacy encoding is currently known.
Where this section uses the term normalized form, this means the form defined in Section 3.2.
Applications or tools transcoding from a legacy encoding to an encoding based on UCS MUST ensure that their output is in normalized form.
The producer of text data MUST assure that data is produced or sent out in normalized form. For the purpose of W3C specifications and their implementations, the producer of text data is the sender of the data in the case of protocols. In the case of formats, it is the tool that produces the data.
Implementors of producer software in the above sense are encouraged to delegate normalization to their respective data sources wherever possible. (Examples of data sources would be: operating system, libraries, keyboard drivers.)
If any intermediate recipient of text data applies any operations, it MUST assure that the results of these operations is again in normalized form, provided the incoming data is in that form. Intermediate recipients may provide additional normalization towards the normalization form, as a side-effect of their operations and/or as an additional service.
If intermediate recipients do not touch the data but just pass it on, they are not required to check normalization or to normalize data. (Example: caching proxies)
The recipients of text data SHOULD assume that data is normalized. Recipients MAY provide normalization as an add-on service and safety measure. Recipients used in connection with text generation tools SHOULD NOT provide normalization.
Example: An authoring tool should normalize all the text that is produced, but must not provide normalization for the operations that are the same as in a browser, in order to catch potential problems early.
Tools or operations that just do string identity matching, and that have both strings to be matched available in the same encoding, SHOULD do so by binary comparison.
Note: Generators, intermediate recipients, and transcoders must support a repertoire of Unicode codepoints that is complete with respect to normalization, i.e. if any arbitrary sequence of codepoints in the repertoire is normalized, the codepoints needed for the normalization must also be part of the repertoire.
String identity matching on the World Wide Web is based on the following steps:
Conversion to UCS, and to the same encoding for both strings, assures that text strings and not just bytes are compared. Early normalization gives the responsibility of avoiding duplicate encodings to the data producer; ensures that a minimum of effort is spent on solving the problem.
This specification does not address compatibility equivalents. Compatibility as listed in the Unicode database covers a wide range of similarities/distinctions. Depending on the situation, some distinctions are needed, and others will be confusing. To specify all these situations in a single place seems premature. In the absence of any further specifications, implementations are advised to generate the non-compatibility equivalent if they do not explicitly need the compatibility character. A compatibility character here is a character that disappears when applying Unicode Compatibility Composition (Normalization Form CC of Unicode [TR #15]). A non-compatibility equivalent is the character resulting from applying Unicode Compatibility Composition.
Specifications are advised to exclude compatibility characters in the syntactic elements of the formats they define if this is reasonable (e.g. exclusion of compatibility characters for GIs in XML). In the future, compatibility characters should be replaced by appropriate style or markup information wherever possible.
This specification does not address any further equivalents, such as case equivalents, the equivalence between katakana and hiragana, the equivalence between accented and un-accented characters, the equivalence between full characters and fallbacks (e.g. "ö" vs. "oe"), and the equivalence between various spellings and morphological forms (e.g. color vs. colour). Such equivalence is on a higher level; whether and where it is needed depends on the language, the application, and the preferences of the user.
This specification does not address all characters with control functions (see Section 3.2 for some). In general, applications are advised to reduce the use of control characters to a minimum. Specifications are advised to exclude control characters in the syntactic elements of the formats they define if this is reasonable (e.g. exclusion of control characters for GIs in XML). Control characters should be replaced by appropriate markup or style information wherever possible.
On many occasions, in order to access a substring or a character, it is necessary to identify positions (between "characters") in a text string/sequence/array. Where such indices are exchanged between components of the WWW, there is a need for a uniform definition of string indexing in order to insure consistent behavior. The requirements for string indexing are discussed in [CharReq, section 4].
Because of the wide variability of scripts and characters, and because of tradeoffs between user friendliness and implementation efficiency, different operations may be required to work at different levels of aggregation or subdivision. At least the following levels can be distinguished:
Note: In many cases, it is highly preferable to use non-numeric ways of identifying substrings. The specification of string indexing for the WWW should not be seen as a general recommendation for the use of string indexing for substring identification. As an example, in the case of translation of a document from one language to another, identification of substrings based on document structure can be expected to be much more stable than identification based on string indexing.
Note: The issue of indexing origin, i.e. whether the first character in a string is indexed as character number 0 or as character number 1, will not be addressed here. In general, even individual characters should be understood and processed as substrings, identified by a position before and a position after the substring. In this case, starting with an index of 0 for the position at the start of the string is the best solution.
According to the current definition [RFC2396], URIs are restricted to a subset of US-ASCII. There is also an escaping mechanism to encode arbitrary byte values using the %HH convention, but because in general, the mapping from characters to bytes is not defined, this is of limited use. To avoid future incompatibilities, W3C specifications MUST include the following paragraph by reference:
For all syntactic elements in the format/protocol which are being interpreted as URIs, characters that are syntactically not allowed by the generic URI syntax (i.e. all non-ASCII characters plus the excluded characters [RFC2396, Section 2.4.3]) MUST be treated as follows: Each such character is represented in UTF-8 as one or more bytes, each of these bytes is escaped with the URI escaping mechanism (i.e. converted to %HH, where HH is the hexadecimal notation of the byte value), and the original character is replaced by the resulting character sequence.
Example: In the URI <http://www.w3.org/People/Dürst/>, the character "ü" is not allowed. The representation of "ü" in UTF-8 consists of two bytes with the values 0xC3 and 0xBC. The URI is therefore converted to <http://www.w3.org/People/D%C3%BCrst/>.
Note: The intent of this is not to freeze the definitions of URIs to a subset of US-ASCII characters forever, but to assure that W3C technology correctly and predictably interacts with systems that are based on the current definition of URIs while not inhibiting a future extension of the URI definition.
Note: Current W3C specifications already contain provisions in accordance with the above. For [XML 1.0], please see Section 4.2.2, External Entities. For [HTML 4.0], please see Appendix B.2.1: Non-ASCII characters in URI attribute values, which also contains some provisions for backwards compatibility. Further information and links can be found at [I18NURI].
This glossary does not provide exact definitions of terms but gives some background on how certain words are used in this document.