From Wikidna.org

This working abstract gather most of the significant texts of all the IDNA Drafts. Removed are tables, legalese, references, acknowledgments, notes to the editor, etc.

.

Contents 1 IAB 1.1 1. Introduction 1.1.1 1.1. APIs 1.2 2. Use of Non-DNS Protocols 1.3 3. Use of Non-ASCII in DNS 1.4 4. Recommendations 1.5 5. Security Considerations 1.6 6. IANA Considerations 2 DEFINITIONS 2.1 1. Introduction 2.1.1 1.1. IDNA2008 2.1.1.1 1.1.1. Audiences 2.1.1.2 1.1.2. Normative Language 2.1.2 1.2. Discussion Forum 2.1.3 1.3. Roadmap of IDNA2008 Documents 2.2 2. Definitions and Terminology 2.2.1 2.1. Characters and Character Sets 2.2.2 2.2. DNS-related Terminology 2.2.3 2.3. Terminology Specific to IDNA 2.2.3.1 2.3.1. LDH-label 2.2.3.2 2.3.2. Terms for IDN Label Codings 2.2.3.2.1 2.3.2.1. IDNA-valid strings, A-label, and U-label 2.2.3.2.2 2.3.2.2. NR-LDH-label and Internationalized Label 2.2.3.2.3 2.3.2.3. Internationalized Domain Name 2.2.3.2.4 2.3.2.4. Label Equivalence 2.2.3.2.5 2.3.2.5. ACE Prefix 2.2.3.2.6 2.3.2.6. Domain Name Slot 2.2.3.3 2.3.3. Order of Characters in Labels 2.2.3.4 2.3.4. Punycode is an Algorithm, not a Name or Adjective 2.3 3. IANA Considerations 2.4 4. Security Considerations 2.4.1 4.1. General Issues 2.4.2 4.2. Local Character Set Issues 2.4.3 4.3. Visually Similar Characters 2.4.4 4.4. IDNA Lookup, Registration, and the Base DNS Specifications 2.4.5 4.5. Legacy IDN Label Strings 2.4.6 4.6. Security Differences from IDNA2003 2.4.7 4.7. Summary 3 IDNA RATIONALE 3.1 1. Introduction 3.1.1 1.1. Context and Overview 3.1.2 1.2. Discussion Forum 3.1.3 1.3. Terminology 3.1.3.1 1.3.1. DNS "Name" Terminology 3.1.3.2 1.3.2. New Terminology and Restrictions 3.1.4 1.4. Objectives 3.1.5 1.5. Applicability and Function of IDNA 3.1.6 1.6. Comprehensibility of IDNA Mechanisms and Processing 3.2 2. Processing in IDNA2008 3.3 3. Permitted Characters: An Inclusion List 3.3.1 3.1. A Tiered Model of Permitted Characters and Labels 3.3.1.1 3.1.1. PROTOCOL-VALID 3.3.1.2 3.1.2. CONTEXTUAL RULE REQUIRED 3.3.1.2.1 3.1.2.1. Contextual Restrictions 3.3.1.2.2 3.1.2.2. Rules and Their Application 3.3.1.3 3.1.3. DISALLOWED 3.3.1.4 3.1.4. UNASSIGNED 3.3.2 3.2. Registration Policy 3.3.3 3.3. Layered Restrictions: Tables, Context, Registration, Applications 3.4 4. Issues that Constrain Possible Solutions 3.4.1 4.1. Display and Network Order 3.4.2 4.2. Entry and Display in Applications 3.4.3 4.3. Linguistic Expectations: Ligatures, Digraphs, and Alternate 3.4.4 4.4. Case Mapping and Related Issues 3.4.5 4.5. Right to Left Text 3.5 5. IDNs and the Robustness Principle 3.6 6. Front-end and User Interface Processing for Lookup 3.7 7. Migration from IDNA2003 and Unicode Version Synchronization 3.7.1 7.1. Design Criteria 3.7.1.1 7.1.1. Summary and Discussion of IDNA Validity Criteria 3.7.1.2 7.1.2. Labels in Registration 3.7.1.3 7.1.3. Labels in Lookup 3.7.2 7.2. Changes in Character Interpretations 3.7.3 7.3. Character Mapping 3.7.4 7.4. The Question of Prefix Changes 3.7.4.1 7.4.1. Conditions Requiring a Prefix Change 3.7.4.2 7.4.2. Conditions Not Requiring a Prefix Change 3.7.4.3 7.4.3. Implications of Prefix Changes 3.7.5 7.5. Stringprep Changes and Compatibility 3.7.6 7.6. The Symbol Question 3.7.7 7.7. Migration Between Unicode Versions: Unassigned Code Points 3.7.8 7.8. Other Compatibility Issues 3.8 8. Name Server Considerations 3.8.1 8.1. Processing Non-ASCII Strings 3.8.2 8.2. DNSSEC Authentication of IDN Domain Names 3.8.3 8.3. Root and other DNS Server Considerations 3.9 9. Internationalization Considerations 3.10 10. IANA Considerations 3.10.1 10.1. IDNA Character Registry 3.10.2 10.2. IDNA Context Registry 3.10.3 10.3. IANA Repository of IDN Practices of TLDs 3.11 11. Security Considerations 3.11.1 11.1. General Security Issues with IDNA 4 MAPPING 4.1 1. Introduction 4.2 2. The General Procedure 4.3 3. IANA Considerations 4.4 4. Security Considerations 5 PROTOCOL 5.1 1. Introduction 5.1.1 1.1. Discussion Forum 5.2 2. Terminology 5.3 3. Requirements and Applicability 5.3.1 3.1. Requirements 5.3.2 3.2. Applicability 5.3.2.1 3.2.1. DNS Resource Records 5.3.2.2 3.2.2. Non-domain-name Data Types Stored in the DNS 5.4 4. Registration Protocol 5.4.1 4.1. Input to IDNA Registration Process 5.4.2 4.2. Permitted Character and Label Validation 5.4.2.1 4.2.1. Input Format 5.4.2.2 4.2.2. Rejection of Characters that are not Permitted 5.4.2.3 4.2.3. Label Validation 5.4.2.3.1 4.2.3.1. Consecutive Hyphens 5.4.2.3.2 4.2.3.2. Leading Combining Marks 5.4.2.3.3 4.2.3.3. Contextual Rules 5.4.2.3.4 4.2.3.4. Labels Containing Characters Written Right to Left 5.4.2.4 4.2.4. Registration Validation Summary 5.4.3 4.3. Registry Restrictions 5.4.4 4.4. Punycode Conversion 5.4.5 4.5. Insertion in the Zone 5.5 5. Domain Name Lookup Protocol 5.5.1 5.1. Label String Input 5.5.2 5.2. Conversion to Unicode 5.5.3 5.3. A-label Input 5.5.4 5.4. Validation and Character List Testing 5.5.5 5.5. Punycode Conversion 5.5.6 5.6. DNS Name Resolution 5.6 6. Security Considerations 5.7 7. IANA Considerations 6 BIDI 6.1 1. Introduction 6.1.1 1.1. Purpose and applicability 6.1.2 1.2. Background and history 6.1.3 1.3. Layout of the rest of this document 6.1.4 1.4. Terminology 6.2 2. A replacement for the RFC 3454 BIDI rule 6.3 3. A requirement set for the BIDI rule 6.4 4. Examples of issues found 6.4.1 4.1. Dhivehi 6.4.2 4.2. Yiddish 6.4.3 4.3. Strings with numbers 6.5 5. Troublesome situations and guidelines 6.6 6. Other issues in need of resolution 6.7 7. Compatibility considerations 6.7.1 7.1. Backwards compatibility considerations 6.7.2 7.2. Forward compatibility considerations 6.8 8. IANA Considerations 6.9 9. Security Considerations 7 TABLES 7.1 1. Introduction 7.2 2. Category definitions Used to Calculate Derived Property Value 7.2.1 2.1. LetterDigits (A) 7.2.2 2.2. Unstable (B) 7.2.3 2.3. IgnorableProperties (C) 7.2.4 2.4. IgnorableBlocks (D) 7.2.5 2.5. LDH (E) 7.2.6 2.6. Exceptions (F) 7.2.7 2.7. BackwardCompatible (G) 7.2.8 2.8. JoinControl (H) 7.2.9 2.9. OldHangulJamo (I) 7.2.10 2.10. Unassigned (J) 7.3 3. Calculation of the Derived Property 7.4 4. Codepoints 7.5 5. IANA Considerations 7.5.1 5.1. IDNA derived property value registry 7.5.2 5.2. IDNA Context Registry 7.6 6. Security Considerations 7.7 Appendix A. Contextual Rules Registry 7.8 Appendix A.1. ZERO WIDTH NON-JOINER 7.9 Appendix A.2. ZERO WIDTH JOINER 7.10 Appendix A.3. MIDDLE DOT 7.11 Appendix A.4. GREEK LOWER NUMERAL SIGN (KERAIA) 7.12 Appendix A.5. HEBREW PUNCTUATION GERESH 7.13 Appendix A.6. HEBREW PUNCTUATION GERSHAYIM 7.14 Appendix A.7. KATAKANA MIDDLE DOT 7.15 Appendix A.8. ARABIC-INDIC DIGITS 7.16 Appendix A.9. EXTENDED ARABIC-INDIC DIGITS 7.17 Appendix B. Codepoints 0x0000 - 0x10FFFF 7.18 Appendix B.1. Codepoints in Unicode Character Database (UCD) format

[edit]
IAB

This document explores issues with Internationalized Domain Names (IDNs) that result from the use of various encoding schemes such as Punycode and UTF-8.

[edit] 1. Introduction

The goal of this document is to explore what can be learned from the current difficulties in implementing Internationalized Domain Names (IDNs). Although some elements of this exploration may immediately feed back into current IETF work it is explicitly not the intention for this document to influence any current working group charter.

An Internationalized Domain Name (IDN) is a name that contains one or more non-ASCII characters. An IDN can be encoded in various ways.

Punycode [RFC3492] is a mechanism for encoding a Unicode string in ASCII characters using only letters, digits, and hypens. When an IDN is encoded with Punycode, it is prefixed with "xn--", which assumes that ASCII names do not start with this prefix. While this assumption is not necessarily true, taking this limitation is seen to be acceptable.

The term "ToASCII" refers to the combination of a non-reversible character mapping operation (e.g., converting upper case characters to lower case characters), plus a reversible Unicode-to-Punycode conversion. Similarly, the term "ToUnicode" refers to the combination of a non-reversible character mapping operation, plus a reversible Punycode-to-Unicode conversion.

ISO-2022-JP [RFC1468] is a mechanism for encoding a string of ASCII and Japanese characters, where an ASCII character is preserved as-is.

UTF-8 [RFC3629] is a mechanism for encoding a Unicode character in a variable number of 8-bit octets, where an ASCII character is preserved as-is. A UTF-8 string is thus a string of UTF-8 characters.

UTF-16 [RFC2781] is a mechanism for encoding a Unicode character in one or two 16-bit integers. A UTF-16 string is thus a string of UTF-16 characters.

UTF-32 (formerly UCS-4) ([UNICODE] section 3.10) is a mechanism for encoding a Unicode character in a single 32-bit integer. A UTF-32 string is thus a string of UTF-32 characters.

Different applications, APIs, and protocols use different encoding schemes today. Historically, many of them were originally defined to use only ASCII. Internationalizing Domain Names in Applications (IDNA) [RFC3490] defined a mechanism that required changes to applications, but not APIs or servers, and specifies that Punycode is to be used.

[RFC3490] section 1.3 states:

The IDNA protocol is contained completely within applications. It is not a client-server or peer-to-peer protocol: everything is done inside the application itself. When used with a DNS resolver library, IDNA is inserted as a "shim" between the application and the resolver library. When used for writing names into a DNS zone, IDNA is used just before the name is committed to the zone.

Figure 1 depicts a simplistic architecture that a naive reader might assume from the paragraph quoted above. (A variant of this same picture appears in [RFC3490] section 6, further strengthening this assumption.)

+-----------------------------------------+
|Host                                     |
|             +-------------+             |
|             | Application |             |
|             +------+------+             |
|                    |                    |
|               +----+----+               |
|               |   DNS   |               |
|               | Resolver|               |
|               | Library |               |
|               +----+----+               |
|                    |                    |
+-----------------------------------------+
                     |
            _________|_________
           /                   \
          /                     \
         /                       \
        |         Internet        |
         \                       /
          \                     /
           \___________________/

Simplistic Architecture

Figure 1

There are, however, two problems with this simplistic architecture that cause it to differ from reality.

First, resolver APIs on OS's today (MacOS, Windows, Linux, etc.) are not DNS-specific. They typically provide a layer of indirection so that the application can work independent of the name resolution mechanism, which could be DNS, mDNS [I-D.cheshire-dnsext-multicastdns], LLMNR [RFC4795], NetBIOS-over-TCP [RFC1001][RFC1002], etc/hosts file [RFC0952], NIS [NIS], or anything else. For example, RFC 3493 [RFC3493] specifies the getaddrinfo() API and contains many phrases like "For example, when using the DNS" and "any type of name resolution service (for example, the DNS)".

Importantly, DNS is mentioned only as an example, and the application has no knowledge as to whether DNS or some other protocol will be used.

Second, even with the DNS protocol, private name spaces (sometimes referred to as "split DNS"), do not necessarily use the same character set encoding scheme as the public Internet name space.

We will discuss each of the above issues in subsequent sections. For reference, Figure 2 depicts a more realistic architecture on typical hosts today. More generally, the host may be multi-homed to one or more local networks, each of which may or may not be connected to the public Internet and may or may not have a private name space.

+-----------------------------------------+
|Host                                     |
|             +-------------+             |
|             | Application |             |
|             +------+------+             |
|                    |                    |
|             +------+------+             |
|             |   Sockets   |             |
|             |   Library   |             |
|             +------+------+             |
|                    |                    |
|   +-----+------+---+--+-------+-----+   |
|   |     |      |      |       |     |   |
| +-+-++--+--++--+-++---+---++--+--++-+-+ |
| |DNS||LLMNR||mDNS||NetBIOS||hosts||...| |
| +---++-----++----++-------++-----++---+ |
|                                         |
+-----------------------------------------+
                    |
              ______|______
             /             \
            /               \
           /      local      \
           \     network     /
            \               /
             \_____________/
                    |
           _________|_________
          /                   \
         /                     \
        /                       \
       |         Internet        |
        \                       /
         \                     /
          \___________________/

Realistic Architecture

Figure 2

[edit] 1.1. APIs

[RFC3490] section 6.2 states:

It is expected that new versions of the resolver libraries in the future will be able to accept domain names in other charsets than ASCII, and application developers might one day pass not only domain names in Unicode, but also in local script to a new API for the resolver libraries in the operating system. Thus the ToASCII and ToUnicode operations might be performed inside these new versions of the resolver libraries.

Resolver APIs such as getaddrinfo() and its predecessor gethostbyname() were defined to accept "char *" arguments, meaning they accept a string of bytes, terminated with a NULL (0) byte. This is sufficient for ASCII strings, Punycode strings, and even ISO-2022-JP and UTF-8 strings (unless an implementation artificially precludes them), but not UTF-16 or UTF-32 strings. Several operating systems historically used in Japan will accept (and expect) ISO-2022-JP strings in such APIs. Some platforms used worldwide also have new versions of the APIs (e.g., GetAddrInfoW() on Windows) that accept other encoding schemes such as UTF-16.

It is worth noting that an API using "char *" arguments can distinguish between ASCII, Punycode, ISO-2022-JP, and UTF-8 strings as follows:

• if the string contains an ESC (0x1B) byte the string is ISO-2022-JP; otherwise, * if any byte in the string has the high bit set, the string is UTF-8; otherwise, * if the string starts with "xn--" then it is Punycode; otherwise, * the string is ASCII.

Again this assumes that ASCII names never start with "xn--", and also that UTF-8 strings never contain an ESC character.

[edit] 2. Use of Non-DNS Protocols

As noted earlier, typical name resolution libraries are not DNS- specific. Furthermore, some protocols are defined to use encoding schemes other than Punycode. For example, mDNS specifies that UTF-8 be used. Indeed, the IETF policy on character sets and languages [RFC2277] states:

Protocols MUST be able to use the UTF-8 charset, which consists of the ISO 10646 coded character set combined with the UTF-8 character encoding scheme, as defined in [10646] Annex R (published in Amendment 2), for all text. Protocols MAY specify, in addition, how to use other charsets or other character encoding schemes for ISO 10646, such as UTF-16, but lack of an ability to use UTF-8 is a violation of this policy; such a violation would need a variance procedure ([BCP9] section 9) with clear and solid justification in the protocol specification document before being entered into or advanced upon the standards track. For existing protocols or protocols that move data from existing datastores, support of other charsets, or even using a default other than UTF-8, may be a requirement. This is acceptable, but UTF-8 support MUST be possible.

Applications that convert an IDN to Punycode before calling getaddrinfo() will result in name resolution failures if the Punycode name is directly used in such protocols. Having libraries or protocols to convert from Punycode to the encoding scheme defined by the protocol (e.g., UTF-8) would require changes to APIs and/or servers, which IDNA was intended to avoid.

As a result, applications that assume that non-ASCII names are resolved using the public DNS and blindly convert them to Punycode without knowledge of what protocol will be selected by the name resolution library have problems. Furthermore, name resolution libraries often try multiple protocols, until one succeeds, because they are defined to use a common name space. For example, the hosts file ([RFC0952] and [RFC1123] section 2.1), DNS ([RFC1034] section 2.1), and NetBIOS-over-TCP ([RFC1001] section 11.1.1) are all defined by RFC to be able to share a common name space. This means that when an application passes a name to be resolved, resolution may in fact be attempted using multiple protocols, each with a potentially different encoding scheme. For this to work successfully, the name must be converted to the appropriate encoding scheme only after the choice is made to use that protocol. In general, this cannot be done by the application since the choice of protocol is not made by the application.

[edit] 3. Use of Non-ASCII in DNS

A common misconception is that DNS only supports names that can be expressed using letters, digits, and hyphens.

This misconception originally stemed from the definition of an "Internet host name" in [RFC0952], published in 1985, which defines the use of the hosts file. An Internet host name was defined therein as including only letters, digits, and hyphens, where upper and lower case letters were to be treated as identical. For DNS, [RFC1034] section 3.5 entitled "Preferred name syntax" then repeated this definition in 1987, saying that this "syntax will result in fewer problems with many applications that use domain names (e.g., mail, TELNET)".

The confusion was thus left as to whether the "preferred" name syntax was a mandatory restriction, or merely "preferred". [RFC1123] section 2.1 updated the definition of an Internet host name as defined in [RFC0952], to allow starting with a digit (to support IPv4 addresses in dotted-decimal form). Section 6.1 of that RFC discusses the use of DNS (and the hosts file) for resolving host names to IP addresses and vice versa. This led to confusion as to whether all names in DNS are "host names", or whether a "host name" is merely a special case of a DNS name.

By 1997, things had progressed to a state where it was necessary to clarify these areas of confusion. "Clarifications to the DNS Specification" [RFC2181] section 11 clarifies:

The DNS itself places only one restriction on the particular labels that can be used to identify resource records. That one restriction relates to the length of the label and the full name.

The length of any one label is limited to between 1 and 63 octets.

A full domain name is limited to 255 octets (including the separators). The zero length full name is defined as representing the root of the DNS tree, and is typically written and displayed as ".". Those restrictions aside, any binary string whatever can be used as the label of any resource record. Similarly, any binary string can serve as the value of any record that includes a domain name as some or all of its value (SOA, NS, MX, PTR, CNAME, and any others that may be added). Implementations of the DNS protocols must not place any restrictions on the labels that can be used.

Hence, it clarified that the restriction to letters, digits, and hyphens does not apply to DNS names in general, nor to records that include "domain names". Hence the "preferred" name syntax specified in [RFC1034] is indeed merely "preferred", not mandatory.

Since there is no restriction even to ASCII, let alone letter-digit- hyphen use, DNS is in conformance with the requirement in [RFC2277] to allow UTF-8.

However, this requirement is complicated by the fact that in an 8-bit clean protocol, one has to have some way of knowing whether a binary string is encoded in UTF-8, UTF-16, UTF-32, or some other encoding.

While implementations of the DNS protocol must not place any restrictons on the labels that can be used, applications that use the DNS are free to impose whatever restrictions they like, and many have. The above rules permit a domain name label that contains unusual characters, such as embedded spaces which many applications would consider a bad idea. For example, the SMTP protocol [RFC5321] originally constrained the character set usable in email addresses and now has an effort underway to extend SMTP to support email address internationalization.

Shortly after [RFC2181] and [RFC2277] were written, the need for internationalized names within private name spaces (i.e., within enterprises) arose. The current (and past, predating Punycode) practice within enterprises that support other languages is to put UTF-8 names in their internal DNS servers in a private name space.

For example, [I-D.skwan-utf8-dns-00] was first written in 1997, and was then widely deployed in Windows. The use of UTF-8 names in DNS was similarly implemented and deployed in MacOS. Within a private name space, and especially in light of [RFC2277], it was reasonable to assume within a private name space that binary strings were encoded in UTF-8.

[EDITOR'S NOTE: There are also normalization/mapping issues which the next version of this document may explore. Currently we only explore encoding issues.]

Five years after UTF-8 was already in use in private name spaces in DNS, Punycode began to be developed ([I-D.ietf-idn-punycode-00] began in 2002, culminating in the publication of [RFC3492] in 2003) for use in the public DNS name space. This publication thus resulted in having to use different encodings for different name spaces (where UTF-8 for private name spaces was already deployed). Hence, referring back to Figure 2, a different encoding scheme may be in use on the Internet vs. a local network.

In general a host may be connected to zero or more networks using private name spaces, plus potentially the public name space.

Applications that convert an IDN to Punycode before calling getaddrinfo() will result in name resolution failures if the name is actually registered in a private name space in some other encoding (e.g., UTF-8). Having libraries or protocols convert from Punycode to the encoding used by a private name space (e.g., UTF-8) would require changes to APIs and/or servers, which IDNA was intended to avoid.

Some examples of cases that can happen in existing implementations today (where {non-ASCII} below represents some user-entered non-ASCII string) are:

1. User types in {non-ASCII}.{non-ASCII}.com, and the application passes it, in the form of a UTF-8 string, to getaddrinfo or gethostbyname or equivalent.

• The DNS resolver passes the (UTF-8) string unmodified to a DNS server.

2. User types in {non-ASCII}.{non-ASCII}.com, and the application passes it to a name resolution API that accepts strings in some other encoding such as UTF-16, e.g., GetAddrInfoW on Windows.

• The name resolution API decides to pass the string to DNS (and possibly other protocols).

• The DNS resolver converts the name from UTF-16 to UTF-8 and passes the query to a DNS server.

3. User types in {non-ASCII}.{non-ASCII}.com, but the application first converts it to Punycode such that the name that is passed to name resolution APIs is (say) xn--e1afmkfd.xn-- 80akhbyknj4f.com.

• The name resolution API decides to pass the string to DNS (and possibly other protocols).

• The DNS resolver passes the string unmodified to a DNS server.

• If the name is not found in DNS, the name resolution API decides to try another protocol, say mDNS.

• The query goes out in mDNS, but since mDNS specified that names are to be registered in UTF-8, the name isn't found since it was Punycode encoded in the query.

4. User types in {non-ASCII}, and the application passes it, in the form of a UTF-8 string, to getaddrinfo or equivalent.

• The name resolution API decides to pass the string to DNS (and possibly other protocols).

• The DNS resolver will append suffixes in the suffix search list, which may contain UTF-8 characters if the local network uses a private name space.

• Each FQDN in turn will then be sent in a query to a DNS server, until one succeeds.

5. User types in {non-ASCII}, but the application first converts it to Punycode, such that the name that is passed to getaddrinfo or equivalent is (say) xn--e1afmkfd.

• The name resolution API decides to pass the string to DNS (and possibly other protocols).

• The DNS stub resolver will append suffixes in the suffix search list, which may contain UTF-8 characters if the local network uses a private name space, resulting in (say) xn--e1afmkfd.{non-ASCII}.com

• Each FQDN in turn will then be sent in a query to a DNS

server, until one succeeds.

• Since the private name space in this case uses UTF-8, the above queries fail, since the Punycode version of the name was not registered in that name space.

6. User types in {non-ASCII1}.{non-ASCII2}.{non-ASCII3}.com, where {non-ASCII3}.com is a public name space using Punycode, but {non- ASCII2}.{non-ASCII3}.com is a private name space using UTF-8, which is accessible to the user. The application passes the name, in the form of a UTF-8 string, to getaddrinfo or equivalent.

• The name resolution API decides to pass the string to DNS (and possibly other protocols).

• The DNS resolver tries to locate the authoritative server, but fails the lookup because it cannot find a server for the UTF-8 encoding of {non-ASCII3}.com, even though it would have access to the private name space. (To make this work, the private name space would need to include the UTF-8 encoding of {non-ASCII3}.com.)

When users use multiple applications, some of which do Punycode conversion prior to passing a name to name resolution APIs, and some of which do not, odd behavior can result which at best violates the principle of least surprise, and at worst can result in security vulnerabilities.

First consider two competing applications, such as web browsers, that are designed to achieve the same task. If the user types the same name into each browser, one may successfully resolve the name (and hence access the desired content) because the encoding scheme was correct, while the other may fail name resolution because the encoding scheme was incorrect. Hence the issue can incent users to switch to another application (which in some cases means switching to an IDNA application, and in other cases means switching away from an IDNA application).

Next consider two separate applications where one is designed to be launched from the other, for example a web browser launching a media player application when the link to a media file is clicked. If both types of content (web pages and media files in this example) are hosted at the same IDN in a private name space, but one application converts to Punycode before calling name resolution APIs and the other does not, the user may be able to access a web page, click on the media file causing the media player to launch and attempt to retrieve the media file, which will then fail because the IDN encoding scheme was incorrect. Or even worse, if an attacker was able to register the same name in the other encoding scheme, may get the content from the attacker's machine. This is similar to a normal phishing attack, except that the two names represent exactly the same Unicode characters.

[edit] 4. Recommendations

Taking into account the issues above, it would seem inappropriate for an application to convert a name to Punycode when it does not know whether DNS will be used by the name resolution library, or whether the name exists in a private name space that uses UTF-8, or in the global DNS that uses Punycode.

Instead, conversion to Punycode, UTF-8, or whatever other encoding, should be done only by an entity that knows which protocol will be used (e.g., the DNS resolver, or getaddrinfo upon deciding to pass the name to DNS), rather than by general applications that call protocol-independent name resolution APIs. Similarly, even when DNS is used, the conversion to Punycode should be done only by an entity that knows which name space will be used.

That is, a more intelligent DNS resolver would be more liberal inept from an application and be able to query for both a Punycode name (e.g., over the Internet) and a UTF-8 name (e.g., over a corporate network with a private name space) in case the server only recognized one. However, we might also take into account that the various resolution behaviors discussed earlier could also occur with record updates (e.g., with Dynamic Update [RFC2136]), resulting in some names being registered in a local network's private name space by applications doing Punycode conversion, and other names being registered using UTF-8. Hence a name might have to be queried with both encodings to be sure to succeed without changes to DNS servers.

Similarly, a more intelligent stub resolver would also be more liberal in what it would accept from a response as the value of a record (e.g., PTR) in that it would accept either UTF-8 or Punycode and convert them to whatever encoding is used by the application APIs to return strings to applications.

Indeed the choice of conversion within the resolver libraries is consistent with the quote from [RFC3490] section 6.2 stating that Punycode conversion "might be performed inside these new versions of the resolver libraries".

That said, some application-layer protocols may be defined to use Punycode rather than UTF-8 as recommended by [RFC2277]. In this case, an application may receive a Punycode name and want to pass it to name resolution APIs. Again the recommendation is that a resolver library be more liberal in what it would accept from an application would mean that such a name would be accepted and re-encoded as needed, rather than requiring the application to do so.

Finally, the question remains about what a DNS server should do to handle cases where some existing applications or hosts do Punycode queries within the local network using a private name space, and other existing applications or hosts send UTF-8 queries. It is undesirable to store different records for different encodings of the same name, since this introduces the possibility for inconsistency between them. Instead, a new DNS server could treat encoding- conversion in the same way as case-insensitive comparison which a DNS server is already required to do. Two encodings are, in this sense, two representations of the same name, just as two case-different strings are. However, whereas case comparison of non-ASCII characters is complicated by ambiguities (see [RFC4690]), encoding conversion between Punycode and UTF-8 is unambiguous.

[EDITOR'S NOTE: There are also normalization/mapping issues which the next version of this document may explore. Currently we only explore encoding issues.]

[edit] 5. Security Considerations

Having applications convert names to Punycode before calling name resolution can result in security vulnerabilities. If the name is resolved by protocols or in zones for which records are registered using other encoding schemes, an attacker can claim the Punycode version of the same name and hence trick the victim into accessing a different destination. This can be done for any non-ASCII name, even when there is no possible confusion due to case, language, or other issues. Other types of confusion beyond those resulting simply from the choice of encoding scheme are discussed in [RFC4690].

[edit] 6. IANA Considerations

[RFC Editor: please remove this section prior to publication.] This document has no IANA Actions.

[edit]
DEFINITIONS

This document is one of a collection that, together, describe the protocol and usage context for a revision of Internationalized Domain Names for Applications (IDNA), superseding the earlier version. It describes the document collection and provides definitions and other material that are common to the set.

[edit] 1. Introduction

[edit] 1.1. IDNA2008

This document is one of a collection that, together, describe the protocol and usage context for a revision of Internationalized Domain Names for Applications (IDNA) that was largely completed in 2008, known within the series and elsewhere as IDNA2008. The series replaces an earlier version of IDNA, described in [RFC3490] and [RFC3491]. It continues to use the Punycode algorithm [RFC3492] and ACE (ASCII-compatible encoding) prefix from that earlier version.

The document collection is described in Section 1.3. As indicated there, this document provides definitions and other material that are common to the set.

[edit] 1.1.1. Audiences

While many IETF specifications are directed exclusively to protocol implementers, the character of IDNA requires that it be understood and properly used by those whose responsibilities include making decisions about

• what names are permitted in DNS zone files, * policies related to names and naming, and * the handling of domain name strings in files and systems, even with no immediate intention of looking them up.

This document and those concerned with the protocol definition, rules for handling strings that include characters written right-to-left, and the actual list of characters and categories will be of primary interest to protocol implementers. This document and the one containing explanatory material will be of primary interest to others, although they may have to fill in some details by reference to other documents in the set.

This document and the associated ones are written from the perspective of an IDNA-aware user, application, or implementation.

While they may reiterate fundamental DNS rules and requirements for the convenience of the reader, they make no attempt to be comprehensive about DNS principles and should not be considered as a substitute for a thorough understanding of the DNS protocols and specifications.

[edit] 1.1.2. Normative Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].

[edit] 1.2. Discussion Forum

RFC Editor: please remove this section. IDNA2008 is being discussed in the IETF "idnabis" Working Group and on the mailing list idna-update@alvestrand.no

[edit] 1.3. Roadmap of IDNA2008 Documents

IDNA2008 consists of the following documents:

• This document, containing definitions and other material that are needed for understanding other documents in the set. It is referred to informally in other documents in the set as "Defs" or "Definitions".
• A document [IDNA2008-Rationale] that provides an overview of the protocol and associated tables together with explanatory material and some rationale for the decisions that led to IDNA2008. That document also contains advice for registry operations and those who use internationalized domain names. It is referred to informally in other documents in the set as "Rationale". It is not normative.
• A document [IDNA2008-Protocol] that describes the core IDNA2008 protocol and its operations. In combination with the "Bidi" document described immediately below, it explicitly updates and replaces RFC 3490. It is referred to informally in other documents in the set as "Protocol".
• A document [IDNA2008-Bidi] that specifies special rules ("Bidi") for labels that contain characters that are written from right to left.
• A specification [IDNA2008-Tables] of the categories and rules that identify the code points allowed in a label written in native character form (defined more specifically as a "U-label" in Section 2.3.2.1 below), based on Unicode 5.1 [Unicode51] code point assignments and additional rules unique to IDNA2008. The Unicode-based rules are expected to be stable across Unicode updates and hence independent of Unicode versions. That specification obsoletes RFC 3941 and IDN use of the tables to which it refers. It is referred to informally in other documents in the set as "Tables".
• A document [IDNA2008-Mapping] that discusses the issue of mapping characters into other characters and that provides guidance for doing so when that is appropriate. This document provides advice; it is not a required part of IDNA.

[edit] 2. Definitions and Terminology

[edit] 2.1. Characters and Character Sets

A code point is an integer value in the codespace of a coded character set. In Unicode, these are integers from 0 to 0x10FFFF.

Unicode [Unicode51] is a coded character set with about 100,000 characters assigned to code points as of version 5.1. A single Unicode code point is denoted in these documents by "U+" followed by four to six hexadecimal digits, while a range of Unicode code points is denoted by two four to six digit hexadecimal numbers separated by "..", with no prefixes.

ASCII means US-ASCII [ASCII], a coded character set containing 128 characters associated with code points in the range 0000..007F.

Unicode is a superset of ASCII and may be thought of as a generalization of it; it includes all the ASCII characters and associates them with equivalent code points.

"Letters" are, informally, generalizations from the ASCII and common- sense understanding of that term, i.e., characters that are used to write text that are not digits, symbols, or punctuation. Formally, they are characters with a Unicode General Category value starting in "L" (see Section 4.5 of [Unicode51]).

[edit] 2.2. DNS-related Terminology

When discussing the DNS, this document generally assumes the terminology used in the DNS specifications [RFC1034] [RFC1035] as modified by [RFC1123] and [RFC2181]. The term "lookup" is used to describe the combination of operations performed by the IDNA2008 protocol and those actually performed by a DNS resolver. The process of placing an entry into the DNS is referred to as "registration", similar to common contemporary usage in other contexts.

Consequently, any DNS zone administration is described as a "registry", and the terms "registry" and "zone administrator" are used interchangeably, regardless of the actual administrative arrangements or level in the DNS tree. More detail about that relationship is included in the "Rationale" document.

The term "LDH code point" is defined in this document to refer to the code points associated with ASCII letters (Unicode code points 0041..005A and 0061..007A), digits (0030..0039), and the hyphen-minus (U+002D). "LDH" is an abbreviation for "letters, digits, hyphen".

The base DNS specifications [RFC1034] [RFC1035] discuss "domain names" and "host names", but many people use the terms interchangeably, as do sections of these specifications. Lack of clarity about that terminology has contributed to confusion about intent in some cases. These documents generally use the term "domain name". When they refer to, e.g., host name syntax restrictions, they explicitly cite the relevant defining documents. The remaining definitions in this subsection are essentially a review: if there is any perceived difference between those definitions and the definitions in the base DNS documents or those cited below, the definitions in the other documents take precedence.

A label is an individual component of a domain name. Labels are usually shown separated by dots; for example, the domain name "www.example.com" is composed of three labels: "www", "example", and "com". (The zero-length root label described in RFC 1123 [RFC1123], which can be explicit as in "www.example.com." or implicit as in "www.example.com", is not considered in this specification.) IDNA extends the set of usable characters in labels that are treated as text (as distinct from the binary string labels discussed in RFC 1035 and RFC 2181 [RFC2181] and the bitstring ones described in RFC 2673 [RFC2673]) , but only in certain contexts. The different contexts for different sets of usable characters are outlined in the next section. For the rest of this document and in the related ones, the term "label" is shorthand for "text label", and "every label" means "every text label", including the expanded context.

[edit] 2.3. Terminology Specific to IDNA

This section defines some terminology to reduce dependence on terms and definitions that have been problematic in the past. The relationships among these definitions are illustrated in Figure 1 and Figure 2.

[edit] 2.3.1. LDH-label

This is the classical label form used in host names [RFC0952] and described as the preferred form in RFC 1035 [RFC1035]. It is a string consisting of ASCII letters, digits, and the hyphen with the further restriction that the hyphen cannot appear at the beginning or end of the string. Like all DNS labels, its total length must not exceed 63 octets.

LDH-labels include the specialized labels used by IDNA (described as "A-labels" below) and some additional restricted forms (also described below).

To facilitate clear description, two new subsets of LDH-labels are created by the introduction of IDNA. These are called Reserved LDH labels (R-LDH labels) and Non-Reserved LDH labels (NR-LDH labels).

Reserved LDH labels, known as "tagged domain names" in some other contexts, have the property that they contain "--" in the third and fourth characters but which otherwise conform to LDH-label rules.

Only a subset of the R-LDH labels can be used in IDNA-aware applications. That subset consists of the class of labels that begin with the prefix "xn--" (case independent), but otherwise conform to the LDH-label rules. That subset is called "XN-labels" in this set of documents. XN-labels are further divided in those whose remaining characters (after the "xn--") are valid output of the Punycode algorithm RFC 3492 [RFC3492]. Such labels are known as "A-labels" if they also meet the other criteria for IDNA-validity described below.

Because LDH-labels (and, indeed, any DNS label) must not be more than 63 characters in length, the Punycode-derived portion of XN-labels is limited to no more than 59 characters. Non-reserved LDH labels are the set of valid LDH labels that do not have "--" in the third and fourth positions.

Some labels that are prefixed with "xn--" may not be the output of the Punycode algorithm, or may fail the other tests outlined below or violate other IDNA restrictions and thus are also not valid IDNA- labels. They are called FAKE A-Labels for convenience.

Labels within the class of R-LDH labels that are not prefixed with "xn--" are also not valid IDNA-labels. To allow for future use of mechanisms similar to IDNA, those labels MUST NOT be processed as ordinary LDH-labels by IDNA-conforming programs and SHOULD NOT be mixed with IDNA-labels in the same zone.

These distinctions among possible LDH labels are only of significance for software that is "IDNA-aware" or for future extensions that use extensions based on the same "prefix and encoding" model. For IDNA- aware systems, the valid label types are: A-labels, U-labels and NR- LDH labels.

IDNA-labels come in two flavors: An ACE-encoded form and a Unicode (native character) form. These are referred to as A-labels and U-labels respectively and are described in detail in the next section.

[[anchor10: Note in Draft: Figure 1, including all of the notes and annotation text, is 60-odd lines long, much longer than the 49 text- content lines permitted by RFCs. Consequently, the xml processor is breaking it across page boundaries. Once we have agreement that it is right, I'll try to reformat it again, possibly removing the notes to text blocks, to make it all fit on a page. Doing that before things solidify is not just a waste of time but a menace to future editing. This will be done in version -11, after WG Last Call. --JcK (Ed.)]] The figure on this page illustrates the relationships among some of the terms defined above. The parenthesized numbers refer to the notes below the figure.

ASCII-LABEL

----------------------------------------------------------------
|                                                                |
|                 LDH-LABEL (1) (4)                              |
|     _______________________________________________________    |
|    |                                                       |   |
|    |                                                       |   |
|    |  __________________________________                   |   |
|    |  |IDN Reserved LDH Labels          |                  |   |
|    |  | ("??--") or R-LDH LABELS        | ______________   |   |
|    |  |                                 | |NON-RESERVED |  |   |
|    |  | ------------------------------- | | LDH LABELS  |  |   |
|    |  | |       XN LABELS             | | | (NR-LDH-    |  |   |
|    |  | | _____________   ___________ | | |   labels)   |  |   |
|    |  | | |           |   |          || | |NR-LDH LABELS|  |   |
|    |  | | | A-labels  |   | Fake (3) || | |             |  |   |
|    |  | | | "xn--"(2) |   | A-labels || | |_____________|  |   |
|    |  | | |___________|   |__________|| |                  |   |
|    |  | |_____________________________| |                  |   |
|    |  |_________________________________|                  |   |
|    |_______________________________________________________|   |
|                                                                |
|                    NON-LDH-LABEL                               |
|       _________________________________                        |
|       |                                |                       |
|       |      ----------------------    |                       |
|       |      | Underscore labels  |    |                       |
|       |      |  e.g. _tcp         |    |                       |
|       |      |--------------------|    |                       |
|       |      | leading            |    |                       |
|       |      | or trailing        |    |                       |
|       |      | hyphens "-abcd"    |    |                       |
|       |      | or "xyz-"          |    |                       |
|       |      | or "-uvw-"         |    |                       |
|       |      |--------------------|    |                       |
|       |      | Other non-LDH      |    |                       |
|       |      | ASCII chars        |    |                       |
|       |      | e.g. #$%_          |    |                       |
|       |      ----------------------    |                       |
|       |--------------------------------|                       |
|                                                                |
|________________________________________________________________|

(1) ASCII letters (upper and lower case), digits, hyphen. Hyphen may not appear in first or last position. Less than 63 characters.

(2) Note that the string following "xn--" must be the valid output of the Punycode algorithm and must be convertible into valid U-label form.

(3) Note that a Fake A-Label has a prefix "xn--" but the remainder of the label is NOT the valid output of the Punycode algorithm.

(4) LDH-LABEL subtypes are indistinguishable to IDNA-unaware applications.

Figure 1: IDNA and Related DNS Terminology Space -- ASCII labels

__________________________
|  Non-ASCII             |
|                        |
|    ___________________ |
|    | U-label (5)     | |
|    |_________________| |
|    |                 | |
|    |  Binary Label   | |
|    | (including      | |
|    |  high bit on)   | |
|    |_________________| |
|    |                 | |
|    | Bit String      | |
|    |   Label         | |
|    |_________________| |
|________________________|

(5) To IDNA-unaware applications, U-labels are indistinguishable from Binary ones.

Figure 2: Non-ASCII labels

[edit] 2.3.2. Terms for IDN Label Codings

[edit] 2.3.2.1. IDNA-valid strings, A-label, and U-label

For IDNA-aware applications, valid labels include "A-labels", "U-labels", and "NR-LDH-labels", each of which is defined below. The relationships among them are illustrated in Figure 1 and Figure 2.

• A string is "IDNA-valid" if it meets all of the requirements of these specifications for an IDNA label. IDNA-valid strings may appear in either of the two forms, defined immediately below, or may be drawn from the NR-LDH-label subset. IDNA-valid strings must also conform to all basic DNS requirements for labels. These documents make specific reference to the form appropriate to any context in which the distinction is important.
• An "A-label" is the ASCII-Compatible Encoding (ACE, see Section 2.3.2.5) form of an IDNA-valid string. It must be a complete label: IDNA is defined for labels, not for parts of them and not for complete domain names. This means, by definition, that every A-label will begin with the IDNA ACE prefix, "xn--" (see Section 2.3.2.5), followed by a string that is a valid output of the Punycode algorithm [RFC3492] and hence a maximum of 59 ASCII characters in length. The prefix and string together must conform to all requirements for a label that can be stored in the DNS including conformance to the rules for the preferred form described in RFC 1034, RFC 1035, and RFC 1123. A string meeting the above requirements is still not an A-label unless it can be decoded into a U-label.
• A "U-label" is an IDNA-valid string of Unicode characters, in normalization form NFC and including at least one non-ASCII character, expressed in a standard Unicode Encoding Form (in an Internet transmission context this will normally be UTF-8). It is also subject to the constraints about permitted characters that are specified in the Protocol and Tables documents, the Bidi constraints in that document if it contains any character from scripts that are written right to left, and the symmetry constraint described immediately below. Conversions between U-labels and A-labels are performed according to the "Punycode" specification [RFC3492], adding or removing the ACE prefix as needed.

[[anchor12: Note in draft: Insert section number references to Protocol and maybe Tables and Bidi above before handoff to RFC Editor and warn them to make sure they stay consistent.]] To be valid, U-labels and A-labels must obey an important symmetry constraint. While that constraint may be tested in any of several ways, an A-label must be capable of being produced by conversion from a U-label and a U-label must be capable of being produced by conversion from an A-label. Among other things, this implies that both U-labels and A-labels must be strings in Unicode NFC [Unicode-UAX15] normalized form. These strings MUST contain only characters specified elsewhere in this document series, and only in the contexts indicated as appropriate.

Any rules or conventions that apply to DNS labels in general, such as rules about lengths of strings, apply to whichever of the U-label or A-label would be more restrictive. For the U-label, constraints imposed by existing protocols and their presentation forms make the length restriction apply to the length in octets of the UTF-8 form of those labels (which will always be greater than or equal to the length in code points). The exception to this, of course, is that the restriction to ASCII characters does not apply to the U-label.

For context, IDNA-unaware applications treat all LDH-labels as valid for appearance in DNS zone files and queries. IDNA-aware applications permit only A-labels and NR-LDH-labels to appear in zone files and queries. U-labels can appear, along with the other two, in presentation and user interface forms and in selected protocols other than those of the DNS itself.

Specifically, for IDNA-aware applications, the three allowed categories are A-label, U-label, and NR-LDH-label. Of the reserved LDH labels (R-LDH-labels) only A-labels are valid for IDNA use.

Strings that appear to be A-labels or U-labels are processed in various of the operations of [IDNA2008-Protocol]. Those strings are not yet demonstrably conformant with the conditions outlined above, because they are in the process of validation. Such strings may be referred to as "unvalidated", "putative", or "apparent", or as being "in the form of" one of the label types to indicate that they have not been verified to meet the specified conformance requirements.

Unvalidated A-labels are known only to be XN Labels, while Fake A-labels have been demonstrated to fail some of the A-label tests.

Similarly, unvalidated U-labels are simply Non-ASCII labels that may or may not meet the requirements for U-labels.

[edit] 2.3.2.2. NR-LDH-label and Internationalized Label

These specifications use the term "NR-LDH-label" strictly to refer to an all-ASCII label that obeys the preferred syntax (often known as "hostname" (from RFC 952 [RFC0952]) or "LDH") conventions and that is neither an IDN nor a label form reserved by IDNA (R-LDH-label). It should be stressed that an A-label obeys the "hostname" rules and is sometimes described as "LDH-conformant".

[edit] 2.3.2.3. Internationalized Domain Name

An "internationalized domain name" (IDN) is a domain name that contains at least one A-label or U-label, but that otherwise may contain any mixture of NR-LDH-labels, A-labels, or U-labels. Just as has been the case with ASCII names, some DNS zone administrators may impose restrictions, beyond those imposed by DNS or IDNA, on the characters or strings that may be registered as labels in their zones. Because of the diversity of characters that can be used in a U-label and the confusion they might cause, such restrictions are mandatory for IDN registries and zones even though the particular restrictions are not part of these specifications (the issue is discussed in more detail in [IDNA2008-Protocol] [[anchor15: Insert section number]]. Because these restrictions, commonly known as "registry restrictions", only affect what can be registered and not lookup processing, they have no effect on the syntax or semantics of DNS protocol messages; a query for a name that matches no records will yield the same response regardless of the reason why it is not in the zone. Clients issuing queries or interpreting responses cannot be assumed to have any knowledge of zone-specific restrictions or conventions. See the section on registration policy in [IDNA2008-Rationale] for additional discussion.

"Internationalized label" is used when a term is needed to refer to a single label of an IDN, i.e., one that might be any of an NR-LDH- label, A-label, or U-label. There are some standardized DNS label formats, such as the "underscore labels" used for service location (SRV) records [RFC2782], that do not fall into any of the three categories and hence are not internationalized labels.

[edit] 2.3.2.4. Label Equivalence

In IDNA, equivalence of labels is defined in terms of the A-labels.

If the A-labels are equal in a case-independent comparison, then the labels are considered equivalent, no matter how they are represented.

Because of the isomorphism of A-labels and U-labels in IDNA2008, it is possible to compare U-labels directly; see [IDNA2008-Protocol] for details. Traditional LDH labels already have a notion of equivalence: within that list of characters, upper case and lower case are considered equivalent. The IDNA notion of equivalence is an extension of that older notion but, because there is no mapping, the only equivalents are:

• Exact (bit-string identity) matches between a pair of U-labels.

• Matches between a pair of A-labels, using normal DNS matching rules.
• Equivalence between a U-label and an A-label determined by translating the U-label form into an A-label form and then testing for an exact match between the A-labels.

[edit] 2.3.2.5. ACE Prefix

The "ACE prefix" is defined in this document to be a string of ASCII characters "xn--" that appears at the beginning of every A-label.

"ACE" stands for "ASCII-Compatible Encoding".

[edit] 2.3.2.6. Domain Name Slot

A "domain name slot" is defined in this document to be a protocol element or a function argument or a return value (and so on) explicitly designated for carrying a domain name. Examples of domain name slots include the QNAME field of a DNS query; the name argument of the gethostbyname() or getaddrinfo() standard C library functions; the part of an email address following the at-sign (@) in the parameter to the SMTP MAIL or RCPT commands or the "From:" field of an email message header; and the host portion of the URI in the src attribute of an HTML <IMG> tag. A string that has the syntax of a domain name but that appears in general text is not in a domain name slot. For example, a domain name appearing in the plain text body of an email message is not occupying a domain name slot.

An "IDN-aware domain name slot" is defined for this set of documents to be a domain name slot explicitly designated for carrying an internationalized domain name as defined in this document. The designation may be static (for example, in the specification of the protocol or interface) or dynamic (for example, as a result of negotiation in an interactive session).

An "IDN-unaware domain name slot" is defined for this set of documents to be any domain name slot that is not an IDN-aware domain name slot. Obviously, this includes any domain name slot whose specification predates IDNA. Note that the requirements of some protocols that use the DNS for data storage prevent the use of IDNs.

For example, the format required for the underscore labels used by the service location protocol[RFC2782] precludes representation of a non-ASCII label in the DNS using A-labels because those SRV-related labels must start with underscores. Of course, non-ASCII IDN labels may be part of a domain name that also includes underscore labels.

[edit] 2.3.3. Order of Characters in Labels

Because IDN labels may contain characters that are read, and preferentially displayed, from right to left, there is a potential ambiguity about which character in a label is "first". For the purposes of these specifications, labels are considered, and characters numbered, strictly in the order in which they appear "on the wire". That order is equivalent to the leftmost character being treated as first in a label that is read left-to-right and to the rightmost character being first in a label that is read right-to- left. The "Bidi" specification contains additional discussion of the conditions that influence reading order.

[edit] 2.3.4. Punycode is an Algorithm, not a Name or Adjective

There has been some confusion about whether a "Punycode string" does or does not include the ACE prefix and about whether it is required that such strings could have been the output of the ToASCII operation (see RFC 3490, Section 4 [RFC3490]). This specification discourages the use of the term "Punycode" to describe anything but the encoding method and algorithm of [RFC3492]. The terms defined above are preferred as much more clear than the term "Punycode string".

[edit] 3. IANA Considerations

Actions for IANA are specified in other documents in this series [IDNA2008-Protocol] [IDNA2008-Tables]. An overview of the relationships among the various IANA registries appears in [IDNA2008-Rationale]. This document does not specify any actions for IANA.

[edit] 4. Security Considerations

[edit] 4.1. General Issues

Security on the Internet partly relies on the DNS. Thus, any change to the characteristics of the DNS can change the security of much of the Internet.

Domain names are used by users to identify and connect to Internet hosts and other network resources. The security of the Internet is compromised if a user entering a single internationalized name is connected to different servers based on different interpretations of the internationalized domain name. In addition to characters that are permitted by IDNA2003 and its mapping conventions (See Section 4.5), the current specification changes the interpretation of a few characters that were mapped to others in the earlier version; zone administrators should be aware of the problems that might raise and take appropriate measures. The context for this issue is discussed in more detail in [IDNA2008-Rationale]).

In addition to the Security Considerations material that appears in this document, [IDNA2008-Bidi] contains a discussion of security issues specific to labels containing characters from scripts that are normally written right to left.

[edit] 4.2. Local Character Set Issues

When systems use local character sets other than ASCII and Unicode, these specifications leave the problem of converting between the local character set and Unicode up to the application or local system. If different applications (or different versions of one application) implement different rules for conversions among coded character sets, they could interpret the same name differently and contact different servers. This problem is not solved by security protocols, such as Transport Layer Security (TLS) [RFC5246], that do not take local character sets into account.

[edit] 4.3. Visually Similar Characters

To help prevent confusion between characters that are visually similar, it is suggested that implementations provide visual indications where a domain name contains multiple scripts, especially when the scripts contain characters that are easily confused visually, such as an omicron in Greek mixed with Latin text. Such mechanisms can also be used to show when a name contains a mixture of simplified and traditional Chinese characters, or to distinguish zero and one from upper-case "O" and lower-case "L". DNS zone administrators may impose restrictions (subject to the limitations identified elsewhere in these documents) that try to minimize characters that have similar appearance or similar interpretations.

It is worth noting that there are no comprehensive technical solutions to the problems of confusable characters. One can reduce the extent of the problems in various ways, but probably never eliminate it. Some specific suggestions about identification and handling of confusable characters appear in a Unicode Consortium publication [Unicode-UTR36].

[edit] 4.4. IDNA Lookup, Registration, and the Base DNS Specifications

The Protocol specification [IDNA2008-Protocol] describes procedures for registering and looking up labels that are not compatible with the preferred syntax described in the base DNS specifications (STD13 [RFC1034] [RFC1035] and Host Requirements [RFC1123]) because they contain non-ASCII characters. These procedures depend on the use of a special ASCII-compatible encoding form that contains only characters permitted in host names by those earlier specifications.

The encoding used is Punycode [RFC3492]. No security issues such as string length increases or new allowed values are introduced by the encoding process or the use of these encoded values, apart from those introduced by the ACE encoding itself.

Domain names (or portions of them) are sometimes compared against a set of domains to be given special treatment if a match occurs, e.g., treated as more privileged than others or blocked in some way. In such situations, it is especially important that the comparisons be done properly, as specified in the Requirements section of [IDNA2008-Protocol]. For labels already in ASCII form, the proper comparison reduces to the same case-insensitive ASCII comparison that has always been used for ASCII labels although IDNA-aware applications are expected to look up only A-labels and NR-LDH-labels, i.e., to avoid looking up R-LDH-labels that are not A-labels.

The introduction of IDNA meant that any existing labels that start with the ACE prefix would be construed as A-labels, at least until they failed one of the relevant tests, whether or not that was the intent of the zone administrator or registrant. There is no evidence that this has caused any practical problems since RFC 3490 was adopted, but the risk still exists in principle.

[edit] 4.5. Legacy IDN Label Strings

The URI Standard [RFC3986] and a number of application specifications (e.g., [RFC5321], [RFC2616]) do not permit non-ASCII labels in DNS names used with those protocols, i.e., only the A-label form of IDNs is permitted in those contexts. If only A-labels are used, differences in interpretation between IDNA2003 and this version arise only for characters whose interpretation have actually changed (e.g., characters, such as ZWJ and ZWNJ, that were mapped to nothing in IDNA2003 and that are considered legitimate by these specifications).

Despite that prohibition, there are a significant number of files and databases on the Internet in which domain name strings appear in native-character form; a subset of those strings use native-character labels that require IDNA2003 mapping to produce valid A-labels. The treatment of such labels will vary by types of applications and application-designer preference: in some situations, warnings to the user or outright rejection may be appropriate; in others, it may be preferable to attempt to apply the earlier mappings if lookup strictly conformant to these specifications fails or even to do lookups under both sets of rules. This general situation is discussed in more detail in [IDNA2008-Rationale]. However, in the absence of care by registries about how strings that could have different interpretations under IDNA2003 and the current specification are handled, it is possible that the differences could be used as a component of name-matching or name-confusion attacks.

Such care is therefore appropriate.

[edit] 4.6. Security Differences from IDNA2003

The registration and lookup models described in this set of documents change the mechanisms available for lookup applications to determine the validity of labels they encounter. In some respects, the ability to test is strengthened. For example, putative labels that contain unassigned code points will now be rejected, while IDNA2003 permitted them (see [IDNA2008-Rationale] for a discussion of the reasons for this). On the other hand, the protocol specification no longer assumes that the application that looks up a name will be able to determine, and apply, information about the protocol version used in registration. In theory, that may increase risk since the application will be able to do less pre-lookup validation. In practice, the protection afforded by that test has been largely illusory for reasons explained in RFC 4690 [RFC4690] and elsewhere in these documents.

Any change to the Stringprep [RFC3454] procedure that is profiled and used in IDNA2003, or, more broadly, the IETF's model of the use of internationalized character strings in different protocols, creates some risk of inadvertent changes to those protocols, invalidating deployed applications or databases, and so on. But these specifications do not change Stringprep at all; they merely bypass it. Because these documents do not depend on Stringprep, the question of upgrading other protocols that do have that dependency can be left to experts on those protocols: the IDNA changes and possible upgrades to security protocols or conventions are independent issues.

[edit] 4.7. Summary

No mechanism involving names or identifiers alone can protect against a wide variety of security threats and attacks that are largely independent of the naming or identification system. These attacks include spoofed pages, DNS query trapping and diversion, and so on.

[edit]
IDNA RATIONALE

Several years have passed since the original protocol for Internationalized Domain Names (IDNs) was completed and deployed.

During that time, a number of issues have arisen, including the need to update the system to deal with newer versions of Unicode. Some of these issues require tuning of the existing protocols and the tables on which they depend. This document provides an overview of a revised system and provides explanatory material for its components.

[edit] 1. Introduction

[edit] 1.1. Context and Overview

Internationalized Domain Names in Applications (IDNA) is a collection of standards that allow client applications to convert some Unicode mnemonics to an ASCII-compatible encoding form ("ACE") which is a valid DNS label containing only letters, digits, and hyphens. The specific form of ACE label used by IDNA is called an "A-label". A client can look up an exact A-label in the existing DNS, so A-labels do not require any extensions to DNS, upgrades of DNS servers or updates to low-level client libraries. An A-label is recognizable from the prefix "xn--" before the characters produced by the Punycode algorithm [RFC3492], thus a user application can identify an A-label and convert it into Unicode (or some local coded character set) for display.

On the registry side, IDNA allows a registry to offer Internationalized Domain Names (IDNs) for registration as A-labels.

A registry may offer any subset of valid IDNs, and may apply any restrictions or bundling (grouping of similar labels together in one registration) appropriate for the context of that registry.

Registration of labels is sometimes discussed separately from lookup, and is subject to a few specific requirements that do not apply to lookup.

DNS clients and registries are subject to some differences in requirements for handling IDNs. In particular, registries are urged to register only exact, valid A-labels, while clients might do some mapping to get from otherwise-invalid user input to a valid A-label.

The first version of IDNA was published in 2003 and is referred to here as IDNA2003 to contrast it with the current version, which is known as IDNA2008 (after the year in which IETF work started on it).

IDNA2003 consists of four documents: the IDNA base specification [RFC3490], Nameprep [RFC3491], Punycode [RFC3492], and Stringprep [RFC3454]. The current set of documents, IDNA2008, are not dependent on any of the IDNA2003 specifications other than the one for Punycode encoding. References to "these specifications" or "these documents" are to the entire IDNA2008 set listed in [IDNA2008-Defs]. The characters that are valid in A-labels are identified from rules listed in the Tables document [IDNA2008-Tables], but validity can be derived from the Unicode properties of those characters with a very few exceptions.

Traditionally, DNS labels are matched case-insensitively [RFC1034][RFC1035]. That convention was preserved in IDNA2003 by a case-folding operation that generally maps capital letters into lower-case ones. However, if case rules are enforced from one language, another language sometimes loses the ability to treat two characters separately. Case-sensitivity is treated slightly differently in IDNA2008.

IDNA2003 used Unicode version 3.2 only. In order to keep up with new characters added in new versions of UNICODE, IDNA2008 decouples its rules from any particular version of UNICODE. Instead, the attributes of new characters in Unicode, supplemented by a small number of exception cases, determine how and whether the characters can be used in IDNA labels.

This document provides informational context for IDNA2008, including terminology, background, and policy discussions.

[edit] 1.2. Discussion Forum

RFC Editor: please remove this section. IDNA2008 is being discussed in the IETF "idnabis" Working Group and on the mailing list idna-update@alvestrand.no

[edit] 1.3. Terminology

Terminology for IDNA2008 appears in [IDNA2008-Defs]. That document also contains a roadmap to the IDNA2008 document collection. No attempt should be made to understand this document without the definitions and concepts that appear there.

[edit] 1.3.1. DNS "Name" Terminology

In the context of IDNs, the DNS term "name" has introduced some confusion as people speak of DNS labels in terms of the words or phrases of various natural languages. Historically, many of the "names" in the DNS have been mnemonics to identify some particular concept, object, or organization. They are typically rooted in some language because most people think in language-based ways. But, because they are mnemonics, they need not obey the orthographic conventions of any language: it is not a requirement that it be possible for them to be "words".

This distinction is important because the reasonable goal of an IDN effort is not to be able to write the great Klingon (or language of one's choice) novel in DNS labels but to be able to form a usefully broad range of mnemonics in ways that are as natural as possible in a very broad range of scripts.

[edit] 1.3.2. New Terminology and Restrictions

These documents introduce new terminology, and precise definitions (in [IDNA2008-Defs]), for the terms "U-label", "A-Label", LDH-label (to which all valid pre-IDNA host names conformed), Reserved-LDH- label (R-LDH-label), XN-label, Fake-A-Label, and Non-Reserved-LDH- label (NR-LDH-label).

In addition, the term "putative label" has been adopted to refer to a label that may appear to meet certain definitional constraints but has not yet been sufficiently tested for validity.

These definitions are also illustrated in Figure 1 of the Definitions Document [IDNA2008-Defs]. R-LDH-labels contain "--" in the third and fourth character from the beginning of the label. In IDNA-aware applications, only a subset of these reserved labels is permitted to be used, namely the A-label subset. A-labels are a subset of the R-LDH-labels that begin with the case-insensitive string "xn--".

Labels that bear this prefix but which are not otherwise valid fall into the "Fake-A-label" category. The non-reserved labels (NR-LDH- labels) are implicitly valid since they do not trigger any resemblance to IDNA-landr NR-LDH-labels.

The creation of the Reserved-LDH category is required for three reasons:

• to prevent confusion with pre-IDNA coding forms;
• to permit future extensions that would require changing the prefix, no matter how unlikely those might be (see Section 7.4); and
• to reduce the opportunities for attacks via the Punycode encoding algorithm itself.

As with other documents in the IDNA2008 set, this document uses the term "registry" to describe any zone in the DNS. That term, and the terms "zone" or "zone administration", are interchangeable.

[edit] 1.4. Objectives

These are the main objectives in revising IDNA.

• Use a more recent version of Unicode, and allow IDNA to be independent of Unicode versions, so that IDNA2008 need not be updated for implementations to adopt codepoints from new Unicode versions.

• Fix a very small number of code-point categorizations that have turned out to cause problems in the communities that use those code-points.
• Reduce the dependency on mapping, in order that the pre-mapped forms (which are not valid IDNA labels) tend to appear less often in various contexts, in favor of valid A-labels.
• Fix some details in the bidirectional codepoint handling algorithms.

[edit] 1.5. Applicability and Function of IDNA

The IDNA specification solves the problem of extending the repertoire of characters that can be used in domain names to include a large subset of the Unicode repertoire.

IDNA does not extend DNS. Instead, the applications (and, by implication, the users) continue to see an exact-match lookup service. Either there is a single exactly-matching (subject to the base DNS requirement of case-insensitive ASCII matching) name or there is no match. This model has served the existing applications well, but it requires, with or without internationalized domain names, that users know the exact spelling of the domain names that are to be typed into applications such as web browsers and mail user agents. The introduction of the larger repertoire of characters potentially makes the set of misspellings larger, especially given that in some cases the same appearance, for example on a business card, might visually match several Unicode code points or several sequences of code points.

The IDNA standard does not require any applications to conform to it, nor does it retroactively change those applications. An application can elect to use IDNA in order to support IDN while maintaining interoperability with existing infrastructure. If an application wants to use non-ASCII characters in public DNS domain names, IDNA is the only currently-defined option. Adding IDNA support to an existing application entails changes to the application only, and leaves room for flexibility in front-end processing and more specifically in the user interface (see Section 6).

A great deal of the discussion of IDN solutions has focused on transition issues and how IDNs will work in a world where not all of the components have been updated. Proposals that were not chosen by the original IDN Working Group would have depended on updating of user applications, DNS resolvers, and DNS servers in order for a user to apply an internationalized domain name in any form or coding acceptable under that method. While processing must be performed prior to or after access to the DNS, IDNA requires no changes to the DNS protocol or any DNS servers or the resolvers on user's computers.

IDNA allows the graceful introduction of IDNs not only by avoiding upgrades to existing infrastructure (such as DNS servers and mail transport agents), but also by allowing some limited use of IDNs in applications by using the ASCII-encoded representation of the labels containing non-ASCII characters. While such names are user- unfriendly to read and type, and hence not optimal for user input, they can be used as a last resort to allow rudimentary IDN usage.

For example, they might be the best choice for display if it were known that relevant fonts were not available on the user's computer.

In order to allow user-friendly input and output of the IDNs and acceptance of some characters as equivalent to those to be processed according to the protocol, the applications need to be modified to conform to this specification.

This version of IDNA uses the Unicode character repertoire, for continuity with the original version of IDNA.

[edit] 1.6. Comprehensibility of IDNA Mechanisms and Processing

One goal of IDNA2008, which is aided by the main goal of reducing the dependency on mapping, is to improve the general understanding of how IDNA works and what characters are permitted and what happens to them. Comprehensibility and predictability to users and registrants are important design goals for this effort. End-user applications have an important role to play in increasing this comprehensibility.

Any system that tries to handle international characters encounters some common problems. For example, a UI cannot display a character if no font for that character is available. In some cases, internationalization enables effective localization while maintaining some global uniformity but losing some universality.

It is difficult to even make suggestions for end-user applications to cope when characters and fonts are not available. Because display functions are rarely controlled by the types of applications that would call upon IDNA, such suggestions will rarely be very effective.

Converting between local character sets and normalized Unicode, if needed, is part of this set of user agent issues. This conversion introduces complexity in a system that is not Unicode-native. If a label is converted to a local character set that does not have all the needed characters, or that uses different character-coding principles, the user agent may have to add special logic to avoid or reduce loss of information.

The major difficulty may lie in accurately identifying the incoming character set and applying the correct conversion routine. Even more difficult, the local character coding system could be based on conceptually different assumptions than those used by Unicode (e.g., choice of font encodings used for publications in some Indic scripts). Those differences may not easily yield unambiguous conversions or interpretations even if each coding system is internally consistent and adequate to represent the local language and script.

IDNA2008 shifts responsibility for character mapping and other adjustments from the protocol (where it was located in IDNA2003) to pre-processing before invoking IDNA itself. The intent is that this change will lead to greater usage of fully-valid A-Labels or U-labels in display, transit and storage, which should aid comprehensibility and predictability. A careful look at pre-processing raises issues about what that pre-processing should do and at what point pre- processing becomes harmful, how universally consistent pre-processing algorithms can be, and how to be compatible with labels prepared in a IDNA2003 context. Those issues are discussed in Section 6 and in the separate document [IDNA2008-Mapping].

[edit] 2. Processing in IDNA2008

These specifications separate Domain Name Registration and Lookup in the protocol specification. Although most steps in the two processes are similar, the separation reflects current practice in which per- registry (DNS zone) restrictions and special processing are applied at registration time but not during lookup. Another significant benefit is that separation facilitates incremental addition of permitted character groups to avoid freezing on one particular version of Unicode.

The actual registration and lookup protocols for IDNA2008 are specified in [IDNA2008-Protocol].

[edit] 3. Permitted Characters: An Inclusion List

IDNA2008 adopts the inclusion model. A code-point is assumed to be invalid for IDN use unless it is included as part of a Unicode property-based rule or, in rare cases, included individually by an exception. When an implementation moves to a new version of Unicode, the rules may indicate new valid code-points.

This section provides an overview of the model used to establish the algorithm and character lists of [IDNA2008-Tables] and describes the names and applicability of the categories used there. Note that the inclusion of a character in the first category group (Section 3.1.1) does not imply that it can be used indiscriminately; some characters are associated with contextual rules that must be applied as well.

The information given in this section is provided to make the rules, tables, and protocol easier to understand. The normative generating rules that correspond to this informal discussion appear in [IDNA2008-Tables] and the rules that actually determine what labels can be registered or looked up are in [IDNA2008-Protocol].

[edit] 3.1. A Tiered Model of Permitted Characters and Labels

Moving to an inclusion model involves a new specification for the list of characters that are permitted in IDNs. In IDNA2003, character validity is independent of context and fixed forever (or until the standard is replaced). However, globally context- independent rules have proved to be impractical because some characters, especially those that are called "Join_Controls" in Unicode, are needed to make reasonable use of some scripts but have no visible effect in others. IDNA2003 prohibited those types of characters entirely by discarding them. We now have a consensus that under some conditions, these "joiner" characters are legitimately needed to allow useful mnemonics for some languages and scripts. In general, context-dependent rules help deal with characters (generally characters that would otherwise be prohibited entirely) that are used differently or perceived differently across different scripts, and allow the standard to be applied more appropriately in cases where a string is not universally handled the same way.

IDNA2008 divides all possible Unicode code-points into four categories: PROTOCOL-VALID, CONTEXTUAL RULE REQUIRED, DISALLOWED and UNASSIGNED.

[edit] 3.1.1. PROTOCOL-VALID

Characters identified as "PROTOCOL-VALID" (often abbreviated "PVALID") are permitted in IDNs. Their use may be restricted by rules about the context in which they appear or by other rules that apply to the entire label in which they are to be embedded. For example, any label that contains a character in this category that has a "right-to-left" property must be used in context with the "Bidi" rules (see [IDNA2008-Bidi]).

The term "PROTOCOL-VALID" is used to stress the fact that the presence of a character in this category does not imply that a given registry need accept registrations containing any of the characters in the category. Registries are still expected to apply judgment about labels they will accept and to maintain rules consistent with those judgments (see [IDNA2008-Protocol] and Section 3.3).

Characters that are placed in the "PROTOCOL-VALID" category are expected to never be removed from it or reclassified. While theoretically characters could be removed from Unicode, such removal would be inconsistent with the Unicode stability principles (see [Unicode51], Appendix F) and hence should never occur.

[edit] 3.1.2. CONTEXTUAL RULE REQUIRED

Some characters may be unsuitable for general use in IDNs but necessary for the plausible support of some scripts. The two most commonly-cited examples are the zero-width joiner and non-joiner characters (ZWJ, U+200D and ZWNJ, U+200C) but other characters may require special treatment because they would otherwise be DISALLOWED (typically because Unicode considers them punctuation or special symbols) but need to be permitted in limited contexts. Other characters are given this special treatment because they pose exceptional danger of being used to produce misleading labels or to cause unacceptable ambiguity in label matching and interpretation.

[edit] 3.1.2.1. Contextual Restrictions

Characters with contextual restrictions are identified as "CONTEXTUAL RULE REQUIRED" and associated with a rule. The rule defines whether the character is valid in a particular string, and also whether the rule itself is to be applied on lookup as well as registration.

A distinction is made between characters that indicate or prohibit joining and ones similar to them (known as "CONTEXT-JOINER" or "CONTEXTJ") and other characters requiring contextual treatment ("CONTEXT-OTHER" or "CONTEXTO"). Only the former require full testing at lookup time.

It is important to note that these contextual rules cannot prevent all uses of the relevant characters that might be confusing or problematic. What they are expected do is to confine applicability of the characters to scripts (and narrower contexts) where zone administrators are knowledgeable enough about the use of those characters to be prepared to deal with them appropriately. For example, a registry dealing with an Indic script that requires ZWJ and/or ZWNJ as part of the writing system is expected to understand where the characters have visible effect and where they do not and to make registration rules accordingly. By contrast, a registry dealing primarily with Latin or Cyrillic script might not be actively aware that the characters exist, much less about the consequences of embedding them in labels drawn from those scripts.

[edit] 3.1.2.2. Rules and Their Application

Rules have descriptions such as "Must follow a character from Script XYZ", "Must occur only if the entire label is in Script ABC", or "Must occur only if the previous and subsequent characters have the DFG property". The actual rules may be DEFINED or NULL. If present, they may have values of "True" (character may be used in any position in any label), "False" (character may not be used in any label), or may be a set of procedural rules that specify the context in which the character is permitted.

Examples of descriptions of typical rules, stated informally and in English, include "Must follow a character from Script XYZ", "Must occur only if the entire label is in Script ABC", "Must occur only if the previous and subsequent characters have the DFG property".

Because it is easier to identify these characters than to know that they are actually needed in IDNs or how to establish exactly the right rules for each one, a rule may have a null value in a given version of the tables. Characters associated with null rules are not permitted to appear in putative labels for either registration or lookup. Of course, a later version of the tables might contain a non-null rule.

The actual rules and their descriptions are in [IDNA2008-Tables].

anchor9: ??? Section number would be good here. That document also specifies the creation of a registry for future rules.

[edit] 3.1.3. DISALLOWED

Some characters are inappropriate for use in IDNs and are thus excluded for both registration and lookup (i.e., IDNA-conforming applications performing name lookup should verify that these characters are absent; if they are present, the label strings should be rejected rather than converted to A-labels and looked up. Some of these characters are problematic for use in IDNs (such as the FRACTION SLASH character, U+2044), while some of them (such as the various HEART symbols, e.g., U+2665, U+2661, and U+2765, see Section 7.6) simply fall outside the conventions for typical identifiers (basically letters and numbers).

Of course, this category would include code points that had been removed entirely from Unicode should such removals ever occur.

Characters that are placed in the "DISALLOWED" category are expected to never be removed from it or reclassified. If a character is classified as "DISALLOWED" in error and the error is sufficiently problematic, the only recourse would be either to introduce a new code point into Unicode and classify it as "PROTOCOL-VALID" or for the IETF to accept the considerable costs of an incompatible change and replace the relevant RFC with one containing appropriate exceptions.

There is provision for exception cases but, in general, characters are placed into "DISALLOWED" if they fall into one or more of the following groups:

• The character is a compatibility equivalent for another character. In slightly more precise Unicode terms, application of normalization method NFKC to the character yields some other character.
• The character is an upper-case form or some other form that is mapped to another character by Unicode casefolding.
• The character is a symbol or punctuation form or, more generally, something that is not a letter, digit, or a mark that is used to form a letter or digit.

[edit] 3.1.4. UNASSIGNED

For convenience in processing and table-building, code points that do not have assigned values in a given version of Unicode are treated as belonging to a special UNASSIGNED category. Such code points are prohibited in labels to be registered or looked up. The category differs from DISALLOWED in that code points are moved out of it by the simple expedient of being assigned in a later version of Unicode (at which point, they are classified into one of the other categories as appropriate).

The rationale for restricting the processing of UNASSIGNED characters is simply that the properties of such code points cannot be completely known until actual characters are assigned to them. If, for example, such a code point was permitted to be included in a label to be looked up, and the code point was later to be assigned to a character that required some set of contextual rules, un-updated instances of IDNA-aware software might permit lookup of labels containing the previously-unassigned characters while updated versions of IDNA-aware software might restrict their use in lookup, depending on the contextual rules. It should be clear that under no circumstance should an UNASSIGNED character be permitted in a label to be registered as part of a domain name.

[edit] 3.2. Registration Policy

While these recommendations cannot and should not define registry policies, registries should develop and apply additional restrictions as needed to reduce confusion and other problems. For example, it is generally believed that labels containing characters from more than one script are a bad practice although there may be some important exceptions to that principle. Some registries may choose to restrict registrations to characters drawn from a very small number of scripts. For many scripts, the use of variant techniques such as those as described in RFC 3843 [RFC3743] and RFC 4290 [RFC4290], and illustrated for Chinese by the tables described in RFC 4713 [RFC4713] may be helpful in reducing problems that might be perceived by users.

In general, users will benefit if registries only permit characters from scripts that are well-understood by the registry or its advisers. If a registry decides to reduce opportunities for confusion by constructing policies that disallow characters used in historic writing systems or characters whose use is restricted to specialized, highly technical contexts, some relevant information may be found in Section 2.4 "Specific Character Adjustments", Table 4 "Candidate Characters for Exclusion from Identifiers" of [Unicode-UAX31] and Section 3.1. "General Security Profile for Identifiers" in [Unicode-Security].

The requirement (in [IDNA2008-Protocol] [[anchor10: ?? Section number]]) that registration procedures use only U-labels and/or A-labels is intended to ensure that registrants are fully aware of exactly what is being registered as well as encouraging use of those canonical forms. That provision should not be interpreted as requiring that registrant need to provide characters in a particular code sequence. Registrant input conventions and management are part of registrant-registrar interactions and relationships between registries and registrars and are outside the scope of these standards.

It is worth stressing that these principles of policy development and application apply at all levels of the DNS, not only, e.g., TLD or SLD registrations and that even a trivial, "anything permitted that is valid under the protocol" policy is helpful in that it helps users and application developers know what to expect.

[edit] 3.3. Layered Restrictions: Tables, Context, Registration, Applications

The character rules in IDNA2008 are based on the realization that there is no single magic bullet for any of the security, confusability, or other issues associated with IDNs. Instead, the specifications define a variety of approaches. The character tables are the first mechanism, protocol rules about how those characters are applied or restricted in context are the second, and those two in combination constitute the limits of what can be done in the protocol. As discussed in the previous section (Section 3.2), registries are expected to restrict what they permit to be registered, devising and using rules that are designed to optimize the balance between confusion and risk on the one hand and maximum expressiveness in mnemonics on the other.

In addition, there is an important role for user agents in warning against label forms that appear problematic given their knowledge of local contexts and conventions. Of course, no approach based on naming or identifiers alone can protect against all threats.

[edit] 4. Issues that Constrain Possible Solutions

[edit] 4.1. Display and Network Order

Domain names are always transmitted in network order (the order in which the code points are sent in protocols), but may have a different display order (the order in which the code points are displayed on a screen or paper). When a domain name contains characters that are normally written right to left, display order may be affected although network order is not. It gets even more complicated if left to right and right to left labels are adjacent to each other within a domain name. The decision about the display order is ultimately under the control of user agents --including Web browsers, mail clients, hosted Web applications and many more -- which may be highly localized. Should a domain name abc.def, in which both labels are represented in scripts that are written right to left, be displayed as fed.cba or cba.fed? Applications that are in deployment today are already diverse, and one can find examples of either choice.

The picture changes once again when an IDN appears in a Internationalized Resource Identifier (IRI) [RFC3987]. An IRI or Internationalized Email address contains elements other than the domain name. For example, IRIs contain protocol identifiers and field delimiter syntax such as "http://" or "mailto:" while email addresses contain the "@" to separate local parts from domain names.

An IRI in network order begins with "http://" followed by domain labels in network order, thus "http://abc.def".

User agents are not required to display and allow input of IRIs directly but often do so. Implementors have to choose whether the overall direction of these strings will always be left to right (or right to left) for an IRI or email address. The natural order for a user typing a domain name on a right to left system is fed.cba.

Should the R2L user agent reverse the entire domain name each time a domain name is typed? Does this change if the user types "http://" right before typing a domain name, thus implying that the user is beginning at the beginning of the network order IRI? Experience in the 1980s and 1990s with mixing systems in which domain name labels were read in network order (left to right) and those in which those labels were read right to left would predict a great deal of confusion.

If each implementation of each application makes its own decisions on these issues, users will develop heuristics that will sometimes fail when switching applications. However, while some display order conventions, voluntarily adopted, would be desirable to reduce confusion, such suggestions are beyond the scope of these specifications.

[edit] 4.2. Entry and Display in Applications

Applications can accept and display domain names using any character set or character coding system. The IDNA protocol does not necessarily affect the interface between users and applications. An IDNA-aware application can accept and display internationalized domain names in two formats: the internationalized character set(s) supported by the application (i.e., an appropriate local representation of a U-label), and as an A-label. Applications may allow the display of A-labels, but are encouraged to not do so except as an interface for special purposes, possibly for debugging, or to cope with display limitations. In general, they should allow, but not encourage, user input of A-labels. A-labels are opaque, ugly, and malicious variations on them are not easily detected by users.

Where possible, they should thus only be exposed when they are absolutely needed. Because IDN labels can be rendered either as A-labels or U-labels, the application may reasonably have an option for the user to select the preferred method of display. Rendering the U-label should normally be the default.

Domain names are often stored and transported in many places. For example, they are part of documents such as mail messages and web pages. They are transported in many parts of many protocols, such as both the control commands of SMTP and associated message body parts, and in the headers and the body content in HTTP. It is important to remember that domain names appear both in domain name slots and in the content that is passed over protocols.

In protocols and document formats that define how to handle specification or negotiation of charsets, labels can be encoded in any charset allowed by the protocol or document format. If a protocol or document format only allows one charset, the labels must be given in that charset. Of course, not all charsets can properly represent all labels. If a U-label cannot be displayed in its entirety, the only choice (without loss of information) may be to display the A-label.

Where a protocol or document format allows IDNs, labels should be in whatever character encoding and escape mechanism the protocol or document format uses at that place. This provision is intended to prevent situations in which, e.g., UTF-8 domain names appear embedded in text that is otherwise in some other character coding.

All protocols that use domain name slots (See Section 2.3.1.6 in [IDNA2008-Defs]) already have the capacity for handling domain names in the ASCII charset. Thus, A-labels can inherently be handled by those protocols.

These documents do not specify required mappings between one character or code point and others. An extended discussion of mapping issues occurs in Section 6 and specific recommendations appear in [IDNA2008-Mapping]. In general, IDNA2008 prohibits characters that would be mapped to others by normalization or other rules. As examples, while mathematical characters based on Latin ones are accepted as input to IDNA2003, they are prohibited in IDNA2008. Similarly, upper-case characters, double-width characters, and other variations are prohibited as IDNA input although mapping them as needed in user interfaces is strongly encouraged.

Since the rules in [IDNA2008-Tables] have the effect that only strings that are not transformed by NFKC are valid, if an application chooses to perform NFKC normalization before lookup, that operation is safe since this will never make the application unable to look up any valid string. However, as discussed above, the application cannot guarantee that any other application will perform that mapping, so it should be used only with caution and for informed users.

In many cases these prohibitions should have no effect on what the user can type as input to the lookup process. It is perfectly reasonable for systems that support user interfaces to perform some character mapping that is appropriate to the local environment. This would normally be done prior to actual invocation of IDNA. At least conceptually, the mapping would be part of the Unicode conversions discussed above and in [IDNA2008-Protocol]. However, those changes will be local ones only -- local to environments in which users will clearly understand that the character forms are equivalent. For use in interchange among systems, it appears to be much more important that U-labels and A-labels can be mapped back and forth without loss of information.

One specific, and very important, instance of this strategy arises with case-folding. In the ASCII-only DNS, names are looked up and matched in a case-independent way, but no actual case-folding occurs.

Names can be placed in the DNS in either upper or lower case form (or any mixture of them) and that form is preserved, returned in queries, and so on. IDNA2003 approximated that behavior for non-ASCII strings by performing case-folding at registration time (resulting in only lower-case IDNs in the DNS) and when names were looked up.

As suggested earlier in this section, it appears to be desirable to do as little character mapping as possible as long as Unicode works correctly (e.g., NFC mapping to resolve different codings for the same character is still necessary although the specifications require that it be performed prior to invoking the protocol) in order to make the mapping between A-labels and U-labels idempotent. Case-mapping is not an exception to this principle. If only lower case characters can be registered in the DNS (i.e., be present in a U-label), then IDNA2008 should prohibit upper-case characters as input even though user interfaces to applications should probably map those characters.

Some other considerations reinforce this conclusion. For example, in ASCII case-mapping for individual characters, uppercase(character) must be equal to uppercase(lowercase(character)). That may not be true with IDNs. In some scripts that use case distinctions, there are a few characters that do not have counterparts in one case or the other. The relationship between upper case and lower case may even be language-dependent, with different languages (or even the same language in different areas) expecting different mappings. User agents can meet the expectations of users who are accustomed to the case-insensitive DNS environment by performing case folding prior to IDNA processing, but the IDNA procedures themselves should neither require such mapping nor expect them when they are not natural to the localized environment.

[edit] 4.3. Linguistic Expectations: Ligatures, Digraphs, and Alternate

Character Forms Users have expectations about character matching or equivalence that are based on their own languages and the orthography of those languages. These expectations may not always be met in a global system, especially if multiple languages are written using the same script but using different conventions. Some examples:

• A Norwegian user might expect a label with the ae-ligature to be treated as the same label as one using the Swedish spelling with a-diaeresis even though applying that mapping to English would be astonishing to users.

• A user in German might expect a label with an o-umlaut and a label that had "oe" substituted, but was otherwise the same, treated as equivalent even though that substitution would be a clear error in Swedish.
• A Chinese user might expect automatic matching of Simplified and Traditional Chinese characters, but applying that matching for Korean or Japanese text would create considerable confusion.
• An English user might expect "theater" and "theatre" to match.

A number of languages use alphabetic scripts in which single phonemes are written using two characters, termed a "digraph", for example, the "ph" in "pharmacy" and "telephone". (Such characters can also appear consecutively without forming a digraph, as in "tophat".) Certain digraphs may be indicated typographically by setting the two characters closer together than they would be if used consecutively to represent different phonemes. Some digraphs are fully joined as ligatures. For example, the word "encyclopaedia" is sometimes set with a U+00E6 LATIN SMALL LIGATURE AE. When ligature and digraph forms have the same interpretation across all languages that use a given script, application of Unicode normalization generally resolves the differences and causes them to match. When they have different interpretations, matching must utilize other methods, presumably chosen at the registry level, or users must be educated to understand that matching will not occur.

The nature of the problem can be illustrated by many words in the Norwegian language, where the "ae" ligature is the 27th letter of a 29-letter extended Latin alphabet. It is equivalent to the 28th letter of the Swedish alphabet (also containing 29 letters), U+00E4 LATIN SMALL LETTER A WITH DIAERESIS, for which an "ae" cannot be substituted according to current orthographic standards. That character (U+00E4) is also part of the German alphabet where, unlike in the Nordic languages, the two-character sequence "ae" is usually treated as a fully acceptable alternate orthography for the "umlauted a" character. The inverse is however not true, and those two characters cannot necessarily be combined into an "umlauted a". This also applies to another German character, the "umlauted o" (U+00F6 LATIN SMALL LETTER O WITH DIAERESIS) which, for example, cannot be used for writing the name of the author "Goethe". It is also a letter in the Swedish alphabet where, like the "a with diaeresis", it cannot be correctly represented as "oe" and in the Norwegian alphabet, where it is represented, not as "o with diaeresis", but as "slashed o", U+00F8.

Some of the ligatures that have explicit code points in Unicode were given special handling in IDNA2003 and now pose additional problems in transition. See Section 7.2.

Additional cases with alphabets written right to left are described in Section 4.5.

Matching and comparison algorithm selection often requires information about the language being used, context, or both -- information that is not available to IDNA or the DNS. Consequently, these specifications make no attempt to treat combined characters in any special way. A registry that is aware of the language context in which labels are to be registered, and where that language sometimes (or always) treats the two- character sequences as equivalent to the combined form, should give serious consideration to applying a "variant" model [RFC3743][RFC4290], or to prohibiting registration of one of the forms entirely, to reduce the opportunities for user confusion and fraud that would result from the related strings being registered to different parties.

[edit] 4.4. Case Mapping and Related Issues

In the DNS, ASCII letters are stored with their case preserved.

Matching during the query process is case-independent, but none of the information that might be represented by choices of case has been lost. That model has been accidentally helpful because, as people have created DNS labels by catenating words (or parts of words) to form labels, case has often been used to distinguish among components and make the labels more memorable.

Since DNS servers do not get involved in parsing IDNs, they cannot do case-independent matching. Thus, keeping the cases separate in lookup or registration, and doing matching at the server, is not feasible with IDNA or any similar approach. Case-matching must be done, if desired, by IDN clients even though it wasn't done by ASCII- only DNS clients. That situation was recognized in IDNA2003 and nothing in these specifications fundamentally changes it or could do so. In IDNA2003, all characters are case-folded and mapped by clients in a standardized step.

Some characters do not have upper case forms. For example the Unicode case folding operation maps Greek Final Form Sigma (U+03C2) to the medial form (U+03C3) and maps Eszett (German Sharp S, U+00DF) to "ss". Neither of these mappings is reversible because the upper case of U+03C3 is the Upper Case Sigma (U+03A3) and "ss" is an ASCII string. IDNA2008 permits, at the risk of some incompatibility, slightly more flexibility in this area by avoiding case folding and treating these characters as themselves. Approaches to handling one- way mappings are discussed in Section 7.2.

Because IDNA2003 maps Final Sigma and Eszett to other characters, and the reverse mapping is never possible, that in some sense means that neither Final Sigma nor Eszett can be represented in a IDNA2003 IDN.

With IDNA2008, both characters can be used in an IDN and so the A-label used for lookup for any U-label containing those characters, is now different. See Section 7.1 for a discussion of what kinds of changes might require the IDNA prefix to change; after extended discussions, the WG came to consensus that the change for these characters did not justify a prefix change.

[edit] 4.5. Right to Left Text

In order to be sure that the directionality of right to left text is unambiguous, IDNA2003 required that any label in which right to left characters appear both starts and ends with them and that it not include any characters with strong left to right properties (that excludes other alphabetic characters but permits European digits).

Any other string that contains a right to left character and does not meet those requirements is rejected. This is one of the few places where the IDNA algorithms (both in IDNA2003 and in IDAN2008) examine an entire label, not just individual characters. The algorithmic model used in IDNA2003 rejects the label when the final character in a right to left string requires a combining mark in order to be correctly represented.

That prohibition is not acceptable for writing systems for languages written with consonantal alphabets to which diacritical vocalic systems are applied, and for languages with orthographies derived from them where the combining marks may have different functionality.

In both cases the combining marks can be essential components of the orthography. Examples of this are Yiddish, written with an extended Hebrew script, and Dhivehi (the official language of Maldives) which is written in the Thaana script (which is, in turn, derived from the Arabic script). IDNA2008 removes the restriction on final combining characters with a new set of rules for right to left scripts and their characters. Those new rules are specified in [IDNA2008-Bidi].

[edit] 5. IDNs and the Robustness Principle

The "Robustness Principle" is often stated as "Be conservative about what you send and liberal in what you accept" (See, e.g., Section 1.2.2 of the applications-layer Host Requirements specification [RFC1123]) This principle applies to IDNA. In applying the principle to registries as the source ("sender") of all registered and useful IDNs, registries are responsible for being conservative about what they register and put out in the Internet. For IDNs to work well, zone administrators (registries) must have and require sensible policies about what is registered -- conservative policies -- and implement and enforce them.

Conversely, lookup applications are expected to reject labels that clearly violate global (protocol) rules (no one has ever seriously claimed that being liberal in what is accepted requires being stupid). However, once one gets past such global rules and deals with anything sensitive to script or locale, it is necessary to assume that garbage has not been placed into the DNS, i.e., one must be liberal about what one is willing to look up in the DNS rather than guessing about whether it should have been permitted to be registered.

If a string cannot be successfully found in the DNS after the lookup processing described here, it makes no difference whether it simply wasn't registered or was prohibited by some rule at the registry.

Application implementors should be aware that where DNS wildcards are used, the ability to successfully resolve a name does not guarantee that it was actually registered.

[edit] 6. Front-end and User Interface Processing for Lookup

Domain names may be identified and processed in many contexts. They may be typed in by users either by themselves or embedded in an identifier such as email addresses, URIs, or IRIs. They may occur in running text or be processed by one system after being provided in another. Systems may try to normalize URLs to determine (or guess) whether a reference is valid or two references point to the same object without actually looking the objects up (comparison without lookup is necessary for URI types that are not intended to be resolved). Some of these goals may be more easily and reliably satisfied than others. While there are strong arguments for any domain name that is placed "on the wire" -- transmitted between systems -- to be in the zero-ambiguity forms of A-labels, it is inevitable that programs that process domain names will encounter U-labels or variant forms.

An application that implements the IDNA protocol [IDNA2008-Protocol] will always take any user input and convert it to a set of Unicode code points. That user input may be acquired by any of several different input methods, all with differing conversion processes to be taken into consideration (e.g., typed on a keyboard, written by hand onto some sort of digitizer, spoken into a microphone and interpreted by a speech-to-text engine, etc.). The process of taking any particular user input and mapping it into a Unicode code point may be a simple one: If a user strikes the "A" key on a US English keyboard, without any modifiers such as the "Shift" key held down, in order to draw a Latin small letter A ("a"), many (perhaps most) modern operating system input methods will produce to the calling application the code point U+0061, encoded in a single octet.

Sometimes the process is somewhat more complicated: a user might strike a particular set of keys to represent a combining macron followed by striking the "A" key in order to draw a Latin small letter A with a macron above it. Depending on the operating system, the input method chosen by the user, and even the parameters with which the application communicates with the input method, the result might be the code point U+0101 (encoded as two octets in UTF-8 or UTF-16, four octets in UTF-32, etc.), the code point U+0061 followed by the code point U+0304 (again, encoded in three or more octets, depending upon the encoding used) or even the code point U+FF41 followed by the code point U+0304 (and encoded in some form). And these examples leave aside the issue of operating systems and input methods that do not use Unicode code points for their character set.

In every case, applications (with the help of the operating systems on which they run and the input methods used) need to perform a mapping from user input into Unicode code points.

The original version of the IDNA protocol [RFC3490] used a model whereby input was taken from the user, mapped (via whatever input method mechanisms were used) to a set of Unicode code points, and then further mapped to a set of Unicode code points using the Nameprep profile specified in [RFC3491]. In this procedure, there are two separate mapping steps: First, a mapping done by the input method (which might be controlled by the operating system, the application, or some combination) and then a second mapping performed by the Nameprep portion of the IDNA protocol. The mapping done in Nameprep includes a particular mapping table to re-map some characters to other characters, a particular normalization, and a set of prohibited characters.

Note that the result of the two step mapping process means that the mapping chosen by the operating system or application in the first step might differ significantly from the mapping supplied by the Nameprep profile in the second step. This has advantages and disadvantages. Of course, the second mapping regularizes what gets looked up in the DNS, making for better interoperability between implementations which use the Nameprep mapping. However, the application or operating system may choose mappings in their input methods, which when passed through the second (Nameprep) mapping result in characters that are "surprising" to the end user.

The other important feature of the original version of the IDNA protocol is that, with very few exceptions, it assumes that any set of Unicode code points provided to the Nameprep mapping can be mapped into a string of Unicode code points that are "sensible", even if that means mapping some code points to nothing (that is, removing the code points from the string). This allowed maximum flexibility in input strings.

The present version of IDNA differs significantly in approach from the original version. First and foremost, it does not provide explicit mapping instructions. Instead, it assumes that the application (perhaps via an operating system input method) will do whatever mapping it requires to convert input into Unicode code points. This has the advantage of giving flexibility to the application to choose a mapping that is suitable for its user given specific user requirements, and avoids the two-step mapping of the original protocol. Instead of a mapping, the current version of IDNA provides a set of categories that can be used to specify the valid code points allowed in a domain name.

In principle, an application ought to take user input of a domain name and convert it to the set of Unicode code points that represent the domain name the user intends. As a practical matter, of course, determining user intent is a tricky business, so an application needs to choose a reasonable mapping from user input. That may differ based on the particular circumstances of a user, depending on locale, language, type of input method, etc. It is up to the application to make a reasonable choice.

[edit] 7. Migration from IDNA2003 and Unicode Version Synchronization

[edit] 7.1. Design Criteria

As mentioned above and in RFC 4690, two key goals of the IDNA2008 design are

• to enable applications to be agnostic about whether they are being run in environments supporting any Unicode version from 3.2 onward,
• to permit incrementally adding new characters, character groups, scripts, and other character collections as they are incorporated into Unicode, doing so without disruption and, in the long term, without "heavy" processes (an IETF consensus process is required by the IDNA2008 specifications and is expected to be required and used until significant experience accumulates with IDNA operations and new versions of Unicode).

[edit] 7.1.1. Summary and Discussion of IDNA Validity Criteria

The general criteria for a label to be considered IDNA-valid are (the actual rules are rigorously defined in the "Protocol" and "Tables" documents):

• The characters are "letters", marks needed to form letters, numerals, or other code points used to write words in some language. Symbols, drawing characters, and various notational characters are intended to be permanently excluded. There is no evidence that they are important enough to Internet operations or internationalization to justify expansion of domain names beyond the general principle of "letters, digits, and hyphen". (Additional discussion and rationale for the symbol decision appears in Section 7.6).

• Other than in very exceptional cases, e.g., where they are needed to write substantially any word of a given language, punctuation characters are excluded. The fact that a word exists is not proof that it should be usable in a DNS label and DNS labels are not expected to be usable for multiple-word phrases (although they are certainly not prohibited if the conventions and orthography of a particular language cause that to be possible).
• Characters that are unassigned (have no character assignment at all) in the version of Unicode being used by the registry or application are not permitted, even on lookup. The issues involved in this decision are discussed in Section 7.7.
• Any character that is mapped to another character by a current version of NFKC is prohibited as input to IDNA (for either registration or lookup). With a few exceptions, this principle excludes any character mapped to another by Nameprep [RFC3491].

The principles above drive the design of rules that are specified exactly in [IDNA2008-Tables]. Those rules identify the characters that are IDNA-valid. The rules themselves are normative, and the tables are derived from them, rather than vice versa.

[edit] 7.1.2. Labels in Registration

Any label registered in a DNS zone must be validated -- i.e., the criteria for that label must be met -- in order for applications to work as intended. This principle is not new. For example, since the DNS was first deployed, zone administrators have been expected to verify that names meet "hostname" requirements [RFC0952] where those requirements are imposed by the expected applications. Other applications contexts, such as the later addition of special service location formats [RFC2782] imposed new requirements on zone administrators. For zones that will contain IDNs, support for Unicode version-independence requires restrictions on all strings placed in the zone. In particular, for such zones:

• Any label that appears to be an A-label, i.e., any label that starts in "xn--", must be IDNA-valid, i.e., they must be valid A-labels, as discussed in Section 2 above.
• The Unicode tables (i.e., tables of code points, character classes, and properties) and IDNA tables (i.e., tables of contextual rules such as those that appear in the Tables document), must be consistent on the systems performing or validating labels to be registered. Note that this does not require that tables reflect the latest version of Unicode, only that all tables used on a given system are consistent with each other.

Under this model, registry tables will need to be updated (both the Unicode-associated tables and the tables of permitted IDN characters) to enable a new script or other set of new characters. The registry will not be affected by newer versions of Unicode, or newly- authorized characters, until and unless it wishes to support them.

The zone administrator is responsible for verifying IDNA-validity as well as its local policies -- a more extensive set of checks than are required for looking up the labels. Systems looking up or resolving DNS labels, especially IDN DNS labels, must be able to assume that applicable registration rules were followed for names entered into the DNS.

[edit] 7.1.3. Labels in Lookup

Anyone looking up a label in a DNS zone is required to

• Maintain IDNA and Unicode tables that are consistent with regard to versions, i.e., unless the application actually executes the classification rules in [IDNA2008-Tables], its IDNA tables must be derived from the version of Unicode that is supported more generally on the system. As with registration, the tables need not reflect the latest version of Unicode but they must be consistent.
• Validate the characters in labels to be looked up only to the extent of determining that the U-label does not contain "DISALLOWED" code points or code points that are unassigned in its version of Unicode.

• Validate the label itself for conformance with a small number of whole-label rules. In particular, it must verify that * there are no leading combining marks, * the "bidi" conditions are met if right to left characters appear, * any required contextual rules are available, and * any contextual rules that are associated with Joiner Controls (and "CONTEXTJ" characters more generally) are tested.
• Do not reject labels based on other contextual rules about characters, including mixed-script label prohibitions. Such rules may be used to influence presentation decisions in the user interface, but not to avoid looking up domain names.

Lookup applications that following these rules, rather than having their own criteria for rejecting lookup attempts, are not sensitive to version incompatibilities with the particular zone registry associated with the domain name except for labels containing characters recently added to Unicode.

An application or client that processes names according to this protocol and then resolves them in the DNS will be able to locate any name that is registered, as long as those registrations are IDNA- valid and its version of the IDNA tables is sufficiently up-to-date to interpret all of the characters in the label. Messages to users should distinguish between "label contains an unallocated code point" and other types of lookup failures. A failure on the basis of an old version of Unicode may lead the user to a desire to upgrade to a newer version, but will have no other ill effects (this is consistent with behavior in the transition to the DNS when some hosts could not yet handle some forms of names or record types).

[edit] 7.2. Changes in Character Interpretations

In those scripts that make case distinctions, there are a few characters for which an obvious and unique upper case character has not historically been available to match a lower case one or vice versa. For those characters, the mappings used in constructing the Stringprep tables for IDNA2003, performed using the Unicode CaseFold operation (See Section 5.8 of the Unicode Standard [Unicode51]), generate different characters or sets of characters. Those operations are not reversible and lose even more information than traditional upper case or lower case transformations, but are more useful than those transformations for comparison purposes. Two notable characters of this type are the German character Eszett (Sharp S, U+00DF) and the Greek Final Form Sigma (U+03C2). The former is case-folded to the ASCII string "ss", the latter to a medial (Lower Case) Sigma (U+03C3).

The decision to eliminate mandatory and standardized mappings, including case folding, from the IDNA2008 protocol in order to make A-labels and U-labels idempotent made these characters problematic.

If they were to be disallowed, important words and mnemonics could not be written in orthographically reasonable ways. If they were to be permitted as distinct characters, there would be no information loss and registries would have more flexibility, but IDNA2003 and IDNA2008 lookups might result in different A-labels.

With the understanding that there would be incompatibility either way but a judgment that the incompatibility was not significant enough to justify a prefix change, the WG concluded that Eszett and Final Form Sigma should be treated as distinct and Protocol-Valid characters.

Registries, especially those maintaining zones for third parties, must decide how to introduce a new service in a way that does not create confusion or significantly weaken or invalidate existing identifiers. This is not a new problem; registries were faced with similar issues when IDNs were introduced and when other new forms of strings have been permitted as labels.

There are several approaches to problems of this type. Without any preference or claim to completeness, some of these, all of which have been used by registries in the past for similar transitions, are:

• Do not permit use of the newly-available character at the registry level. This might cause lookup failures if a domain name were to be written with the expectation of the IDNA2003 mapping behavior, but would eliminate any possibility of false matches.
• Hold a "sunrise"-like arrangement in which holders of labels containing "ss" in the Eszett case or Lower Case Sigma are given priority (and perhaps other benefits) for registering the corresponding string containing Eszett or Final Sigma respectively.
• Adopt some sort of "variant" approach in which registrants obtain labels with both character forms.
• Adopt a different form of "variant" approach in which registration of additional names is either not permitted at all or permitted only by the registrant who already has one of the names.

[edit] 7.3. Character Mapping

As discussed at length in Section 6, IDNA2003, via Nameprep (see Section 7.5), mapped many characters into related ones. Those mappings no longer exist as requirements in IDNA2008. These specifications strongly prefer that only A-labels or U-labels be used in protocol contexts and as much as practical more generally.

IDNA2008 does anticipate situations in which some mapping at the time of user input into lookup applications is appropriate and desirable.

The issues are discussed in Section 6 and specific recommendations are made in [IDNA2008-Mapping].

[edit] 7.4. The Question of Prefix Changes

The conditions that would require a change in the IDNA ACE prefix ("xn--" for the version of IDNA specified in [RFC3490]) have been a great concern to the community. A prefix change would clearly be necessary if the algorithms were modified in a manner that would create serious ambiguities during subsequent transition in registrations. This section summarizes our conclusions about the conditions under which changes in prefix would be necessary and the implications of such a change.

[edit] 7.4.1. Conditions Requiring a Prefix Change

An IDN prefix change is needed if a given string would be looked up or otherwise interpreted differently depending on the version of the protocol or tables being used. An IDNA upgrade would require a prefix change if, and only if, one of the following four conditions were met:

1. The conversion of an A-label to Unicode (i.e., a U-label) yields one string under IDNA2003 (RFC3490) and a different string under IDNA2008.

2. In a significant number of cases, an input string that is valid under IDNA2003 and also valid under IDNA2008 yields two different A-labels with the different versions. This condition is believed to be essentially equivalent to the one above except for a very small number of edge cases which may not justify a prefix change (See Section 7.2).

Note that if the input string is valid under one version and not valid under the other, this condition does not apply. See the first item in Section 7.4.2, below.

3. A fundamental change is made to the semantics of the string that is inserted in the DNS, e.g., if a decision were made to try to include language or script information in the encoding in addition to the string itself.

4. A sufficiently large number of characters is added to Unicode so that the Punycode mechanism for block offsets can no longer reference the higher-numbered planes and blocks. This condition is unlikely even in the long term and certain not to arise in the next several years.

[edit] 7.4.2. Conditions Not Requiring a Prefix Change

As a result of the principles described above, none of the following changes require a new prefix:

1. Prohibition of some characters as input to IDNA. This may make names that are now registered inaccessible, but does not change those names.

2. Adjustments in IDNA tables or actions, including normalization definitions, that affect characters that were already invalid under IDNA2003.

3. Changes in the style of the IDNA definition that does not alter the actions performed by IDNA.

[edit] 7.4.3. Implications of Prefix Changes

While it might be possible to make a prefix change, the costs of such a change are considerable. Registries could not convert all IDNA2003 ("xn--") registrations to a new form at the same time and synchronize that change with applications supporting lookup. Unless all existing registrations were simply to be declared invalid (and perhaps even then) systems that needed to support both labels with old prefixes and labels with new ones would first process a putative label under the IDNA2008 rules and try to look it up and then, if it were not found, would process the label under IDNA2003 rules and look it up again. That process could significantly slow down all processing that involved IDNs in the DNS especially since a fully-qualified name might contain a mixture of labels that were registered with the old and new prefixes. That would make DNS caching very difficult. In addition, looking up the same input string as two separate A-labels creates some potential for confusion and attacks, since the labels could map to different targets and then resolve to different entries in the DNS.

Consequently, a prefix change is to be avoided if at all possible, even if it means accepting some IDNA2003 decisions about character distinctions as irreversible and/or giving special treatment to edge cases.

[edit] 7.5. Stringprep Changes and Compatibility

The Nameprep [RFC3491] specification, a key part of IDNA2003, is a profile of Stringprep [RFC3454]. While Nameprep is a Stringprep profile specific to IDNA, Stringprep is used by a number of other protocols. Were Stringprep to be modified by IDNA2008, those changes to improve the handling of IDNs could cause problems for non-DNS uses, most notably if they affected identification and authentication protocols. Several elements of IDNA2008 give interpretations to strings prohibited under IDNA2003 or prohibit strings that IDNA2003 permitted. Those elements include the proposed new inclusion tables [IDNA2008-Tables], the reduction in the number of characters permitted as input for registration or lookup (Section 3), and even the proposed changes in handling of right to left strings [IDNA2008-Bidi]. IDNA2008 does not use Nameprep or Stringprep at all, so there are no side-effect changes to other protocols.

It is particularly important to keep IDNA processing separate from processing for various security protocols because some of the constraints that are necessary for smooth and comprehensible use of IDNs may be unwanted or undesirable in other contexts. For example, the criteria for good passwords or passphrases are very different from those for desirable IDNs: passwords should be hard to guess, while domain names should normally be easily memorable. Similarly, internationalized SCSI identifiers and other protocol components are likely to have different requirements than IDNs.

[edit] 7.6. The Symbol Question

One of the major differences between this specification and the original version of IDNA is that the original version permitted non- letter symbols of various sorts, including punctuation and line- drawing symbols, in the protocol. They were always discouraged in practice. In particular, both the "IESG Statement" about IDNA and all versions of the ICANN Guidelines specify that only language characters be used in labels. This specification disallows symbols entirely. There are several reasons for this, which include:

1. As discussed elsewhere, the original IDNA specification assumed that as many Unicode characters as possible should be permitted, directly or via mapping to other characters, in IDNs. This specification operates on an inclusion model, extrapolating from the original "hostname" rules (LDH, see [IDNA2008-Defs]) -- which have served the Internet very well -- to a Unicode base rather than an ASCII base.

2. Symbol names are more problematic than letters because there may be no general agreement on whether a particular glyph matches a symbol; there are no uniform conventions for naming; variations such as outline, solid, and shaded forms may or may not exist; and so on. As just one example, consider a "heart" symbol as it might appear in a logo that might be read as "I love...". While the user might read such a logo as "I love..." or "I heart...", considerable knowledge of the coding distinctions made in Unicode is needed to know that there more than one "heart" character (e.g., U+2665, U+2661, and U+2765) and how to describe it. These issues are of particular importance if strings are expected to be understood or transcribed by the listener after being read out loud.

3. Design of a screen reader used by blind Internet users who must listen to renderings of IDN domain names and possibly reproduce them on the keyboard becomes considerably more complicated when the names of characters are not obvious and intuitive to anyone familiar with the language in question.

4. As a simplified example of this, assume one wanted to use a "heart" or "star" symbol in a label. This is problematic because those names are ambiguous in the Unicode system of naming (the actual Unicode names require far more qualification). A user or would-be registrant has no way to know -- absent careful study of the code tables -- whether it is ambiguous (e.g., where there are multiple "heart" characters) or not. Conversely, the user seeing the hypothetical label doesn't know whether to read it -- try to transmit it to a colleague by voice -- as "heart", as "love", as "black heart", or as any of the other examples below.

5. The actual situation is even worse than this. There is no possible way for a normal, casual, user to tell the difference between the hearts of U+2665 and U+2765 and the stars of U+2606 and U+2729 or the without somehow knowing to look for a distinction. We have a white heart (U+2661) and few black hearts. Consequently, describing a label as containing a heart hopelessly ambiguous: we can only know that it contains one of several characters that look like hearts or have "heart" in their names. In cities where "Square" is a popular part of a location name, one might well want to use a square symbol in a label as well and there are far more squares of various flavors in Unicode than there are hearts or stars.

The consequence of these ambiguities is that symbols are a very poor basis for reliable communication. Consistent with this conclusion, the Unicode standard recommends that strings used in identifiers not contain symbols or punctuation [Unicode-UAX31]. Of course, these difficulties with symbols do not arise with actual pictographic languages and scripts which would be treated like any other language characters; the two should not be confused.

[edit] 7.7. Migration Between Unicode Versions: Unassigned Code Points

In IDNA2003, labels containing unassigned code points are looked up on the assumption that, if they appear in labels and can be mapped and then resolved, the relevant standards must have changed and the registry has properly allocated only assigned values.

In the protocol described in these documents, strings containing unassigned code points must not be either looked up or registered.

In summary, the status of an unassigned character with regard to the DISALLOWED, PROTOCOL-VALID, and CONTEXTUAL RULE REQUIRED categories cannot be evaluated until a character is actually assigned and known.

There are several reasons for this, with the most important ones being:

• Tests involving the context of characters (e.g., some characters being permitted only adjacent to others of specific types) and integrity tests on complete labels are needed. Unassigned code points cannot be permitted because one cannot determine whether particular code points will require contextual rules (and what those rules should be) before characters are assigned to them and the properties of those characters fully understood.
• It cannot be known in advance, and with sufficient reliability, whether a newly-assigned code point will be associated with a character that would be disallowed by the rules in [IDNA2008-Tables] (such as a compatibility character). In IDNA2003, since there is no direct dependency on NFKC (many of the entries in Stringprep's tables are based on NFKC, but IDNA2003 depends only on Stringprep), allocation of a compatibility character might produce some odd situations, but it would not be a problem. In IDNA2008, where compatibility characters are DISALLOWED unless character-specific exceptions are made, permitting strings containing unassigned characters to be looked up would violate the principle that characters in DISALLOWED are not looked up.
• The Unicode Standard specifies that an unassigned code point normalizes (and, where relevant, case folds) to itself. If the code point is later assigned to a character, and particularly if the newly-assigned code point has a combining class that determines its placement relative to other combining characters, it could normalize to some other code point or sequence.

It is possible to argue that the issues above are not important and that, as a consequence, it is better to retain the principle of looking up labels even if they contain unassigned characters because all of the important scripts and characters have been coded as of Unicode 5.1 and hence unassigned code points will be assigned only to obscure characters or archaic scripts. Unfortunately, that does not appear to be a safe assumption for at least two reasons. First, much the same claim of completeness has been made for earlier versions of Unicode. The reality is that a script that is obscure to much of the world may still be very important to those who use it. Cultural and linguistic preservation principles make it inappropriate to declare the script of no importance in IDNs. Second, we already have counterexamples in, e.g., the relationships associated with new Han characters being added (whether in the BMP or in Unicode Plane 2).

Independent of the technical transition issues identified above, it can be observed that any addition of characters to an existing script to make it easier to use or to better accommodate particular languages may lead to transition issues. Such changes may change the preferred form for writing a particular string, changes that may be reflected, e.g., in keyboard transition modules that would necessarily be different from those for earlier versions of Unicode where the newer characters may not exist. This creates an inherent transition problem because attempts to access labels may use either the old or the new conventions, requiring registry action whether the older conventions were used in labels or not. The need to consider transition mechanisms is inherent to evolution of Unicode to better accommodate writing systems and is independent of how IDNs are represented in the DNS or how transitions among versions of those mechanisms occur. The requirement for transitions of this type is illustrated by the addition of Malayalam Chillu in Unicode 5.1.0.

[edit] 7.8. Other Compatibility Issues

The 2003 IDNA model includes several odd artifacts of the context in which it was developed. Many, if not all, of these are potential avenues for exploits, especially if the registration process permits "source" names (names that have not been processed through IDNA and Nameprep) to be registered. As one example, since the character Eszett, used in German, is mapped by IDNA2003 into the sequence "ss" rather than being retained as itself or prohibited, a string containing that character but that is otherwise in ASCII is not really an IDN (in the U-label sense defined above) at all. After Nameprep maps the Eszett out, the result is an ASCII string and so does not get an xn-- prefix, but the string that can be displayed to a user appears to be an IDN. The newer version of the protocol eliminates this artifact. A character is either permitted as itself or it is prohibited; special cases that make sense only in a particular linguistic or cultural context can be dealt with as localization matters where appropriate.

[edit] 8. Name Server Considerations

[edit] 8.1. Processing Non-ASCII Strings

Existing DNS servers do not know the IDNA rules for handling non- ASCII forms of IDNs, and therefore need to be shielded from them.

All existing channels through which names can enter a DNS server database (for example, master files (as described in RFC 1034) and DNS update messages [RFC2136]) are IDN-unaware because they predate IDNA. Other sections of this document provide the needed shielding by ensuring that internationalized domain names entering DNS server databases through such channels have already been converted to their equivalent ASCII A-label forms.

Because of the distinction made between the algorithms for Registration and Lookup in [IDNA2008-Protocol] (a domain name containing only ASCII codepoints can not be converted to an A-label), there can not be more than one A-label form for any given U-label.

As specified in RFC 2181 [RFC2181], the DNS protocol explicitly allows domain labels to contain octets beyond the ASCII range (0000..007F), and this document does not change that. However, although the interpretation of octets 0080..00FF is well-defined in the DNS, many application protocols support only ASCII labels and there is no defined interpretation of these non-ASCII octets as characters and, in particular, no interpretation of case-independent matching for them (see, e.g., [RFC4343]). If labels containing these octets are returned to applications, unpredictable behavior could result. The A-label form, which cannot contain those characters, is the only standard representation for internationalized labels in the DNS protocol.

[edit] 8.2. DNSSEC Authentication of IDN Domain Names

DNS Security (DNSSEC) [RFC2535] is a method for supplying cryptographic verification information along with DNS messages.

Public Key Cryptography is used in conjunction with digital signatures to provide a means for a requester of domain information to authenticate the source of the data. This ensures that it can be traced back to a trusted source, either directly or via a chain of trust linking the source of the information to the top of the DNS hierarchy.

IDNA specifies that all internationalized domain names served by DNS servers that cannot be represented directly in ASCII MUST use the A-label form. Conversion to A-labels MUST be performed prior to a zone being signed by the private key for that zone. Because of this ordering, it is important to recognize that DNSSEC authenticates a domain name containing A-labels or conventional LDH-labels, not U-labels. In the presence of DNSSEC, no form of a zone file or query response that contains a U-label may be signed or the signature validated.

One consequence of this for sites deploying IDNA in the presence of DNSSEC is that any special purpose proxies or forwarders used to transform user input into IDNs must be earlier in the lookup flow than DNSSEC authenticating nameservers for DNSSEC to work.

[edit] 8.3. Root and other DNS Server Considerations

IDNs in A-label form will generally be somewhat longer than current domain names, so the bandwidth needed by the root servers is likely to go up by a small amount. Also, queries and responses for IDNs will probably be somewhat longer than typical queries historically, so EDNS0 [RFC2671] support may be more important (otherwise, queries and responses may be forced to go to TCP instead of UDP).

[edit] 9. Internationalization Considerations

DNS labels and fully-qualified domain names provide mnemonics that assist in identifying and referring to resources on the Internet.

IDNs expand the range of those mnemonics to include those based on languages and character sets other than Western European and Roman- derived ones. But domain "names" are not, in general, words in any language. The recommendations of the IETF policy on character sets and languages, (BCP 18 [RFC2277]) are applicable to situations in which language identification is used to provide language-specific contexts. The DNS is, by contrast, global and international and ultimately has nothing to do with languages. Adding languages (or similar context) to IDNs generally, or to DNS matching in particular, would imply context dependent matching in DNS, which would be a very significant change to the DNS protocol itself. It would also imply that users would need to identify the language associated with a particular label in order to look that label up. That knowledge is generally not available because many labels are not words in any language and some may be words in more than one.

[edit] 10. IANA Considerations

This section gives an overview of IANA registries required for IDNA.

The actual definitions of, and specifications for, the first two, which must be newly-created for IDNA2008, appear in [IDNA2008-Tables]. This document describes the registries but does not specify any IANA actions.

[edit] 10.1. IDNA Character Registry

The distinction among the major categories "UNASSIGNED", "DISALLOWED", "PROTOCOL-VALID", and "CONTEXTUAL RULE REQUIRED" is made by special categories and rules that are integral elements of [IDNA2008-Tables]. While not normative, an IANA registry of characters and scripts and their categories, updated for each new version of Unicode and the characters it contains, will be convenient for programming and validation purposes. The details of this registry are specified in [IDNA2008-Tables].

[edit] 10.2. IDNA Context Registry

IANA will create and maintain a list of approved contextual rules for characters that are defined in the IDNA Character Registry list as requiring a Contextual Rule (i.e., the types of rule described in Section 3.1.2). The details for those rules appear in [IDNA2008-Tables].

[edit] 10.3. IANA Repository of IDN Practices of TLDs

This registry, historically described as the "IANA Language Character Set Registry" or "IANA Script Registry" (both somewhat misleading terms) is maintained by IANA at the request of ICANN. It is used to provide a central documentation repository of the IDN policies used by top level domain (TLD) registries who volunteer to contribute to it and is used in conjunction with ICANN Guidelines for IDN use.

It is not an IETF-managed registry and, while the protocol changes specified here may call for some revisions to the tables, these specifications have no direct effect on that registry and no IANA action is required as a result.

[edit] 11. Security Considerations

[edit] 11.1. General Security Issues with IDNA

This document is purely explanatory and informational and consequently introduces no new security issues. It would, of course, be a poor idea for someone to try to implement from it; such an attempt would almost certainly lead to interoperability problems and might lead to security ones. A discussion of security issues with IDNA, including some relevant history, appears in [IDNA2008-Defs].

[edit]
MAPPING

In the original version of the Internationalized Domain Names in Applications (IDNA) protocol, any Unicode code points taken from user input were mapped into a set of Unicode code points that "make sense", which were then encoded and passed to the domain name system (DNS). The current version of IDNA presumes that the input to the protocol comes from a set of "permitted" code points, which it then encodes and passes to the DNS, but does not specify what to do with the result of user input. This document describes the actions taken by an implementation between user input and passing permitted code points to the new IDNA protocol.

[edit] 1. Introduction

This document describes the operations that can be applied to user input in order to get it into a form acceptable by the Internationalized Domain Names in Applications (IDNA) protocol [I-D.ietf-idnabis-protocol]. The document describes a general implementation procedure for mapping in section 2.

It should be noted that this document does not specify the behavior of a protocol that appears "on the wire". It describes an operation that is to be applied to user input in order to prepare that user input for use in an "on the network" protocol. As unusual as this may be for an IETF protocol document, it is a necessary operation to maintain interoperability.

[edit] 2. The General Procedure

This section defines a general algorithm that applications ought to implement in order to produce Unicode code points that will be valid under the IDNA protocol. An application might implement the full mapping as described below, or can choose a different mapping. In fact, an application might want to implement a full mapping that is substantially compatible with the original IDNA protocol instead of the algorithm given here.

The general algorithm that an application (or the input method provided by an operating system) ought to use is relatively straightforward:

1. Upper case characters are mapped to their lower case equivalents by using the algorithm for mapping Unicode characters.

2. Full-width and half-width characters (those defined with Decomposition Types <wide> and <narrow>) are mapped to their decomposition mappings as shown in the Unicode character database.

3. All characters are mapped using Unicode Normalization Form C (NFC).

4. [I-D.ietf-idnabis-protocol] is specified such that the protocol acts on the indvidual labels of the domain name. If an implementation of this mapping is also performing the step of separation of the parts of a domain name into labels by using the FULL STOP character (U+002E), the following character can be mapped to the FULL STOP before label separation occurs:

• IDEOGRAPHIC FULL STOP (U+3002)

There are other characters that are used as "full stops" that one could consider mapping as label separators, but their use as such has not been investigated thoroughly.

Definitions for the rules in this algorithm can be found in [Unicode51]. Specifically:

• Unicode Normalization Form C can be found in Annex #15 of [Unicode51].
• In order to map upper case characters to their lower case equivalents (defined in section 3.13 of [Unicode51]), first map characters to the "Lowercase_Mapping" property (the "<lower>" entry in the second column) in <http://www.unicode.org/Public/UNIDATA/SpecialCasing.txt>, if any.

Then, map characters to the "Simple_Lowercase_Mapping" property (the fourteenth column) in <http://www.unicode.org/Public/UNIDATA/UnicodeData.txt>, if any.

• In order to map full-width and half-width characters to their decomposition mappings, map any character whose "Decomposition_Type" (contained in the first part of of the sixth column) in <http://www.unicode.org/Public/UNIDATA/UnicodeData.txt> is either "<wide>" or "<narrow>" to the "Decomposition_Mapping" of that character (contained in the second part of the sixth column) in <http://www.unicode.org/Public/UNIDATA/UnicodeData.txt>.
• The <http://www.unicode.org/Public/UNIDATA/UCD.html> web page has useful descriptions of the contents of these files.

If this mappings in this document are applied to versions of Unicode later than Unicode 5.1, the later versions of the Unicode Standard should be consulted.

These are a minimal set of mappings that an application should strongly consider doing. Of course, there are many others that might be done.

[edit] 3. IANA Considerations

This memo includes no request to IANA.

[edit] 4. Security Considerations

This document suggests creating mappings that might cause confusion for some users while alleviating confusion in other users. Such confusion is not covered in any depth in this document (nor in the other IDNA-related documents).

[edit]
PROTOCOL

This document is the revised protocol definition for internationalized domain names (IDNs). The rationale for changes, the relationship to the older specification, and important terminology are provided in other documents. This document specifies the protocol mechanism, called Internationalizing Domain Names in Applications (IDNA), for registering and looking up IDNs in a way that does not require changes to the DNS itself. IDNA is only meant for processing domain names, not free text.

[edit] 1. Introduction

This document supplies the protocol definition for internationalized domain names. Essential definitions and terminology for understanding this document and a road map of the collection of documents that make up IDNA2008 appear in [IDNA2008-Defs].

Appendix A discusses the relationship between this specification and the earlier version of IDNA (referred to here as "IDNA2003") and the rationale for these changes, along with considerable explanatory material and advice to zone administrators who support IDNs is provided in another document, [IDNA2008-Rationale].

IDNA works by allowing applications to use certain ASCII string labels (beginning with a special prefix) to represent non-ASCII name labels. Lower-layer protocols need not be aware of this; therefore IDNA does not changes any infrastructure. In particular, IDNA does not depend on any changes to DNS servers, resolvers, or protocol elements, because the ASCII name service provided by the existing DNS can be used for IDNA.

IDNA applies only to DNS labels. The base DNS standards [RFC1034] [RFC1035] and their various updates specify how to combine labels into fully-qualified domain names and parse labels out of those names.

This document describes two separate protocols, one for IDN registration (Section 4) and one for IDN lookup (Section 5), that share some terminology, reference data and operations. [[anchor2:

Note in draft: See the note in the introduction to.]]Section 5

[edit] 1.1. Discussion Forum

anchor4: RFC Editor: please remove this section. This work is being discussed in the IETF IDNABIS WG and on the mailing list idna-update@alvestrand.no

[edit] 2. Terminology

Terminology used in IDNA, but also in Unicode or other character set standards and the DNS, appears in [IDNA2008-Defs]. Terminology that is required as part of the IDNA definition, including the definitions of "ACE", appears in that document as well. Readers of this document are assumed to be familiar with [IDNA2008-Defs] and with the DNS- specific terminology in RFC 1034 [RFC1034].

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [RFC2119].

[edit] 3. Requirements and Applicability

[edit] 3.1. Requirements

IDNA makes the following requirements:

1. Whenever a domain name is put into an IDN-unaware domain name slot (see Section 2 and [IDNA2008-Defs]), it MUST contain only ASCII characters (i.e., must be either an A-label or an NR-LDH- label), unless the DNS application is not subject to historical recommendations for "hostname"-style names (see [RFC1034] and Section 3.2.1).

2. Labels MUST be compared using equivalent forms: either both A-Label forms or both U-Label forms. Because A-labels and U-labels can be transformed into each other without loss of information, these comparisons are equivalent. A pair of A-labels MUST be compared as case-insensitive ASCII (as with all comparisons of ASCII DNS labels). U-labels must be compared as-is, without case-folding or other intermediate steps. Note that it is not necessary to validate labels in order to compare them. In many cases, validation may be important for other reasons and SHOULD be performed.

3. Labels being registered MUST conform to the requirements of Section 4. Labels being looked up and the lookup process MUST conform to the requirements of Section 5.

[edit] 3.2. Applicability

IDNA applies to all domain names in all domain name slots in protocols except where it is explicitly excluded. It does not apply to domain name slots which do not use the Letter/Digit/Hyphen (LDH) syntax rules.

Because it uses the DNS, IDNA applies to many protocols that were specified before it was designed. IDNs occupying domain name slots in those older protocols MUST be in A-label form until and unless those protocols and implementations of them are explicitly upgraded to be aware of IDNs in Unicode. IDNs actually appearing in DNS queries or responses MUST be A-labels.

IDNA is not defined for extended label types (see RFC 2671, Section 3 [RFC2671]).

[edit] 3.2.1. DNS Resource Records

IDNA applies only to domain names in the NAME and RDATA fields of DNS resource records whose CLASS is IN. See RFC 1034 [RFC1034] for precise definitions of these terms.

The application of IDNA to DNS resource records depends entirely on the CLASS of the record, and not on the TYPE except as noted below.

This will remain true, even as new types are defined, unless a new type defines type-specific rules. Special naming conventions for SRV records (and "underscore names" more generally) are incompatible with IDNA coding. The first two labels on a SRV type record (the ones required to start in "_") MUST NOT be A-labels or U-labels, because conversion to an A-label would lose information (since the underscore is not a letter, digit, or hyphen and is consequently DISALLOWED in IDNs). Of course, those labels may be part of a domain that uses IDN labels at higher levels in the tree.

[edit] 3.2.2. Non-domain-name Data Types Stored in the DNS

Although IDNA enables the representation of non-ASCII characters in domain names, that does not imply that IDNA enables the representation of non-ASCII characters in other data types that are stored in domain names, specifically in the RDATA field for types that have structured RDATA format. For example, an email address local part is stored in a domain name in the RNAME field as part of the RDATA of an SOA record (hostmaster@example.com would be represented as hostmaster.example.com). IDNA does not update the existing email standards, which allow only ASCII characters in local parts. Even though work is in progress to define internationalization for email addresses [RFC4952], changes to the email address part of the SOA RDATA would require action in, or updates to, other standards, specifically those that specify the format of the SOA RR.

[edit] 4. Registration Protocol

This section defines the procedure for registering an IDN. The procedure is implementation independent; any sequence of steps that produces exactly the same result for all labels is considered a valid implementation.

Note that, while the registration and lookup protocols (Section 5) are very similar in most respects, they are different and implementers should carefully follow the appropriate steps.

[edit] 4.1. Input to IDNA Registration Process

Registration processes, especially processing by entities, such as "registrars" who deal with registrants before the request actually reaches the zone manager ("registry") are outside the scope of these protocols and may differ significantly depending on local needs. By the time a string enters the IDNA registration process as described in this specification, it is expected to be in Unicode and MUST be in Unicode Normalization Form C (NFC [Unicode-UAX15]). Entities responsible for zone files ("registries") are expected to accept only the exact string for which registration is requested, free of any mappings or local adjustments. They SHOULD avoid any possible ambiguity by accepting registrations only for A-labels, possibly paired with the relevant U-labels so that they can verify the correspondence.

[edit] 4.2. Permitted Character and Label Validation

[edit] 4.2.1. Input Format

The registry SHOULD permit submission of labels in A-label form and is encouraged to accept both the A-label form and the U-label one.

If it does so, it MUST perform a conversion to a U-label, perform the steps and tests described below, and verify that the A-label produced by the step in Section 4.4 matches the one provided as input. In addition, if a U-label was provided, that U-label and the one obtained by conversion of the A-label MUST match exactly. If, for some reason, these tests fail, the registration MUST be rejected. If the conversion to a U-label is not performed, the registry MUST still verify that the A-label is superficially valid, i.e., that it does not violate any of the rules of Punycode [RFC3492] encoding such as the prohibition on trailing hyphen-minus, appearance of non-basic characters before the delimiter, and so on. Fake A-labels, i.e., invalid strings that appear to be A-labels but are not, MUST NOT be placed in DNS zones that support IDNA.

[edit] 4.2.2. Rejection of Characters that are not Permitted

The candidate Unicode string MUST NOT contain characters in the "DISALLOWED" and "UNASSIGNED" lists specified in [IDNA2008-Tables].

[edit] 4.2.3. Label Validation

The proposed label (in the form of a Unicode string, i.e., a string that at least superficially appears to be a U-label) is then examined, performing tests that require examination of more than one character. Character order is considered to be the on-the-wire order, not the display order.

[edit] 4.2.3.1. Consecutive Hyphens

The Unicode string MUST NOT contain "--" (two consecutive hyphens) in the third and fourth character positions.

[edit] 4.2.3.2. Leading Combining Marks

The Unicode string MUST NOT begin with a combining mark or combining character (see The Unicode Standard, Section 2.11 [Unicode] for an exact definition).

[edit] 4.2.3.3. Contextual Rules

The Unicode string MUST NOT contain any characters whose validity is context-dependent, unless the validity is positively confirmed by a contextual rule. To check this, each code-point marked as CONTEXTJ and CONTEXTO in [IDNA2008-Tables] MUST have a non-null rule. If such a code-point is missing a rule, it is invalid. If the rule exists but the result of applying the rule is negative or inconclusive, the proposed label is invalid.

[edit] 4.2.3.4. Labels Containing Characters Written Right to Left

If the proposed label contains any characters that are written from right to left it MUST meet the "bidi" criteria [IDNA2008-BIDI].

[edit] 4.2.4. Registration Validation Summary

Strings that contain at least one non-ASCII character, have been produced by the steps above, whose contents pass all of the tests in Section 4.2, and are 63 or fewer characters long in ACE form (see Section 4.4), are U-labels.

To summarize, tests are made in Section 4.2 for invalid characters, invalid combinations of characters, for labels that are invalid even if the characters they contain are valid individually, and for labels that do not conform to the restrictions for strings containing right to left characters.

[edit] 4.3. Registry Restrictions

In addition to the rules and tests above, there are many reasons why a registry could reject a label. Registries at all levels of the DNS, not just the top level, establish policies about label registrations. Policies are likely to be informed by the local languages and may depend on many factors including what characters are in the label (for example, a label may be rejected based on other labels already registered). See [IDNA2008-Rationale] for a discussion and recommendations about registry policies.

The string produced by the steps in Section 4.2 is checked and processed as appropriate to local registry restrictions. Application of those registry restrictions may result in the rejection of some labels or the application of special restrictions to others.

[edit] 4.4. Punycode Conversion

The resulting U-label is converted to an A-label (defined in [IDNA2008-Defs] anchor13: Insert section number). The A-label is the encoding of the U-label according to the Punycode algorithm [RFC3492] with the ACE prefix "xn--" added at the beginning of the string. The resulting string must, of course, conform to the length limits imposed by the DNS. This document updates RFC 3492 only to the extent of replacing the reference to the discussion of the ACE prefix. The ACE prefix is now specified in this document rather than as part of RFC 3490 or Nameprep [RFC3491] but is the same in both sets of documents.

The failure conditions identified in the Punycode encoding procedure cannot occur if the input is a U-label as determined by the steps above.

[edit] 4.5. Insertion in the Zone

The A-label is registered in the DNS by insertion into a zone.

[edit] 5. Domain Name Lookup Protocol

Lookup is different from registration and different tests are applied on the client. Although some validity checks are necessary to avoid serious problems with the protocol, the lookup-side tests are more permissive and rely on the assumption that names that are present in the DNS are valid. That assumption is, however, a weak one because the presence of wild cards in the DNS might cause a string that is not actually registered in the DNS to be successfully looked up.

The two steps described in Section 5.2 are required.

[edit] 5.1. Label String Input

The user supplies a string in the local character set, typically by typing it or clicking on, or copying and pasting, a resource identifier, e.g., a URI [RFC3986] or IRI [RFC3987] from which the domain name is extracted. Alternately, some process not directly involving the user may read the string from a file or obtain it in some other way. Processing in this step and the next two are local matters, to be accomplished prior to actual invocation of IDNA.

[edit] 5.2. Conversion to Unicode

The string is converted from the local character set into Unicode, if it is not already Unicode. Depending on local needs, this conversion may involve mapping some characters into other characters as well as coding conversions. Those issues are discussed in [IDNA2008-Mapping] and the mapping-related sections of [IDNA2008-Rationale]. [[anchor14:

Supply section number.]] A Unicode string may require normalization as discussed in Section 4.1. The result MUST be a Unicode string in NFC form.

[edit] 5.3. A-label Input

If the input to this procedure appears to be an A-label (i.e., it starts in "xn--"), the lookup application MAY attempt to convert it to a U-label and apply the tests of Section 5.4 and the conversion of Section 5.5 to that form. If the label is converted to Unicode (i.e., to U-label form) using the Punycode decoding algorithm, then the processing specified in those two sections MUST be performed, and the label MUST be rejected if the resulting label is not identical to the original. See the Name Server Considerations section of [IDNA2008-Rationale] for additional discussion on this topic.

That conversion and testing SHOULD be performed if the domain name will later be presented to the user in native character form (this requires that the lookup application be IDNA-aware). If those steps are not performed, the lookup process SHOULD at least make tests to determine that the string is actually an A-label, examining it for the invalid formats specified in the Punycode decoding specification.

Applications that are not IDNA-aware will obviously omit that testing; others MAY treat the string as opaque to avoid the additional processing at the expense of providing less protection and information to users.

[edit] 5.4. Validation and Character List Testing

As with the registration procedure described in Section 4, the Unicode string is checked to verify that all characters that appear in it are valid as input to IDNA lookup processing. As discussed above and in [IDNA2008-Rationale], the lookup check is more liberal than the registration one. Labels that have not been fully evaluated for conformance to the applicable rules are referred to as "putative" labels as discussed in [IDNA2008-Defs][[anchor15: ??? Insert section number -- 2.2.3 as of Defs-09]]. Putative labels with any of the following characteristics MUST BE rejected prior to DNS lookup:

• Labels containing code points that are unassigned in the version of Unicode being used by the application, i.e.,in the UNASSIGNED category of [IDNA2008-Tables].
• Labels that are not in NFC form as defined in [Unicode-UAX15].
• Labels containing "--" (two consecutive hyphens) in the third and fourth character positions.
• Labels containing prohibited code points, i.e., those that are assigned to the "DISALLOWED" category in the permitted character table [IDNA2008-Tables].
• Labels containing code points that are identified in [IDNA2008-Tables] as "CONTEXTJ", i.e., requiring exceptional contextual rule processing on lookup, but that do not conform to that rule. Note that this implies that a rule must be defined, not null: a character that requires a contextual rule but for which the rule is null is treated in this step as having failed to conform to the rule.
• Labels containing code points that are identified in [IDNA2008-Tables] as "CONTEXTO", but for which no such rule appears in the table of rules. Applications resolving DNS names or carrying out equivalent operations are not required to test contextual rules for "CONTEXTO" characters, only to verify that a rule is defined (although they MAY make such tests to provide better protection or give better information to the user).
• Labels whose first character is a combining mark (see Section 4.2.3.2).

In addition, the application SHOULD apply the following test.

• Verification that the string is compliant with the requirements for right to left characters, specified in [IDNA2008-BIDI].

This test may be omitted in special circumstances, such as when the lookup application knows that the conditions are enforced elsewhere, because an attempt to look up and resolve such strings will almost certainly lead to a DNS lookup failure except when wildcards are present in the zone. However, applying the test is likely to give much better information about the reason for a lookup failure -- information that may be usefully passed to the user when that is feasible -- than DNS resolution failure information alone. In any event, lookup applications should avoid attempting to resolve labels that are invalid under that test.

For all other strings, the lookup application MUST rely on the presence or absence of labels in the DNS to determine the validity of those labels and the validity of the characters they contain. If they are registered, they are presumed to be valid; if they are not, their possible validity is not relevant. While a lookup application may reasonably issue warnings about strings it believes may be problematic, applications that decline to process a string that conforms to the rules above (i.e., does not look it up in the DNS) are not in conformance with this protocol.

[edit] 5.5. Punycode Conversion

The string that has now been validated for lookup is converted to ACE form using the Punycode algorithm (with the ACE prefix added). With the understanding that this summary is not normative (the steps above are), the string is either

• in Unicode NFC form that contains no leading combining marks, contains no DISALLOWED or UNASSIGNED code points, has rules associated with any code points in CONTEXTJ or CONTEXTO, and, for those in CONTEXTJ, to satisfies the conditions of the rules; or
• in A-label form, was supplied under circumstances in which the U-label conversions and tests have not been performed (see Section 5.3).

[edit] 5.6. DNS Name Resolution

That resulting validated string is looked up in the DNS, using normal DNS resolver procedures. That lookup can obviously either succeed (returning information) or fail.

[edit] 6. Security Considerations

Security Considerations for this version of IDNA, except for the special issues associated with right to left scripts and characters, are described in [IDNA2008-Defs]. Specific issues for labels containing characters associated with scripts written right to left appear in [IDNA2008-BIDI].

[edit] 7. IANA Considerations

IANA actions for this version of IDNA are specified in [IDNA2008-Tables] and discussed informally in [IDNA2008-Rationale].

The components of IDNA described in this document do not require any IANA actions.

[edit]
BIDI

The use of right-to-left scripts in internationalized domain names has presented several challenges. This memo discusses some problems with these scripts, and some shortcomings in the 2003 IDNA BIDI criterion. Based on this discussion, it proposes a new BIDI rule for IDNA labels.

[edit] 1. Introduction

[edit] 1.1. Purpose and applicability

The purpose of this document is to establish a rule that can be applied to Internationalized Domain Name (IDN) labels in Unicode form (U-labels) containing characters from scripts that are written from right to left.

When labels satisfy the rule, and when certain other conditions are satisfied, they can be used with a minimal chance of these labels being displayed in a confusing way by a bidirectional display algorithm.

This specification is not intended to place any requirements on domain names that do not contain right-to-left characters.

[edit] 1.2. Background and history

The IDNA specification "Stringprep" [RFC3454] makes the following statement in its section 6 on the BIDI algorithm:

3) If a string contains any RandALCat character, a RandALCat character MUST be the first character of the string, and a RandALCat character MUST be the last character of the string.

(A RandALCat character is a character with unambiguously right-to- left directionality.) The reasoning behind this prohibition was to ensure that every component of a displayed domain name has an unambiguously preferred direction. However, this makes certain words in languages written with right-to-left scripts invalid as IDN labels, and in at least one case means that all the words of an entire language are forbidden as IDN labels.

This is illustrated below with examples taken from the Dhivehi and Yiddish languages, as written with the Thaana and Hebrew scripts, respectively.

In investigating this problem, it was realized that the RFC 3454 specification did not explicitly state the requirement to be fulfilled, and therefore, it was impossible to tell whether a simple relaxation of the rule would continue to fulfil the requirement. A further investigation led to the conclusion that there was one reasonable set of requirements that IDNA2003's BIDI restriction did not fulfil. This document therefore proposes replacing the RFC 3454 BIDI requirement in its entirety.

While the document proposes completely new text, most reasonable labels that were allowed under the old criterion will also be allowed under the new criterion, so the operational impact of the rule change is limited.

[edit] 1.3. Layout of the rest of this document

Section 2 defines a test, the "BIDI test", that can be used on a domain name label (no matter what the direction of the label is) to check how safe it is to use in a domain name of possibly mixed directionality. One user of that test is the IDNA2008 protocol[I-D.ietf-idnabis-protocol].

Section 3 sets out the requirements for defining a BIDI rule.

Section 4 gives detailed examples that serve as justification for the change proposed here.

Section 5 to Section 9 describe various situations that can occur when dealing with domain names with characters of different directionality.

Only Section 1.4 and Section 2 are normative.

[edit] 1.4. Terminology

In this memo, we use "network order" to describe the sequence of characters as transmitted on the wire or stored in a file; the terms "first", "next", "previous", "before" and "after" are used to refer to the relationship of characters and labels in network order.

We use "display order" to talk about the sequence of characters as imaged on a display medium; the terms "left" and "right" are used to refer to the relationship of characters and labels in display order.

Most of the time, the examples use the abbreviations for the Unicode BIDI classes to denote the directionality of the characters; the example string CS L consists of one character of class CS and one character of class L. In some examples, the convention that uppercase characters are of class R or AL, and lowercase characters are of class L is used - thus, the example string ABC.abc would consist of 3 right-to-left characters and 3 left-to-right characters, The directionality of such examples is determined by context - for instance, in the sentence "ABC.abc is displayed as CBA.abc", the first example string is in network order, the second example string is in display order.

The term "paragraph" is used in the sense of the Unicode BIDI specification [UAX9] - it means "a block of text that has an overall direction, either left-to-right or right-to-left", approximately; see UAX 9 for the details.

"LTR" and "RTL" are abbreviations for "right to left" and "left to right", respectively.

An RTL label is a label that contains at least one character of type R or AL.

An LTR label is any label that is not an RTL label.

A "Bidi domain name" is a domain name that contains at least one RTL label.

The terminology used for the BIDI properties of Unicode characters is taken from the Unicode Standard[Unicode] For reference, here are the values that the Unicode BIDI property can have:

• L - Left-to-right - most letters in LTR scripts
• R - Right-to-left - most letters in non-Arabic RTL scripts
• AL - Arabic letters - most letters in the Arabic script
• EN - European Number (0-9, and Extended Arabic-Indic numbers)
• ES - European Number Separator (+ and -)
• ET - European Number Terminator (currency symbols, the hash sign, the percent sign and so on)
• AN - Arabic Number; this encompasses the Arabic-Indic numbers, but not the Extended Arabic-Indic numbers
• CS - Common Number Separator (. , / : et al)
• NSM - Non spacing Mark - most combining accents
• BN - Boundary Neutral - control characters
• B - Paragraph Separator
• S - Segment Separator
• WS - Whitespace, including the SPACE character
• ON - Other Neutrals, including @, &, parentheses, MIDDLE DOT
• LRE, LRO, RLE, RLO, PDF - these are "directional control characters", and are not used in IDNA labels.

The other terminology used to describe IDNA concepts is defined in [I-D.ietf-idnabis-defs]

[edit] 2. A replacement for the RFC 3454 BIDI rule

The following test has been developed for labels in BIDI domain names. The requirements that this test satisifies are described in Section 3.

1. The first character must be a character with BIDI property L, R or AL. If it has the R or AL property, it is an RTL label; if it has the L property, it is an LTR label.

2. In an RTL label, only characters with the BIDI properties R, AL, AN, EN, ES, CS, ET, ON, BN and NSM are allowed.

3. In an RTL label, the end of the label must be a character with BIDI property R, AL, EN or AN, followed by zero or more characters with BIDI property NSM.

4. In an RTL label, if an EN is present, no AN may be present, and vice versa.

5. In an LTR label, only characters with the BIDI properties L, En, ES, CS. ET, ON and NSM are allowed.

6. In an LTR label, the end of the label must be a character with BIDI property L or EN, followed by zero or more characters with BIDI property NSM.

The following guarantees can be made based on the above:

• In a domain name consisting of only labels that pass the test, the requirements of Section 3 are satisfied. Note that even LTR labels and pure ASCII labels have to be tested.
• In a domain name consisting of only LDH-labels and labels that pass the test, the requirements of Section 3 are satisfied as long as a label that starts with an ASCII digit does not come after a right-to-left label.

No guarantee is given for other combinations.

[edit] 3. A requirement set for the BIDI rule

One issue with RFC 3454 was that it did not give an explicit justification for the BIDI rule, thus it was hard to tell if a modified rule would continue to fulfil the purpose for which the RFC 3454 rule was written.

This document proposes an explicit justification, by stating a set of requirements for which it is possible to test whether or not the modified rule fulfils the requirement.

All the text in this document assumes that text containing the labels under consideration will be displayed using the Unicode bidirectional algorithm [UAX9].

The requirements proposed are these:

• Label Uniqueness: No two labels, when presented in display order in the same paragraph, should have the same sequence of characters without also having the same sequence of characters in network order, both when the paragraph has LTR direction and when the paragraph has RTL direction. (This is the criterion that is explicit in RFC 3454). (Note that a label displayed in an RTL paragraph may display the same as a different label displayed in an LTR paragraph, and still satisfy this criterion.) * Character Grouping: When displaying a string of labels, using the Unicode BIDI algorithm to reorder the characters for display, the characters of each label should remain grouped between the characters delimiting the labels, both when the string is embedded in a paragraph with LTR direction and when it is embedded in a paragraph with RTL direction.

Several stronger statements were considered and rejected, because they seem to be impossible to fulfil within the constraints of the Unicode bidirectional algorithm. These include:

• The appearance of a label should be unaffected by its embedding context. This proved impossible even for ASCII labels; the label "123-A" will have a different display order in an RTL context than in an LTR context. (This particular example is, however, disallowed anyway.)

• The sequence of labels should be consistent with network order. This proved impossible - a domain name consisting of the labels (in network order) L1.R1.R2.L2 will be displayed as L1.R2.R1.L2 in an LTR context. (In a RTL context, it will be displayed as L2.R2.R1.L1).

• No two domain names should be displayed the same, even under differing directionality. This was shown to be unsound, since the domain name (in network order) ABC.abc will have display order CBA.abc in an LTR context and abc.CBA in an RTL context, while the domain name (network) abc.ABC will have display order abc.CBA in an LTR context and CBA.abc in an RTL context.

One specific requirement was thought to be problematic, but turned out to be satisfied by a string that obeys the proposed rules:

• The Character Grouping requirement should be satisfied when directional controls (LRE, RLE, RLO, LRO, PDF) are used in the same paragraph (outside of the labels). Because these controls affect presentation order in non-obvious ways, by affecting the "sor" and "eor" properties of the Unicode BIDI algorithm, the conditions above require extra testing in order to figure out whether or not they influence the display of the domain name.

Testing found that for the strings allowed under the rule presented in this document, directional controls do not influence the display of the domain name.

In the following descriptions, first-level bullets are used to indicate rules or normative statements; second-level bullets are commentary.

The Character Grouping requirement can be more formally stated as:

• Let "Delimiterchars" be a set of characters with the Unicode BIDI properties CS, WS, ON. (These are commonly used to delimit labels * both the FULL STOP and the space are included. They are not allowed in domain names.) * ET, though it commonly occurs next to domain names in practice, is problematic: the context R CS L EN ET (for instance A.a1%) makes the label L EN not satisfy the character grouping requirement.

• ES commonly occurs in labels as HYPHEN-MINUS, but could also be used as a delimiter (for instance, the plus sign). It is left out here.

• Let "unproblematic label" be a label that either satisfies the requirements, or does not contain any character with the BIDI properties R, AL or AN, and does not begin with a character with the BIDI property EN. (Informally, "it does not start with a number".) A label X satisfies the Character Grouping requirement when, for any Delimiter Character D1 and D2, and for any label S1 and S2 that is an unproblematic label or an empty string, the following holds true:

If the string formed by concatenating S1, D1, X, D2 and S2 is reordered according to the BIDI algorithm, then all the characters of X in the reordered string are between D1 and D2, and no other characters are between D1 and D2, both if the overall paragraph direction is LTR and if the overall paragraph direction is RTL.

Note that the definition is self-referential, since S1 and S2 are constrained to be "legal" by this definition; this makes testing changes to proposed rules a little complex, but does not create problems for testing whether or not a given proposed rule satisfies the criterion.

The "zero-length" case represents the case where a domain name is next to something that isn't a domain name, separated by a delimiter character.

The Label Uniqueness requirement can be formally stated as:

If two non-identical labels X and Y, embedded as for the test above, displayed in paragraphs with the same directionality, are reordered by the BIDI algorithm into the same sequence of codepoints, at most one of the labels X and Y is a legal label.

[edit] 4. Examples of issues found

[edit] 4.1. Dhivehi

Dhivehi, the official language of the Maldives, is written with the Thaana script. This displays some of the characteristics of Arabic script, including its directional properties, and the indication of vowels by the diacritical marking of consonantal base characters.

This marking is obligatory, and both double vowels and syllable-final consonants are indicated by the marking of special unvoiced characters. Every Dhivehi word therefore ends with a combining mark.

The word for "computer", which is romanized as "konpeetaru", is written with the following sequence of Unicode code points:

U+0786 THAANA LETTER KAAFU (AL)

U+07AE THAANA OBOFILI (NSM)

U+0782 THAANA LETTER NOONU (AL)

U+07B0 THAANA SUKUN (NSM)

U+0795 THAANA LETTER PAVIYANI (AL)

U+07A9 THAANA LETTER EEBEEFILI (AL)

U+0793 THAANA LETTER TAVIYANI (AL)

U+07A6 THAANA ABAFILI (NSM)

U+0783 THAANA LETTER RAA (AL)

U+07AA THAANA UBIUFILI (NSM)

The directionality class of U+07AA in the Unicode database [Unicode] is NSM (non-spacing mark), which is not R or AL; a conformant implementation of the IDNA2003 algorithm will say that "this is not in RandALCat", and refuse to encode the string.

[edit] 4.2. Yiddish

Yiddish is one of several languages written with the Hebrew script (others include Hebrew and Ladino). This is basically a consonantal alphabet (also termed an "abjad") but Yiddish is written using an extended form that is fully vocalic. The vowels are indicated in several ways, of which one is by repurposing letters that are consonants in Hebrew. Other letters are used both as vowels and consonants, with combining marks, called "points", used to differentiate between them. Finally, some base characters can indicate several different vowels, which are also disambiguated by combining marks. Pointed characters can appear in word-final position and may therefore also be needed at the end of labels. This is not an invariable attribute of a Yiddish string and there is thus greater latitude here than there is with Dhivehi.

The organization now known as the "YIVO Institute for Jewish Research" developed orthographic rules for modern Standard Yiddish during the 1930s on the basis of work conducted in several venues since earlier in that century. These are given in, "The Standardized Yiddish Orthography: Rules of Yiddish Spelling" [SYO], and are taken as normatively descriptive of modern Standard Yiddish in any context where that notion is deemed relevant. They have been applied exclusively in all formal Yiddish dictionaries published since their establishment, and are similarly dominant in academic and bibliographic regards.

It therefore appears appropriate for this repertoire also to be supported fully by IDNA. This presents no difficulty with characters in initial and medial positions, but pointed characters are regularly used in final position as well. All of the characters in the SYO repertoire appear in both marked and unmarked form with one exception: the HEBREW LETTER PE (U+05E4). The SYO only permits this with a HEBREW POINT DAGESH (U+05BC), providing the Yiddish equivalent to the Latin letter "p", or a HEBREW POINT RAFE (U+05BF), equivalent to the Latin letter "f". There is, however, a separate unpointed allograph, the HEBREW LETTER FINAL PE (U+05E3), for the latter character when it appears in final position. The constraint on the use of the SYO repertoire resulting from the proscription of combining marks at the end of RTL strings thus reduces to nothing more, or less, than the equivalent of saying that a string of Latin characters cannot end with the letter "p". It must also be noted that the HEBREW LETTER PE with HEBREW POINT DAGESH is characteristic of almost all traditional Yiddish orthographies that predate (or remain in use in parallel to) the SYO, being the first pointed character to appear in any of them.

A more general instantiation of the basic problem can be seen in the representation of the YIVO acronym. This is written with the Hebrew letters YOD YOD HIRIQ VAV VAV ALEF QAMATS, where HIRIQ and QAMATS are combining points:

U+05D9 HEBREW LETTER YOD (R)

U+05B4 HEBREW POINT HIRIQ (NSM)

U+05D5 HEBREW LETTER VAV (R)

U+05D0 HEBREW LETTER ALEF (R)

U+05B8 HEBREW POINT QAMATS (NSM)

The directionality class of U+05B8 HEBREW POINT QAMATS in the Unicode database is NSM, which again causes the IDNA2003 algorithm to reject the string.

It may also be noted that all of the combined characters mentioned above exist in precomposed form at separate positions in the Unicode chart. However, by invoking Stringprep, the IDNA2003 algorithm also rejects those codepoints, for reasons not discussed here.

[edit] 4.3. Strings with numbers

By requiring that the first or last character of a string be category R or AL, RFC 3454 prohibited a string containing right-to-left characters from ending with a number.

Consider the strings ALEF 5 (HEBREW LETTER ALEF + DIGIT FIVE) and 5 ALEF. Displayed in an LTR context, the first one will be displayed from left to right as 5 ALEF (with the 5 being considered right-to- left because of the leading ALEF), while 5 ALEF will be displayed in exactly the same order (5 taking the direction from context).

Clearly, only one of those should be permitted as a registered label, but barring them both seems to require justification.

[edit] 5. Troublesome situations and guidelines

There are situations in which labels that satisfy the rule above will be displayed in a surprising fashion. The most important of these is the case where a label ending in a character with BIDI property AL, AN or R occurs before a label beginning with a character of BIDI property EN. In that case, the number will appear to move into the label containing the right-to-left character, violating the Character Grouping requirement.

If the label that occurs after the right-to-left label itself satisfies the BIDI criterion, the requirements will be satisfied in all cases (this is the reason why the criterion talks about strings containing L in some cases). However, the WG concluded that this could not be required for several reasons:

• There is a large current deployment of ASCII domain names starting with digits. These cannot possibly be invalidated.
• Domain names are often constructed piecemeal, for instance by combining a string with the content of a search list. This may occur after IDNA processing, and thus in part of the code that is not IDNA-aware, making detection of the undesirable combination impossible.
• Even if a label is registered under a "safe" label, there may be a DNAME [RFC2672]with an "unsafe" label that points to the "safe" label, thus creating seemingly-valid names that would not satisfy the criterion.
• Wildcards create the odd situation where a label is "valid" (can be looked up successfully) without the zone owner knowing that this label exists. So an owner of a zone whose name starts with a digit and contains a wildcard has no way of controlling whether or not names with RTL labels in them are looked up in his zone.

So rather than trying to suggest rules that disallow all such undesirable situations, this document merely warns about the possibility.

[edit] 6. Other issues in need of resolution

This document concerns itself only with the rules that are needed when dealing with domain names with characters that have differing BIDI properties, and considers characters only in terms of their BIDI properties. All other issues with these scripts have to be considered in other contexts.

One such issue is the need to keep numbers separate; several scripts, are used with multiple sets of numbers - most commonly they use Latin numbers and a script-specific set of numbers, but in the case of Arabic, there are 2 sets of "Arabic-Indic" digits involved.

The algorithm in this document disallows occurrences of AN-class characters ("Arabic-Indic digits", U+0660 to U+0669) together with EN-class characters (which includes "European" digits, U+0030 to U+0039 and "extended Arabic-Indic digits", U+06F0 to U+06F9), but does not help in preventing the mixing of, for instance, Bengali digits (U+09E6 to U+09EF) and Gujarati digits (U+0AE6 to U+0AEF), both of which have BIDI class L. A registry or script community that wishes to create rules for the mixing of digits in a label will be able to specify these restrictions at the registry level. Rules can also be specified at the protocol level, but while the example above involves right-to-left characters, this is not inherently a BIDI problem.

Another set of issues concerns the proper display of IDNs with a mixture of LTR and RTL labels, or only RTL labels.

It is unrealistic to expect that applications will display domain names using embedded formatting codes between their labels (for one thing, no reliable algorithms for identifying domain names in running text exist); thus, the display order will be determined by the bidirectional algorithm. Thus, a sequence (in network order) of R1.R2.ltr will be displayed in the order 2R.1R.ltr in an LTR context, which might surprise someone expecting to see labels displayed in hierarchical order. Again, this memo does not attempt to suggest a solution to this problem.

[edit] 7. Compatibility considerations

[edit] 7.1. Backwards compatibility considerations

As with any change to an existing standard, it is important to consider what happens with existing implementations when the change is introduced. The following troublesome cases have been noted:

• Old program used to input the newly-allowed string. If the old program checks the input against RFC 3454, the string will not be allowed, and that domain name will remain inaccessible.
• Old program is asked to display the newly-allowed string, and checks it against RFC 3454 before displaying. The program will perform some kind of fallback, most likely displaying the string in A-label form.
• Old program tries to display the newly-allowed string. If the old program has code for displaying the last character of a string that is different from the code used to display the characters in the middle of the string, display may be inconsistent and cause confusion.

One particular example of the last case is if a program chooses to examine the last character (in network order) of a string in order to determine its directionality, rather than its first. If it finds an NSM character and tries to display the string as if it was a left-to- right string, the resulting display may be interesting, but not useful.

The editors believe that these cases will have less harmful impact in practice than continuing to deny the use of words from the languages for which these strings are necessary as IDN labels.

This specification does not forbid using leading European numbers in ASCII-only labels, since this would in conflict with a large installed base of such labels, and would increase the scope of the specification from RTL labels to all labels. The harm resulting from this limitation of scope is described in Section 5. Registries and private zone managers can check for this particular condition before they allow registration of any string with right-to-left characters in it. Generally it is best to not allow registration of any right- to-left strings in a zone where the label at the level above begins with a digit.

[edit] 7.2. Forward compatibility considerations

This text is, intentionally, specified strictly in terms of the Unicode BIDI properties. The determination that the condition is sufficient to fulfil the criteria depends on the Unicode BIDI algorithm; it is unlikely that drastic changes will be made to this algorithm.

However, the determination of validity for any string depends on the Unicode BIDI property values, which are not declared immutable by the Unicode Consortium. Furthermore, the behaviour of the algorithm for any given character is likely to be linguistically and culturally sensitive, so that it is not unlikely that later versions of the Unicode standard may change the BIDI properties assigned to certain Unicode characters.

This memo does not propose a solution for this problem.

[edit] 8. IANA Considerations

This document makes no request of IANA.

Note to RFC Editor: this section may be removed on publication as an RFC.

[edit] 9. Security Considerations

This modification will allow some strings to be used in IDNA contexts that are not allowed today. It is possible that differences in the interpretation of the specification between old and new implementations could pose a security risk, but it is difficult to envision any specific instantiation of this.

Any rational attempt to compute, for instance, a hash over an identifier processed by IDNA would use network order for its computation, and thus be unaffected by the changes proposed here.

While it is not believed to pose a problem, if display routines had been written with specific knowledge of the RFC 3454 IDNA prohibitions, it is possible that the potential problems noted under "backwards compatibility" could cause new kinds of confusion.

[edit]
TABLES

This document specifies rules for deciding whether a code point, considered in isolation, is a candidate for inclusion in an Internationalized Domain Name.

It is part of the specification of IDNA2008.

[edit] 1. Introduction

RFC 4690 [RFC4690]suggests an inclusion based approach for selecting the code points from The Unicode Standard [Unicode51] that should be included in the list of code points that may be used in Internationalized Domain Names.

Specifically, RFC 4690 [RFC4690] says the following:

The IAB has concluded that there is a consensus within the broader community that lists of code points should be specified by the use of an inclusion-based mechanism (i.e., identifying the characters that are permitted), rather than by excluding a small number of characters from the total Unicode set as Stringprep [RFC3454] and Nameprep [RFC3491] do today. That conclusion should be reviewed by the IETF community and action taken as appropriate.

This document reviews and classifies the collections of code points in the Unicode character set by examining various properties of the code points. It then defines an algorithm for determining a derived property value. It specifies a procedure and not a table of code points so that the algorithm can be used to determine code point sets independent of the version of Unicode that is in use.

This document is not intended to specify precisely how these property values are to be applied in IDN labels. That information appears in [IDNA2008-protocol], but it is important to understand that the assignment of a value of this property to a particular character is not sufficient to determine whether it can be used in a given label.

In particular, some combinations of allowed code points are not advisable for use in IDNs due to rules specific to a script or class of characters. The requirement for such rules is linked to the operations in [IDNA2008-protocol] and especially to the characters designated as requiring contextual rules.

The value of the property is to be interpreted approximately as follows.

• PROTOCOL VALID: Those that are allowed to be used in IDNs. Code points with this property value are permitted for general use in IDNs. However, that a label consists only of code points that have this property value does not imply that the label can be used in DNS. See [IDNA2008-protocol] for algorithms to make decisions about labels in domain names. The abbreviated term PVALID is used to refer to this value in the rest of this document.
• CONTEXTUAL RULE REQUIRED: Some characteristics of the character, such as it being invisible in certain contexts or problematic in others, requires that it not be used in labels unless specific other characters or properties are present. The abbreviated term CONTEXT is used to refer to this value in the rest of this document. There are two subdivisions of CONTEXTUAL RULE REQUIRED, one for Join_controls (called CONTEXTJ) and for other characters (called CONTEXTO). These are discussed in more detail below and in [IDNA2008-protocol].
• DISALLOWED: Those that should clearly not be included in IDNs. Code points with this property value are not permitted in IDNs.
• UNASSIGNED: Those code points that are not designated (i.e. are unassigned) in the Unicode Standard.

The mechanisms described here allow determination of the value of the property for future versions of Unicode (including characters added after Unicode 5.1). This is suitable for any newer versions of Unicode as well. Changes in Unicode properties that do not affect the outcome of this process do not affect IDN. For example, a character can have its Unicode General_Category value change from So to Sm, or from Lo to Ll, without affecting the table results.

Moreover, even if such changes were to result, the BackwardCompatible list (Section 2.7) can be adjusted to ensure the stability of the results.

Some code points need to be allowed in exceptional circumstances, but should be excluded in all other cases; these rules are also described in other documents. The most notable of these are the the Join Control characters, U+200D ZERO WIDTH JOINER and U+200C ZERO WIDTH NON-JOINER. Both of them have the derived property value CONTEXTJ.

A character with the derived property value CONTEXTJ or CONTEXTO (CONTEXTUAL RULE REQUIRED) is not to be used unless an appropriate rule has been established and the context of the character is consistent with that rule. It is invalid to either register a string containing these characters or even to look one up unless such contextual rule is found and satisfied. Please see Appendix A, The Contextual Rules Registry, for more information.

This document is part of a series that, together, constitute a proposal for updating the IDNA standards to resolve issues uncovered in recent years, cover a broader range of scripts, and provide for migration to newer versions of Unicode. See [IDNA2008-rationale] for a broader discussion.

[edit] 2. Category definitions Used to Calculate Derived Property Value

The derived property obtains its value based on a two-step procedure.

First, characters are placed in one or more character categories based on either core properties defined by the Unicode Standard or by treating the codepoint as an exception and addressing the codepoint by its codepoint value. These categories are not mutually exclusive.

In the second step, set operations are used with these categories to determine the values for an IDN-specific property. Those operations are specified in Section 3.

Unicode property names and property value names may have short abbreviations, such as gc for the General_Category property, and Ll for the Lowercase_Letter property value of the gc property.

In the following specification of categories, the operation which returns the value of a particular Unicode character property for a code point is designated by using the formal name of that property (from PropertyAliases.txt) followed by '(cp)'. For example, the value of the General_Category property for a code point is indicated by General_Category(cp).

[edit] 2.1. LetterDigits (A)

A: General_Category(cp) is in {Ll, Lu, Lo, Nd, Lm, Mn, Mc}

These rules identifies characters commonly used in mnemonics and often informally described as "language characters". In general, only code points assigned to this category are suitable for use in IDN.

For more information, see section 4.5 of The Unicode Standard [Unicode5].

The categories used in this rule are:

• Ll - Lowercase_Letter
• Lu - Uppercase_Letter
• Lo - Other_Letter
• Nd - Decimal_Number
• Lm - Modifier_Letter
• Mn - Nonspacing_Mark
• Mc - Spacing_Mark

[edit] 2.2. Unstable (B)

B: toNFKC(toCaseFold(toNFKC(cp))) != cp

This category is used to group the characters that are not stable under NFKC normalization and casefolding. In general, these code points are not suitable for use for IDN.

The toCaseFold() operation is defined in Section 3.13 of The Unicode Standard [Unicode5].

The toNFKC() operation returns the code point in normalization form KC. For more information, see Section 5 of Unicode Standard Annex

1. 15 [TR15].

[edit] 2.3. IgnorableProperties (C)

C: Default_Ignorable_Code_Point(cp) = True or White_Space(cp) = True or Noncharacter_Code_Point(cp) = True

This category is used to group code points that are not recommended for use in identifiers. In general, these code points are not suitable for use for IDN.

The definition for Default_Ignorable_Code_Point can be found in DerivedCoreProperties.txt [1] and is at the time of Unicode 5.1:

Other_Default_Ignorable_Code_Point + Cf (Format characters) + Variation_Selector - White_Space - FFF9..FFFB (Annotation Characters) - 0600..0603, 06DD, 070F (exceptional Cf characters that should be visible)

[edit] 2.4. IgnorableBlocks (D)

D: Block(cp) is in {Combining Diacritical Marks for Symbols, Musical Symbols, Ancient Greek Musical Notation}

This category is used to identifying code points that are not useful in mnemonics or that are otherwise impractical for IDN use. In general, these code points are not suitable for use for IDN.

[edit] 2.5. LDH (E)

E: cp is in {002D, 0030..0039, 0061..007A}

This category is used in the second step to preserve the traditional "hostname" (LDH) characters ('-', 0-9 and a-z). In general, these code points are suitable for use for IDN. Note that there are other rules regarding the code point U+002D HYPHEN-MINUS that are specified in the IDNA Protocol Specification [IDNA2008-protocol].

[edit] 2.6. Exceptions (F)

F: cp is in {00B7, 00DF, 0375, 03C2, 05F3, 05F4, 0640, 0660,
             0661, 0662, 0663, 0664, 0665, 0666, 0667, 0668,
             0669, 06F0, 06F1, 06F2, 06F3, 06F4, 06F5, 06F6,
             06F7, 06F8, 06F9, 06FD, 06FE, 07FA, 0F0B, 3007,
             302E, 302F, 3031, 3032, 3033, 3034, 3035, 303B,
             30FB}

This category explicitly lists code points for which the category cannot be assigned using only the core property values that exist in the Unicode standard. The values are according to the table below:

00DF; PVALID     # LATIN SMALL LETTER SHARP S
03C2; PVALID     # GREEK SMALL LETTER FINAL SIGMA 
06FD; PVALID     # ARABIC SIGN SINDHI AMPERSAND 
06FE; PVALID     # ARABIC SIGN SINDHI POSTPOSITION MEN 
0F0B; PVALID     # TIBETAN MARK INTERSYLLABIC TSHEG 
3007; PVALID     # IDEOGRAPHIC NUMBER ZERO 
00B7; CONTEXTO   # MIDDLE DOT 
0375; CONTEXTO   # GREEK LOWER NUMERAL SIGN (KERAIA) 
05F3; CONTEXTO   # HEBREW PUNCTUATION GERESH 
05F4; CONTEXTO   # HEBREW PUNCTUATION GERSHAYIM 
30FB; CONTEXTO   # KATAKANA MIDDLE DOT 
0660; CONTEXTO   # ARABIC-INDIC DIGIT ZERO 
0661; CONTEXTO   # ARABIC-INDIC DIGIT ONE 
0662; CONTEXTO   # ARABIC-INDIC DIGIT TWO 
0663; CONTEXTO   # ARABIC-INDIC DIGIT THREE 
0664; CONTEXTO   # ARABIC-INDIC DIGIT FOUR 
0665; CONTEXTO   # ARABIC-INDIC DIGIT FIVE 
0666; CONTEXTO   # ARABIC-INDIC DIGIT SIX 
0667; CONTEXTO   # ARABIC-INDIC DIGIT SEVEN 
0668; CONTEXTO   # ARABIC-INDIC DIGIT EIGHT 
0669; CONTEXTO   # ARABIC-INDIC DIGIT NINE 
06F0; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT ZERO 
06F1; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT ONE 
06F2; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT TWO 
06F3; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT THREE 
06F4; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT FOUR 
06F5; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT FIVE 
06F6; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT SIX 
06F7; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT SEVEN 
06F8; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT EIGHT 
06F9; CONTEXTO   # EXTENDED ARABIC-INDIC DIGIT NINE 
0640; DISALLOWED # ARABIC TATWEEL 
07FA; DISALLOWED # NKO LAJANYALAN 
302E; DISALLOWED # HANGUL SINGLE DOT TONE MARK 
302F; DISALLOWED # HANGUL DOUBLE DOT TONE MARK 
3031; DISALLOWED # VERTICAL KANA REPEAT MARK 
3032; DISALLOWED # VERTICAL KANA REPEAT WITH VOICED SOUND MARK 
3033; DISALLOWED # VERTICAL KANA REPEAT MARK UPPER HALF 
3034; DISALLOWED # VERTICAL KANA REPEAT WITH VOICED SOUND MARK UPPER HALF 
3035; DISALLOWED # VERTICAL KANA REPEAT MARK LOWER HALF 
303B; DISALLOWED # VERTICAL IDEOGRAPHIC ITERATION MARK 

[edit] 2.7. BackwardCompatible (G)

G: cp is in {}

This category includes the code points that property values in versions of Unicode after 5.1 have changed in such a way that the derived property value would no longer be PVALID or DISALLOWED. If changes are made to future versions of Unicode so that code points might change property value from PVALID or DISALLOWED, then this table can be updated and keep special exception values so that the property values for code points stay stable.

[edit] 2.8. JoinControl (H)

H: Join_Control(cp) = True

This category consists of Join Control characters (i.e., they are not in LetterDigits (Section 2.1)) but are still required in IDN labels under some circumstances. They require extended special treatment in Lookup and Resolution.

[edit] 2.9. OldHangulJamo (I)

I: Hangul_Syllable_Type(cp) is in {L, V, T}

This category consists of all conjoining Hangul Jamo (Leading Jamo, Vowel Jamo, and Trailing Jamo).

Elimination of conjoining Hangul Jamos from the set of PVALID characters results in restricting the set of Korean PVALID characters just to preformed, modern Hangul syllable characters. Old Hangul syllables, which must be spelled with sequences of conjoining Hangul Jamos, are not PVALID for IDNs.

[edit] 2.10. Unassigned (J)

This category consists of code points in the Unicode character set that are not (yet) assigned. It should be noted that Unicode distinguishes between 'unassigned code points' and 'unassigned characters'. The unassigned code points are all but (Cn - Noncharacters), while the unassigned *characters* are all but (Cn + Cs).

[edit] 3. Calculation of the Derived Property

As described above (Section 1) and in more detail in the "IDNA Protocol" document [IDNA2008-protocol], possible values of the IDN property are:

• PVALID
• CONTEXTJ
• CONTEXTO
• DISALLOWED
• UNASSIGNED

The algorithm to calculate the value of the derived property is as follows. If the names of a rule (such as Exception) is used, that implies the set of codepoints that the rule define, while the same name as a function call (such as Exception(cp)) imply the value cp has in the Exceptions table.

If .cp. .in. Exceptions Then Exceptions(cp);

Else If .cp. .in. BackwardCompatible Then BackwardCompatible(cp);

Else If .cp. .in. Unassigned Then UNASSIGNED;

Else If .cp. .in. LDH Then PVALID;

Else If .cp. .in. JoinControl Then CONTEXTJ;

Else If .cp. .in. Unstable Then DISALLOWED;

Else If .cp. .in. IgnorableProperties Then DISALLOWED;

Else If .cp. .in. IgnorableBlocks Then DISALLOWED;

Else If .cp. .in. OldHangulJamo Then DISALLOWED;

Else If .cp. .in. LetterDigits Then PVALID;

Else DISALLOWED;

[edit] 4. Codepoints

The Categories and Rules defined in Section 2 and Section 3 apply to all Unicode code points. The table in Appendix B shows, for illustrative purposes, the consequences of the categories and classification rules, and the resulting property values.

The list of code points that can be found in Appendix B is non- normative. Section 2 and Section 3 are normative.

[edit] 5. IANA Considerations

[edit] 5.1. IDNA derived property value registry

IANA is to keep a list of the derived property for the versions of Unicode that is released after (and including) version 5.1. The derived property value is to be calculated according to the specifications in sections Section 2 and Section 3 and not by copying the non-normative table found in Appendix B. Changes to the rules, including BackwardCompatible (Section 2.7) (a set that is at release of this document is empty), require IETF Review, as described in [RFC5226]

[edit] 5.2. IDNA Context Registry

For characters that are defined in the IDNA Character Registry list as CONTEXTO or CONTEXTJ and therefore requiring a contextual rule IANA will create and maintain a list of approved contextual rules.

Additions or changes to these rules require IETF Review, as described in [RFC5226].

A table from which that registry can be initialized, and some further discussion, appears in Appendix A.

[edit] 6. Security Considerations

The security issues associated with this work are discussed in [IDNA2008-rationale] and [IDNA2008-protocol].

[edit] Appendix A. Contextual Rules Registry

As discussed in Section 5.2 and in the IANA Considerations section of [IDNA2008-rationale], a registry of rules that define the contexts in which particular PROTOCOL-VALID characters, characters associated with a requirement for Contextual Information, are permitted. These rules are expressed as tests on the label in which the characters appear (all, or any part of, the label may be tested).

The grammatical rules are expressed in pseudo code. The conventions used for that pseudo code are explained here.

Each rule is constructed as a Boolean expression that evaluates to either True or False. A simple "True;" or "False;" rule sets the default result value for the rule set. Subsequent conditional rules that evaluate to True or False may re-set the result value.

A special value "Undefined" is used to deal with any error conditions, such as an attempt to test a character before the start of a label or after the end of a label. If any term of a rule evaluates to Undefined, further evaluation of the rule immediately terminates, as the result value of the rule will itself be Undefined.

cp represents the codepoint to be tested.

FirstChar is a special term which denotes the first codepoint in a label.

LastChar is a special term which denotes the last codepoint in a label.

.eq. represents the equality relation.

A .eq. B evaluates to True if A equals B.

.is. represents checking position in a label.

A .is. B evaluates to True if A and B have same position in the same label.

.ne. represents the non-equality relation.

A .ne. B evaluates to True if A is not equal to B.

.in. represents the set inclusion relation.

A .in. B evaluates to True if A is a member of the set B.

A functional notation, Function_Name(cp), is used to express either string positions within a label, Boolean character property tests of a codepoint, or a regular expression match. When such function names refer to Boolean character property tests, the function names use the exact Unicode character property name for the property in question, and "cp" is evaluated as the Unicode value of the codepoint to be tested, rather than as its position in the label. When such function names refer to string positions within a label, "cp" is evaluated as its position in the label.

RegExpMatch(X) takes as its parameter X a schematic regular expression consisting of a mix of Unicode character property values and literal Unicode codepoints.

Script(cp) returns the value of the Unicode Script property, as defined in Scripts.txt in the Unicode Character Database.

Canonical_Combining_Class(cp) returns the value of the Unicode Canonical_Combining_Class property, as defined in UnicodeData.txt in the Unicode Character Database.

Before(cp) returns the codepoint of the character immediately preceding cp in logical order in the string representing the label. Before(FirstChar) evaluates to Undefined.

After(cp) returns the codepoint of the character immediately following cp in logical order in the string representing the label. After(LastChar) evaluates to Undefined.

Note that "Before" and "After" do not refer to the visual display order of the character in a label, which may be reversed or otherwise modified by the bidirectional algorithm for labels including characters from scripts written right-to-left.

The clauses "Then True" and "Then False" imply exit from the pseudo- code routine with the corresponding result.

Repeated evaluation for all characters in a label makes use of the special construct:

For All Characters:

Expression;

End For;

This construct requires repeated evaluation of "Expression" for each codepoint in the label, starting from FirstChar and proceeding to LastChar.

The different fields in the rules are to be interpreted as follows: Code point:

  The codepoint, or codepoints, that this rule is to be applied to.
  Normally, this implies that if any of the codepoints in a label is
  as defined, then the rules should be applied.  If evaluated to
  True, the codepoint is ok as used; if evaluated to False, it is
  not o.k.

Overview:

  A description of the goal with the rule, in plain english.

Lookup:

  True if application of this rule is recommended at lookup time;
  False otherwise.

Rule Set:

  The rule set itself, as described above.

[edit] Appendix A.1. ZERO WIDTH NON-JOINER

Code point:

  U+200C

Overview:

  This may occur in a formally cursive script (such as Arabic) in a
  context where it breaks a cursive connection as required for
  orthographic rules, as in the Persian language, for example.  It
  also may occur in Indic scripts in a consonant conjunct context
  (immediately following a virama), to control required display of
  such conjuncts.

Lookup:

  True

Rule Set:

  False;
  If Canonical_Combining_Class(Before(cp)) .eq.  Virama Then True;
  If RegExpMatch((Joining_Type:{L,D})(Joining_Type:T)*\u200C
     (Joining_Type:T)*(Joining_Type:{R,D})) Then True;

[edit] Appendix A.2. ZERO WIDTH JOINER

Code point:

  U+200D

Overview:

  This may occur in Indic scripts in a consonant conjunct context
  (immediately following a virama), to control required display of
  such conjuncts.

Lookup:

  True

Rule Set:

  False;
  If Canonical_Combining_Class(Before(cp)) .eq.  Virama Then True;

[edit] Appendix A.3. MIDDLE DOT

Code point:

  U+00B7

Overview:

  Between 'l' (U+006C) characters only, used to permit the Catalan
  character ela geminada to be expressed

Lookup:

  False

Rule Set:

  False;
  If Before(cp) .eq.  U+006C And
     After(cp) .eq.  U+006C Then True;

[edit] Appendix A.4. GREEK LOWER NUMERAL SIGN (KERAIA)

Code point:

  U+0375

Overview:

  The script of the following character MUST be Greek.

Lookup:

  False

Rule Set:

  False;
  If Script(After(cp)) .eq.  Greek Then True;

[edit] Appendix A.5. HEBREW PUNCTUATION GERESH

Code point:

  U+05F3

Overview:

  The script of the preceding character MUST be Hebrew.

Lookup:

  False

Rule Set:

  False;
  If Script(Before(cp)) .eq.  Hebrew Then True;

[edit] Appendix A.6. HEBREW PUNCTUATION GERSHAYIM

Code point:

  U+05F4

Overview:

  The script of the preceding character MUST be Hebrew.

Lookup:

  False

Rule Set:

  False;
  If Script(Before(cp)) .eq.  Hebrew Then True;

[edit] Appendix A.7. KATAKANA MIDDLE DOT

Code point:

  U+30FB

Overview:

  Note that the Script of Katakana Middle Dot is not any of
  "Hiragana", "Katakana" or "Han".  The effect of this rule is to
  require at least one character in the label to be in one of those
  scripts.

Lookup:

  False

Rule Set:

  False;
  For All Characters:
     If Script(cp) .in. {Hiragana, Katakana, Han} Then True;

[edit] Appendix A.8. ARABIC-INDIC DIGITS

Code point:

  0660..0669

Overview:

  Can not be mixed with Extended Arabic-Indic Digits.

Lookup:

  False

Rule Set:

  True;
  For All Characters:
     If cp .in. 06F0..06F9 Then False;
  End For;

[edit] Appendix A.9. EXTENDED ARABIC-INDIC DIGITS

Code point:

  06F0..06F9

Overview:

  Can not be mixed with Arabic-Indic Digits.

Lookup:

  False

Rule Set:

  True;
  For All Characters:
     If cp .in. 0660..0669 Then False;
  End For;

[edit] Appendix B. Codepoints 0x0000 - 0x10FFFF

If one applies the rules (Section 3) to the code points 0x0000 to 0x10FFFF to Unicode 5.1, the result is as follows.

This list is non-normative, and only included for illustrative purposes. Specifically, what is displayed in the third column is not the formal name of the codepoint (as defined in section 4.8 of The Unicode Standard [Unicode51]). The differences exists for example for the codepoints that have the codepoint value as part of the name (example: CJK UNIFIED IDEOGRAPH-4E00) and the naming of Hangul syllables. For many codepoints, what you see is the official name.

[edit] Appendix B.1. Codepoints in Unicode Character Database (UCD) format

  	0000..002C	;	DISALLOWED	#	<control>..COMMA
  	002D	;	PVALID	#	HYPHEN-MINUS
  	002E..002F	;	DISALLOWED	#	FULL STOP..SOLIDUS
  	0030..0039	;	PVALID	#	DIGIT ZERO..DIGIT NINE
  	003A..0060	;	DISALLOWED	#	COLON..GRAVE ACCENT
  	0061..007A	;	PVALID	#	LATIN SMALL LETTER A..LATIN SMALL LETTER Z
  	007B..00B6	;	DISALLOWED	#	LEFT CURLY BRACKET..PILCROW SIGN
  	00B7	;	CONTEXTO	#	MIDDLE DOT