Character encoding

Character encoding is the process of assigning numbers to graphical characters, especially the written characters of human language, allowing them to be stored, transmitted, and transformed using computers.^[1] The numerical values that make up a character encoding are known as code points and collectively comprise a code space, a code page, or character map.

Early character encodings that originated with optical or electrical telegraphy and in early computers could only represent a subset of the characters used in written languages, sometimes restricted to upper case letters, numerals and some punctuation only. Over time, character encodings capable of representing more characters were created, such as ASCII, the ISO/IEC 8859 encodings, various computer vendor encodings, and Unicode encodings such as UTF-8 and UTF-16.

The most popular character encoding on the World Wide Web is UTF-8, which is used in 98.2% of surveyed web sites, as of May 2024.^[2] In application programs and operating system tasks, both UTF-8 and UTF-16 are popular options.^[3]

The history of character codes illustrates the evolving need for machine-mediated character-based symbolic information over a distance, using once-novel electrical means. The earliest codes were based upon manual and hand-written encoding and cyphering systems, such as Bacon's cipher, Braille, international maritime signal flags, and the 4-digit encoding of Chinese characters for a Chinese telegraph code (Hans Schjellerup, 1869). With the adoption of electrical and electro-mechanical techniques these earliest codes were adapted to the new capabilities and limitations of the early machines. The earliest well-known electrically transmitted character code, Morse code, introduced in the 1840s, used a system of four "symbols" (short signal, long signal, short space, long space) to generate codes of variable length. Though some commercial use of Morse code was via machinery, it was often used as a manual code, generated by hand on a telegraph key and decipherable by ear, and persists in amateur radio and aeronautical use. Most codes are of fixed per-character length or variable-length sequences of fixed-length codes (e.g. Unicode).^[4]

Common examples of character encoding systems include Morse code, the

The Baudot code, a five-bit encoding, was created by Émile Baudot in 1870, patented in 1874, modified by Donald Murray in 1901, and standardized by CCITT as International Telegraph Alphabet No. 2 (ITA2) in 1930. The name baudot has been erroneously applied to ITA2 and its many variants. ITA2 suffered from many shortcomings and was often improved by many equipment manufacturers, sometimes creating compatibility issues.

Herman Hollerith invented punch card data encoding in the late 19th century to analyze census data. Initially, each hole position represented a different data element, but later, numeric information was encoded by numbering the lower rows 0 to 9, with a punch in a column representing its row number. Later alphabetic data was encoded by allowing more than one punch per column. Electromechanical tabulating machines represented date internally by the timing of pulses relative to the motion of the cards through the machine.

When

In trying to develop universally interchangeable character encodings, researchers in the 1980s faced the dilemma that, on the one hand, it seemed necessary to add more bits to accommodate additional characters, but on the other hand, for the users of the relatively small character set of the Latin alphabet (who still constituted the majority of computer users), those additional bits were a colossal waste of then-scarce and expensive computing resources (as they would always be zeroed out for such users). In 1985, the average personal computer user's hard disk drive could store only about 10 megabytes, and it cost approximately US$250 on the wholesale market (and much higher if purchased separately at retail),^[8] so it was very important at the time to make every bit count.

The compromise solution that was eventually found and developed into Unicode^[vague] was to break the assumption (dating back to telegraph codes) that each character should always directly correspond to a particular sequence of bits. Instead, characters would first be mapped to a universal intermediate representation in the form of abstract numbers called code points. Code points would then be represented in a variety of ways and with various default numbers of bits per character (code units) depending on context. To encode code points higher than the length of the code unit, such as above 256 for eight-bit units, the solution was to implement variable-length encodings where an escape sequence would signal that subsequent bits should be parsed as a higher code point.

Informally, the terms "character encoding", "character map", "character set" and "code page" are often used interchangeably.^[9] Historically, the same standard would specify a repertoire of characters and how they were to be encoded into a stream of code units — usually with a single character per code unit. However, due to the emergence of more sophisticated character encodings, the distinction between these terms has become important.

• A character is a minimal unit of text that has semantic value.^[9][10]
• A character set is a collection of elements used to represent text.^[9][10] For example, the Latin alphabet and Greek alphabet are both character sets.
• A coded character set is a character set mapped to a set of unique numbers.^[10] For historical reasons, this is also often referred to as a code page.^[9]
• A character repertoire is the set of characters that can be represented by a particular coded character set.^[10][11] The repertoire may be closed, meaning that no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series); or it may be open, allowing additions (as is the case with Unicode and to a limited extent Windows code pages).^[11]
• A code point is a value or position of a character in a coded character set.^[10]
• A code space is the range of numerical values spanned by a coded character set.^[10][12]
• A code unit is the minimum bit combination that can represent a character in a character encoding (in computer science terms, it is the word size of the character encoding).^[10][12] For example, common code units include 7-bit, 8-bit, 16-bit, and 32-bit. In some encodings, some characters are encoded using multiple code units; such an encoding is referred to as a variable-width encoding.

"Code page" is a historical name for a coded character set.

Originally, a code page referred to a specific

The term "code page" is not used in Unix or Linux, where "charmap" is preferred, usually in the larger context of locales. IBM's Character Data Representation Architecture (CDRA) designates entities with coded character set identifiers (CCSIDs), each of which is variously called a "charset", "character set", "code page", or "CHARMAP".^[12]

The code unit size is equivalent to the bit measurement for the particular encoding:

• A code unit in ASCII consists of 7 bits;
• A code unit in UTF-8, EBCDIC and GB 18030 consists of 8 bits;
• A code unit in UTF-16 consists of 16 bits;
• A code unit in UTF-32 consists of 32 bits.

A code point is represented by a sequence of code units. The mapping is defined by the encoding. Thus, the number of code units required to represent a code point depends on the encoding:

• UTF-8: code points map to a sequence of one, two, three or four code units.
• UTF-16: code units are twice as long as 8-bit code units. Therefore, any code point with a scalar value less than U+10000 is encoded with a single code unit. Code points with a value U+10000 or higher require two code units each. These pairs of code units have a unique term in UTF-16: "Unicode surrogate pairs".
• UTF-32: the 32-bit code unit is large enough that every code point is represented as a single code unit.
• GB 18030: multiple code units per code point are common, because of the small code units. Code points are mapped to one, two, or four code units.^[14]

Exactly what constitutes a character varies between character encodings.

For example, for letters with

Exactly how to handle glyph variants is a choice that must be made when constructing a particular character encoding. Some writing systems, such as Arabic and Hebrew, need to accommodate things like graphemes that are joined in different ways in different contexts, but represent the same semantic character.

A coded character set (CCS) is a function that maps characters to code points (each code point represents one character). For example, in a given repertoire, the capital letter "A" in the Latin alphabet might be represented by the code point 65, the character "B" by 66, and so on. Multiple coded character sets may share the same character repertoire; for example ISO/IEC 8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different code points.

A character encoding form (CEF) is the mapping of code points to code units to facilitate storage in a system that represents numbers as bit sequences of fixed length (i.e. practically any computer system). For example, a system that stores numeric information in 16-bit units can only directly represent code points 0 to 65,535 in each unit, but larger code points (say, 65,536 to 1.4 million) could be represented by using multiple 16-bit units. This correspondence is defined by a CEF.

A character encoding scheme (CES) is the mapping of code units to a sequence of octets to facilitate storage on an octet-based file system or transmission over an octet-based network. Simple character encoding schemes include

Finally, there may be a higher-level protocol which supplies additional information to select the particular variant of a Unicode character, particularly where there are regional variants that have been 'unified' in Unicode as the same character. An example is the XML attribute xml:lang.

The Unicode model uses the term "character map" for other systems which directly assign a sequence of characters to a sequence of bytes, covering all of the CCS, CEF and CES layers.^[12]

In Unicode, a character can be referred to as 'U+' followed by its codepoint value in hexadecimal. The range of valid code points (the codespace) for the Unicode standard is U+0000 to U+10FFFF, inclusive, divided in 17

The following table shows examples of code point values:

Consider a string of the letters "ab̲c𐐀"—that is, a string containing a Unicode combining character (U+0332 ̲ COMBINING LOW LINE) as well as a supplementary character (U+10400 𐐀 DESERET CAPITAL LETTER LONG I). This string has several Unicode representations which are logically equivalent, yet while each is suited to a diverse set of circumstances or range of requirements:

• Four composed characters:

  a, b̲, c, 𐐀

• Five graphemes:

  a, b, _, c, 𐐀

• Five Unicode code points:

  U+0061, U+0062, U+0332, U+0063, U+10400

• Five UTF-32 code units (32-bit integer values):

  0x00000061, 0x00000062, 0x00000332, 0x00000063, 0x00010400

• Six UTF-16 code units (16-bit integers)

  0x0061, 0x0062, 0x0332, 0x0063, 0xD801, 0xDC00

• Nine UTF-8 code units (8-bit values, or bytes)

  0x61, 0x62, 0xCC, 0xB2, 0x63, 0xF0, 0x90, 0x90, 0x80

Note in particular that 𐐀 is represented with either one 32-bit value (UTF-32), two 16-bit values (UTF-16), or four 8-bit values (UTF-8). Although each of those forms uses the same total number of bits (32) to represent the glyph, it is not obvious how the actual numeric byte values are related.

As a result of having many character encoding methods in use (and the need for backward compatibility with archived data), many computer programs have been developed to translate data between character encoding schemes, a process known as transcoding. Some of these are cited below.

• character encoding detection
  . On Firefox 3, for example, see the View/Character Encoding submenu.
• iconv – a program and standardized API to convert encodings
• luit – a program that converts encoding of input and output to programs running interactively
• International Components for Unicode – A set of C and Java libraries to perform charset conversion. uconv can be used from ICU4C.

Windows:

• Encoding.Convert – .NET API^[15]
• MultiByteToWideChar/WideCharToMultiByte – to convert from ANSI to Unicode & Unicode to ANSI^[16][17]

This section needs expansion with: Popularity and comparison: Statistics on popularity Especially, a comparison of the advantages and disadvantages of the few 3-5 most common character encodings (e.g. UTF-8, UTF-16 and UTF-32). You can help by adding to it. (June 2024)

The most used character encoding on the web is UTF-8, used in 98.2% of surveyed web sites, as of May 2024.^[2] In application programs and operating system tasks, both UTF-8 and UTF-16 are popular options.^[3][18]

• Percent-encoding
• Alt code
• Character encodings in HTML
• Category:Character encoding – articles related to character encoding in general
• Category:Character sets – articles detailing specific character encodings
• Hexadecimal representations
• Mojibake – character set mismap
• Mojikyō – a system ("glyph set") that includes over 100,000 Chinese character drawings, modern and ancient, popular and obscure
• Presentation layer
• TRON, part of the TRON project, is an encoding system that does not use Han Unification; instead, it uses "control codes" to switch between 16-bit "planes" of characters.
• Universal Character Set characters
• Charset sniffing
  – used in some applications when character encoding metadata is not available

Further reading

• Mackenzie, Charles E. (1980). Coded Character Sets, History and Development (PDF). The Systems Programming Series (1 ed.).
  LCCN 77-90165. Archived
  (PDF) from the original on May 26, 2016. Retrieved August 25, 2019.

External links

Wikimedia Commons has media related to Encodings.

Wikiversity has learning resources about Character encoding

• Character sets registered by Internet Assigned Numbers Authority (IANA)
• Characters and encodings, by Jukka Korpela
• Unicode Technical Report #17: Character Encoding Model
• Decimal, Hexadecimal Character Codes in HTML Unicode – Encoding converter
• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky (Oct 10, 2003)