Standard Compression Scheme for Unicode
You can help expand this article with text translated from the corresponding article in German. (August 2013) Click [show] for important translation instructions.
|
The Standard Compression Scheme for Unicode (SCSU)
History & use
Reuters originally developed SCSU, then under the name RCSU for Reuters Compression Scheme for Unicode.[2][3][4][5]
At first the Unicode Consortium considered it to be a character encoding,[6] but in 1999 changed its mind: although it was still considered a transfer encoding syntax, for a while it was no longer considered a character encoding because different compressors might yield different outputs for the same text.[7] However, in 2004 this decision was reverted and now SCSU is considered a compressing character encoding scheme, as opposed to a simple or compound character encoding scheme.[8][9]
Roman Czyborra (of GNU Unifont) wrote a decompressor.[10] The IBM-contributed decompressor is found in International Components for Unicode, along with a compressor written in Java.[11] Simpler reference codecs are available as attachments to TR6.
The scheme
The following sections briefly describe the anatomy of a compressed SCSU stream. For a full description (matching that of a decompressor), see the UTS #6 document.
Encoding modes
SCSU starts in the single-byte mode, which uses the compressed Window encoding. There exist commands to switch to a UTF-16BE "Unicode" mode, and to switch to the single-byte mode from that mode.
Window encoding
The core of SCSU lies in the windows for which the meanings of bytes 0x80-0xff are defined. There are eight static windows for simpler scripts and punctuation, and 6 types of dynamic windows (plus "half Unicode block" windows and custom Windows for the supplementary planes) for scripts making use of more characters.
Both simple and dynamic windows are selected by special command characters. For individual characters that do not fit into the current block, command characters for quoting are provided.
Comparison with general-purpose plain text compression schemes
Because UTF-16 or UTF-8 text might occupy more space than its equivalent in pre-Unicode encodings did, one might want to use compression such as SCSU to mitigate this problem.[13] In comparison with general-purpose compressors, it is not necessarily advantageous to use SCSU.[5] Also, while it can be used as a text encoding, because of the stateful nature of the algorithm difficulties may arise when using it as an internal text representation since basic text operations become non-trivial.
Treated purely as a compression algorithm, SCSU is inferior to most commonly used general-purpose algorithms for texts of over a few kilobytes.
SCSU does have the advantage that it can usefully compress texts that are only a few characters long, whereas most full-scale compressors need hundreds of bytes of data to break even against their own overhead. In
In HTML
Supporting SCSU in
See also
- Binary Ordered Compression for Unicode (BOCU-1)
- International Components for Unicode A library that can convert between SCSU and other Unicode encodings
References
- ^ "UTS #6: Compression Scheme for Unicode". 2005-05-06. Retrieved 2008-06-13.
SCSU defines a compact encoding, which is sometimes useful. However, Unicode text is much more commonly stored and transmitted in UTF-8 which is less compact (except for ASCII), much simpler, and does not present any security issues. For longer texts, general-purpose compression is effective and common.
- ^ "Ninth International Unicode Conference - Friday - Track B".
- ^ "Tenth International Unicode Conference - Conference Program".
- ^ "Compression Scheme for Unicode".
- ^ a b "A survey of Unicode compression" (PDF).
- ^ "UTR#17: Character Encoding Model".
- ^ https://unicode.org/reports/tr17/tr17-3.html#Transfer Encoding Syntax
- ^ "UTR#17: Character Encoding Model". 2004-07-14.
- ^ "UTR#17: Unicode Character Encoding Model". unicode.org. Retrieved 2023-11-14.
- ^ https://czyborra.com/scsu/scsu.c [bare URL plain text file]
- ^ "International Components for Unicode". GitHub. 22 October 2021.
- ^ "Unicode Compression Implementation (SQL Server 2008 R2 Books Online)". Retrieved 2008-08-18.
- ^ https://unicode.org/versions/Unicode3.0.0/ch05.pdf [bare URL PDF]
- ^ "8.2.2.3. Character encodings". HTML 5.1 Standard. W3C.
- ^ "8.2.2.3. Character encodings". HTML 5 Standard. W3C.
- ^ "12.2.3.3 Character encodings". HTML Living Standard. WHATWG.
- ^ "<meta> - HTML". MDN Web Docs. Mozilla. Archived from the original on 3 October 2018.