Binary Ordered Compression for Unicode
Binary Ordered Compression for Unicode (BOCU) is a MIME compatible Unicode compression scheme. BOCU-1 combines the wide applicability of UTF-8 with the compactness of Standard Compression Scheme for Unicode (SCSU). This Unicode encoding is designed to be useful for compressing short strings, and maintains code point order. BOCU-1 is specified in a Unicode Technical Note.[1]
For comparison SCSU was adopted as standard Unicode compression scheme with a byte/code point ratio similar to language-specific
Both SCSU[3] and BOCU-1[4] are IANA registered charsets.
Details
All numbers in this section are hexadecimal, and all ranges are inclusive.
Code points from U+0000
to U+0020
are encoded in BOCU-1 as the corresponding byte value. All other code points (that is, U+0021
through U+D7FF
and U+E000
through U+10FFFF
) are encoded as a difference between the code point and a normalized version of the most recently encoded code point that was not an ASCII space (U+0020
). The initial state is U+0040
. The normalization mapping is as follows:
Code range | Normalized code point | Notes |
---|---|---|
U+3040 to U+309F
|
U+3070
|
Hiragana |
U+4E00 to U+9FA5
|
U+7711
|
Unihan |
U+AC00 to U+D7A3
|
U+C1D1
|
Hangul |
U+0020
|
encoder state kept as is | Space |
U+hhhh00 to U+hhhh7F (excluding ranges above) |
U+hhhh40
|
middle of 128 |
U+hhhh80 to U+hhhhFF (excluding ranges above) |
U+hhhhC0
|
middle of 128 |
The difference between the current code point and the normalized previous code point is encoded as follows:
Difference range | Byte sequence range (see below) |
---|---|
-10FF9F to -2DD0D
|
21 F0 58 D9 to 21 FF FF FF
|
-2DD0C to -2912
|
22 01 01 to 24 FF FF
|
-2911 to -41
|
25 01 to 4F FF
|
-40 to 3F
|
50 to CF
|
40 to 2910
|
D0 01 to FA FF
|
2911 to 2DD0B
|
FB 01 01 to FD FF FF
|
2DD0C to 10FFBF
|
FE 01 01 01 to FE 19 B4 54
|
Each byte range is
00 07 08 09 0A 0B 0C 0D 0E 0F 1A 1B 20
. For example, the byte sequence FC 06 FF
, coding for a difference of 1156B
, is immediately followed by the byte sequence FC 10 01
, coding for a difference of 1156C
.
Any ASCII input U+0000
to U+007F
excluding space U+0020
resets the encoder to U+0040
. Because the above-mentioned values cover line end code points U+000D
and U+000A
as is (0D 0A
), the encoder is in a known state at the begin of each line. The corruption of a single byte therefore affects at most one line. For comparison, the corruption of a single byte in UTF-8 affects at most one code point, for SCSU it can affect the entire document.
BOCU-1 offers a similar robustness also for input texts without the above-mentioned values with the special reset code 0xFF
. When a decoder finds this octet it resets its state to U+0040
as for a line end. The use of 0xFF
reset bytes is not recommended in the BOCU-1 specification, because it conflicts with other BOCU-1 design goals, notably the binary order.
The optional use of a signature
U+FEFF
FB EE 28
, changes the initial state U+0040
to U+FEC0
. In other words, the signature cannot simply be stripped as in most other Unicode encoding schemes. Adding a reset byte after the signature (FB EE 28 FF
) could avoid this effect, but the BOCU-1 specification does not recommend this practice.
In theory
7FFFFFFF
. BOCU-1 and UTF-16U+0000
to U+10FFFF
. Excluding the thirteen protected code points encoded as single octets BOCU-1 can use octets in multi-byte encodings. BOCU-1 needs at most four bytes consisting of a lead byte and one to three trail bytes. The trail bytes encode a remaining "0xFF
is not protected and can occur as trail byte.
Patent
Prior to 16 November 2022, the general BOCU algorithm was covered by United States Patent #6,737,994, which also mentions the specific BOCU-1 implementation.[5] This patent has now expired.
IBM, which employed both of the inventors of BOCU-1 at the time it was created, stated in the Unicode Technical Note that implementers of a "fully compliant version of BOCU-1" had to contact IBM to request a royalty-free license.[6] BOCU-1 is the only Unicode compression scheme described on the Unicode Web site that is known to have been encumbered with intellectual property restrictions.
By contrast, IBM also filed for a patent on UTF-EBCDIC, but it chose in that case to make the documentation and encoding scheme "freely available to anyone concerned towards making the transformation format as part of the UCS standards", instead of requiring implementers to request a license.[7]
References
- ^ Markus Scherer, Mark Davis (2006-02-04). "UTN #6: BOCU-1". Retrieved 2008-05-18.
- ^ Ewell, Doug (2004-01-30). "UTN #14: A survey of Unicode compression" (PDF). Retrieved 2008-06-13.
- ^ IANA registration record for SCSU
- ^ IANA registration record for BOCU-1
- ^ Davis; et al. (2004-05-18). "United States Patent #6,737,994, "Binary-ordered compression for unicode"". Retrieved 2022-12-28.
- ^ Markus Scherer, Mark Davis (2006-02-04). "UTN #6: BOCU-1". Retrieved 2014-02-05.
- ^ V.S. Umamaheswaran (2002-04-16). "UTR #16: UTF-EBCDIC". Retrieved 2008-11-16.
See also
- UTF-1 contains a comparison of the UTF-1, UTF-8, and BOCU-1 designs
- International Components for Unicode A library that can convert between BOCU-1 and other Unicode encodings