Implementing Cross-Locale CJKV Code Conversion

Ken Lunde

CJKV Type Development Adobe Systems Incorporated
ftp://ftp.oreilly.com/pub/examples/nutshell/ujip/unicode/iuc13-c2-paper.pdf ftp://ftp.oreilly.com/pub/examples/nutshell/ujip/unicode/iuc13-c2-slides.pdf

Code Conversion Basics

Algorithmic code conversion
Within a single locale: Shift-JIS, EUC-JP, and ISO-2022-JP A purely mathematical process
Table-driven code conversion
Required across locales: Chinese Japanese Required when dealing with Unicode Mapping tables are required Can sometimes be faster than algorithmic code conversion depends on the implementation

September 10, 1998

Copyright 1998 Adobe Systems Incorporated
Code Conversion Basics (Contd)
CJKV character set differences
Different number of characters Different ordering of characters Different characters
Character Sets Versus Encodings
Common CJKV character set standards
China: GB 1988-89, GB 2312-80; GB 1988-89, GBK Taiwan: ASCII, Big Five; CNS 5205-1989, CNS 11643-1992 Hong Kong: ASCII, Big Five with Hong Kong extension Japan: JIS X 0201-1997, JIS X 0208:1997, JIS X 0212-1990 South Korea: KS X 1003:1993, KS X 1001:1992, KS X 1002:1991 North Korea: ASCII (?), KPS 9566-97 Vietnam: TCVN 5712:1993, TCVN 5773:1993, TCVN 6056:1995

Common CJKV encodings

Locale-independent: EUC-*, ISO-2022-* Locale-specic: GBK, Big Five, Big Five Plus, Shift-JIS, Johab, Unied Hangul Code Other: UCS-2, UCS-4, UTF-7, UTF-8, UTF-16
September 10, 1998 Copyright 1998 Adobe Systems Incorporated
Chinese Character Relationships
Simplied and Traditional forms
Traditional forms still used in Taiwan and Korea Simplied forms used in Japan Greatly simplied forms used in China Simplied/Traditional relationship not always one-to-one Simplied/Traditional relationship is locale-specic

Variant forms

Alternate character forms with same semantics

Common forms

Luckily, many forms are still common across all CJKV locales:

OPQRSTUYZ

Advantages of Unicode
A common representation for all characters
Only 2n mapping tables required (where n equals the number of supported character set plus encoding combinations) Otherwise, n (n 1) mapping tables would be required.
Mapping tables are readily available at the following URL:

ftp://ftp.unicode.org/

Unicode is used as information interchange code
Handling Common Characters
Trivial effort that goes through Unicode

O P U Z

Consider KS X 1001:1992 to JIS X 0208:1997 conversion:

Glyph KS X 1001:1992

Unicode

JIS X 0208:1997 Glyph

76-73 63-03 79-03 90-76

4E00 5C8840

16-76 27-19 37-36 23-76
Handling Simplied/Traditional
More problematic because simplied/traditional databases are required
A Unicode code point does not reect such relationships!
Consider GB 2312-80 to JIS X 0208:1997 conversion:

Glyph GB 2312-80

Unicode'

26-58 26-26

9ED1 6C49

9ED2 6F22

25-85 20-33
Note how the Unicode representation is altered to acheive a successful conversion Japan and China share many simplied forms (such as =), which can result in more direct mappings
Handling Simplied/Traditional (Contd)
Word-level disambiguation is required Consider the traditional representations of :

itself

Disambiguation can be resolved through context

(miy) (mijn)

Handling Variants Forms
There are often more than one variant form for a given Chinese character Consider the following Japanese-specic character relationships (all in JIS X 0208:1997):
Standard Form Traditional Forms General Variants

E ? ) H

F @ * IJK

G ABCD 0 L

Handling The Compatibility Zone

GB 2312-80 Glyph

KS X 1001:1992 includes 268 duplicate hanja (with multiple readings)encoded in CJK Compatibility Zone Consider KS X 1001:1992 to GB 2312-80 conversion:

KS X 1001:1992

Unicode"

d d d d

49-66 53-05 68-37 72-89

F914 F95C 6A02 F9BF

6A02 6A02 6A02 6A02

4E50 4E50 4E50 4E50

32-54 32-54 32-54 32-54

w w w w

Note the two transformations in the Unicode representation
Normalize to the same code point (0x6A02) Convert to simplied hanzi (0x4E50)
Handling Unmappable Characters
Some characters are simply not available in the target character set plus encoding combination, not even as a variant form
Consider ] (JIS X 0208:1997 55-27, U+5F41) Consider Korean hangul
Output such characters as tags for round-trip code conversion purposes

Such as HTML/XML character references: 彁
Remove such characters from the output altogethernot terribly graceful
Avoiding Code Conversion Pitfalls
Same glyph but different semantics Consider GB 2312-80 to JIS X 0208:1997 conversion:

JIS X 0208:1997

38-88 38-88 38-88
6C14 6C14 6C14 6C23 6C23 6C17

61-67 21-04 61-70

The above illustrates three possible scenarios, depending on how the semantics of u (U+6C14) are interpreted
As a Radical? As a Simplied (standard) form? As a Traditional form? (also in JIS X 0208:1997)
Cross-Locale Code Conversion Tools

CJKVConv.pl

Developed in Perl by Ken Lunde Test vehicle to illustrate cross-locale code conversion issues ftp://ftp.oreilly.com/pub/examples/nutshell/ujip/perl/cjkvconv.pl

Uniconv

Developed by Basis Technology Supports a wide variety of character sets and encodings Provides many transformations http://www.basistech.com/unicode/