Querying Unicode Character Data

Python
import unicodedata

# Basic lookups
unicodedata.name('€')          # 'EURO SIGN'
unicodedata.category('A')      # 'Lu' (Letter, uppercase)
unicodedata.decimal('9')       # 9
unicodedata.numeric('½')       # 0.5

# Normalization (most common real-world use)
s1 = unicodedata.normalize('NFC', 'café')   # composed form
s2 = unicodedata.normalize('NFD', 'café')   # decomposed form
s1 == s2  # False — same visual string, different byte representation!

# Safe comparison of visually-equivalent strings
def unicode_equal(a, b):
    return unicodedata.normalize('NFC', a) == unicodedata.normalize('NFC', b)

Progress:

• Identify the problem type: comparison, validation, stripping, or inspection
• Choose the right normalization form (NFC/NFD/NFKC/NFKD)
• Apply the appropriate function(s)
• Handle ValueError for unassigned/invalid codepoints
• Test with edge cases (combining marks, surrogate pairs, CJK, emoji)

1. Diagnose what's actually needed

2. Pick the right normalization form

• NFC — canonical composition. Default choice for storage/comparison/general use.
• NFD — canonical decomposition. Use before stripping accents or doing per-character analysis.
• NFKC / NFKD — compatibility forms. Use when visually/semantically equivalent but differently-encoded characters (full-width "Ａ" vs "A", ligature "fi" vs "fi") should be treated as the same. Lossy — don't use for round-tripping.

3. Apply functions

Python
unicodedata.lookup('LATIN SMALL LETTER A')   # 'a' — name to char
unicodedata.name(chr(0x1F600))               # char to name
unicodedata.category(c)                      # 'Lu','Ll','Nd','Zs','Cc', etc.
unicodedata.bidirectional(c)                 # 'L','R','AL','EN', etc.
unicodedata.combining(c)                     # canonical combining class (0 = not combining)
unicodedata.mirrored(c)                      # 1 if char is mirrored in bidi text
unicodedata.east_asian_width(c)              # 'W','Na','A','H','F','N'
unicodedata.decomposition(c)                 # raw decomposition mapping string
unicodedata.is_normalized('NFC', s)          # fast check without allocating result

Example 1: Strip accents/diacritics from text Input: "café naïve"

Python
def strip_accents(s):
    nfd = unicodedata.normalize('NFD', s)
    return ''.join(c for c in nfd if unicodedata.category(c) != 'Mn')

strip_accents("café naïve")

Output: "cafe naive"

Example 2: Compare user input for equality regardless of composition Input: "e\u0301" (e + combining acute accent) vs "\u00e9" (é precomposed)

Python
a, b = "e\u0301", "\u00e9"
a == b                                          # False
unicodedata.normalize('NFC', a) == unicodedata.normalize('NFC', b)  # True

Output: False then True

Example 3: Validate that a string is all decimal digits (including non-ASCII) Input: "٣٤٥" (Arabic-Indic digits)

Python
def all_decimal(s):
    try:
        return all(unicodedata.decimal(c) is not None for c in s)
    except ValueError:
        return False

[unicodedata.decimal(c) for c in "٣٤٥"]

Output: [3, 4, 5]

Example 4: Inspect a mystery character for debugging Input: '\u200b' (appeared unexpectedly in scraped text)

Python
c = '\u200b'
unicodedata.name(c), unicodedata.category(c)

Output: ('ZERO WIDTH SPACE', 'Cf') — reveals it's a formatting control char, explains invisible-but-present-length bugs.