Processing Text with Python
Pythonimport re import textwrap from difflib import unified_diff from string import Template # Regex matching match = re.search(r'(\d+)-(\d+)', "range: 10-20") if match: start, end = match.groups() # Text wrapping print(textwrap.fill("A very long line of text that needs wrapping", width=20)) # String templating t = Template("Hello, $name!") print(t.substitute(name="World")) # Diffing diff = unified_diff("line1\nline2\n".splitlines(), "line1\nline3\n".splitlines(), lineterm="") print("\n".join(diff))
Progress:
- Step 1: Identify the text task category (matching, formatting, comparing, normalizing, constants)
- Step 2: Select the right module (see mapping below)
- Step 3: Write minimal, correct code using the module's idiomatic API
- Step 4: Handle edge cases (empty strings, Unicode, unmatched patterns)
- Step 5: Test with representative inputs
Module Selection Map
| Task | Module | Key API |
|---|---|---|
| Pattern matching/extraction | re | re.match, re.search, re.findall, re.sub, re.compile |
| String constants (ascii letters, digits, punctuation) | string | string.ascii_letters, string.digits, string.punctuation |
| Template-based substitution | string.Template | .substitute(), .safe_substitute() |
| Wrapping/indenting paragraphs | textwrap | textwrap.wrap, textwrap.fill, textwrap.dedent, textwrap.indent |
| Comparing sequences/text | difflib | SequenceMatcher, unified_diff, get_close_matches |
| Unicode normalization | unicodedata | unicodedata.normalize, unicodedata.category |
| Internationalized string prep (rare, legacy) | stringprep | used internally by encodings.idna |
| Interactive completion (REPL only) | rlcompleter | rarely used directly in scripts |
Example 1: Extract and validate structured data with re
Input: Extract all email addresses from a block of text.
Pythonimport re text = "Contact: alice@example.com or bob@test.org" emails = re.findall(r'[\w.+-]+@[\w-]+\.[\w.-]+', text)
Output: ['alice@example.com', 'bob@test.org']
Example 2: Wrap CLI help text with textwrap
Input: A long description string that must fit an 80-column terminal, indented as a bullet.
Pythonimport textwrap desc = "This tool processes large datasets efficiently using streaming." wrapped = textwrap.fill(desc, width=40, initial_indent="- ", subsequent_indent=" ")
Output:
- This tool processes large datasets
efficiently using streaming.
Example 3: Fuzzy-match user input against known commands with difflib
Input: User types "stauts", available commands are ["status", "start", "stop"].
Pythonfrom difflib import get_close_matches suggestion = get_close_matches("stauts", ["status", "start", "stop"], n=1)
Output: ['status']
Example 4: Safe templating with missing keys Input: Template with a placeholder not present in the substitution dict.
Pythonfrom string import Template t = Template("$greeting, $name!") result = t.safe_substitute(name="Sam")
Output: '$greeting, Sam!' (no KeyError, unlike .substitute())
Example 5: Normalize Unicode for comparison
Input: Compare "café" (composed) vs "café" (decomposed, e + combining accent).
Pythonimport unicodedata a = unicodedata.normalize('NFC', "café") b = unicodedata.normalize('NFC', "cafe\u0301") a == b
Output: True
- Compile regex once with
re.compile()when reused in loops; use raw strings (r"...") always. - Prefer
re.fullmatchoverre.matchwhen validating an entire string against a pattern. - Use
Template.safe_substituteover.substitute()when input keys aren't guaranteed complete — avoids exceptions in user-facing tools. - Use
textwrap.shorteninstead of manual truncation for building "..." previews with word-boundary awareness. - Normalize Unicode (
NFCtypically) before any string equality check or hashing involving user-supplied international text. - Use
difflib.SequenceMatcher.ratio()for similarity scoring, not justget_close_matches, when you need a numeric score. - Avoid greedy
.*in regex — use.*?or explicit character classes to prevent catastrophic backtracking on large inputs.
- Don't use
refor full HTML/XML parsing — use a real parser (html.parser,xml.etree, orBeautifulSoup); regex breaks on nested/irregular structures. - Don't forget
re.MULTILINEwhen using^/$on multi-line text — default only matches start/end of the whole string. - Don't compare Unicode strings without normalizing first — visually identical strings can differ byte-wise and fail
==. - Don't use
string.Templatefor HTML/code generation with untrusted input without escaping — it does plain substitution, no escaping. - Don't call
textwrap.wrapexpecting it to preserve existing line breaks — it collapses whitespace by default (usereplace_whitespace=Falseto preserve). - Don't rely on
rlcompleterorstringprepdirectly in application code — they're low-level building blocks for other stdlib features (readline, IDNA encoding).