Working with Binary Data

Python
import struct

# Pack integers into binary data (big-endian, 2 ints)
packed = struct.pack('>2i', 1, 2)  # b'\x00\x00\x00\x01\x00\x00\x00\x02'

# Unpack binary data back to Python values
values = struct.unpack('>2i', packed)  # (1, 2)

# Serialize any Python object with pickle
import pickle
data = pickle.dumps({'key': 'value', 'nums': [1, 2, 3]})
obj = pickle.loads(data)

Choose the right module based on the task:

Progress:

• Identify data source/target: fixed-format binary (C structs, network protocols) vs. arbitrary Python objects
• Choose module: struct for binary layouts, pickle for object serialization, shelve for persistent dict-like storage, marshal only for internal .pyc use (never for untrusted/external data)
• For custom classes with pickle, implement __reduce__ or register with copyreg if default pickling fails
• Define format strings (struct) or protocol version (pickle)
• Handle byte order, size, and alignment explicitly for struct
• Never unpickle data from untrusted sources — treat like eval()

struct — Binary layouts

Format strings define byte order + field types:

Common format codes: b/B (signed/unsigned char), h/H (short), i/I (int), l/L (long), q/Q (long long), f/d (float/double), s (char array), ? (bool).

Python
struct.pack('<3sB', b'abc', 255)     # bytes with prefix count
struct.calcsize('>4i')               # size in bytes without packing
struct.unpack_from(buffer, offset=0) # unpack from a slice of a larger buffer

Use struct.Struct(fmt) when reusing the same format repeatedly — precompiles it for speed.

pickle — Object serialization

Python
with open('data.pkl', 'wb') as f:
    pickle.dump(obj, f, protocol=pickle.HIGHEST_PROTOCOL)

with open('data.pkl', 'rb') as f:
    obj = pickle.load(f)

• Use pickle.HIGHEST_PROTOCOL unless cross-version compatibility is required.
• Not all objects are picklable by default (open files, sockets, lambdas, some C extension objects). Implement __getstate__/__setstate__ or __reduce__ on the class.

copyreg — Extending pickle for classes you don't control

Python
import copyreg

def pickle_point(p):
    return (Point, (p.x, p.y))

copyreg.pickle(Point, pickle_point)

shelve — Persistent dict-backed storage

Python
import shelve

with shelve.open('mydata') as db:
    db['key'] = {'complex': 'object'}
    value = db['key']

Values are pickled automatically; keys must be strings.

marshal — Internal use only

Reserved for Python's own .pyc bytecode serialization. Not stable across versions, not safe for untrusted input. Do not use for application-level serialization — use pickle instead.

Example 1: Parsing a binary file header Input: A binary file starts with a 4-byte magic number, 2-byte version (uint16), 4-byte length (uint32), all big-endian. Output:

Python
import struct
HEADER_FMT = '>4sHI'
HEADER_SIZE = struct.calcsize(HEADER_FMT)  # 10 bytes

with open('file.bin', 'rb') as f:
    magic, version, length = struct.unpack(HEADER_FMT, f.read(HEADER_SIZE))

Example 2: Caching computed results to disk across runs Input: An expensive function whose results should persist between program executions, keyed by input string. Output:

Python
import shelve

def get_cached(key, compute_fn):
    with shelve.open('cache.db') as db:
        if key not in db:
            db[key] = compute_fn(key)
        return db[key]

Example 3: Deep-copying a complex nested object graph Input: An object with circular references and custom classes that needs a full deep copy. Output:

Python
import pickle
def deep_copy(obj):
    return pickle.loads(pickle.dumps(obj))