Packing Binary Data with Struct

Python
import struct

# Pack: 1 unsigned short, 1 signed int, 4-byte string
data = struct.pack('>Hi4s', 512, -42, b'ABCD')

# Unpack back
h, i, s = struct.unpack('>Hi4s', data)
# h=512, i=-42, s=b'ABCD'

# Calculate size before allocating buffers
size = struct.calcsize('>Hi4s')  # 10

Progress:

• Identify the binary layout (field order, types, sizes) — from spec/docs/C struct definition
• Choose byte order/alignment prefix
• Build the format string
• Pack or unpack; verify with calcsize
• Handle variable-length data (strings, arrays) explicitly
• Test round-trip (pack → unpack → compare)

1. Choose byte order & alignment prefix

Default recommendation: use < or > explicitly (never rely on @) when the data is persisted to disk or sent over a network — native alignment causes portability bugs.

2. Map format characters

Repeat count prefix applies per-field (e.g. 4h = 4 shorts), except s/p where the count is the field length, not a repeat.

3. Pack / unpack

Python
struct.pack(fmt, v1, v2, ...)        # -> bytes
struct.unpack(fmt, buffer)           # -> tuple, buffer size must match calcsize(fmt)
struct.unpack_from(fmt, buffer, offset=0)  # unpack from a larger buffer without slicing
struct.pack_into(fmt, buffer, offset, *values)  # write into an existing mutable buffer

4. Reuse compiled formats for performance

Python
Header = struct.Struct('<4sHHI')  # magic, ver_major, ver_minor, length
data = Header.pack(b'MAGC', 1, 0, 128)
magic, vmaj, vmin, length = Header.unpack(data)
print(Header.size)  # equivalent to calcsize

Use Struct objects instead of repeated struct.pack(fmt, ...) calls when the same format is used many times (e.g. parsing many records in a loop).

Example 1: Reading a fixed-size binary file header Input: File header spec — 4-byte magic, uint32 version (little-endian), uint32 payload length. Output:

Python
HEADER_FMT = '<4sII'
HEADER_SIZE = struct.calcsize(HEADER_FMT)  # 12

with open('file.bin', 'rb') as f:
    magic, version, length = struct.unpack(HEADER_FMT, f.read(HEADER_SIZE))
    payload = f.read(length)

Example 2: Parsing a network packet with mixed types Input: Packet = 1 byte type, 1 byte flags, 2-byte big-endian sequence number, 8-byte big-endian timestamp. Output:

Python
PKT_FMT = '!BBHQ'
pkt_type, flags, seq, ts = struct.unpack(PKT_FMT, raw_bytes)

Example 3: Packing an array of records Input: 3 points, each (x: float, y: float) little-endian. Output:

Python
fmt = '<' + 'ff' * 3
data = struct.pack(fmt, 0.0, 0.0, 1.5, 2.5, -1.0, 3.0)

Example 4: Unpacking from a larger buffer at an offset Input: A 1024-byte buffer where a 16-byte record starts at offset 100. Output:

Python
record = struct.unpack_from('<4sIII', buffer, offset=100)