AI Skill Report Card
Forging Byte Level Datasets
Quick Start14 / 15
Pythonimport numpy as np from pathlib import Path VECTOR_LEN = 256 PAD_VALUE = 256 # out-of-byte-range sentinel for "no data" (use 0 if strict uint8 needed) def file_to_vectors(path: Path, vec_len: int = VECTOR_LEN) -> np.ndarray: data = path.read_bytes() n_chunks = (len(data) + vec_len - 1) // vec_len padded = data.ljust(n_chunks * vec_len, b"\x00") arr = np.frombuffer(padded, dtype=np.uint8).reshape(n_chunks, vec_len) return arr def forge_dataset(input_dir: str, out_path: str): vectors = [] labels = [] for f in sorted(Path(input_dir).rglob("*")): if f.is_file(): vecs = file_to_vectors(f) vectors.append(vecs) labels.extend([f.parent.name] * len(vecs)) dataset = np.concatenate(vectors, axis=0) np.savez_compressed(out_path, data=dataset, labels=np.array(labels)) return dataset.shape forge_dataset("./corpus", "byteforge_256.npz")
Recommendation▾
Add a concrete example of sliding-window code implementation, not just described output, to match the Quick Start code depth
Workflow14 / 15
Progress:
- Step 1: Inventory input files — total size, file-type distribution, average length
- Step 2: Decide chunking strategy (non-overlapping windows vs. sliding window with stride)
- Step 3: Choose padding/truncation policy for final chunk of each file
- Step 4: Vectorize files into
uint8arrays shaped(n_chunks, 256) - Step 5: Attach metadata (source file, offset, label) alongside each vector
- Step 6: Shuffle and split into train/val/test preserving file-level grouping (never split a file's chunks across sets)
- Step 7: Serialize to disk (
.npz,.npymemmap, or sharded.tfrecord/.parquetfor scale) - Step 8: Validate — spot-check reconstructed bytes match source, confirm shape and dtype
Recommendation▾
Include a small worked example showing the metadata/provenance array structure alongside vectors
Examples16 / 20
Example 1:
Input: Directory ./samples/ with 3 files: a.exe (612 bytes), b.pdf (1050 bytes), c.txt (40 bytes)
Output:
a.exe -> 3 vectors (612 bytes padded to 768, last vector 156 real bytes + 100 zero-pad)
b.pdf -> 5 vectors (1050 bytes padded to 1280, last vector 26 real bytes + 230 zero-pad)
c.txt -> 1 vector (40 real bytes + 216 zero-pad)
Total dataset shape: (9, 256), dtype=uint8
Example 2: Input: Sliding-window mode, stride=64, on a 500-byte file Output:
Window starts: 0, 64, 128, 192, 244 (last window clamped so it doesn't overrun; end-aligned)
5 overlapping vectors of shape (256,) each
Example 3:
Input: Malware classifier prep — benign/ and malicious/ subfolders
Output: Dataset with labels array of "benign"/"malicious" aligned 1:1 with each 256-byte vector; file-level split ensures no data leakage between train/test.
Recommendation▾
Note the PAD_VALUE=256 default conflicts with uint8 dtype used in code (256 overflows uint8) — clarify or fix this inconsistency
Best Practices
- Group by source file when splitting — chunks from the same file are correlated; leaking them across train/test inflates metrics.
- Record provenance — store
(source_path, byte_offset, chunk_index)per vector for debugging and reconstruction. - Prefer zero-padding for the tail chunk, but flag padded positions with an optional mask array if the model needs to distinguish real vs. pad bytes.
- Use memory-mapped
.npyor sharded formats once corpus exceeds available RAM — don't hold everything in one in-memory array. - Deduplicate identical files/chunks before forging to avoid training bias from copies.
- Normalize file discovery order (sorted paths) for reproducible dataset builds across runs.
- Consider overlapping windows (stride < 256) when data is scarce, to increase effective sample count — document the stride used.
Common Pitfalls
- Splitting chunks from the same file across train/val/test — causes data leakage.
- Using
PAD_VALUEoutside[0,255]if downstream expects strictuint8— will silently overflow/wrap. - Forgetting to skip non-regular files (symlinks, directories, empty files) during traversal.
- Loading the entire corpus into memory when it doesn't fit — causes OOM on large forges.
- Ignoring endianness/interpretation assumptions if vectors are later treated as anything other than raw byte sequences (e.g., accidentally reinterpreting as int16).
- Not versioning the forge script/config alongside the output dataset — reproducibility breaks when padding or chunking logic later changes.