Evolving Neural Architectures with LLM Tools

Core principle: use the LLM as a tool for local, constrained decisions, never as an autonomous end-to-end generator. Wrap it inside an algorithmic scaffold that guarantees executability.

Minimal pipeline:

Python
# 1. Mine reusable modules from existing source code (rule-based, no LLM)
module_db = mine_modules_from_source(codebase_paths)  # AST-based extraction

# 2. Represent architecture as a hierarchical tree (not a data-flow graph)
tree = build_tree_from_module(base_model, module_db)

# 3. Evolve: coarse decision by algorithm, fine decision by LLM
for step in range(num_iterations):
    coarse_action = diversity_guided_planner(tree, history_db)   # e.g. "swap backbone.stage2"
    candidate_tree = llm_resolve_transformation(tree, coarse_action, module_db)  # LLM fills in details
    if is_executable(candidate_tree):
        score = train_and_eval(candidate_tree)
        history_db.add(candidate_tree, score)

Progress:

• Step 1: Mine module database from source code via AST parsing
• Step 2: Convert candidate architectures into hierarchical tree representation
• Step 3: Define coarse-level evolutionary operators (structural/topological choices)
• Step 4: Delegate fine-grained parameter/module resolution to LLM
• Step 5: Enforce executability checks before evaluation
• Step 6: Update history database and Bayesian diversity model
• Step 7: Repeat with exploitation/exploration balance

Step 1: Module Mining (Rule-Based, Not LLM)

• Parse source into an Abstract Syntax Tree (AST); never use an LLM for extraction — it must be 100% correct, deterministic, and hallucination-free.
• Identify all classes inheriting the framework's base module type (e.g., torch.nn.Module).
• For each module, record:
  • Search space s_i: every constructor (__init__) argument becomes a tunable hyperparameter.
  • Metadata μ_i: default arg values, number of forward() inputs/outputs, and raw source snippet.
• Store as D_M = {(m_i, s_i, μ_i)}. This database is the sole source of legal building blocks — no free-form code synthesis.

Step 2: Deploy-Friendly Tree Representation

• Represent the full architecture as a tree, not a data-flow graph:
  • Root = model; children = conceptual sub-modules (backbone, neck, head); leaves = scalar hyperparameters (channels, kernel size, bias).
  • Intermediate nodes = modules (can be expanded further); leaf nodes = terminal hyperparameters.
  • Support list-typed nodes for variable-length structures (e.g., stacking N repeated layers).
• This representation mirrors how humans structure code (nested modules), unlike data-flow graphs, making it directly compatible with reused source code and top-down refinement (start rough, add detail later).

Step 3–4: Coarse-to-Fine Evolution

• Coarse level (algorithmic): decide what kind of structural change to make (e.g., replace a subtree, insert a stage, change list length). Governed by a diversity-guided algorithm using Bayesian modeling over historical outcomes (D_A = {(A_i, e_i)}) to balance exploration vs. exploitation.
• Fine level (LLM): given the coarse directive and legal module choices from D_M, the LLM resolves remaining degrees of freedom — which specific module, what hyperparameter values, how to connect it — producing a concrete tree transformation.
• Critically: the LLM operates on a constrained decision template, not open-ended code generation. Its output space is limited to valid tree edits using cataloged modules.

Step 5–7: Execution and Feedback Loop

• Validate every candidate tree is executable (correct shapes, valid arg types) before spending compute on training.
• Convert tree → runnable model code deterministically (tree structure maps directly to nested module instantiation).
• Train/evaluate, then push (A_i, e_i) back into the architecture database to refine future Bayesian-guided coarse planning.