Recursive Language Models

Large Language Models face a fundamental constraint: context windows. Even "long-context" models (1M+ tokens) degrade on tasks requiring dense access to large inputs. The MIT CSAIL paper "Recursive Language Models" (arxiv:2512.24601) proposes a paradigm shift: treat context as an external environment variable, not prompt content.

The key insight: instead of injecting massive context into the prompt, store it as a variable in a REPL environment. The model writes code to navigate the context, using sub-LM calls for semantic understanding. This enables processing 10M+ tokens with comparable cost to standard inference.

2.1 Core Components

Our implementation follows the original RLM architecture:

┌─────────────────────────────────────────────┐
│             RLMSession                       │
│  - Manages the iteration loop               │
│  - Routes to root/sub models                │
│  - Tracks costs                             │
└─────────────────────────────────────────────┘
                    │
                    ▼
┌─────────────────────────────────────────────┐
│           RLMEnvironment                     │
│  - Sandboxed Python REPL                    │
│  - context = <your massive input>           │
│  - llm_query(prompt) → sub-LM call          │
│  - results = {} for findings                │
└─────────────────────────────────────────────┘

RLMEnvironment: Sandboxed Python REPL where context is stored as a variable. Provides:

• context — The input data (can be arbitrarily large)
• llm_query(prompt) — Sub-LM calls for semantic understanding
• results — Dictionary for storing intermediate findings
• chunk_text(), chunk_lines() — Chunking helpers
• Standard library: re, json, print()

2.2 Model Routing

Following the paper's recommendations, we use two models for cost efficiency:

The paper shows Haiku achieves 90% of Sonnet's performance on bounded semantic tasks while costing 10x less.

2.3 Termination Markers

The model signals completion via:

• FINAL(your answer here) — Direct answer
• FINAL_VAR(results) — Return a variable from the environment

We reviewed our implementation against the original alexzhang13/rlm repository, identifying several critical issues.

3.1 Bug: Undefined Client Variable

File: modal_rlm.py:401

response = client.messages.create(...)

response = anthropic_client.messages.create(...)

This would have crashed at runtime in production.

3.2 Bug: FINAL() Regex Limitation

Original pattern:

final_match = re.search(r"FINAL\(([^)]+)\)", response)

Problem: [^)]+ stops at the first ), so:

• FINAL(Answer is (a) and (b)) → captures only "Answer is (a"

final_match = re.search(r"(?:^|\n)FINAL\((.+)\)\s*$", response)

3.3 Bug: FINAL Detection Before Code Execution

Original flow:

1. Get model response
2. Check for FINAL ← Problem: FINAL matched before code runs
3. Execute code blocks
4. Feed results back

Problem: Model outputs code blocks AND FINAL_VAR(results) together, expecting code to populate results first. But we checked for FINAL before executing code, returning empty results.

# Execute code blocks first
code_blocks = re.findall(r"```repl\n(.*?)```", response, re.DOTALL)
for code in code_blocks:
    exec_result = env.execute(code.strip())
    # ... capture output

# NOW check for FINAL (results are populated)
final_match = re.search(r"(?:^|\n)FINAL\((.+)\)\s*$", response)

3.4 Bug: MULTILINE Flag Causing Early Match

final_match = re.search(r"FINAL\((.+)\)\s*$", response, re.MULTILINE)

Problem: re.MULTILINE makes $ match at end of ANY line, not just end of string. FINAL mentioned mid-response matched prematurely.

final_match = re.search(r"(?:^|\n)FINAL\((.+)\)\s*$", response)

3.5 Enhancement: Structured Messages

Original: Flattened conversation to text blob.

Fix: Pass structured messages to API for proper multi-turn handling.

config = ProviderConfig(
    messages=conversation,  # List of {"role": ..., "content": ...}
    ...
)

4.1 Test Case: DRY Violation Analysis

We ran the RLM against our monorepo to find DRY violations.

4.2 Results

Based on RLM findings, we created four shared utilities:

// Before: 20+ files with duplicate fetch
const response = await fetch(
  `${IDENTITY_API}/v1/auth/login`
);

// After: Typed client
const result = await identityClient
  .login({ email, password });

// Before: Duplicate try/catch
try { ... }
catch (err) { console.error(...); }

// After: Wrapped handler
export const POST = catchApiError(
  'ProfileAPI',
  async (event) => { ... }
);

// Before: Repeated patterns
if (records.length === 0) { ... }

// After: Type-safe helpers
if (isEmpty(records)) { ... }
const result = validateStringField(
  body.name, 'name', { required: true }
);

// Before: Console calls
console.log('[ProfileAPI]', email);

// After: Structured logging
const logger = createLogger('ProfileAPI');
logger.info('Fetching', {
  email, correlationId
});

6.1 RLM Effectiveness

Strengths:

• Successfully processed 157K characters (far beyond prompt limits)
• Identified actionable patterns through programmatic filtering
• Cost-effective: $0.03 for comprehensive analysis
• Single iteration completion demonstrates good prompt engineering

Limitations:

• No sub-LM calls used in this task (regex sufficient)
• Model occasionally includes FINAL in first response without exploration
• Requires careful prompt engineering to encourage REPL usage

6.2 Implementation Lessons

6.3 Comparison to Original

Using the RLM

from create_something_agents.rlm import RLMSession, RLMConfig
from create_something_agents.providers.claude import ClaudeProvider

# Your large context
corpus = open("massive_corpus.txt").read()

# Create session
session = RLMSession(
    context=corpus,
    provider=ClaudeProvider(),
    config=RLMConfig(root_model="sonnet", sub_model="haiku")
)

# Run query
result = await session.run("What patterns emerge across all documents?")
print(f"Answer: {result.answer}")
print(f"Cost: ${result.cost_usd:.4f}")

Using the DRY Utilities

// Identity API calls
import { identityClient } from '@create-something/canon/api';
const result = await identityClient.login({ email, password });

// API error handling
import { catchApiError, apiError } from '@create-something/canon/utils';
export const POST = catchApiError('MyAPI', async (event) => { ... });

// Validation
import { isEmpty, validateStringField } from '@create-something/canon/utils';
if (isEmpty(records)) return apiError('Not found', 404);

// Logging
import { createLogger } from '@create-something/canon/utils';
const logger = createLogger('MyService');
logger.info('Processing', { id, correlationId });

We successfully implemented and validated the Recursive Language Models pattern from MIT CSAIL's research. The implementation review against alexzhang13/rlm revealed four critical bugs that we fixed:

1. Undefined client variable (crash at runtime)
2. FINAL regex failing on nested parentheses
3. FINAL detection before code execution (empty results)
4. MULTILINE flag causing premature termination

The RLM demonstrated practical value by analyzing 157K characters of codebase, identifying 165+ DRY violations, and enabling creation of four shared utilities that measurably reduce code duplication.