Research Methodology

What makes CREATE SOMETHING different from AI blogs: we don't just document results—we document the process of building with AI agents.

Every experiment is tracked with automated logging, real costs from APIs, precise time measurements, and intervention documentation. This transforms anecdotes into reproducible experiments.

How We Work

1. Build

Claude Code

Work with AI agents as development partners

2. Track

Auto-Log

Hooks capture every prompt, error, intervention

3. Analyze

Real Data

Actual costs, time, errors from APIs

4. Publish

Honest Results

What worked, what didn't, and why

This is research, not blogging.

Every Experiment Tracked With

Prompts

Real-time logging via Claude Code hooks

47 iterations logged

Errors

Precise counts & resolution times

23 errors, avg fix: 8 min

Costs

Token usage + infrastructure from APIs

$18.50 Claude + $8.30 Cloudflare

Interventions

When AI needed human help, and why

12 manual fixes documented

Time

Session duration, not guesswork

26 hours actual vs 120 estimated

Architecture

Decisions made, alternatives considered

Workflows over Workers (why)

Displaying Evidence

Tracking data isn't enough—how you display that data determines whether your research is credible or dismissed. We follow Edward Tufte's principles for quantitative information design.

"Excellence in statistical graphics consists of complex ideas communicated with clarity, precision, and efficiency." — Edward Tufte, The Visual Display of Quantitative Information

Maximize Data-Ink Ratio

Every pixel should convey information. Remove decoration that obscures data.

In our analytics: Sparklines show trends in 12 pixels instead of full charts with axes and labels.

Show Data Variation

Display change and patterns, not just static numbers.

In our analytics: Inline trends on metrics show movement over time, not just current values.

High Data Density

Maximize meaningful information per unit area.

In our analytics: Tables show 10 experiments with counts + percentages instead of 3 with decoration.

Integrate Text & Data

Labels, numbers, and context should appear together.

In our analytics: Property stats show count, percentage, and trend inline—no separate legend needed.

Small Multiples

Show multiple dimensions side-by-side for comparison.

In our analytics: 7-day grid lets you spot patterns across the week at a glance.

Remove Chartjunk

Eliminate visual noise that distracts from data.

In our analytics: Subtle borders and minimal backgrounds—data stands out, not design.

Why Visualization Standards Matter

Credibility: Poor visualization makes readers question your data. If metrics are buried under decoration, they assume you're hiding something.

Reproducibility: Clear presentation lets others verify your work. Sparklines and small multiples make patterns obvious without interpretation.

Efficiency: Researchers need to extract insights quickly. High data density means less scrolling, more understanding.

These principles are applied throughout our analytics dashboard and experiment results.

View Analytics Dashboard

Three Tracking Modes

Real-Time Tracking

Ideal

Start tracking from day one. Get complete data on every iteration, error, and decision.

Data Quality:

High confidence, precise metrics

Use Case:

New experiments starting from scratch

Mid-Flight Tracking

Practical

Start tracking on an in-progress project. Combine real-time data with git history reconstruction.

Data Quality:

Mixed: estimates for past work, precise for future

Use Case:

Active projects you realize are experiment-worthy

Retroactive Documentation

Still Valuable

Document already-deployed projects. Reconstruct from git, APIs, and memory.

Data Quality:

Lower confidence, acknowledged limitations

Use Case:

Completed projects with production data

Why This Matters

Without Tracking

  • "I built X with AI" (anecdote)
  • No reproducibility
  • Can't verify claims
  • Just another AI blog

With Tracking

  • "I built X: 26 hrs, $27, 78% savings" (data)
  • Others can replicate experiments
  • Transparent methodology
  • Scientific research platform

The tracking methodology transforms "prompting and hoping" into systematic evaluation with reproducible results. This is what separates research from blogging.

For Researchers: Use This Methodology

Want to adopt this approach for your own AI-native development research? The experiment tracking system is available as a Claude Code Skill.

1

Install the Skill

Add experiment tracking to your Claude Code setup

2

Build & Track

Work with Claude Code while automatic logging captures everything

3

Generate Papers

Transform tracked data into reproducible research

View on GitHub See Example Experiment

Methodology in Action

Example from Experiment #1: Zoom Transcript Automation

26
Hours
47
Errors
12
Interventions
78%
Time Savings

Data sources: Real-time prompt logging via hooks, Claude Code Analytics API, Cloudflare billing API, git commit history

Reproducibility: Starting prompt, tracking logs, and architecture decisions documented

Read the full experiment

Canon & Influences

CREATE SOMETHING's methodology doesn't exist in isolation. We build on foundational work from researchers, practitioners, and thinkers who established standards for rigorous knowledge production.

Masters

Edward R. Tufte

Professor Emeritus, Yale University

Data Visualization

Work: The Visual Display of Quantitative Information (1983)

Influence on CREATE SOMETHING: Tufte's principles for displaying quantitative data define how we present experiment results, analytics dashboards, and research findings. His concept of the data-ink ratio—maximizing information while minimizing decoration—is fundamental to our visualization standards.

Core Principles We Apply:

  • Maximize data-ink ratio
  • Show data variation, not design variation
  • Remove chartjunk

Implemented In:

  • Analytics dashboard design
  • Experiment metrics visualization
  • Research paper data presentation
Full Canon Entry Official Site

How Work Enters the Canon

A "Master" in the CREATE SOMETHING canon represents someone whose work has:

  • Foundational impact — Established principles we build upon
  • Practical application — Not just theory; implemented in our work
  • Evidence-based approach — Research-backed, not opinion
  • Timeless relevance — Principles that withstand technological change

The canon grows through practice, not planning. We don't add influences speculatively—only after we've applied their work and proven its value in our methodology.

The complete canon—with full biographies, principles, quotes, and resources—lives on CREATE SOMETHING.ltd

Explore All Masters

Dieter Rams • Mies van der Rohe • Edward Tufte