Data Agent

The Problem

Every company has data. Almost nobody can query it.

The people who understand the business — PMs, ops leads, founders — can't write SQL. The people who can write SQL — data engineers, analysts — are bottlenecked with request queues. A product manager waiting three days for "how many users signed up last week" is a product manager making decisions blind.

Semantic layers and BI dashboards try to bridge this gap, but they require months of setup, constant maintenance, and someone who understands both the data model and the business logic. Most teams don't have that person. The dashboard gets stale, the semantic layer drifts, and people go back to pinging the data team on Slack.

The real question: can an AI agent sit between a plain English question and a SQL warehouse, generate correct queries, and get better over time — without a semantic layer?

What We Built

A full-stack data agent that takes natural language questions, generates dialect-aware SQL, executes it against your warehouse, scores its own accuracy, and learns from corrections. Point it at a dbt project or a raw database schema, and it starts answering questions immediately.

The Pipeline

Every question runs through a seven-step pipeline:

Question → Context Sub-Agent → Knowledge Store → SQL Agent → Executor → Self-Scoring → Response
                                                                              ↓
                                                                     Recovery Loop (if needed)

Context Sub-Agent — Before generating SQL, a lightweight agent analyzes the question against your schema. It identifies relevant tables, columns, join paths, and filter conditions. This narrows the context window for the main SQL agent so it's not reasoning over hundreds of tables.

Knowledge Store — A hybrid vector + keyword search layer built on pgvector. It retrieves three types of context: quirks (learned corrections from past mistakes), metric definitions (what "revenue" or "active user" actually means in your schema), and annotations (inline @agent: comments in your dbt SQL files). This is how the system gets smarter.

SQL Agent — Claude Sonnet generates dialect-aware SQL (Postgres, Snowflake, or BigQuery) with full schema context, retrieved knowledge, and the sub-agent's table/column recommendations. The prompt is engineered for correctness over cleverness — simple joins, explicit column references, no unnecessary subqueries.

Executor — Runs the query against your warehouse with timeout handling, result formatting, and error classification. Errors are categorized (syntax, permission, timeout, schema mismatch) so the recovery loop knows what kind of fix to attempt.

Self-Scoring — Every query gets a composite confidence score across three dimensions:

Structural (45%) — SQL validity, anti-pattern detection, schema validation. Does the query reference real tables and columns? Does it use appropriate join types? Are there cartesian joins or missing WHERE clauses?
Execution (35%) — Did it run without errors? Are the row counts reasonable? Did it finish in a reasonable time?
Alignment (20%) — Does the SQL actually answer the question asked? An LLM-scored check comparing the question intent against the query logic and result shape.

Recovery Loop — Queries scoring below 0.7 trigger automatic retry. The system has three recovery strategies: SQL fix (rewrite the query based on the error), context refresh (re-run the context sub-agent with broader scope), and scope reduction (simplify the question into answerable parts). Max 2 retries before returning the best attempt with a confidence warning.

Response Formatter — Generates a natural language summary with key numbers, trends, and visualization suggestions. The user sees an answer, not a result set.

The Knowledge Store

This is the part that makes the system compound over time.

When a user corrects an answer — "actually, you need to exclude test accounts" — the correction goes through a quirk extraction pipeline. It generalizes the fix into a reusable rule: "When querying the users table, always filter WHERE is_test = false unless explicitly asked about test accounts."

That quirk gets embedded and stored. Next time anyone asks a question involving the users table, the knowledge store retrieves it. The agent doesn't make the same mistake twice.

Three knowledge types feed the pipeline:

Quirks — Learned corrections. "Revenue means order_total - refunds, not just order_total." Extracted automatically from user corrections, stored with embeddings for semantic retrieval.

Metric Definitions — Canonical definitions for business terms. Managed through the admin dashboard. "Active user = logged in within the last 30 days AND completed at least one action."

Annotations — Inline comments in dbt SQL files using the @agent: prefix. Data engineers can leave notes for the agent directly in the code: -- @agent: this table is partitioned by date, always include a date filter.

dbt Integration

Point the agent at a dbt project directory and it automatically parses model definitions, column descriptions, tests, source definitions, and ref() dependencies. The schema context the SQL agent receives isn't just raw information_schema — it's enriched with the business logic your data team already documented.

This means teams that already invest in dbt documentation get immediate value. The agent understands column descriptions, model relationships, and test constraints without any additional setup.

Multi-Dialect SQL

The SQL agent generates dialect-specific syntax for Postgres, Snowflake, and BigQuery. Date functions, string operations, array handling, and NULL semantics differ across dialects — the agent handles these automatically based on the configured SQL_DIALECT.

Slack Bot

The Slack integration turns the agent into a team-wide data analyst. Mention it in any channel or DM it directly. It supports threaded conversations for follow-up questions, a correction flow (reply with "actually..." to teach it), and emoji-based feedback for tracking answer quality over time.

Admin Dashboard

A management interface for the data team to monitor and tune the system:

Overview — Query volume, average confidence scores, correction rate, knowledge store size
Knowledge — Browse, edit, and manage quirks, metrics, and annotations
Sources — Connected data sources and schema status
Queries — Full query lifecycle history with scoring breakdowns, recovery attempts, and user feedback

Demo Mode

A zero-config demo mode seeds a local SQLite database with ~10K rows of sample e-commerce data (users, orders, products, events). Set DEMO_MODE=true and an Anthropic API key — the full pipeline works out of the box.

Architecture

The system is a Next.js 14 application with three layers:

Frontend — Chat interface for asking questions, admin dashboard for management. Built with shadcn/ui components and Tailwind CSS. Conversations are threaded, with full history and the ability to drill into scoring details.

API Layer — Next.js API routes handle chat requests, corrections, and admin operations. The chat endpoint orchestrates the full seven-step pipeline. Slack events route through dedicated webhook and Socket Mode handlers.

Intelligence Layer — The core pipeline: context sub-agent, knowledge store (Supabase + pgvector), SQL agent (Claude Sonnet), executor (multi-dialect database connectors), scorer (structural + execution + alignment), and recovery loop. Each component is independently testable and swappable.

Data Layer — Supabase for the knowledge store, query history, and user corrections. pgvector for embedding-based semantic search. The actual data warehouse is accessed read-only through configured database connectors.

┌─────────────────────────────────────────┐
│  Frontend (Next.js App Router)          │
│  ├── Chat Interface                     │
│  ├── Admin Dashboard                    │
│  └── Slack Bot (Bolt SDK)               │
├─────────────────────────────────────────┤
│  API Layer                              │
│  ├── /api/chat (pipeline orchestrator)  │
│  ├── /api/correct (quirk extraction)    │
│  └── /api/admin/* (management)          │
├─────────────────────────────────────────┤
│  Intelligence Layer                     │
│  ├── Context Sub-Agent                  │
│  ├── Knowledge Store (pgvector)         │
│  ├── SQL Agent (Claude Sonnet)          │
│  ├── Executor (Postgres/SF/BQ)          │
│  ├── Self-Scorer (3-dimension)          │
│  └── Recovery Loop (max 2 retries)      │
├─────────────────────────────────────────┤
│  Data Layer                             │
│  ├── Supabase (knowledge, history)      │
│  └── Customer Warehouse (read-only)     │
└─────────────────────────────────────────┘

Why This Matters

Most companies spend $120-150K per analyst hire. A 20-50 person startup with a 2-3 person data team is spending $300-500K/year on salaries — and still has a request queue. The data agent handles 80%+ of ad-hoc queries without an analyst in the loop, and it gets smarter every time someone corrects it.

This architecture is production-proven. The pattern was first built at Notion's data team, where it replaced a hiring plan for 4-5 analysts. We've productized it into a turnkey system that works with any dbt-based data stack.

How We Deploy It

Setup engagement (3 weeks) — We connect the agent to your dbt repo, warehouse (Postgres, Snowflake, or BigQuery), and Slack. We configure the context sub-agent for your schema, seed the knowledge store with your core metric definitions, and tune the scoring pipeline against your actual query patterns. By week 3, your team is asking questions in Slack and getting SQL-backed answers in seconds.

Ongoing tuning — The system improves passively through user corrections, but active maintenance accelerates it. We monitor confidence scores, review low-scoring queries, add metric definitions for new data models, and expand the quirks store as edge cases surface. The result is an agent that understands your data better every month.

Who It's For

Series A-C startups with a dbt repo, 1-3 analysts who are bottlenecked, and business teams asking the same questions in Slack every week. The ICP is a Head of Data or technical CEO who'd rather deploy an agent than hire analyst #4.

The Competitive Gap

Approach	Problem
Static semantic layers (Cube.dev, MetricFlow)	Months of setup, constant maintenance, brittle
BI tools with AI bolted on (ThoughtSpot, Mode)	Shallow integration, no learning, expensive
Internal builds	Requires a senior data engineer, 3+ weeks, no ongoing support
Data Agent	3-week setup, self-improving, production-proven architecture

The key insight: context management beats semantic layers. Instead of pre-defining every metric and relationship upfront, the agent investigates your schema per-question and learns from corrections. No semantic layer to maintain. No dashboard to keep updated. The agent adapts to your data, not the other way around.

What We Learned

Schema context is everything. The difference between a good and bad NLP-to-SQL system is how much schema context the model sees. Raw information_schema dumps are noisy and miss business logic. dbt descriptions, annotations, and learned quirks dramatically improve first-attempt accuracy. The context sub-agent exists specifically to filter the schema down to what's relevant.

Self-scoring changes the user experience. Without confidence scores, every answer looks equally authoritative. With scoring, the system can say "I'm 85% confident in this" vs. "I'm 55% confident — here's what I'm unsure about." Users trust the system more when it's honest about uncertainty, and the recovery loop means low-confidence answers often self-correct before the user even sees them.

Corrections are the best training data. Every user correction is a labeled example of what the system got wrong and why. The quirk extraction pipeline turns these into reusable knowledge that compounds. After 20-30 corrections, the agent understands your schema's quirks better than most new analysts would in their first month.

No semantic layer is a feature. Traditional BI tools require defining every possible metric and dimension upfront. The data agent works with raw schema + learned knowledge, which means it can answer questions nobody anticipated. The tradeoff is occasional errors — but the self-scoring and correction loop handle that gracefully.

Data Agent is available as a service from Parallel Studio. We set it up on your stack in 3 weeks, and it gets smarter every month. Tell us about your data.

At a Glance