Case study · Ontraport

Shipping Ontraport's first AI initiative

How a 60-person SaaS company went from zero AI to an in-app assistant with agents over customer CRM data, and the gateway, evaluation framework, and feedback loop designed to make it trustworthy.

Role

Product Director, end-to-end owner

Scope

Assistant, agents, LLM gateway, evals

Team impact

50%+ faster idea to spec

Status

In progress

Context

Ontraport is a CRM and marketing automation platform. Customers run their businesses on it: contacts, pipelines, payments, campaigns, messages. That means every account holds a dense, structured picture of a business, and most customers only ever see a fraction of what their own data could tell them.

When LLMs made conversational access to data practical, the question wasn't whether to build, it was what could be trusted in front of customers whose businesses live on the platform. I owned that question end to end: strategy, architecture decisions with engineering, design, and delivery.

The three decisions that shaped it

1. Read-only agents, by design

The assistant's agents can query and explain an account's data, but they cannot mutate it. That single constraint did more for trust than any prompt engineering: the worst case of a wrong answer is a correction, never a corrupted campaign or a wrongly charged customer. It also sharply reduced the evaluation surface we had to cover, a large part of why V1 came together rather than stalling in review.

2. Rebuild the LLM gateway before scaling features

Every AI feature routes through one gateway rather than calling model providers directly. That gave us a single place where the operational questions live: which model serves which task, what it costs, how fast it responds, and what gets logged for review. Models change monthly; the gateway means we exchange them without rewriting features, and cost and latency tradeoffs are decisions we make once, explicitly, instead of accidents scattered across the codebase.

3. Make quality measurable before making it better

The uncomfortable truth of AI products is that everyone has an opinion about answer quality and nobody has a number. We built an evaluation framework first: define what a good answer looks like for each kind of question, score against it, and re-score whenever a model or prompt changes. An in-product feedback loop feeds real customer interactions back into that framework, so the eval set grows where users actually struggle, not where we guessed they would.

How we worked

A 60-person company doesn't get an AI lab, so the initiative had to change how the whole product team works, not just add a feature. I introduced AI prototyping into our discovery process: working prototypes in front of stakeholders in days, not mock flows in weeks. Paired with new spec workflows, that cut the time from idea to buildable spec by more than half, and those workflows are now standard across the product team, well beyond the AI work.

• Prototype first. Real prompts against real (sandboxed) data beat abstract debates about what the model "would" do.
• Specs with evals attached. A feature isn't specced until we can say how we'll know its answers are good.
• Build the smallest trustworthy thing. A working read-only V1 beats a write-capable V2 stuck in review.

What we built

• An in-app chat assistant that answers questions over account data through read-only agents.
• A rebuilt LLM gateway built to underpin AI features beyond the assistant: routing, cost and latency visibility, and logging in one place.
• An evaluation framework and feedback loop, so quality is a number we track, not a feeling we argue about.
• AI prototyping and spec workflows adopted across the product team: 50%+ faster from idea to spec.

Where it goes next

What comes next, and when, is Ontraport's story to tell. The principle behind it is the one that shaped V1: trust is sequenced, not assumed. Each expansion is gated on the evaluation framework proving the last one is solid.