We are training an agent that can build product. Doing this requires training signal, and for interfaces and product that is not trivial, because good is a matter of human judgment and of how the product makes people feel. A coding agent can check its work by running the tests. An agent that builds interfaces can only check its work by watching what people do. The best known way to get verifiable training signal is to launch each product change as an A/B test to hundreds of thousands of users and measure what happens to their behavior. Running this feedback loop is very expensive, as hundreds of thousands of users are difficult to come by.
In an ideal world we would stay in the realm of data and let the agent loop on its own, proposing a change, launching it, reading the result, and proposing the next. Anyone who runs experiments knows the constraint, which is traffic. A product has only so many users, and only so much real estate that can be changed at once, so every experiment has to be as good as it can be before it goes live. The rest of this page is how we get an experiment to that point, using one hypothetical food-delivery app as the example.
We start in the event data. The company's events and code show how people move through the app and where they stop. In the delivery app, many people open a restaurant page and then leave the app before ordering. The data shows where they leave. It cannot show why. That is what the interviews are for.
A good experiment starts with a good idea, and the best source of ideas we have found is users telling the story of what they were doing in the product. In a discovery interview we ask a customer of the delivery app to walk us through their last order, and somewhere in the story they mention that they left the app to check reviews of the restaurant elsewhere. That sentence is the idea for a change that brings the reviews and photos they went looking for onto the restaurant page. The interviews are run by our AI interviewer, by voice, inside the app, so they can be triggered at the right moment and run in parallel. Talking is the cheapest way for a person to tell us what happened.
We keep a prototype version of the app in which changes can be launched immediately. The agent takes the idea there and iterates on it, building a variant, showing it to users in usability interviews, and building the next variant from what they say, until the people testing it can choose a restaurant without leaving the app. That variant goes to the company's product team for review as a finalized experiment, in the form of a proposal stating the hypothesis, the metric it should move, the guardrails, and what we will learn either way, and a pull request that puts the change behind an experiment flag in the company's codebase.[1] We make sure each experiment measures one concrete thing, so that the result teaches us something specific whether it wins or loses.
The experiment then goes out in front of tens of thousands to millions of people, and the A/B test measures the actual change in their behavior at scale. That measurement tells the company whether to keep the change, and it tells us whether the agent's judgment about interfaces is getting better, one experiment at a time.
A read-only account in the data warehouse, whichever it is, where we build our own views of the event data; access to the code, which explains what the events mean; a way to show the interview invitation when an event fires; and, if we ship the change, permission to open pull requests. Which analytics tool produces the events does not matter, since PostHog, Amplitude, and Mixpanel events all land in BigQuery, Snowflake, or Databricks in the same shape.