AI Pre-Production Tester

Executive Summary

This is a pre-production testing harness for AI-powered real-estate experiences. It lets a researcher configure an AI scenario, choose a user persona, run it, and score the result against a structured rubric, without standing up a live production environment. It's a single Railway-deployed Flask app hitting the Anthropic API.

The motivating insight is simple: AI experience quality is hard to evaluate without a structured rubric and a repeatable test runner. Chat tools can produce a response, but they can't tell you whether it was the right response for this user, this task, and this risk profile. This tool makes that judgment structured, repeatable, and exportable. It runs at the live demo.

Problem Statement

AI-powered listing features (description generators, search assistants, mortgage Q&A) ship with unknown failure modes. There's no standard pre-production tooling for evaluating conversational AI UX before release, and manual QA doesn't scale: each scenario needs different personas and different success criteria.

Research teams need a structured, repeatable way to surface the failures that matter (confidence miscalibration, hallucinated listings or data, and inappropriate recommendations) before users see them. That's a methodology problem as much as a tooling one, and this harness is built around the methodology.

Tool Architecture

Scenario configuration

Five built-in scenario types (home-search assistant, mortgage Q&A, neighborhood comparison, listing-description generator, and a custom scenario) each configure the AI system prompt and the evaluation rubric automatically. Custom mode takes a free-text description for ad-hoc testing.

Persona system

Four personas (first-time buyer, real-estate investor, renter exploring ownership, relocating professional) modify the simulated query context and shift the success-criteria weighting.

Confidence score & batch mode

A 0–100 confidence score is generated by claude-sonnet evaluating the response against the six dimensions, shown with a per-criterion breakdown. Batch mode ("run all scenarios") executes the four built-in scenarios in sequence with a progress indicator and collects them into a single export.

Export & stack

A single run exports JSON (scenario config, persona, response, score, criteria); a batch exports CSV, one row per scenario. The stack is Flask on Railway with the Anthropic API (claude-sonnet for evaluation reasoning, claude-haiku for structured JSON), with httpx==0.27.0 pinned alongside anthropic (a Railway boot fix), plus a PropertyData API and FEMA NFHL for live listing context. The run log persists in session for cross-run comparison.

Design Decisions

Rubric-first evaluation

The six criteria are defined before the AI response is generated, not derived from it. That prevents the evaluator from rationalizing a bad response as acceptable, and it makes the bar explicit to the tester before the run.

Persona as a test variable

Switching personas is a first-class action, not a form field. The same scenario run against a first-time buyer vs. an investor should produce meaningfully different responses. If it doesn't, that's a signal the AI isn't contextualizing the user.

Batch run as a regression suite

The four built-in scenarios cover distinct failure modes: hallucination risk (listing description), confidence calibration (mortgage Q&A), scope creep (neighborhood comparison), task completion (home search). Running them in sequence is effectively a regression suite for AI experience quality.

Export as the artifact

JSON/CSV export isn't a nice-to-have; it's the primary output. Research teams need shareable, archivable evidence of a pre-production quality gate.

What This Demonstrates

This is a full evaluation framework, not a chat wrapper: structured rubric design, persona modeling, and batch regression, built with research methodology in mind. The whole thing was built and iterated through an AI-native, bootstrap-file workflow with Claude Code, and it speaks directly to the research-tooling gap around AI-powered real-estate experiences: a structured, repeatable way to generate evidence of quality before release. Production-quality, built in days.

Live Demo