Agentix

How to grade an AI agent before you deploy it in your business

By Jason Colapietro · Suede AI, West Palm Beach, Florida ·

Most AI agents that fail inside a business do not fail during the demo. They pass the demo, go live, and then start giving confident wrong answers on a busy afternoon when nobody is reading the transcripts.

Jason Colapietro built Agentix, the agent grader and earnings tracker from Suede AI, because that pattern shows up in almost every agent project: a team tests an agent once, then trusts it forever. This is the grading routine the Suede AI team in West Palm Beach, Florida runs before an agent is allowed near a customer, and how the same five dimensions Agentix scores keep working after launch.

Write the job description before the prompt

You cannot grade an agent against a goal nobody wrote down. Before any testing, write one paragraph that names the inputs the agent accepts, the output it must return, what counts as done, and the two or three things it must never do. A quote-intake agent for a Jupiter pool service, for example, might be allowed to collect an address and photos but never allowed to promise a price.

If you cannot write that paragraph, stop. The agent is not ready to grade, and it is certainly not ready to deploy. At Suede, this one paragraph becomes the rubric every later test is scored against; Jason Colapietro has used the same habit since the Suede Labs days, before the company took the Suede AI name.

The five dimensions Agentix grades, and their pre-launch versions

Agentix scores every tracked agent from S to F across five weighted dimensions: revenue (30%), growth (20%), consistency (15%), reliability (20%) and activity (15%). A score of 90 or above is an S, 80 is an A, 68 is a B and anything under 40 is an F. Those are live signals. Before launch you grade a proxy for each one:

  • Reliability: run 50 to 100 realistic inputs, including malformed and off-topic ones, and count failures. In Agentix a 20% error rate drives the reliability score to zero, and 5% already costs a quarter of it. Treat anything above 5% in testing as a launch blocker.
  • Consistency: send the same input five times. If the answers disagree on facts (hours, service area, price ranges), the agent is guessing. Wording can vary; facts cannot.
  • Activity: decide what alive means for this agent and how you will know within a day if it stops. Agentix decays the activity score after one, three, seven and fourteen days without a paid call, so a quiet agent shows up fast.
  • Revenue and growth: put a number on the value of one good call, in dollars earned or minutes saved, and estimate volume. If the number is tiny, a perfect agent is still a poor investment.

Agentix also caps the grade by status: a draft agent cannot score above 38, an agent whose endpoint is down cannot score above 55, and a paused agent cannot score above 62. Copy that rule in testing. If the agent cannot reach a tool it depends on, such as a calendar or a pricing sheet, its grade is capped no matter how good its answers sounded.

Build the test set from real conversations

Synthetic test questions are too polite. Pull 30 real questions from your inbox, call notes and website chat, then add five adversarial ones: a request outside your service area, a request for a discount, a question in Spanish, a customer who is angry, and a question with a missing detail. For a Palm Beach County business, include at least one service-area edge case, such as a caller in Tequesta asking whether you cover Juno Beach and Jupiter.

Keep this set in a spreadsheet with the expected answer in one column and the agent's answer in the next. Run it again every time the prompt, the model or a tool changes. That spreadsheet is worth more than any single test run.

Grade the failure, not only the answer

An agent that says it is not sure and hands the customer to a person has passed. An agent that invents a confident answer has failed, even when the invented answer happens to be right. Score three outcomes separately: correct, safely escalated and wrong. A launch-ready agent has close to zero wrong answers and an escalation rate the team can actually staff.

Set the pass line before you look at results

Decide the threshold in advance and write it next to the job description. Jason Colapietro's rule is blunt: the pass line is set by the person who will answer the angry phone call, not by the person who built the agent. A reasonable starting line is a B or better on the Agentix scale (68 and up) for a limited launch, with no wrong answers on the five adversarial questions. If you set the line after seeing the results, you will find a way to pass.

Where this grading habit comes from

Jason Colapietro learned to distrust untested code the slow way. He has had 54 pull requests merged across 46 open-source repositories he does not maintain, including Jest, Adobe React Spectrum, Backstage and the Polkadot SDK. Maintainers of projects like those merge on evidence: tests, reproductions and clear failure cases. The same discipline runs through two of his five books on Amazon, The Claude Code Bible and Codex in Production.

It is also why Suede AI keeps receipts. Suede AI is a member of Claude for Startups, Anthropic's startup program, and the claims Suede makes about itself are listed with proof on the Suede AI evidence page. Suede AI ranks among Clutch's Top 30 AI Consultants in Florida and Top 50 AI Consultants nationwide (November 2026). An agent grade should work the same way: a number you can trace back to the calls that produced it.

After launch, keep grading

Pre-launch testing tells you the agent can do the job. Only live data tells you it keeps doing it. In Agentix you register the agent, log its daily calls, errors and earnings, and the grade updates from your own entries, saved in your browser. A grade that slips from A to C over two weeks is the early warning that a prompt drifted or a tool broke. The Code Reviewer example profile shows what the breakdown looks like for one agent.

If you want a second opinion on whether your agent is findable by customers at all, the Suede Scan AI readiness check grades the site the agent lives on, and Suede AI's developer docs cover the tool and MCP side.

Your next step

Write the one-paragraph job description for the agent you are considering this week, then build the 35-question test set from real conversations. If you run a business in West Palm Beach, Jupiter or anywhere in Palm Beach County and want Suede AI to grade the agent and the website around it, start with the free Suede AI assessment. To see who you would be working with, read Jason Colapietro's founder page.