AI agent testing that verifies the actual outcome.
Discipline
AI agent testing
Engineering
Jarek Trepkowski Principal Software Engineer
Focus
Python / AI / Automation
AI agent and voice agent testing: conversation scenarios, API and CRM outcome verification, regression testing and prioritised issue reports before release.
Before the first release, when accepting a supplier’s implementation and after changes to the model, instructions, tools or integrations. I work with businesses, software companies and agencies that need verifiable acceptance criteria alongside a conversation demo.
Scope
What do the tests cover?
Conversation and system state
I compare what the agent says with the actual result: a booking, a CRM entry or another agreed action. Where available, I verify the outcome through APIs or test environment data.
Changed plans and missing details
Scenarios include rescheduling, cancellation, incomplete information and repeated requests. I look for conflicting data, unwanted duplicates and lost context.
Integration failure
I check behaviour when an API fails, times out or denies access. The agent should communicate the actual execution status and follow the agreed rules for handing a task over to a person.
Regression after updates
Selected scenarios can run again after a change. We agree repetition counts and evaluation criteria with model variability and execution costs in mind.
Voice and text agent testing
Your agent says “done”. I verify the outcome.
I check whether your AI agent actually did what it promised. You get an issue report and recommended fixes before launch.
How it works
The conversation is only half the test.
Illustrative example, not a client project result
Customer
Book an appointment for Tuesday at 10:00.
AI agent
Done, your appointment is booked.
I check the system
Calendar
×No booking found
×Test failed
The agent confirmed an action it never completed.
Calendar
✓Tuesday, 10:00 · saved
✓Test passed
The booking matches the date and details agreed in the conversation.
Sample issue report
A result your team can act on.
AI-014 · Appointment booking
High priority
Reproduce
Ask the agent for Tuesday at 10:00.
Expected
A booking saved in the calendar.
Observed
Verbal confirmation, no saved booking.
Next step
Fix API response handling and repeat the test.
AI-014 · Appointment booking
Fix verified
Reproduce
Ask the agent for Tuesday at 10:00.
Expected
A booking saved in the calendar.
Observed
The date and details are correct.
Next step
Keep this scenario in the regression suite.
Explore returns in the interactive lab: change the workflow blocks and their values.
Indicative prices, excluding VAT. NBP: 9 Nov 2026 · 1 EUR = 4.3873 PLN. Rounded to the nearest EUR 50.
Base prices are net, in PLN. Scope, currency and final price are fixed in your quote. Model, telephony and platform costs are separate, within an agreed limit.
Testing requires a working agent and an agreed test environment. Larger fixes are quoted separately.
Working together
From acceptance criteria to an actionable report
Agree the goal and boundaries
We choose the workflow, integrations, environment and permitted actions. We agree expected outcomes, test data and spending limits for model or telephony usage.
Run the agreed scenarios
I exercise the expected workflow and edge cases. I repeat selected scenarios to check for inconsistent behaviour and document the execution and outcome within the agreed access scope.
Review the findings
You receive prioritised issues with reproduction steps and expected outcomes. We discuss the order of fixes and which checks to repeat afterwards.
Handover
What do you receive?
Agreed scenarios and pass criteria.
Prioritised issues with reproduction steps.
A results walkthrough and recommended order of fixes.
For automation engagements: regression tests, reports and run instructions for the agreed environment.
Engagement
A clear scope and verifiable results
Testing covers the agreed scenarios and accessible integrations. It is not a security certification. Repetition counts, system access and CI/CD scope are set at the start, using test data and environments.
How do we choose the scope?
We usually start with a one-off audit. Once the scenarios prove useful, we automate regression checks or arrange ongoing verification after changes.
Yes. The scope covers the conversation and actions in connected systems. We agree scenarios, access to telephony and platform costs before work begins.
How is this different from scoring an answer?
Answer scoring evaluates what the agent said. Task testing also verifies the operation, the saved data and how the agent handled failures.
What determines the price?
The number of scenarios, integrations, environment access and automation scope. Packages and base prices are on this page; the final scope is confirmed in your quote.
Start with the problem
Let’s talk.
Tell me what your agent needs to do. Include the workflow, platform, integrations and intended acceptance date in your message. That is enough to start defining a useful scope.
