I got python tests running against my SQL agent

Published: August 5th 2025

That is deterministic tests running against my non-deterministic SQL Agent pipeline

How does it work?

You setup the queries you want to test and run them against your agent

You should know what records the agent is expected to return based on the query

So you just check the returned records against the expected results. Just like you would on any normal integration test against the db.

You do this for all queries in your dataset, and you now have a suite of tests to quickly do regressions tests against if you change anything in your agent (e.g the prompt).

Finding user queries on production that are not giving good results? Add them to your test pipeline, and get them working.

Getting these tests running gives you confidence to mark this part and query set of your project done, so you can move on to the next part step, knowing you have some safety checks behind you

Very similar to unit/integrations tests, but arguably more important because of the non-deterministic nature of LLMs, that extra confidence hits that much more