QA vs QE: Engineering Quality Into Non-Deterministic AI — feat. Janna Loeffler (Quality Engineering Leader)

When your system gives a different answer every time, "assuring" quality is a trap — you have to engineer it in. Janna Loeffler joins Alex and Kuba to unpack why the shift from quality assurance to quality engineering (and even "quality enablement") matters more than ever in the age of LLMs and agents. She argues that AI didn't kill the tester's job — it made human insight, risk judgement and context-driven thinking essential, because non-deterministic output turns testing from clean pass/fail into questions of accuracy, relevance and intent. Janna makes the case for an "army" of single-purpose agents that handle the mundane — regression, accessibility, OWASP Top 10 security checks — so testers can focus on outcomes over outputs. The conversation digs into the real danger of eroding guardrails, including a chilling example of a system talked past its safety limits within minutes, and why model providers like Anthropic and OpenAI are pushing safety responsibility down onto the customers who deploy them. Along the way she shares how she vibe-coded a unit-test-writing agent in Cursor in a single day, why diverse test teams are the antidote to biased models, and why the emerging "AI architect" role will decide whether we get one agent or an orchestrated fleet of hundreds. \*\*In this episode:\** \- Quality Assurance vs Quality Engineering vs Quality Enablement \- Testing Non-Deterministic And Probabilistic Systems \- Automating The Mundane With Single-Purpose Agents \- Outcome Over Output: Validating Intent, Not Exact Strings \- Guardrails, Context Buffers And Adversarial Manipulation \- Bias, Diverse Test Teams And Context-Driven Testing \- Auditability, Traceability And Observability In Production \- The Emerging AI Architect Role And Agent Orchestration \*\*Guest:\** Janna Loeffler — Quality Engineering Leader / Director, 20+ years in QE, testing and platform reliability \*\*Host:\** Alex Belotsky (CEO, TestSavant.AI) with co-host Jakub "Kuba" Fiatkevich > "yes, everybody should be responsible for quality, but you need somebody accountable for that quality." 🎧 \*Subscribe to Ctrl-Alt-Deploy on Apple Podcasts, Spotify and YouTube for conversations on building and shipping reliable AI to production, QA, testing and AI governance.* \## Hashtags #CtrlAltDeploy #AI #SoftwareTesting #QA #QualityEngineering #QualityEnablement #AITesting #ContextDrivenTesting #Agents #LLM #Guardrails #AIGovernance #TestAutomation #NonDeterministic