Design and own the end-to-end QA strategy for the Conversational Banking Platform, covering functional, regression, performance, security, and AI-specific evaluation.
Build and maintain golden datasets, eval suites, and LLM-as-judge frameworks to validate conversational quality across intents, languages, and tenants.
Define the tenant onboarding QA gate, the certification checklist every new business unit must pass before going live.
Establish regression strategies for prompt changes, model upgrades, retrieval index updates, and guardrail policy changes.
Use Langfuse traces to drive evaluation: mine production failures, convert them into test cases, and close the loop with engineering.
Test NeMo Guardrails configurations against jailbreaks, prompt injection, off-topic drift, and false-positive over-blocking.
Validate governance and compliance behaviors: data residency, PII handling, regul...