Job Title: Quality Engineer - AI/GenAI Validation & Testing
Location: Toronto, ON (Hybrid, 3-4 days onsite per week)
Contract Duration: 12 Months (Strong possibility of extension)
Overview
We are seeking a highly experienced Quality Engineer specializing in Artificial Intelligence (AI), Machine Learning (ML), and Generative AI (GenAI) solution validation. This role will be responsible for defining and executing comprehensive testing strategies that ensure AI-powered applications, agents, and intelligent workflows are accurate, reliable, secure, performant, and production-ready.
The ideal candidate will combine deep expertise in Quality Engineering with hands-on experience evaluating Large Language Models (LLMs), AI agents, AI-assisted workflows, and enterprise AI platforms. You will work closely with Engineering, Architecture, Security, Risk, and Product teams to provide evidence-based assessments and go/no-go recommendations for production deployment.
Key Responsibilities
- Define, develop, and execute end-to-end testing strategies for AI/ML and Generative AI solutions.
- Validate the accuracy, reliability, safety, and performance of AI models, AI agents, and LLM-powered applications.
- Design and maintain automated test frameworks using Python and modern test automation tools.
- Create and manage AI-specific test datasets, prompts, evaluation criteria, and benchmark scenarios.
- Conduct testing focused on:
- Groundedness and response relevance
- Hallucination detection and mitigation
- Bias, fairness, and safety validation
- Regression testing
- API and integration testing
- Functional and non-functional testing
- Performance and scalability testing
- Security and penetration testing
- Evaluate retrieval-augmented generation (RAG) implementations, agentic workflows, and enterprise LLM integrations.
- Develop auditable test evidence and quality metrics to support governance, risk, compliance, and architecture review processes.
- Analyze defects, trends, and quality indicators, providing actionable recommendations for remediation.
- Collaborate with development, product, cybersecurity, privacy, and risk teams throughout the software delivery lifecycle.
- Produce detailed quality assessments and deployment readiness recommendations.
Required Qualifications
- 10+ years of experience in Quality Assurance, Quality Engineering, or Software Testing.
- 2+ years of hands-on experience evaluating AI, Machine Learning, or Generative AI solutions.
- Strong programming and scripting skills in Python.
- Extensive experience with: Test automation frameworks, API testing, Integration testing, Regression testing, non-functional testing, Performance and load testing
- Experience designing prompts, test suites, and evaluation datasets for AI systems.
- Deep understanding of: LLM evaluation methodologies, Hallucination assessment, Response quality measurement, Groundedness and relevance testing, Bias and safety evaluation, Experience working within Agile delivery environments.
- Strong analytical, troubleshooting, and communication skills.
Desired Attributes
- Strong attention to detail and quality mindset.
- Ability to assess complex AI systems and communicate risks effectively.
- Experience producing executive-ready quality reports and recommendations.
- Comfortable working with cross-functional stakeholders across engineering, architecture, security, privacy, and business teams.
- Proven ability to drive testing strategy for complex enterprise technology initiatives.
