Senior AI QA / Evaluation Engineer

About the role

We are seeking a Senior AI QA / Evaluation Engineer to own quality assurance for production AI systems in a regulated enterprise environment. The role will focus on testing, evaluating, and safeguarding large-language-model (LLM) and machine-learning (ML) applications.

Key responsibilities

• Build versioned golden datasets and evaluation harnesses for AI agents.
• Gate production releases by running evaluation baselines in CI pipelines.
• Re‑validate agents whenever model versions change and run quality control on recurring reports.
• Conduct adversarial tests for prompt injection, data leakage, and unsafe tool usage.
• Triage support incidents and maintain audit evidence packs.

Required profile

• Based in Baku or willing to relocate.
• 4+ years of QA or test‑engineering experience with test‑automation ownership.
• Proven experience evaluating LLM or ML outputs.
• Strong Python programming skills for building evaluation tooling.
• Fluent English communication.

Required skills

• Python, pytest, API testing, Playwright, CI/CD pipelines.
• Evaluation frameworks such as Ragas, DeepEval, promptfoo.
• Adversarial testing tools: OWASP Top 10 for LLMs, garak, PyRIT.
• Observability platforms like Langfuse or Arize Phoenix.
• Formal defect‑management processes.