➡️ Apply here: LLM Test Automation Engineer
👩💼 Want to stand out? Improve your resume to appeal to recruiters, hiring managers, and Applicant Tracking Systems. ➡️ Improve your resume
We are looking for a skilled **LLM Test Automation Engineer** to design and implement an advanced testing framework for a Retrieval-Augmented Generation (RAG) Knowledge Base deployed in AWS.
This role is crucial in ensuring accuracy, relevance, safety, and scalability by leveraging cutting-edge metrics and techniques like RAGAS and LLM-as-a-judge.
*Experience the freedom of remote work from anywhere in Georgia, whether from the comfort of your home, our modern offices in Tbilisi and Batumi or a coworking space in Kutaisi.*
**Responsibilities**
* Audit the current LLM and RAG test automation framework, identify gaps, and propose improved architecture
* Implement automated testing pipelines incorporating RAGAS metrics and LLM-as-a-judge methods for quality assessment
* Integrate workflows with AWS services (S3, Lambda, CloudWatch, OpenSearch, RDS, SQS) for seamless operations
* Define clear evaluation rubrics, metric thresholds, regression alerts, and reporting dashboards to improve monitoring
* Ensure reproducible, scalable testing solutions within CI/CD pipelines
**Requirements**
* 3+ years of experience in LLM and RAG evaluation, including RAGAS and prompt-based judging automation
* Strong competency in AWS services (compute, storage, orchestration, monitoring)
* Background in vector databases and semantic similarity metrics
* Ability to establish KPIs and translate them into automated testing logic
* Proficiency in Python or Java
**Nice to have**
* Familiarity with LangChain, LlamaIndex, or similar RAG frameworks
* Understanding of CI/CD processes
**We offer**
* **We connect like-minded people**
* Delivering innovative solutions to industry leaders, making a global impact
* Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
* Opportunity to work abroad for up to two months per year
* Relocation opportunities within our offices in 55+ countries
* Corporate and social events
* **We invest in your growth**
* Leadership development, career advising, soft skills and well-being programs
* Certifications, including GCP, Azure and AWS
* Unlimited access to LinkedIn Learning and Get Abstract
* Free English classes with certified teachers
* **We cover it all**
* Participation in the Employee Stock Purchase Plan
* Monetary bonuses for engaging in the referral program
* Comprehensive medical & family care package
* Five trust days per year (sick leave without a medical certificate)
* Benefits package (sports activities, a variety of stores and services)
EPAM Georgia is a team of innovators united by a passion for technology. The dynamic and inclusive culture we embrace helps positively impact our communities, clients, and employees. Here you will collaborate with multi-national teams, contribute to numerous cutting-edge projects, deliver the most creative solutions, and have an opportunity to learn. Our people are at the heart of our success, and we are proud to provide talents with a solid ground to develop and grow.