Back to all jobs
Engineering & Development
Full-time
November 13, 2026

Senior AI Quality & Evaluation Engineer

๐ŸŒ USA, Polandโฐ Full-time๐Ÿ“… Apply by May 18, 2027
๐Ÿ’ป Engineering & DevelopmentFull-time
Solvd ยท Direct from employer
USA, Poland
NEW Join our Discord for daily job alerts! โ†’

Solvd is hiring a

Senior AI Quality & Evaluation Engineer

Description

Solvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.

Following the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end deliveryโ€”from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively.

We are looking for a Senior AI Quality & Evaluation Engineer to own the independent proof that retrieval works and that permissions hold. Youโ€™ll build and run the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates โ€” so the engineers who built the system are not grading their own work.

This is a 50% allocation role across weeks 3โ€“10, drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day-to-day with the retrieval engineer.

What youโ€™ll do

  • Build the golden question suite from top contact drivers and needs workshops โ€” question, expected source, expected passage, and acceptance thresholds for relevance, grounding, and citation accuracy, versioned alongside the content it tests.

  • Build the negative entitlement test set โ€” queries issued under unentitled identities that must return no restricted passage, with any failure raised as a severity-one alarm.

  • Build freshness and retirement tests โ€” source-modified-to-index-available lag per item against the agreed threshold, and verification that retired items disappear on the next sync.

  • Automate the harness to run on schedule and after every content, configuration, or model change โ€” publishing scores to the quality dashboard with trend lines and regression alarms.

  • Define the QA gate content analysts apply to remediated content.

  • Run the suite before every pilot change in weeks 7โ€“9 and report misses with root cause โ€” content, chunking, ranking, or permissions โ€” to the right owner.

  • Write the evaluation runbook and train client stewards and platform owners to add questions, re-run the suite, and interpret regressions.

Required skills & experience

  • 3+ years in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule.

  • Hands-on evaluation of a search or RAG system in production โ€” writing ground truth sets, scoring relevance and grounding, and tracking drift over time.

  • Experience designing security-focused negative tests โ€” authorization, entitlement, data exposure โ€” and reporting them at the severity a client security team expects.

  • Ability to work in AWS โ€” invoking Bedrock and OpenSearch APIs, reading CloudWatch, deploying a scheduled job with infrastructure code.

  • Clear written reporting; the suiteโ€™s output is read by client stewards, not only engineers.

Requirements

  • AWS Certified AI Practitioner or Machine Learning Engineer Associate.

  • Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.

  • Contact center or knowledge management domain exposure.

  • Familiarity with SOC 2 or ISO 27001 evidence practices.

When you join Solvd, you'llโ€ฆ

  • Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation.

  • Be part of a global team with equal opportunities for collaboration across continents and cultures.

  • Thrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.

Ready to make an impact?

If you're excited to build things that matter, champion responsible AI, and grow with some of the industryโ€™s sharpest minds. Apply today and letโ€™s innovate together.

Solvd is an equal opportunity employer.

Apply for this job

You will be sent to the employer's application page.

Questions about this role

Where can I work from for this Senior AI Quality & Evaluation Engineer role?

Solvd lists the location as "USA, Poland". If your country is not named, check the job description for eligibility rules before applying.

Is this a fully remote job?

Yes. Every role on Euro Remote Talent is remote. This one is full-time, for candidates based in USA, Poland.

What does this role pay?

Solvd has not published a salary for this role. Ask about the range in your first conversation; our salary calculator helps you set expectations.

How do I apply to Solvd?

Click Apply now. Create an account if you don't have one, or log in to your account, then apply to the job. Applications close on May 18, 2027.

Solvd

AI, Fintech, Ecommerce

Apply now

About Solvd

Solvd builds AI systems for enterprises, including core AI, AI data and cloud, and agentic software engineering solutions.

They work with clients across various industries, including e-commerce, healthcare, and finance, to deliver production-ready AI systems.

๐Ÿญ Industry
AI, Fintech, Ecommerce
All jobs at Solvd โ†’
Job summary
  • TypeFull-time
  • LocationUSA, Poland
  • CategoryEngineering & Development
  • PostedNovember 13, 2026
  • DeadlineMay 18, 2027
About the company

Solvd builds AI systems for enterprises, including core AI, AI data and cloud, and agentic software engineering solutions. They work with clients across various industries, including e-commerce, healthcare, and finance, to deliver production-ready AI systems.

  • IndustryAI, Fintech, Ecommerce
  • Websitesolvd.com

Solvd

Senior AI Quality & Evaluation Engineer

Apply now
Share this job
00
November 13, 2026
May 18, 2027

Section heading style, do not delete

Sub heading style, do not delete

apply
Julien Sicard
Book A Call
Get started today

For Employers

For Job Seekers

Remote Jobs by Category

Remote Jobs by Country

Resources

Collaboration

Legal

Compare

About Euro Remote Talent

Connect

Address

Euro Remote Talent LLC
30 N Gould St Ste N
Sheridan, WY 82801, USA