About this role
Summary: This role builds and operates evaluation tooling, test suites, and automated quality checks to measure and protect the quality of MSC AI systems. The engineer provides reliable evidence for release and governance decisions by implementing LLM-as-judge evaluators, adversarial testing, and continuous integration pi...
Job Purpose
The role builds and operates the evaluation tooling, test suites and automated quality checks used to measure and protect the quality of MSC AI systems. It is a hands-on individual-contributor role that provides reliable evidence for release, engineering and governance decisions within approved quality criteria and decision authorities.
Key Accountabilities
• Build and maintain evaluation harnesses that score the quality of MSC AI services, including LLM-as-judge evaluators, deterministic retrieval and tool-call assertions, and the supporting datasets.
• Implement and calibrate approved judging criteria used to score AI output, analysing variance and documenting results to support stable and meaningful evaluation.
• Execute and maintain scheduled evaluation runs that support production-canary monitoring for AI services, including the approved synthetic user personas used in those runs.
• Build and maintain adversarial and red-team test suites that probe AI services for policy breaches, prompt injection, data disclosure, persona breaks and off-topic behaviour.
• Implement and run the evaluation checks that support release decisions for prompt, model and retrieval changes, and provide comparison evidence to the designated release authority.
• Contribute technical analysis to the definition of acceptance criteria and configure the approved criteria in automated evaluation workflows.
• Integrate evaluation checks into the delivery pipeline so that relevant quality controls run automatically on the code paths they cover.
• Build and maintain reporting that makes AI quality visible, including dashboards, trend analysis and alerts for quality regressions.
• Turn production incidents and reported failures into regression cases within the evaluation datasets.
• Work with the engineering team to improve AI-output quality based on evaluation findings.
• Produce and maintain documentation of the evaluation approach, approved criteria and results, including evidence required to support AI-governance and regulatory processes.
Qualifications
• English as the primary working language.
• Degree in Computer Science, Information Technology, Engineering or a related field.
• 4-7 years of total professional experience.
• Hands-on experience in software engineering, test automation or quality engineering, including Python.
• Strong Python development skills; this is a hands-on engineering role focused on building and maintaining evaluation tooling, not only executing test plans.
• Practical experience testing or evaluating LLM-based systems, including LLM-as-judge approaches and handling non-deterministic output.
• Understanding of how AI systems fail in production, including hallucination, retrieval gaps, prompt injection, jailbreaks and unsafe or off-brand output.
• Experience designing test datasets and regression suites and maintaining them as products change.
• Experience integrating automated testing into continuous-integration pipelines.
• Analytical mindset, with the ability to distinguish a genuine quality regression from normal variance in a non-deterministic system.
• Good communication and interpersonal skills, including the ability to present quality evidence and constructively challenge engineering and product decisions.
• Familiarity with LLM frameworks such as LangChain, LangGraph or the Microsoft Agent Framework is an advantage.
• Experience with LLM tracing and evaluation tooling such as Langfuse, DeepEval or comparable technologies is an advantage.
• Experience evaluating retrieval-augmented generation systems, including retrieval-quality measurement, is an advantage.
• Experience with adversarial testing or AI red-teaming is an advantage.
• Familiarity with AI-governance or regulatory regimes such as the EU AI Act is an advantage.
• Background in travel, hospitality or another large-scale consumer business is an advantage.
VISA Requirements
Right to work in Italy
What we offer
In compliance with the applicable pay transparency legislation, the following information is provided regarding the initial compensation foreseen for the position.
Contract
• Type of contract: Permanent
• Hybrid working model: 2 days per week remotely
• Applicable Collective Labour Agreement (CCNL): “Addetti agli uffici ed ai terminals delle società di navigazione che esercitano l’armamento privato” – sez. personale di terra
• Gross annual salary (RAL): between €40.000 – €47.000 gross per year, paid over 14 monthly installments
• Meal vouchers: worth €8 per on-site working day
The offer in terms of salary will be determined based on the candidate’s professional experience, technical competencies, and the level of autonomy and responsibility assigned to the role.
Welfare&Wellbeing
We offer a range of Welfare&Wellbeing Initiatives as well as benefits including Productivity award, Health Insurance, Learning opportunities, flexible working hours, Employee Assistance Programme, Discounts & Partnerships within the MSC group.
Our commitment
We are committed to building a future that values diverse perspectives, embraces the world beyond borders, and fosters an inclusive environment where every individual feels valued, respected and empowered to be their authentic selves. Our commitment extends to taking meaningful, measurable actions that have a long-term positive impact on our guests, our employees and our planet.
Ready to turn your passion into something extraordinary? Join us at MSC Cruises, where new opportunities await. Apply today to be part of a global team that is pushing boundaries and achieving something remarkable. Your journey starts here!