About this role
Summary: Internship focused on training vision-language models (VLMs) for table understanding using reinforcement learning without manual annotations. The role involves implementing new approaches, evaluating models on benchmarks, and documenting results within a research environment.
### Subject
Based in Saclay (Essonne), the LIST is one of the two institutes of CEA Tech, the Technological Research Division of the CEA. Dedicated to intelligent digital systems, its mission is to carry out technological developments of excellence on behalf of industrial partners, in order to create value.
Within the LIST, the Laboratory of Textual and Visual Semantic Analysis (LASTI) conducts its research in the field of natural language processing and computer vision to extract, classify and generate information. The laboratory's research themes include learning with few data, trustworthiness and multimodality.
### Job description
As an intern at the CEA, you will have the opportunity to work within a world-renowned research environment. Our teams consist of passionate and dedicated experts, providing an ideal setting for learning and collaboration. You will have access to state-of-the-art equipment and top-tier research resources to carry out your assignments. The work conducted may potentially lead to a scientific publication.
Context :
The team trains vision-language models (VLMs) that read an image of a table and output its structure. Manual table annotation is expensive, which limits classical supervised approaches.
This internship focuses on training VLMs with reinforcement learning (GRPO), using reward signals that do not depend on human annotations: for example, checking the geometric consistencies or generating synthetic tables whose structure is known by construction. The central question is which annotation-free signals correlate well with the real quality of a prediction, and how to combine them to improve the model on real documents.
What do we expect from you?
Get started with the team's training and evaluation environment
Analyse automatic evaluation methods
Propose and implement new approaches, and validate them through experiments
Work on the performance of the pipeline and the robustness of the models
Evaluate the models on public benchmarks
Document the results
The intern works on an existing pipeline, with access to a GPU cluster (Python, PyTorch, HuggingFace ecosystem, distributed training, experiment tracking). Supervision research engineers of the team, with a weekly meeting.
#*Cea-List*
### Applicant Profile
Students in their final year of studies (M2 or engineering school)
Python and PyTorch, Linux
Knowledge of deep learning and statistics
Good level of English
Appreciated: experience with HuggingFace Transformers, notions of reinforcement learning (GRPO, DPO, …)