CLEF Lab – Call for Participation – Living Labs for IR Evaluation (LL4IR)

The Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab

September, 2015, Toulouse, France

Register at:

[Registration closes: 30 April, 2015]


This lab provides a benchmarking platform for researchers to evaluate their ranking systems in a live setting with real users in their natural task environments. The lab acts as a proxy between commercial organizations (live environments) and lab participants (experimental systems).

The lab focuses on three use-cases (using a shared API):

Task 1: product search (on the REGIO JÁTÉK e-commerce site)
Task 2: local domain search (on the University of Amsterdam’s website)
Task 3: web search (through Seznam, a major commercial web search engine)


What is in it for you as a lab participant?


– Access to privileged commercial data (click through, etc.).
– Access to new static TREC-style collections.
– Access to the living labs evaluation environments.
– Opportunity to test your IR systems on real live systems with real live users.


Current Important dates


31 Mar, 2015: Training period ends

1-15 Apr, 2015: Uploading test runs

15 Apr, 2015: Testing period begins

15 May, 2015: Testing period ends


Further Information


Further information is available on the lab website ( or by emailing the lab organisers.


Lab Organisers


Krisztian Balog – University of Stavanger, Norway (krisztian.balog (at)

Liadh Kelly – Trinity College Dublin, Ireland (liadh.kelly (at)

Anne Schuth – University of Amsterdam, The Netherlands (anne.schuth (at)

Source: IRList

NLP News