CLEF 2018 technologically assisted reviews in empirical medicine overview

Evangelos Kanoulas, Dan Li, Leif Azzopardi, Rene Spijker

Research output: Contribution to journalConference articlepeer-review

13 Citations (Scopus)
9 Downloads (Pure)


Conducting a systematic review is a widely used method to obtain an overview over the current scientific consensus on a topic of interest, by bringing together multiple studies in a reliable, transparent way. The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying all relevant studies in an unbiased way both complex and time consuming to the extent that jeopardizes the validity of their findings and the ability to inform policy and practice in a timely manner. The CLEF 2018 e-Health Technology Assisted Reviews in Empirical Medicine task aims at evaluating search algorithms that seek to identify all studies relevant for conducting a systematic review in empirical medicine. The task had a focus on Diagnostic Test Accuracy (DTA) reviews, and consisted of two subtasks: 1) given a number of relevance criteria as described in a systematic review protocol, search a large medical database of article abstracts (PubMed) to find the studies to be included in the review, and 2) given the article abstracts retrieved by a carefully designed Boolean Query, prioritize them to reduce the effort required by experts to screen the abstracts for inclusion in the review. Seven teams participated in the task, with a total of 12 runs submitted for subtask 1 and 19 runs for subtask 2. This paper reports both the methodology used to construct the benchmark collection, and the results of the evaluation.

Original languageEnglish
Number of pages34
JournalCEUR Workshop Proceedings
Publication statusPublished - 24 Jul 2018
Event19th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF 2018 - Avignon, France
Duration: 10 Sep 201814 Sep 2018


  • active learning
  • benchmarking
  • Cochrane
  • diagnostic test accuracy
  • DTA
  • e-health
  • evaluation
  • high recall
  • information retrieval
  • PubMed
  • relevance feedback
  • systematic reviews
  • TAR
  • technology assisted reviews
  • test collection
  • text classification


Dive into the research topics of 'CLEF 2018 technologically assisted reviews in empirical medicine overview'. Together they form a unique fingerprint.

Cite this