Evaluation

• TREC 2001 QA Track (3rd year running)

• Corpus:

  • ~ 3 gigs of newspaper text
  • ~ 106 articles

• Test Questions: 500 questions

  • Mined from Web query logs

• Evaluation: performed manually by judges

• Score: MRR (mean reciprocal rank)

  • Score for each question is reciprocal of the rank in which the first correct answer appears.
  • Averaged over all questions.

Notes: