ZURÜCK Relevance Judgements

Relevance judgements are made after the experiments using a pooling method: The top 100 documents of all lists submitted for a query are taken together to form a "document pool". This pool is intellectually checked for relevant documents by an expert. This method was used to reduce the number of documents to be checked. It assumes that most relevant documents are ranked within the top 100 documents by at least one of the retrieval systems. This is realistic only for a large number of participating systems. If the result lists are very similar, the probability that a relevant document is missed by the pooling method increases.

The quality of the pooling method for relevance judgement was tested in TREC 3: In addition to the pool with 100 top ranked documents a second pool with the 200 top ranked documents was used for relevance judgement. Whereas in the first pool on average 146 relevant documents were found, this number was 196 for the second pool. This shows, that a remarkable number of relevant documents was probably missed. (More detailed results are given in fig .) This leads to an overestimation of recall and a underestimation of precision. But this is true for all participating systems.

ZUGANGAbb. 24: Size of Document Pools Used for Relevance Assessment (from 1995 - WWW, 1996 - WWW)


© 1998 / HTML-Version 17. 11. 1998: R. Ferber