How Consistent Are Human Judgments of Whether an Open Resource is Educational Material?

Harris, M. C., Thom, J. A. and Scholer, F.

Systems that filter web search results to return open educational resources need evaluation. The Cranfield method, which is widely used in information retrieval evaluation, can be used as the basis of a model for evaluating such systems. The Cranfield method requires a collection of resources with associated judgments. In this paper, we describe an experiment to collect judgments of whether open resources are educational material. We show experimentally that judges can agree on what resources are educational material, even in the absence of an educational context, and demonstrate that displaying the query used to retrieve the resource makes a judge less likely to rate a resource as educational.

Cite as: Harris, M. C., Thom, J. A. and Scholer, F. (2010). How Consistent Are Human Judgments of Whether an Open Resource is Educational Material?. In Proc. 21st Australasian Database Conference (ADC 2010) Brisbane, Australia. CRPIT, 104. Shen H.T. and Bouguettaya, A. Eds., ACS. 9-18

(from crpit.com) (local if available)