The effect of variations in relevance assessments in comparative experimental tests of index languages

It was desired to check whether the unexpected test results obtained in Cranfield II had been influenced by the relevance decisions. Three new sets of relevance decisions were therefore obtained, and the Cranfield II results were re-calculated for nineteen index languages on the basis of the new relevance judgements. A rank order of the index languages was found; in no case did the correlation co-efficient of any of the three new rank orders fall below 0.921 when compared with the original Cranfield II results. The findings of a similar type of test by Lesk and Salton are considered, and the conclusion is reached that the results of Cranfield II had showed that the superiority of the single term index languages were not significantly affected by relevance judgements.