Ulysses-RFSQ: improving information retrieval through relevance feedback for similar queries

Abstract

The use of Relevance Feedback can enhance the Information Retrieval (IR) performance, but this method is often used only to improve the retrieval for a specific query: the one currently being processed. When there is available relevance information from past searches, this information may be useful to help future searches. If two queries are sufficiently similar, the relevant documents for one may also be relevant for the other. However, only a few studies were found in the literature dealing with this use of relevance information from past queries, as there is a lack of benchmark datasets containing this information for similar queries. In this sense, this study presents Ulysses-RFSQ, a novel IR method that aims to improve the results for future queries by using the Relevance Feedback information from past similar ones. It works by re-ranking the list of documents retrieved by a base IR algorithm through the addition of a bonus or a penalty to the documents’ score. Therefore, it can be used with any algorithm that computes a score for the documents, such as BM25 or Sentence-BERT models. To evaluate the Ulysses-RFSQ method, a Relevance Feedback dataset, called Ulysses RFCorpus, was built together with the Brazilian Chamber of Deputies and made available to the community. Besides Ulysses-RFCorpus, the proposed method was also evaluated in larger dataset (the Preliminary Search corpus) provided by the Chamber, which could not be made available. The method’s evaluation in the legislative scenario is justified by the fact that most of the queries used in the Brazilian legislative process are redundant. As results, the findings pointed out that Ulysses-RFSQ can use the past feedback information from similar queries to improve the base algorithm’s performance for future queries. Improvements in MAP, MRP, MRR, and nDCG showed that the proposed method could re-rank the retrieved documents list in a way that can rearrange the relevant documents in the first positions while fetching relevant documents not retrieved by the base IR algorithm. The improvements could be better seen in scenarios in which the base IR algorithm did not achieve great results and while using a larger set of stored queries. For instance, the observed improvements in the MAP results ranged from 0.0384 to 0.0773 for the Preliminary Search corpus — in some cases, more than doubling the baseline’s performance.