publications

2024

1. EMNLP’24

   Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model

   Nilanjan Sinhababu, Andrew Parry, Debasis Ganguly, Debasis Samanta, and Pabitra Mitra

   EMNLP 2024 Findings, 2024
2. ACL’24

   Exploiting Positional Bias for Query-Agnostic Generative Content in Search

   Andrew Parry, Sean MacAvaney, and Debasis Ganguly

   In Findings of the Association for Computational Linguistics ACL 2024 , Aug 2024

   Abs

   In recent years, research shows that neural ranking models (NRMs) substantially outperform their lexical counterparts in text retrieval. In traditional search pipelines, a combination of features leads to well-defined behaviour. However, as neural approaches become increasingly prevalent as the final scoring component of engines or as standalone systems, their robustness to malicious text and, more generally, semantic perturbation needs to be better understood. We posit that the transformer attention mechanism can induce exploitable defects in search models through sensitivity to token position within a sequence, leading to an attack that could generalise beyond a single query or topic. We demonstrate such defects by showing that non-relevant text–such as promotional content–can be easily injected into a document without adversely affecting its position in search results. Unlike previous gradient-based attacks, we demonstrate the existence of these biases in a query-agnostic fashion. In doing so, without the knowledge of topicality, we can still reduce the negative effects of non-relevant content injection by controlling injection position. Our experiments are conducted with simulated on-topic promotional text automatically generated by prompting LLMs with topical context from target documents. We find that contextualisation of a non-relevant text further reduces negative effects whilst likely circumventing existing content filtering mechanisms. In contrast, lexical models are found to be more resilient to such content injection attacks. We then investigate a simple yet effective compensation for the weaknesses of the NRMs in search, validating our hypotheses regarding transformer bias.

3. SIGIR’24

   "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"

   Andrew Parry, Debasis Ganguly, and Manish Chandra

   SIGIR 2024, May 2024

   arXiv:2405.01116 [cs]

   Abs PDF

   With the increasing ability of large language models (LLMs), in-context learning (ICL) has evolved as a new paradigm for natural language processing (NLP), where instead of fine-tuning the parameters of an LLM specific to a downstream task with labeled examples, a small number of such examples is appended to a prompt instruction for controlling the decoder’s generation process. ICL, thus, is conceptually similar to a non-parametric approach, such as \k\-NN, where the prediction for each instance essentially depends on the local topology, i.e., on a localised set of similar instances and their labels (called few-shot examples). This suggests that a test instance in ICL is analogous to a query in IR, and similar examples in ICL retrieved from a training set relate to a set of documents retrieved from a collection in IR. While standard unsupervised ranking models can be used to retrieve these few-shot examples from a training set, the effectiveness of the examples can potentially be improved by re-defining the notion of relevance specific to its utility for the downstream task, i.e., considering an example to be relevant if including it in the prompt instruction leads to a correct prediction. With this task-specific notion of relevance, it is possible to train a supervised ranking model (e.g., a bi-encoder or cross-encoder), which potentially learns to optimally select the few-shot examples. We believe that the recent advances in neural rankers can potentially find a use case for this task of optimally choosing examples for more effective downstream ICL predictions.

4. SIGIR’24

   "Axiomatic Guidance for Efficient and Controlled Neural Search"

   Andrew Parry

   SIGIR 2024, May 2024
5. ECIR’24

   Analyzing Adversarial Attacks on Sequence-to-Sequence Relevance Models

   Andrew Parry, Maik Fröbe, Sean MacAvaney, Martin Potthast, and Matthias Hagen

   ECIR, May 2024

   Book Title: Advances in Information Retrieval ISBN: 9783031560590 9783031560606 Place: Cham Publisher: Springer Nature Switzerland

   Abs PDF

   Modern sequence-to-sequence relevance models like monoT5 can effectively capture complex textual interactions between queries and documents through cross-encoding. However, the use of natural language tokens in prompts, such as Query, Document, and Relevant for monoT5, opens an attack vector for malicious documents to manipulate their relevance score through prompt injection, e.g., by adding target words such as true. Since such possibilities have not yet been considered in retrieval evaluation, we analyze the impact of query-independent prompt injection via manually constructed templates and LLM-based rewriting of documents on several existing relevance models. Our experiments on the TREC Deep Learning track show that adversarial documents can easily manipulate different sequence-to-sequence relevance models, while BM25 (as a typical lexical model) is not affected. Remarkably, the attacks also affect encoder-only relevance models (which do not rely on natural language prompt tokens), albeit to a lesser extent.

6. ReNeuIR@SIGIR’24

   Top-Down Partitioning for Efficient List-Wise Ranking

   Andrew Parry, Sean MacAvaney, and Debasis Ganguly

   ReNeuIR@SIGIR 2024, May 2024

   Abs PDF

   Large Language Models (LLMs) have significantly impacted many facets of natural language processing and information retrieval. Unlike previous neural approaches, the enlarged context window of these generative models allows for ranking multiple documents at once, commonly called list-wise ranking. However, there are still limits to the number of documents that can be ranked in a single inference of the model, leading to the broad adoption of a sliding window approach to identify the 𝑘 most relevant items in a ranked list. We argue that the sliding window approach is not well-suited for list-wise re-ranking because it (1) cannot be parallelized in its current form, (2) wastes compute repeatedly re-scoring the best set of documents as it works its way up the initial ranking, and (3) prioritizes the lowest-ranked items for scoring rather than the highest-ranked items by taking a bottom-up approach. We investigate the precision of recent list-wise rankers within a single inference and find that when few relevant documents are found, it is preferable that these documents be at the start of a model’s context window. Motivated by this finding and the shortcomings of a sliding window approach, we propose a novel algorithm that partitions a ranking to depth 𝑘 and processes documents top-down. Unlike sliding window approaches, our algorithm is inherently parallelizable due to the use of a pivot element, which can be compared to documents down to an arbitrary depth concurrently. In doing so, we reduce the number of expected inference calls by around 33% when ranking at depth 100 while matching the performance of prior approaches across multiple strong re-rankers.

2023

1. TREC’23

   Generative Relevance Feedback and Convergence of Adaptive Re-Ranking: University of Glasgow Terrier Team at TREC DL 2023

   Andrew Parry, Thomas Jaenich, Sean MacAvaney, and I. Ounis

   In , May 2023

   Abs PDF

   This paper describes our participation in the TREC 2023 Deep Learning Track. We submitted runs that apply generative relevance feedback from a large language model in both a zero-shot and pseudo-relevance feedback setting over two sparse retrieval approaches, namely BM25 and SPLADE. We couple this first stage with adaptive re-ranking over a BM25 corpus graph scored using a monoELECTRA cross-encoder. We investigate the efficacy of these generative approaches for different query types in first-stage retrieval. In re-ranking, we investigate operating points of adaptive re-ranking with different first stages to find the point in graph traversal where the first stage no longer has an effect on the performance of the overall retrieval pipeline. We find some performance gains from the application of generative query reformulation. However, our strongest run in terms of P@10 and nDCG@10 applied both adaptive re-ranking and generative pseudo-relevance feedback, namely uogtr_b_grf_e_gb.