The Perspective Argument Retrieval Shared Task

Latest News:

Registration and more:

You can register for the shared task using this registration form

You can download the data and evaluation script from this repository

Join our slack channel for any information or to ask questions here

Or write us an e-mail perspectiveargretrieval@gmail.com

About

The "Perspective Argument Retrieval" task addresses the often-overlooked challenge of incorporating socio-cultural factors (such as political views, occupation, age, gender) in argument retrieval. By focusing on these aspects, we acknowledge their potential latent influence on argumentation.

With this shared task, we invite the community to develop methods that concentrate on this crucial area and advance state-of-the-art retrieval models by considering the perspective of societal diversity.

Task Description

Argument retrieval is the task of retrieving a set of top-k relevant arguments out of a corpus given a specific query. With this shared task, we formulate perspective argument retrieval as an expansion of argument retrieval considering sociocultural factors. Concretely, this task proposes three scenarios of varying difficulties to considering socio-cultural profiles. Therefore, we want to foster approaches taking into account latent aspects of argumentation beyond semantic features, such as personal attitude. We consider these aspects both during retrieval and evaluation.

Retrieval Scenarios

Scenario 1: Baseline retrieval of relevant arguments given a specific query from a given corpus. Therefore, we evaluate the general abilities of a system to retrieve relevant arguments.

Example query: Are you in favor of the introduction of a tax on foods containing sugar (sugar tax)?

Example relevant candidates: The reduction of sugar in food should be pushed. Not every food needs additional sugar as a supplement.

Scenario 2: Explicit perspectivism extends the baseline task by explicitly adding socio-cultural information to the query and the corpus and limiting the relevant candidates to arguments from authors matching the corresponding socio-cultural background. With this second scenario, we test whether a retrieval system can consider socio-cultural properties when explicitly mentioned in the query and the candidates.

Example query: Given a left attitude, are you in favor of the introduction of a tax on foods containing sugar (sugar tax)?

Example relevant candidates: With a left attitude, reducing sugar in food should be pushed. Not every food needs additional sugar as a supplement.

Scenario 3: Implicit perspectivism This scenario is similar to explicit perspectivism, but we only add socio-cultural information to the query. Therefore, we test the ability of a retrieval system to account for latently encoded socio-cultural information within the argument.

Example query: Given a liberal attitude, are you in favor of the introduction of a tax on foods containing sugar (sugar tax)?

Example relevant candidates: Eating is an individual decision. It doesn't need a nanny state.

Retrieval Evaluation

We employ for all three scenarios a two-folded evaluation for a comprehensive measure of the retrieval quality. Concretely, we distinguish between relevance and diversity / fairness:

With relevance, we focus on the ability of a retrieval system to select the relevant candidates, for example, all arguments addressing the queried issue for the baseline scenario or arguments that additionally match specific demographic properties for explicit or implicit perspectivism.

Using diversity, we account for the influence of perspectivism in the evaluation by measuring to what extend a retrieval system retrieval system diversifies the relevant arguments regarding stance distribution or other socio-cultural factors, such as age or education.

Data

The data and evaluation script can be downloaded from this repository

This shared task is grounded on the x-stance dataset (Vamvas & Sennrich, 2020), providing arguments annotated with their stance regarding different political issues gathered from the voting recommendation platform https://www.smartvote.ch/. This platform provides voting suggestions based on a questionnaire that politicians and voters fill out. Therein, politicians can argue why they are in favor or against specific political issues.

We use the arguments covering the 2019 Swiss Federal elections as a corpus and the political issues as queries. Afterward, we enrich these arguments with eight socio-cultural properties, either provided by the voting platform itself (gender, age, party, …) or derived from the filled-out questionnaire of the politicians (political attitude, important political aspects, …). This collection encapsulates 26,335 arguments for 45 political aspects from German, French, and Italian.

We generate the train and development splits by considering 35 political aspects for training and 10 for development, while the argument corpus is used for both sets. Apart from the queries for the baseline scenario, we will also provide queries for the perspectivism scenarios, including socio-cultural information. As the x-stance dataset is publicly available, final evaluation data consist of secret test sets.

Submission Policy

You may use any external data source for pre-training your models. However, we do not accept submissions using proprietary LLMs (e.g., GPT-4). Please do not input any of the data into the Chat-GPT online interface to avoid data leakage. You are allowed to use any open-source LLMs. Have a look at this website for a list of frequently used open-source LLMs in case you want to use those.

There are three test sets for the evaluation of the shared task. The first test set is taken from the election of 2019, the second from the year 2023 and the third is a suprise test set. The final evaluation will be on the 8th of May. You can submit all final predictions until the 7th of May, 11.59 pm UTC -12h (“anywhere on Earth”). If you upload predictions before you will see the results of those predictions on the leaderboard (once on the 24th of April and once on the 30th of April). You can change the predictions for all test sets, all scenarios, until the final deadline. You can also submit partial results, however keep in mind that for the final ranking, all predictions will be considered. (average across all test sets and scenarios).

Important Dates

• Release Training Data: 4th March
• Release Test Set 1: 17 March
• First Evaluation Cycle (Test set 1): 24 April
• Second Evaluation Cycle (Test set 2): 2 May
• Third Evaluation Cycle (Surprise Test set): 13 May
• Submission Final Run: 17 May
• Evaluation Final Run: 18 May
• Submission System Description Paper: 27 May
• Camera-ready papers due: 1st July
• Workshop / Shared Task: 15th of August 2024

Leaderboards

First Evaluation Circle (Test set 1, election 2019)

Relevance - Scenario Baseline

Diversity - Scenario Baseline

Relevance - Scenario Explicit

Diversity - Scenario Explicit

Relevance - Scenario Implicit

Diversity - Scenario Implicit

Second Evaluation Circle (Test set 2, election 2023)

Relevance - Scenario Baseline

Diversity - Scenario Baseline

Relevance - Scenario Explicit

Diversity - Scenario Explicit

Relevance - Scenario Implicit

Diversity - Scenario Implicit

Sponsor

A special thank you to Summetix for their generous support! Thanks to our sponsor Summetix we will be able to provide prizes in form of Amazon vouchers for the three best teams in each track: relevance and diversity.

• 1st place: 130 $
• 2nd place: 80 $
• 3rd place: 40 $

Organizing Committee

Andreas Waldis

Neele Falk

Gabriella Lapesa

Iryna Gurevych

Policy

We abide by the ACL anti-harassment policy.