About ALTARS
Augmented Intelligence (AL) is “a subsection of AI machine learning developed to enhance human intelligence rather than operate independently of or outright replace it. It is designed to do so by improving human decision-making and, by extension, actions taken in response to improved decisions.” In this sense, users are supported, not replaced, in the decision-making process by the filtering capabilities of the Augmented Intelligence solutions, but the final decision will always be taken by the users who are still accountable for their actions.
In this field, Technology-Assisted Review Systems (TARS) use a kind of human-in-the-loop approach where classification and/or ranking algorithms are continuously trained according to the relevance feedback from expert reviewers, until a substantial number of the relevant documents are identified. This approach has been shown to be more effective and more efficient than traditional e-discovery and systematic review practices, which typically consists of a mix of keyword search and manual review of the search results.
Given these premises, ALTARS will focus on High-recall Information Retrieval (IR) systems which tackle challenging tasks that require the finding of (nearly) all the relevant documents in a collection. Electronic discovery (eDiscovery) and systematic review systems are probably the most important examples of such systems where the search for relevant information with limited resources, such as time and money, is necessary.
CALL FOR PAPERS ALTARS 2024
In this workshop, we aim to fathom the effectiveness of these systems which is a research challenge itself. In fact, despite the number of evaluation measures at our disposal to assess the effectiveness of a "traditional" retrieval approach, there are additional dimensions of evaluation for TAR systems.
For example, it is true that an effective high-recall system should be able to find the majority of relevant documents using the least number of assessments. However, this type of evaluation discards the resources used to achieve this goal, such as the total time spent on those assessments, or the amount of money spent for the experts judging the documents.
The topics include, but are not restricted to:
- Novel evaluation approaches and measures for Systematic reviews;
- Reproducibility of experiments with test collections;
- Design and evaluation of interactive high-recall retrieval systems;
- Study of evaluation measures;
- User studies in high-recall retrieval systems;
- Novel evaluation protocols for continuous Active Learning;
- Evaluation of sampling bias.
SUBMISSION FORMAT
Research papers, describing original ideas on the listed topics and on other fundamental aspects of Technology-Assisted Reviews methodologies and technologies, are solicited. Moreover, short papers on early research results, new results on previously published works, and extended abstract on previously published works are also welcome.
- Research papers presenting original works should be in the 9 - 10 pages range,
- short papers should be in the 6 - 7 pages range
- posters should be 3 - 4 pages long
- extended abstracts should be 2 pages long
Templates:
The accepted papers will be published in the ALTARS 2024 Proceedings. The Proceedings will be published by CEUR-WS, which is gold open access and indexed by SCOPUS and DBLP.
SUBMISSION MODE
Authors must submit their papers via Easychair: https://easychair.org/conferences/?conf=altars2024.
Important Dates
Submission Deadline):
Acceptance Notification:
Workshop:
Organization
Workshop Chairs
- Giorgio Maria Di Nunzio, University of Padua (Italy)
- Evangelos Kanoulas, University of Amsterdam (The Netherlands)
- Prasenjit Majumder, DAIICT, Gandhinagar and TCG CREST, Kolkata (India)
Program Committee
- Angela Condello , University of Messina (Italy)
- Faezeh Ensan , Toronto Metropolitan University (Canada)
- Parth Mehta , Parmonic (USA)
- Harrisen Scells , Lepizig University (Germany)
- Fabrizio Sebastiani , CNR-ISTI (Italy)
- Ricky Sethi , Fitchburg State University (USA)
- Rene Spijker , Cochrane (The Netherlands)
- Mark Stevenson , University of Sheffield (UK)
- Wojciech Kusa , TU Wien (Austria)
- Jyothi Vinjumar , Walmart (USA)
- Eugene Yang , Johns Hopkins University (USA)
PROGRAM
Local (GMT) Times
Breakfast Coffee
Setup Video/Audio
Opening
Invited talk
Title: A knowledge modeling approach for AI applications based on legal reasoning in the semantic Web
Enrico Francesconi
Research Director IGSG - CNR Institute of Legal Informatics and Judicial Systems National Research Council of Italy
Abstract:
Transforming the Law as machine readable code represents a precondition for developing advanced information services in the legal domain endowed with reasoning facilities. In this talk we present an approach for legal knowledge representation and reasoning in the Semantic Web. It is based on the distinction between provisions and norms and it is able to provide reasoning facilities (like Hohfeldian reasoning) for advanced legal information retrieval, as well as legal compliance checking for deontic norms. It is also shown how the approach can handle norm defeasibility. Such methodology is implemented with decidable fragments of OWL 2, while legal reasoning is implemented through available decidable reasoners.
Coffee Break
Papers
An Interdisciplinary Approach to Legal Terminology: Challenges of the “Algorithmic Turn” in Legal Science
Angela Condello, Giorgio Maria Di Nunzio
Stopping Methods Based on Point Processes: Recent Developments
Mark Stevenson and Reem Bin-Hezam
Towards Explainable Total Recall in TAR for eDiscovery Using Retraining of the Underlying Neural Network Model
Charles Courchaine, Corey Wade, Tasnova Tabassum, Stetson Daisy and Ricky J. Sethi
Lunch Break
Papers
Factors Affecting the Performance of Reviewers in a Large-Scale Technology-Assisted Review Project
Maura Grossman, Gordon Cormack, Andrew Harbison, Tom O'Halloran and Bronagh McManus
Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study
Lena Schmidt, Kaitlyn Hair, Sergio Graziozi, Fiona Campbell, Claudia Kapp, Alireza Khanteymoori, Dawn Craig, Mark Engelbert and James Thomas
Leveraging Cochrane Systematic Literature Reviews for Prospective Evaluation of Large Language Models
Wojciech Kusa, Harrisen Scells, Moritz Staudinger and Allan Hanbury
Coffee Break
New Topics
Title: Extracting information from scientific articles to compile data on ecological impacts of biological invasions
Franck Courchamp
Senior Researcher - CNRS, France
Abstract:
Despite the importance of the ecological impacts of biological invasions, our understanding remains limited due to their diversity, which prevents any generalization. There are indeed at least 16 different types of ecological impacts, from local extinction to habitat degradation, and from trophic web modification to ecosystem service loss, and these are difficult to compare and impossible to compile. It is therefore crucial to develop better methods for assessing these impacts globally and understanding their underlying mechanisms. The aim of our project is to fill this critical gap by carrying out an integrated assessment of these impacts, which we will achieve by standardizing and quantifying all these impacts through a unified metrics. This will be obtained through the application of this metric to all known impact that have been published in the literature, to populate a database of quantified ecological impacts and their various descriptors. In order to achieve this, we need to obtain a dataset of all published studies on ecological impacts in the scientific literature, and to extract information related to these impacts from the pdf of these studies. As there is an estimated 25,000 such studies, we can only hope to achieve this through AI and AL.
Building a Network and a Research Project in TAR Systems
Open discussion and closing
Worskhop Venue
The 3rd ALTARS Wokrshop is colocated with ECIR 2024 and will be take place at the Radisson Blu Hotel, Glasgow