The Legal Bias Flagging Challenge is a new feature of DMAIL 2025, designed to advance the field of responsible and explainable AI in law. This competition invites participants to tackle the important task of flagging bias in legal texts, an issue that lies at the heart of fairness and accountability in automated legal systems.

Challenge Overview

Participants will develop models that automatically detect and explain potential bias in passages from legal case texts. Each task instance will include:
  • a highlighted passage potentially containing bias,
  • the full case text for broader contextual understanding,
  • and a language column.

The model must output a binary decision (Y/N) indicating whether the passage contains bias, along with an automatically generated explanation justifying the prediction. This dual-output task emphasizes both performance and interpretability.

For this challenge, bias is defined as follows (based on an excerpt from this dictionary https://dictionary.law.com/Default.aspx?selected=61): the predisposition of a judge, arbitrator, prospective juror, or anyone making a judicial decision, against or in favor of one of the parties or a class of persons. This can be shown by remarks, decisions contrary to fact, reason or law, or other unfair conduct.

The bias flagging task does not imply a judgment about the fairness of the entire legal proceeding. The annotations are based solely on the provided text snippets, which are only partial representations of the full judicial documents. The flag indicates that expert annotators identified potentially biased language in the snippet, without making any claims about the overall fairness of the judicial proceeding.

Dataset and Setup

The dataset consists of legal cases from a range of jurisdictions, including the United States, Germany, Japan, Vietnam, and Korea. To encourage innovative and adaptable approaches, the challenge will not include any labeled training data. Participants are expected to develop their models using their own methodologies, publicly available resources, and general knowledge.

Evaluation and Awards

All submissions will be evaluated against a hidden ground truth curated by the DMAIL organizing team. The primary evaluation metric will be the F2-score, which places greater emphasis on recall for detecting biased passages.

Winners will be announced in two categories:

  • The Overall Winner, awarded to the top-performing model on the challenge dataset.
  • The Low-Resource Setting Winner, awarded to the best-performing system operating under constrained computing resources.

To qualify for the low-resource category, participants must use publicly available pre-trained models (e.g., from HuggingFace or Ollama), and meet at least one of the following criteria:

  • the model must contain no more than 100 million parameters, or
  • it must run using no more than 16 GB of VRAM.

Submissions in this category must also include a short description of the experimental setup, including model architecture and size, hardware and VRAM usage, the used prompt(s) and overall methodology.

Why Participate?

This challenge offers participants the opportunity to work with diverse, multilingual legal datasets while addressing a critical ethical issue in legal AI. It provides a platform to benchmark models in a realistic and challenging scenario and gain visibility at an international workshop co-located with IEEE ICDM.

Participants are also encouraged to submit a short paper describing their method, results, and insights. Papers should be no more than 4 pages in length and must follow the standard two-column U.S. letter IEEE conference format. See the IEEE Proceedings Author Guidelines for formatting instructions. Authors of accepted papers must present their work at the DMAIL 2025 workshop to have their paper included in the official workshop proceedings.

Participation Requirements

To participate in the Legal Bias Flagging Challenge, each team must complete a formal registration process prior to gaining access to the dataset.

Participation requires:

Only teams that have completed the registration form and signed the license agreement will be granted access to the dataset and permitted to submit challenge results.

To register, please contact: sabine[dot]wehnert[at]ovgu[dot]de

For submission of challenge results, please use the general inquiry email listed at the bottom of this page.

Submission Instructions

Each participant must submit two plain text (.txt) files:

  • Predictions File A plain text file with one line per prediction in the following format: instance_id, label, explanation
    • instance_id: Provided with each task example
    • label: “Y” (biased) or “N” (not biased)
    • explanation: A brief, automatically generated justification
  • System Description File A separate .txt file describing:
    • Prompt (if used)
    • Model name, size, and availability
    • Hardware setup and VRAM usage
    • A concise summary of the overall methodology

All files must be submitted via email by the deadline.

Legal Bias Challenge Paper Submission Guideline and Reviewing

Submission Instructions:

  • Step 1: Login and enter DMAIL 2025 workshop in Cyberchair. Website: https://wi-lab.com/cyberchair/2025/icdm25/scripts/submit.php?subarea=S16
  • Step 2: Select your role as "author". From the top menu, click the "New Submission" button, and then select "DMAIL 2025: The First International Workshop on Data Mining and AI for Law" to continue.
  • Step 3: Enter your paper information and then use the "Submit" button at the bottom of the form.

Paper Length, Formatting, and Reviewing:

  • The length of each paper submitted should be no more than 4 pages, and formatted following the standard 2-column U.S. letter style of IEEE Conference template. See the IEEE Proceedings Author Guidelines for further information and instructions.
  • All legal bias challenge papers will be reviewed by the Organizing Committee on the basis of technical quality, originality, significance, and clarity.
  • Authors are also encouraged to submit supplementary materials, i.e., providing the source code and data through a GitHub-like public repository to support the reproducibility of their research results.

Important Dates

  • Challenge Registration Opens: August 10, 2025
  • Dataset Release: August 12, 2025
  • Submission Deadline: August 30, 2025
  • Results Announcement: September 1, 2025
  • Extended Results Announcement: September 5, 2025
  • Challenge Paper Submission Deadline: September 5, 2025
  • Extended Challenge Paper Submission Deadline: September 10, 2025
  • Notification to Authors: September 18, 2025
  • Notification to Authors: September 19, 2025
  • Camera-Ready Deadline: September 25, 2025
  • Workshop Date: November 12, 2025