XAICBR 2022 – Explainable AI Challenge

Download the full XCBR challenge call.

GOAL

The objective of this challenge is to highlight your expertise, skills and experience in applying Explainable AI (XAI) techniques to a “blackbox” AI model, so that the predictions of the model can be understood, improved or challenged by the users interacting with or impacted by the model.

Different user personas can be defined by the participants, each of them with a distinct objective or intent for explainability.

Participants will also need to provide how the different explanations can be evaluated by these personas and conduct an evaluation of their explanation strategies. These evaluations can be automated or assessed manually by a survey performed on a users group.

A set of standard explainers will be provided and available as standard web services.  Participants can also use their own explainers and document how they have been used. Note that an explanation strategy may combine different explainers.

RULES

  • This challenge is open to all ICCBR 2022 participants.
  • This challenge is available at no cost and no registration fee is required.
  • Those who are interested in participating should register for the ICCBR conference and to the XCBR workshop. In case of a participating team, at least one member from each team should register for the conference and workshop.
  • The participation can be either individual or in groups (up to 6 members). Members can come either from schools, universities, students or industry.
  • The submission must be received by the organisers by the 26th of August 2022.
  • A max 5 page presentation summarising the explanation technique and evaluation, with associated source codes of all programs developed,  is expected.
  • Participants will also be asked to deliver a 5-minute live presentation of their submission in a session during the conference. A recorded video can also be used for participants who can’t attend the session.
  • The organisers reserve the right to reject submissions that do not fulfil the above-mentioned criteria.

SUBMISSION, REGISTRATION and CONTACTS

If you or your team wants to join the challenge, please register by sending an email to hello@isee4xai.com. Then you will be added to the GitHub repositories to receive all the material required for the challenge.  

Enquiries related to this challenge and any questions should be also sent to the following email address: hello@isee4xai.com 

Website: https://isee4xai.com/xcbr_challenge_2022/

What the organisers will provide

  • A set of “blackbox” models trained for a variety of different tasks: image classification, natural language processing, classification of tabular data. These models will be provided as standard web services that can be queried from any programming language.
  • An evaluation dataset (unlabeled data) for each model  that will have to be explained.
  • A set of standard explainers that will be available as web services. Note that participants can also develop their own explainers and/or use other XAI libraries if needed.
  • These will be shared through private GitHub repositories that will be shared to those who register interest by emailing hello@isee4xai.com
  • Organizers will provide and publish the data set, AI black box models and explainers catalog by 26th May 2022.

EVALUATION CRITERIA

Each submission will be evaluated and noted by a panel of expert reviewers and by the community with a vote. The evaluation will take place during the ICCBR 2022 conference.

  • The submission must consist of the following deliverables:
    • a max 5-pager document summarising the project, explainability techniques used and their evaluation.
    • the source codes of all software, librairies, and programs used by the participant.
  • An oral presentation will be delivered during the XCBR conference (or a recorded Youtube video if you can not attend the ICCBR conference or the XCBR workshop).
  • The 5-pager summary will have the option to be published in the XCBR proceedings (using CEUR-WS.org).
  • Key criteria that will be evaluated:
    • Performance of the explanations and their evaluation
    • Novelty and innovativeness of the solution
    • Use of a CBR approach in the reasoning
    • Reusability of the solution in other contexts or domains.

PRIZES

  • 3-month funding (covering accommodation and travel costs to an iSee partner organisation: Spain, Scotland, Ireland or France) to work on a satellite project related to the iSee project. This will be a joint work with the iSee members.
  • an Apple iPad
  • $100 Amazon gift card
  • Champagne bottle from the Nancy area.

ORGANISING COMMITTEE

Belén Díaz-Agudo
Bruno Fleisch
David Leake
Anne Liret
Kyle Martin
Juan A. Recio García
Anjana Wijekoon

F A Q

When will the data be published / provided ?

We will provide and publish the data set, AI black box models and explainers catalog by 26th May 2022.

What will the winner get out of the satellite project ?

Participants from Universities will be able to publish on the work done and applied to iSee : This work could describe a new explainer, a new explanation strategy and how it suits to a use case, and/or how reusable this explainer is on different cases.

Participants from Industry will have the option to work closely with iSee team and become one industry partner : this work will provide the opportunity for them to evaluate their existing explainer or one explainer from the iSee library, once their case designed in the iSee cockpit and evaluation of explanation utilisation performed, they could be granted a certification of explainer “explainability compliance” at no fees, and finally (optionally) they could benefit from the integration of their explainer into iSee library as a recognition.

Is the challenge opened to work in progress and young researcher ?

Yes, they are very welcome ! XCBR challenge is designed to be attractive to early career researcher and/or industry organisation.

We are a team participating to the challenge, who can benefit from the satellite project visit ?

Anyone from the participating team can receive it, providing it is to work and contributing with iSee. The budget is provided as a whole and can be split among 1-2 visitors and for a duration of at least 2 months. The first prize would go either to the team leader, or to another team member selected by mutual agreement between the project site and the team leader.

Which country will I travel to if I win the first prize ?

Regarding the country hosting the satellite project visit, we will recommend the most suitable one to the winner, according to the contribution and synergies of their work with iSee members. On the hand, if the participating team is introducing a new explainer in the community, then we would suggest to go to Spain or Ireland where iSee members developing CBR engine and Explanation Experience case similarity measures. On the other hand, if the participating team is introducing a new use case or application domain need for explainability, then we  would suggest to go to Scotland or France where iSee members are focusing on Evaluating explainability case and developing conversational evaluation layer and representing new cases. 

Can I participate without publishing my submission ?

Participants will have the option to accept or not the publication of the 5-pagers submission in the XCBR proceedings. Publication consent is not required to participate to the Challenge.