So how do we make it better? Tooploox at ECAI 2024

  • Scope:
  • Artificial Intelligence
So how do we make it better? Tooploox at ECAI 2024
Date: October 16, 2024 Author: Konrad Budek 5 min read

Explaining the reasoning behind AI-driven decisions is one of the greatest challenges seen in the AI field. Not only might systems struggle with biased datasets leading to false conclusions, but also machine logic may have nothing in common with logic itself.

Machine Learning and AI development was a paradigm shift in computer science. In a traditional approach, it is the job of a software engineer to write code in which the input, output, and the rules in between are strictly defined. In Machine Learning, the system is trained on data to deliver an outcome, yet what’s in between is extremely hard to determine. 

That’s why counterfactual explanations are sometimes necessary. 

What are counterfactual explanations?

In the context of Machine Learning, a counterfactual explanation is the demonstration of how a change in a particular input may change the outcome of an AI system. Depending on the nature of the system, a counterfactual explanation can be based on multiple factors, providing the user with hints on how to get their desired result. 

Counterfactual explanations originated in philosophy, where sages analyzed their theories using examples and modified them to stretch and stress-test their statements. 

What AI can be used for (and where counterfactual explanations can help)

Artificial Intelligence-based solutions are renowned for their ability to process and analyze vast amounts of data, including volumes completely incomprehensible to humans. Yet gathering the data and processing it is only the first step – the next one is making informed decisions. 

Arguably, all AI-based systems are based on computers making a decision, with Machine Learning broadening the categories where computer-aided decision making can be applied. Examples include: 

  • Credit scoring – AI systems can make decisions about whether one should or should not be granted credit and how risky a particular decision is.
  • Hiring – AI-based systems can (and are) used to scan through resumes sent by candidates.
  • Diagnostics – the system can be super-helpful in the diagnosis process, supporting physicians in providing their patients with sufficient healthcare.

The challenge regarding this approach is that the AI decisions can be hard to explain or justify, making the system into a “black box AI.”

What is Black Box AI (and why should we avoid it)

Black box AI is a term that has been circulating in the engineering world for a while. First mentioned in 1945 in electronic circuit theory, where the circuits were regarded as a “blackbox” where the only information to be gathered is their output on a particular input to the ports. Later the term was applied to describe every system, where the observer has no idea how it worked, yet its operations could be made observable when entering information and getting results with no prior knowledge on the operation or construction of the system. 

Initially the “black box” referred to a situation where there was an unknown system to be reverse-engineered, or in addressing a lack of knowledge regarding the technical details of a functioning solution.

The situation changed after the introduction of Machine Learning solutions, where systems were trained on data to deliver expected results. For example, a system can be trained on thousands of images of cats to recognize if an image depicts a cat or something else.

Yet the exact details of the process are more obscure. The data scientist who trained the model has little to no knowledge regarding the ways the system reasons and delivers its output. For example, it is extremely challenging to determine if a system takes an image’s background into account. Sometimes it’s irrelevant. The system reasoning behind fashion item recognition in a picture is not that important, as long as the system works. On the other hand, if a system is used to support healthcare processes, the reasoning becomes vital. 

Probabilistically Plausible Counterfactual Explanations with Normalizing Flows (PPCEF)

Counterfactual explanations (briefly “counterfactuals”) are focused on designing approaches that provide information on how input values should be changed to alter the model’s decisions. There are two ways of using counterfactuals in practice: 

  • By interacting with the model using counterfactuals, the user can get more information on the system, inducing some transparency into the black box.
  • A good counterfactual provides the user with information about what to change in order to get a desired effect. 

The challenge regarding counterfactuals in the case of Machine Learning systems is the fact that the logic inside is not delivered by a human being, who rather only supervised at best. As such, the system may produce completely absurd counterfactuals that are useless to users. 

Example:

When asked about improving the chance of a 19-year old getting a loan for a second-hand car, the system may reply that launching some joint-venture with Microsoft may be a good starting point to increase income and improve their chances. 

PPCEF balances factors like validity and probabilistic plausibility using normalizing flows.  Allowing the solution to produce results applicable in real-life scenarios, providing users with more detailed information about possible adjustments to be made in their input to get their desired output. 

Details about this solution can be found in the Arxiv paper

Results

The approach is suitable for handling large-dataset challenges and problems, making it suitable for applications in banking, healthcare, and businesses of all kinds. By applying this approach, companies and institutions may gain:

  • Greater transparency – with normalizing flows, the company may provide users with more information on the factors that are taken into account by the AI-powered system.
  • Trustworthiness – following transparency is also greater trustworthiness, as companies don’t appear to hide the reasons behind their decisions, nor over-rely on their machines and their (sometimes flawed) judgments.
  • Compliance – AI-related legal frameworks strongly discourage “black box” and unexplainable AI. As such, counterfactual explanations will only rise in importance. 

Improving chances

Once you have all the reasoning in place, and a system capable of explaining its motivations, you can modify your input to get a better result. The real-life implications of this are immense, including: 

  • Improving the chances of getting a loan – if the system is used to decide on the granting of a loan or the rates the loan is given for an individual, having information about its reasoning can be used to improve one’s chances or negotiate better terms.
  • Recruitment – if the AI system is managing a recruitment process, information on ways to boost chances can be vital to both job seekers and companies.
  • Discovering new drugs – if a system aims to determine the properties of a new chemical mixture to be used in treatments, the researcher may play and tinker with counterfactual explanations to find the perfect (or at least a good enough) drug .

In answer to this issue, researchers affiliated with Tooploox decided to introduce Probabilistically Plausible Counterfactual Explanations with Normalizing Flows – a solution that extracts information about the reasoning behind  particular outputs from a neural network, as well as provide hints on ways to boost one’s chances of getting the desired results. 

Summary

The research was delivered by a team consisting of Patryk Wielopolski, Oleksii Furman, Jerzy Stefanowski and Maciej Zięba of the Wrocław University of Technology, Poznań University of Technology, and Tooploox. The research paper will be presented during the ECAI 2024 conference held in Santiago de Compostela.