Unpredictable and subjective behavior of machines is a good ingredient for a thriller — good food for thought, but a bad ingredient for our peace of mind. Luckily, there is very little progress in research on generalized intelligence, sometimes called strong AI.

AI can be used for the 'good' and for the 'bad', like many things in life. So, how do we make sure humanity gets the benefits of AI technology and minimize the risks and downsides? Life is not perfect, people are not perfect either; there can be risks around every corner. Technology is one way to overcome our limitations and minimize risks.

AI may play an important role in our journey. AI may help improve human decision making. But first, AI must become smarter and better integrated. In my first post on this topic, I explained that we need decisions with explanations to:

• decide the best way to use an AI model;


• prevent making decisions based on biases;


• uncover decision biases;


• help people trust the outcome of the machine-made decision; and


• include more common-sense knowledge in decision support systems.

XAI is narrow AI (applied to automate a specific task) but used in such a way that there is a feedback loop with the environment. The feedback loop may involve human intervention because a human expert or professional understands the scope of the narrow AI solution. Therefore, the expert can adjust the solution when the task at hand requires more knowledge or when we are warned in a meaningful way that the task at hand does not fit in the scope of the AI solution.

I have explained in my previous articles:

• why we need XAI;


• what is AI; and


• what is AI not.

What sets XAI apart from other IT or AI solutions?

Of course, it is about using AI techniques, and we also want it to support a well-defined decision-making task (in AI terminology, named 'narrow AI'). Further, we expect that the solution provides an explanation to justify its decision to humans and includes common-sense knowledge so that our XAI solution is not a black box based on big data alone. Finally, the solution should be part of a feedback loop, including operational-, tactical-, and strategic-level information. When decision biases or norm violations are detected the solution must be easily corrected.

The remainder of this article zooms in on the 6 characteristics of XAI solutions:

1. Supports a well-defined decision-making task

We make decisions all day long and most of them are unconscious. Think about the number of decisions that you make in the first hour after waking up: you decided to snooze the alarm; then you decided to not snooze again but to get out of bed; maybe you reminded yourself of an important meeting and about being on time; you decided how long to shower, what to wear, the strength of your coffee, your mode of transportation to the office, the best time to leave to be on time … it's amazing, if you think about it.

Normally we don't think about all these details, but if we want to automate a task, we need to in order to engineer it. Traditionally, engineering is based on rigorous methods applied to well-defined situations. So, we need to know the possible outcomes of a decision, the criteria, and in what context a decision is made.

Given that many decisions are made unconsciously, it may be a challenge to define a task sufficiently well. Luckily, we have thousands of years of engineering experience providing us with knowledge representation techniques, psychological methods, and statistics to help us. It all starts by realizing that defining the task in detail is an inevitable and vital step for XAI.

2. Provides an explanation

To follow a decision without having the potential of getting an explanation is like being guided by an oracle. An explanation should give you the understanding of what input(s) and criteria have been used to arrive at a decision.

The funny thing is that the decision may be right but the explanation may be wrong. Of course, such a situation should also lead to a 'correction' of the system, similar to the analysis you would do when a decision is, or turns out to be, wrong. The feedback loop consists of continuous communication about the effect of the decision as experienced by humans, their understanding of the decision, and 'corrections' to the system.

An interesting challenge lies in the fact that there is a difference between the detailed working of a decision-making system and the explanation that justifies a decision. Explanations should therefore be phrased using the concepts of the end-user (and that is not necessarily the same as the concepts used by an AI system).

3. Uses AI techniques

Although a game like chess is based on relatively simple and easy-to-understand rules, winning at chess is not just about 'following the rules'. It takes years of experience and lots of talent to become a good chess player because there are so many different possible alternatives. By practicing you learn to recognize successful patterns.

A similar reasoning may be applied for some other decision-making tasks that benefit from AI methods. Such methods explore many possible alternatives, evaluate the result using big-data, store successful patterns, and are able to generate more and more reliable decision outcomes.

We assume that big-data includes knowledge about the correct input-output relationship. However, what this knowledge is about remains in a black box to us; therefore, the machine-made decision maker is comparable to an oracle. In XAI, we still use these AI techniques but add an extra diagnostic feedback mechanism.

4. Includes common sense

At a large airport, forecasts of the number of arriving passengers are based on machine-learning techniques using the flight schedules and past arrival patterns. This resulting model is not able to take into account information we have about today — a holiday or a snowy day — even though everybody understands that those factors may affect arrival patterns.

XAI solutions should have a way to include such common-sense knowledge in decision support systems — not by using more statistics, since one snowy day in 564 days will not affect averages and correlations very much and therefore would not change a machine-learned model!

Instead, we expect a hybrid solution using symbolic reasoning to deal with common-sense knowledge. There should also be a supporting mechanism to help the ecosystem recognize that common knowledge may be missing. We will call this normware and we'll define it later.

5. Excludes biases

The now-famous example of AI-based HR matching software that has a strong gender bias not only demonstrates a weakness in AI techniques but also a weakness in humans. Research has demonstrated that human decision-makers have different kinds of structural biases and may introduce those biases into system solutions. We can conclude that human decision-making is strongly biased [Kahneman & Tversky: Judgement under Uncertainty: heuristics and biases].

Researchers have proposed solutions that have been demonstrated to work. One is to avoid the 'solitary decision maker' but instead use our reasoning skills in a conversation to produce and refute arguments [Mercier & Sperber: the enigma of reason]. Another is to include a system of checks and balances at three levels: Operational, Tactical, and Strategic.

At the operational level, the error between the actual and the desired outcomes must be checked, preferably based on diagnostic feedback. At the tactical level, the plan must be tuned, preferably based on parameters that influence your KPIs. At a strategic level, we should monitor our KPIs to decide when constraints, processes, or guidance need to be adjusted or 'corrected'.

6. Easy to change

A plan-do-check-act cycle (PDCA) is the business variant of a feedback loop in an engineering setting. As you all know, the business variant is, in many organizations, an exercise on paper instead of a real-world practice. What is missing?

When compared to the engineering setting, the PDCA lacks two elements: a clear-cut answer telling us when outcomes are 'an error' and need to be 'repaired', and a mechanism to correct the system based on the error.

What we need in XAI are systems, or ecosystems, in which both elements are clearly present. That is not to say that every step has to be automated. It is perfectly fine when there is 'a human' in the loop. Most important is that the feedback mechanism is monitoring all the norms on operational, tactical, and strategic levels. That is what we call normware, a term introduced by Tom van Engers [https://arxiv.org/abs/1812.02471]. Normware has a well-defined role in (X)AI: the ability to take well-defined measures in an ecosystem supporting the full decision-making process. It is an interface between the guiding agents such as operators, the C-level directors or policy makers, and the guided agents, such as the (X)AI-solution.

Conclusion

The use of AI techniques and the support of a well-defined decision-making task set XAI apart from strong AI initiatives to create general problem solving methods or other IT solutions. We expect that the solution can provide an explanation for a human to justify its decision, which includes common-sense knowledge — knowledge that people already have. This sets XAI apart from black box solutions that trust big data without looking at reality. Finally, there should be a mechanism to prevent or exclude decision biases, by including humans 'in the loop' to guide the system and correct it so XAI solutions must be 'easy to change'.

The resulting picture is an ecosystem that combines three levels of feedback — operational, tactical, and strategic — to support the full decision-making process. We introduce normware to monitor all the norms at the appropriate level and to guide the feedback process.

# # #