Logical validation of neural networks: An explainable AI framework based on formal concept analysis

This paper proposes an XAI framework using Formal Concept Analysis (FCA) to validate neural networks. We generate a ‘local’ context around a prediction, compute its implication basis, and score its ‘local logical consistency’ against the global data. We also suggest using this score as a new loss function.

Keywords

Explainable AI, Logical validation, Local logical consistency, Implication basis, LIME, Neural networks, Loss function

The black-box nature of deep neural networks is a major barrier to their adoption in high-stakes domains. We propose a novel framework for Explainable AI (XAI) that uses Formal Concept Analysis (FCA) to assess the logical consistency of a network’s learned knowledge. Inspired by LIME, our method generates a “local” formal context from perturbations around a specific input instance. We then compute the implication basis of this local context and measure its logical validity with respect to the global knowledge contained in the original training data. This provides a formal score of “local logical consistency” for each prediction. Furthermore, we outline a new training paradigm where this logical validity measure replaces or complements traditional statistical loss functions, paving the way for inherently more robust and interpretable models.

Introduction

While neural networks have achieved superhuman performance on many tasks, their lack of transparency is a critical issue. In fields like medicine or finance, a prediction without a justification is often unusable. The field of Explainable AI (XAI) seeks to address this challenge.

Current XAI methods often provide feature-attribution explanations, highlighting which parts of the input were important for a decision. However, they rarely assess whether the reasoning of the model is logically sound with respect to the underlying data structure.

Formal Concept Analysis (FCA), a mathematical theory for deriving hierarchical structures and logical implications from data, is uniquely suited to fill this gap. In this paper, we introduce a novel FCA-based framework to validate neural network predictions. Our contributions are:

1. A method to generate a local formal context that captures a network’s behavior around a specific prediction.
2. The use of implication bases and formal validity checking to compute a “Local Logical Consistency” (LLC) score for any prediction.
3. A proposal for a new training objective, “Logical consistency optimization,” to create models that are not only accurate but also logically sound.

Methodology and expected results

The core of our methodology is the bridge between the sub-symbolic processing of a neural network and the symbolic reasoning of FCA.

Step 1: Local context generation

Given a trained classifier $f$ and an input object $g$ from a global formal context $\mathbb{K}=(G,M,I)$, we generate a set of $n$ small perturbations of $g$, noted $\{g'_1, \ldots, g'_n\}$. We then construct a local context $\mathbb{K}'_{g}=(G', M \cup C, I')$, where $G'=\{g'_1, \ldots, g'_n\}$, $C$ is the set of output classes, and $I'$ records the attributes of the perturbations and the class predicted by the network, i.e., $I'((g'_i, c)) = 1$ if $f(g'_i) = c$.

Step 2: Logical validity assessment

From $\mathbb{K}'_{g}$, we compute its canonical implication basis, $\Sigma'_{g}$. This basis represents the “local rules” the network has learned in the vicinity of $g$. The main step is to evaluate the validity of each implication $\varphi \in \Sigma'_{g}$ in the original global context $\mathbb{K}$. The Local Logical Consistency (LLC) score is then defined as the proportion of locally valid rules that are also globally valid: $$LLC(g) = \frac{|\{\varphi \in \Sigma'_{g} \mid \mathbb{K} \models \varphi\}|}{|\Sigma'_{g}|}$$ A high LLC score suggests the network’s local reasoning aligns with the global patterns in the data.

Step 3: Logical consistency optimization

We propose a new training paradigm. For a batch of training instances, we can compute their average LLC score. This score can be used as a regularization term in the loss function, or even as the primary objective function, forcing the network to learn logically consistent mappings. The main theoretical result to obtain is a proof that optimizing for LLC leads to models with improved robustness and generalization, particularly on out-of-distribution samples.

Work plan

• Months 1-2: Formalize the LLC score and the logical consistency optimization framework. Implement the core functions for local context generation and validity checking in `fcaR`.
• Months 3-5: Conduct initial experiments on benchmark datasets (e.g., UCI datasets) to validate the methodology. Train simple classifiers (e.g., MLPs) using the new loss function.
• Months 6-8: Apply the framework to a real-world problem, for instance, in medical diagnosis or malware detection, to demonstrate its practical utility.
• Months 9-12: Write up the results for a top-tier AI conference and subsequent journal submission.

Potential target journals and conferences

1. Top AI conferences (NeurIPS, ICML, AAAI): Ideal for presenting the novel training paradigm, which has broad implications for the machine learning community.
2. Expert Systems with Applications (Q1): A perfect venue for the applied part of the work, showcasing a real-world application of the XAI framework.
3. International Journal of Approximate Reasoning (Q2): Suitable for the theoretical aspects connecting fuzzy logic, FCA, and neural network interpretability.

Minimum viable article (MVA) strategy

This work can be split into two main papers: one introducing the XAI framework and another detailing the novel training paradigm.

• Paper 1 (The MVA - the XAI framework):
  • Scope: Introduce the method for generating local contexts and computing the ‘Local Logical Consistency’ (LLC) score. Present this as a novel XAI technique. Apply it to one or two case studies to demonstrate its explanatory power in identifying why a model’s prediction is or isn’t trustworthy.
  • Goal: To establish LLC as a new, logic-based XAI tool for auditing black-box models.
  • Target venue: A top XAI-focused conference or a journal like Expert Systems with Applications.
• Paper 2 (The logical consistency optimization paper):
  • Scope: This paper would be the more ambitious contribution. It would formally propose using the LLC score as a differentiable (or regularizing) term in a loss function. It would require extensive experiments comparing models trained with vs. without this ‘logical loss’ on benchmarks, focusing on robustness and out-of-distribution generalization.
  • Goal: To introduce a new training paradigm that creates inherently more robust and logically sound models.
  • Target venue: A top-tier machine learning conference like NeurIPS or ICML.