In finance, trust is currency. Whether it’s approving a mortgage or flagging a suspicious transaction, decisions made by financial institutions impact people’s lives. Today, many of these decisions are made by AI models. Modern AI models, however, are often opaque ‘black boxes’ that leave consumers and regulators in the dark about how decisions are being made. To gain the trust of stake holders, financial institutions need to ensure that the decisions made by their AI models are intelligible. Fortunately, novel explainable AI or XAI techniques allow us to gain an understanding of how AI models make decisions. These techniques can explain which factors influenced a credit score or why a particular transaction was flagged as fraudulent. The result is not just greater transparency, but also a more accountable and trustworthy decision-making process.
Explainable AI Techniques
While some simple AI models are intrinsically explainable, these models are not as effective, in many settings, as more complex models. Historically, decisions made by complex models have been difficult to interpret and explain, fortunately, novel explainable AI techniques have been developed that allow us to gain an understanding of the process through which complex models make decisions. The most prominent amongst these techniques are SHAP and LIME. SHAP or Shapley Additive Explanations is a technique that deconstructs an AI model’s predictions, showing the exact contribution of each input.
LIME or Local Interpretable Model-agnostic Explanations is a technique that explains individual predictions made by a model, by building a simplified interpretable version of the model. When paired together SHAP and LIME can provide a clear view into the internal workings complex model: LIME provides local interpretability, explaining why a model made a specific prediction, while SHAP provides global interpretability, explaining how each model input affects the models decision.
Lending and Credit Scoring
Financial institutions are amongst the earliest adopters of explainable AI, particularly in the areas of credit scoring and lending. According to the Bank of England, 75% of UK financial firms are already using AI in some capacity, of which 16% are using it for credit rating. Meanwhile, Dataintelo estimates the size of the global credit risk XAI market to be USD 1.92 billion as of 2024, and projects it to grow to USD 9.6 billion by 2033.
Also read: Freshworks’ Sreedhar Gade on AI’s Black Box Dilemma and Need for Traceability
Traditionally, when applying for a loan or credit score, borrowers only received a binary decision (approved / denied) or a single credit score. XAI can transform this process by providing consumers with a clear understanding of how an AI credit scoring or lending model makes a decision. Techniques like SHAP can be leveraged to show applicants which aspects of their profile influenced the outcome of their application. In addition, SHAP can be used to produce counterfactual insights.
For instance, instead of providing a borrower with a flat rejection when they are denied a mortgage, SHAP can be used to notify the applicant that their application would have been approved if the borrower increased their savings by a certain amount. The bank could then work with the applicant on a savings plan, turning what would have been an unexplained rejection into a long-term relationship with the client.
Regulators are also increasingly requiring banks and credit scoring agencies to provide clients and regulators with clear explanations on how their models make decisions. The EU AI Act, for example, classifies credit rating systems as high-risk AI systems, which means these systems must meet strict transparency and accountability requirements. Financial institutions that deploy these systems in the EU, need to provide clear documentation on how their systems work, the limitations of their systems, and the factors that influence outcomes.
They must also ensure human oversight, logging, and bias monitoring, so that individuals denied credit receive intelligible and contestable reasons. In the United States, lenders are also subject to strict oversight laws. The Equal Credit Opportunity Act, for example, obligates creditors to provide specific reasons when denying credit. XAI tools can assist financial institutions in meeting these regulatory requirements. Techniques like SHAP, can be used to produce per-applicant, human-readable attributions and “what-if” explanations that show why a credit decision was made, giving customers intelligible and contestable explanations about their applications. XAI tools can also be used to create auditable records of model decisions which can be checked by humans and reviewed during regulatory inspections.
Fraud Prevention
Banks and credit card providers are increasingly carrying out fraud prevention with the use of AI models. These models can rapidly process vast amounts of transaction data, identifying subtle, non-obvious patterns and anomalies that human analysts might miss. However, fraud prevention algorithms are not infallible, genuine transactions are often incorrectly flagged as fraudulent. Having a transaction blocked or having one’s account frozen can be a frustrating experience for a consumer. Explainable AI tools can provide clear, human-understandable justifications for why a specific transaction was flagged, thereby alleviating consumer frustration and building crucial trust in the automated fraud-prevention system.
LIME can be used to provide a short human-interpretable explanation of a fraud prevention decision by creating a simple, local approximation of the model’s decision for that single transaction. It could highlight and rank the specific reasons why the transaction was flagged as fraudulent, allowing consumers to understand why their transaction was flagged in real time. To illustrate, consider a case where a card holder in Mumbai makes a ₹110,000 online purchase from a merchant based in Eastern Europe using a new device; the card provider’s AI model flags the transaction as fraud. LIME could immediately notify the cardholder that their transaction was blocked because of the foreign merchant location and the use of an unseen device’. If the transaction was genuine, the consumer could use this specific, LIME-generated explanation to quickly unblock the transaction by confirming with the card provider that they recently purchased a new device and authorized the specific foreign purchase. The immediate explanation shortens the resolution time, building a seamless yet trustworthy fraud prevention procedure.
The Road Ahead
The age of the opaque ‘black box’ AI model is nearing its end. As novel techniques like SHAP and LIME mature and regulatory pressures intensifies, the need to explain an AI model’s decision will move from a competitive edge to a fundamental necessity. Financial institutions that proactively embrace Explainable AI (XAI) aren’t just meeting regulatory mandates; they’re investing in building relationships and trust with their customers. Providing applicants with actionable “what-if” insights or instantly clarifying a flagged transaction transforms moments of friction into opportunities for trust-building. The winners in the next decade of finance will be those who not only deploy the most powerful AI but also the ones who can most effectively show their work. The road ahead for AI models in finance is inextricably linked to explainability.
The article has been written by Aman Adukoorie, Quantitative Strategist at a Leading Bank
