AI Scorekeeper: Scotiabank Sharpens the Pencil in Credit Risk

Paul Edwards is helping carry the age-old business of giving loans into the modern era of AI.

Edwards started his career modeling animal behavior as a Ph.D. in numerical ecology. He left his lab coat behind to lead a group of data scientists at Scotiabank, based in Toronto, exploring how machine learning can improve predictions of credit risk.

The team believes machine learning can both make the bank more profitable and help more people who deserve loans get them. They aim to share later this year some of their techniques in hopes of nudging the broader industry forward.

Scorecards Evolve from Pencils to AI

The new tools are being applied to scorecards that date back to the 1950s when calculations were made with paper and pencil. Loan officers would rank applicants’ answers to standard questions, and if the result crossed a set threshold on the scorecard, the bank could grant the loan.

With the rise of computers, banks replaced physical scorecards with digital ones. Decades ago, they settled on a form of statistical modeling called a “weight of evidence logistic regression” that’s widely used today.

One of the great benefits of scorecards is they’re clear. Banks can easily explain their lending criteria to customers and regulators. That’s why in the field of credit risk, the scorecard is the gold standard for explainable models.

“We could make machine-learning models that are bigger, more complex and more accurate than a scorecard, but somewhere they would cross a line and be too big for me to explain to my boss or a regulator,” said Edwards.

Machine Learning Models Save Millions

So, the team looked for fresh ways to build scorecards with machine learning and found a technique called boosting.

They started with a single question on a tiny scorecard, then added one question at a time. They stopped when adding another question would make the scorecard too complex to explain or wouldn’t improve its performance.

The results were no harder to explain than traditional weight-of-evidence models, but often were more accurate.

“We’ve used boosting to build a couple decision models and found a few percent improvement over weight of evidence. A few percent at the scale of all the bank’s applicants means millions of dollars,” he said.

XGBoost Upgraded to Accelerate Scorecards

Edwards’ team understood the potential to accelerate boosting models because they had been using a popular library called XGBoost on an NVIDIA DGX system. The GPU-accelerated code was very fast, but lacked a feature required to generate scorecards, a key tool they needed to keep their models simple.

Griffin Lacey, a senior data scientist at NVIDIA, worked with his colleagues to identify and add the feature. It’s now part of XGBoost in RAPIDS, a suite of open-source software libraries for running data science on GPUs.

As a result, the bank can now generate scorecards 6x faster using a single GPU compared to what used to require 24 CPUs, setting a new benchmark for the bank. “It ended up being a fairly easy fix, but we could have never done it ourselves,” said Edwards.

GPUs speed up calculating digital scorecards and help the bank lift their accuracy while maintaining the models’ explainability. “When our models are more accurate people who are deserving of credit get the credit they need,” said Edwards.

Riding RAPIDS to the AI Age

Looking ahead, Edwards wants to leverage advances from the last few decades of machine learning to refresh the world of scorecards. For example, his team is working with NVIDIA to build a suite of Python tools for scorecards with features that will be familiar to today’s data scientists.

“The NVIDIA team is helping us pull RAPIDS tools into our workflow for developing scorecards, adding modern amenities like Python support, hyperparameter tuning and GPU acceleration,” Edwards said. “We think in six months we could have example code and recipes to share,” he added.

With such tools, banks could modernize and accelerate the workflow for building scorecards, eliminating the current practice of manually tweaking and testing their parameters. For example, with GPU-accelerated hyperparameter tuning, a developer can let a computer test 100,000 model parameters while she is having her lunch.

With a much bigger pool to choose from, banks could select scorecards for their accuracy, simplicity, stability or a balance of all these factors. This helps banks ensure their lending decisions are clear and reliable and that good customers get the loans they need.

Digging into Deep Learning

Data scientists at Scotiabank use their DGX system to handle multiple experiments simultaneously. They tune scorecards, run XGBoost and refine deep-learning models. “That’s really improved our workflow,” said Edwards.

“In a way, the best thing we got from buying that system was all the support we got afterwards,” he added, noting new and upcoming RAPIDS features.

Longer term, the team is exploring use of deep learning to more quickly identify customer needs. An experimental model for calculating credit risk already showed a 20 percent performance improvement over the best scorecard, thanks to deep learning.

In addition, an emerging class of generative models can create synthetic datasets that mimic real bank data but contain no information specific to customers. That may open a door to collaborations that speed the pace of innovation.

The work of Edwards’ team reflects the growing interest and adoption of AI in banking.

“Last year, an annual survey of credit risk departments showed every participating bank was at least exploring machine learning and many were using it day-to-day,” Edwards said.