Compliance is not a sexy topic, and so here’s the nightmare scenario: You are the CFO in a Fortune 500 company, and many internal reports reach your desk on a regular basis. Let’s say your annual internal audit happens in January and an important report lands on your desk in February. If the key information in that report is not dealt with by your office until the next annual audit, then your job may be at risk. The difficulty here is that non-financial risks such as reputation risk, data breach, privacy violations, or even planning errors can lead to losses down the road. These risks are hard to detect because they reach your desk as text, not numbers. Often the are just too many long reports, and key insights are missed. They don’t show up in the books until the crisis is upon you, be it in the form of a fine, a lawsuit, or lost client accounts.

It’s hard to justify in a board meeting why you had the report on your desk (or in your inbox) but never surfaced the issue. In a small company, this just isn’t a problem. There are not enough reports to motivate an AI-based solution. It’s only when you face a tidal wave of reports, that suddenly, the executives can’t handle the volume and variety of risks effectively. These reports are typically curated for decades in a Document Management System (DMS), but don’t reach some analytics tool to generate insights, and don’t get assessed by humans either.

Moreover, reading every single report in detail will still miss the big picture. You can’t see the non-financial performance of business units on a regular basis, leading to a best case scenario of a human-generated annual snapshot, or no picture at all, rather than a live view of the situation in the company, at a detail-oriented level, for whatever keywords you make up on the fly.

For over a year now, our team has been developing and refining an audit artificial intelligence called AuditMap.ai, and I have described the motivation for this tool in past articles (here and here). As I’m writing this, our article on AuditMap.ai is still ranked #1 on Google when you search for internal audit AI.

What I want to do with this article today is to show you the results, rather than focusing on how the artificial intelligence works.

The big idea is to have the AI find key statements in a big set of all your internal audit reports, and expose the underlying information to executives in dashboards, lists, and as action items. This configurable solution assesses individual statements, and distills the information into actionable insights.

Remember, the job of internal audit is to find bad stuff, and provide actionable insights. Let’s have a look together at some key risk statements identified by the artificial intelligence engine in various business areas, after crunching through some massive publicly available document datasets.

The following statements were extracted from a dataset of internal reports, and flagged as risks to various areas of business operations:

Management:

• Current guidance does not require Information System Division PMs to develop a formal project risk assessment.
• The organizations that did not have a risk-based plan were not aware that it was expected.
• Limitation. It has not been possible to specifically identify resources dedicated to the governance function.
• The integrity of the report may be limited to a pro-forma exercise given that Branches have not considered sources of risk consistently, nor conducted control assessments with reliability.

Financial:

• The weaknesses we observed in monitoring expose [entity] to the risk that a project for which it provided funding may fail to comply with agreed conditions, and that [entity] will not become aware of the non-compliance early enough to mitigate the potential environmental or social impact.
• As a result, there is a risk that employees exercising expenditure initiation authority for patient travel may not have been formally delegated these authorities.
• The restriction of analytical work to these countries is a critical shortcoming in the existing body of evidence.
• In addition, we noted that inconsistent levels of documentation existed to support various risk assessments.

Privacy:

• There did not seem to be a clear understanding of the difference between confidential and anonymous.
• Findings. The absence of a national directive on privacy has led to the development of different regional directives that are not necessarily addressing all requirements regarding the protection of personal information.
• Branch contacts interviewed noted that they did not have policies or processes in place to identify all activities within the branch requiring a privacy impact assessment.
• As well, the strategy alone does not meet the practice management requirements set out in Treasury Board’s Privacy Policy.

Data:

• A deficient separation process leads to inappropriate access to information thus compromising its confidentiality and potentially its integrity.
• However, improvements are required to ensure that all required documents are maintained in the central repository and that there is an audit trail in the contract approval process to demonstrate best value, competition, fairness and transparency.
• However, when the evaluation team requested further details to develop arguments around performance, the data provided (after an extensive collection and analysis process by [Team] HQ) differed from what was originally observed.
• In addition, backups of system data are not periodically tested for accuracy and completeness putting the Department at risk of not having complete records of the last known good copy of its system data.

As we see from the statements above, there are lots of non-financial reporting statements that are red flags indicating potential future regulatory or financial losses requiring immediate corrective and preventive actions. Clearly, the AI can read and flag key statements, and it can do it in the context of risk management frameworks and program separation (e.g. PEFA and COSO) and in the context of the company (e.g., a specific audit universe). However, how do we distill key insights for the executive? How do we step back from the identified issues and summarize the big picture?

Let me share with you 3 approaches for how to do that: dashboards, context windows, and risk-control matrices. These are the views that are getting client pull and feature requests.

1) Dashboards

AuditMap is full of dashboards for pulling out high-level insights from the data. I will present 4 dashboards here, to give you a sense of how you can gain insights and obtain the big picture for what’s been going on in your company.

The first view to look at is the ETL job summary. In simple terms, this view tells you how much stuff is in the database, and how much of that stuff was flagged by the AI as relevant. In our latest release, we track risks, controls, findings, recommendations, and observations.

Another dashboard to consider is the audit universe tree. Think of this as the list of all spaces within the organization where an auditor can audit. Every company does this a little differently, as this tree is fitted to the corporate structure and some parts of their immediate business partners.

The audit universe tree gives you a sense of how much data there is in the reports on each part of the company, and how much information the AI has flagged within these internal reports.

Next up, we’ve got a high-level dashboard (above) for identifying gaps in coverage and trends in topics. The upper-left block lets executives find out where and for how long there have been gaps of coverage in their audit universe tree. Starting in 2016, and counting backward, AuditMap identified a spot over time where not a single file had been tied back to this branch of the audit universe tree.

In a real world application, this could mean one of two things. Either:

(1) Everyone is aware that the work performed on this branch of the business was done on a 4 year cycle, Or

(2) Work in here was deprioritized for years and finally made the cut in 2016. This potential 3 years of information blackout is flagged at the program planning stage using AuditMap.

In the same line of terrifying gaps that show up, the upper right-right block of the image above details areas of the audit universe having seen no documented work associated to it. There are 3 areas flagged in the top right, where no reports were found. With the exception of recent additions to an audit universe, it goes without saying that questions need to be raised when this box lists out lines of business. The same sort of widget can be deployed showing when reports are found but risks or controls were not found. This catches the case where reports are sent in for the function every year, but these reports are just smiley faces stapled to blank paper, and therefore don’t expose risks or controls for management review.

The bottom component of this particular dashboard labeled “Entity Trends” tracks the movement of the top mentions across time. This widget tells you what stuff is most mentioned in the reports over time, facilitating some high-level idea generation in the planning process that can lead to insights in the detail-oriented drill-down to specific risks.

The drill-down happens in the explore pages, and the dashboard for those pages show you the very narrow view of a specific program (e.g., Distribution & Logistics > Planning) within a specific context (e.g., Control Activities).

In the dashboard above, you can see that a highly customized view of the risks and controls within a corporate function can be exposed. The tool can do this for any part of the business, for any ERM aspect, and it can also be filtered arbitrarily using the “Search Risks” and “Search Controls” boxes within the widget. And so you can drill down on many dimensions at once to form a thesis of sorts about what’s going on in the organization.

2) Context Windows

For every one of the statements extracted from the documents, AuditMap gives you (the user) the ability to:

1. Get a feel for its context of flagged data at two levels: the paragraph (via the context-text surrounding the bolded sentence in the image below), and the file-level (with the tool’s built in document summarizer).
2. On top of that, Team AuditMap integrated our philosophy of engine transparency, letting people see and even correct the AI predictions to correct mistakes or simply override the AI’s opinion.
3. And here’s the kicker, finding anything of interest doesn’t mean that the fun stops there. These flagged statements can be saved to exportable lists, and then on to generate Risk-Control matrices.

3) Risk-Control Matrices

Audit Managers are more familiar than top management when it comes to Risk-Control Matrix (RCM) design.

The RCM is a key tool for confirming appropriate coverage during audit procedures, so that exploration can flow directly into audit planning. From another perspective, you can use it to build a case about what’s happening in a function or risk area of the company by quickly building relationships between statements.

Risk-Control matrices are, in essence, the list of all controls in scope for a given audit and the visual relation to the risks they’re set to mitigate in the company’s operations. This is yet another view you can use to build a case about what’s happening in some function or risk area of the company. The risk and control statements can be plucked from the views in the explore tab, and set up in a nice matrix format shown below.

The matrix is very compact because a typical RCM can get very large in practice. These are often compiled in excel, and we did think of that, and built an excel export tool.

Take-Home Message: Understand Enterprise with AI

AI can be used to understand a company’s internal reports in a really automated and sophisticated way. The data can be viewed at a high level and sliced and diced to get views of the data across time and in specific contexts. Having the data processed with AI gives you a sense of where the gaps in the data are, and where problems (risks) or effort (controls) are trending upward.

This article originally appeared in Towards Data Science