Why Licensees Measure Everything Except Risk
Licensees have never had more compliance information available to them.
They measure advice quality, complaints, incidents, breaches, continuing professional development, monitoring completion, remediation and audit findings. Technology promises to extend that measurement across entire populations of advice documents, identify thousands of exceptions and produce sophisticated dashboards at unprecedented speed.
Yet more information hasn’t necessarily produced better oversight.
Many licensees still struggle to identify the risks most likely to cause client harm, regulatory intervention or governance failure.
This is the Compliance Measurement Trap.
It occurs when a Licensee mistakes the volume, precision or completion of compliance activity for evidence that its risks are understood and its controls are effective.
Licensees get better and better at monitoring the wrong things
The dashboard may be accurate.
The monitoring program may be complete.
The advice scores may be reassuring.
The conclusion may still be wrong.
Let’s be clear: AI didn’t create this problem, but it can amplify it by allowing organisations to measure the wrong things more efficiently.
The challenge is no longer simply how to collect more compliance information. It’s how to convert that information into governance intelligence: evidence that changes decisions.
If your board pack is dominated by completion rates, percentages and traffic-light dashboards but rarely changes supervision, product governance or risk decisions, your organisation may already be experiencing the Compliance Measurement Trap.
Why do organisations prefer activity metrics?
The Compliance Measurement Trap persists because activity is much easier to measure than effectiveness.
A Licensee can count:
- the number of advice files reviewed;
- the percentage of monitoring activities completed;
- the number of complaints received;
- the number of incidents recorded;
- the percentage of representatives who completed training; and
- the number of remediation actions closed.
These figures are easy to produce, compare and present. They fit neatly into dashboards. They provide apparent certainty. They also allow boards, executives, auditors and regulators to see that something has been done.
Effectiveness is somewhat harder to quantify.
It requires the organisation to determine whether its controls are identifying the right risks, whether recurring problems are being escalated and whether corrective action has changed conduct or client outcomes.
Those questions rarely produce a simple percentage. They require evidence from different sources, contextual interpretation and professional judgment.
There are also structural reasons why organisations favour activity measures.
Compliance teams are assessed against delivery plans. Boards expect concise reporting. Audit processes reward documentary evidence. Business units prefer findings that can be assigned, completed and closed. Technology platforms are designed to count events and track workflows.
The result can be a system that is highly efficient at demonstrating completion without establishing whether the underlying risk has changed.
Activity metrics aren’t useless. They are necessary but insufficient.
Reviewing 500 advice files proves that 500 files were reviewed. It does not prove that the supervision model can identify deteriorating advice quality before clients are harmed.
Completing a monitoring plan proves that the planned activities occurred. It doesn’t prove that the plan was directed at the Licensee’s most material risks.
Closing a remediation action proves that an administrative step was completed. It does not prove that the cause of the problem was removed.
The governance error is not the measurement itself.
It’s treating measurable activity as a proxy for control effectiveness.
How measurement distorts judgement
Consider a dashboard reporting:
Advice quality: 94 per cent
Monitoring completion: 100 per cent
CPD completion: 98 per cent
Governance effectiveness: Green
Each measure may be accurate.
But the dashboard says little about the condition of the business unless those measures are tested against other evidence.
- What if complaints are increasing?
- What if a small group of advisers is responsible for most replacement recommendations?
- What if high-risk products are concentrated among particular client groups?
- What if the same advisers receive repeated findings but remain under ordinary supervision?
- What if incidents are recorded but rarely assessed as reportable?
- What if remediation actions are closed without testing whether behaviour or client outcomes improved?
A dashboard can report every number correctly and still misdescribe the risk.
The problem is seldom false data. It is a false inference.
Three distortions are especially common.
1. Aggregation conceals concentration
Averages reassure because they smooth the variation that governance needs to examine.
An overall advice-quality score may appear strong while concealing a serious cluster of failures involving one adviser, product, client group or business model.
Aggregation can also flatten materially different findings into a single number. A minor documentation issue, a weak advice rationale and evidence of unsuitable advice should not carry equal weight.
A more useful approach is to assess observations according to their severity, recurrence, potential client impact and governance significance.
The objective is not to calculate the average quality of the files reviewed. It’s to determine whether the observations, taken together and weighted by risk, indicate a material weakness in advice, supervision or governance.
2. Scoring creates false precision.
Compliance reporting often treats different forms of evidence as though they are interchangeable.
They are not.
- A complaint classification is categorical data.
- An advice-review score is ordinal data.
- A remediation timeframe is quantitative data.
- A committee discussion is qualitative evidence.
- A pattern of adviser behaviour is contextual evidence.
- A board decision is evidence of action.
Each reveals something different. Each also has limitations.
An average advice score of 4.3 out of 5 may create an illusion of precision. The distance between a score of two and three is not necessarily equivalent to the distance between four and five.
The number means little unless the scoring criteria, reviewer consistency, severity of findings and distribution of results are understood.
Numbers are not neutral, simply because they are numbers.
Their value depends on what is being measured, how it is being measured and what conclusion is drawn from the result.
3. Completion is mistaken for effectiveness.
A green status means that an activity was completed, not that the control was effective.
A low number of reported breaches may suggest strong compliance or indicate weak incident identification, narrow breach assessment and a culture that discourages escalation.
A remediation action may be marked complete even though the relevant behaviour, process weakness or client risk remains unchanged.
Completion tells decision-makers that a task ended.
Effectiveness tells them whether the risk changed.
Those aren’t the same thing.
From compliance information to governance intelligence
At Assured Support, we see compliance information as a progression.
Data records individual observations: a complaint, a failed review, a product recommendation, a breach assessment or a supervisory observation.
Information organises those observations into reports, trends and categories.
Evidence helps demonstrate whether an obligation, control or responsibility is being discharged.
Triangulated evidence tests whether independent sources support or contradict the same conclusion.
Governance intelligence combines multiple independent evidence sources with professional judgement to identify material risks, challenge assumptions, inform decisions and support accountable decision-making.
This final step is where boards, Responsible Managers and compliance professionals create value.
The objective isn’t to produce more reports. It’s to produce information that changes decisions.
What does a practical governance framework look like?
A practical governance framework should ask five questions.
1. What material risk are we trying to understand?
Every metric should be tied to a defined risk.
A file-review completion rate isn’t meaningful on its own. It becomes meaningful when linked to a specific concern, such as unsuitable advice, weak replacement reasoning, poor disclosure or ineffective supervision.
Without that link, reporting measures activity rather than exposure.
2. What independent evidence supports or contradicts the conclusion?
No single source of compliance evidence is sufficient.
Suppose the compliance team reports consistently strong advice quality.
That conclusion should be tested against:
- complaints;
- client attrition;
- replacement activity;
- product concentration;
- supervisory observations;
- incidents;
- audit findings; and
- remediation outcomes.
If complaints are increasing, client attrition has doubled, and replacement advice is concentrated among advisers who have not been directly observed for more than 12 months, the advice score should not be accepted at face value.
The evidence points in different directions.
That inconsistency is itself a risk indicator.
The reverse is also possible. Advice-review scores may decline after a Licensee strengthens its criteria, increases challenge and identifies issues that were previously missed.
If complaints are falling, coaching is improving, and supervisory engagement is increasing, lower scores may reflect a stronger monitoring system rather than deteriorating advice.
The question is not “What does this measure say?” It’s “What conclusion is supported by the totality of the evidence?”
3. Where is the risk concentrated?
Overall results can conceal the people, products, clients or business models driving the exposure.
Governance reporting should therefore identify concentrations rather than relying on averages.
That means asking:
- Which advisers generate recurring findings?
- Which products appear disproportionately in complaints?
- Which client groups experience poor outcomes?
- Which business models produce repeated exceptions?
- Which supervisors are failing to escalate known issues?
The concentration is often more important than the overall result.
4. What recurring themes connect separate events?
Most organisations are better at categorising issues than connecting them.
A complaint classified as “replacement advice” records what happened. It doesn’t explain why it happened, how it happened or what consequences followed.
A failed advice review may identify deficiencies in one file. It does not establish whether the root cause was adviser capability, commercial pressure, weak supervision, unsuitable technology, poor product governance or unclear expectations.
Thematic analysis examines separate events to identify recurring causes and relationships.
A Licensee might compare:
- advice-review findings;
- complaints;
- breach assessments;
- coaching records;
- product concentrations;
- client outcomes;
- audit findings;
- Responsible Manager observations; and
- remediation delays.
Individually, these records may appear unrelated.
Collectively, they may reveal:
- weak articulation of client objectives;
- inadequate consideration of alternatives;
- commercial pressure influencing recommendations;
- repeated use of templated reasoning;
- weak supervisory challenge;
- failure to escalate recurring conduct; or
- ineffective consequence management.
These are governance issues, but they rarely appear as a single dashboard item.
Technology may create real value here, not merely by reviewing more files, but by exposing connections between information held in different systems, teams and reporting structures.
Pattern detection, however, is only the beginning.
A pattern must still be interpreted, challenged and acted on.
5. What decision changed as a result?
The final test of governance intelligence is action.
- Did the evidence change the supervision model?
- Did it lead to targeted file reviews, adviser restrictions, revised training, product controls, escalation or remediation?
- Did the board or Responsible Managers assign accountability and set a timeframe for response?
- Did the organisation later test whether the intervention worked?
If reporting doesn’t change a decision, trigger an intervention or test an assumption, it may be informative.
But it’s not governance intelligence.
The future of compliance isn’t more data.
Technology will continue to expand the volume, speed and precision of compliance measurement.
That’s useful, but it’s not sufficient.
The advantage won’t belong to the Licensees that measure the most, but to those that can:
- distinguish activity from effectiveness;
- connect metrics to material risks;
- identify concentrations and recurring themes;
- test one source of evidence against another;
- recognise when reassuring metrics conflict with emerging risk; and
- turn evidence into accountable action.
That’s the difference between compliance information and governance intelligence.
Compliance information records what happened.
Governance intelligence determines what it means, what must change and who is accountable for acting.
The Compliance Measurement Trap is not a failure of data or technology. It’s a failure to connect evidence, challenge reassuring conclusions and trigger action.
A dashboard can support that work. It cannot think for you.
Found this article useful? You can select Assured Support as a ‘preferred source’ in Google. This may help you see more of our articles in Google Top Stories, AI Mode and AI Overviews when our content is relevant to your search.
Further reading
Frequently Asked Questions
Reviewing more files proves that monitoring occurred—it doesn’t prove the monitoring program is identifying the risks that matter most. A licensee can review hundreds of files, achieve a 100% monitoring completion rate and still miss deteriorating advice quality, weak supervision or emerging conduct issues.
Effective compliance asks whether monitoring changes decisions, identifies material risk earlier and improves client outcomes, not simply whether planned activities were completed.
ASIC’s regulatory expectations focus on maintaining adequate risk management, supervision and compliance arrangements rather than simply demonstrating activity.
While metrics such as completed reviews, training rates and remediation actions are useful, they become meaningful only when they provide evidence that controls are operating effectively and that emerging risks are identified and addressed.
Boards and Responsible Managers should therefore look beyond dashboard statistics to determine whether reporting supports sound governance decisions.
Triangulated evidence means testing one source of information against several independent sources before reaching a governance conclusion. For example, strong advice review scores should be considered alongside complaints, breach assessments, audit findings, client outcomes, supervisory observations and remediation results.
When these sources point to the same conclusion, confidence increases. When they conflict, the inconsistency itself becomes an indicator that warrants further investigation.
Probably not.
AI can review larger populations of advice files, identify recurring themes and detect patterns much faster than traditional monitoring methods.
However, it cannot determine whether those patterns represent material governance risk or what action should follow.
Without professional judgement, AI may simply enable organisations to measure the wrong things more efficiently.
Technology improves analysis; governance still depends on informed human decision-making.
A useful test is to ask what decisions changed because of the information being reported.
If dashboards consistently show green status indicators but supervision, product governance, training, risk controls and accountability rarely change, the reporting may be measuring activity rather than effectiveness.
High-quality governance reporting should identify concentrations of risk, challenge assumptions, connect evidence from multiple sources and lead to measurable action.
