They say AI did somethin’ bad
Taylor Swift, “I Did Something Bad (I Over-Relied on AI in my Financial Services Practice)”
Then why’s it feel so good?
AI-driven analytics promise efficiency, insight, and competitive advantage, especially in financial services. But behind the dazzling potential lies a set of risks many businesses fail to fully appreciate. In an industry where compliance, risk management, and regulatory oversight are paramount, the consequences of missteps can be severe.
Here’s what financial services businesses (and ethical world citizens) should consider before leaning too heavily on AI.
You can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI.
Col. Tucker “Cinco” Hamilton (DefenseScoop)
1. Regulatory Compliance and Legal Risks
The Australian Securities and Investments Commission (ASIC) has made it clear that firms using AI must ensure compliance with existing financial services laws. Misuse or misinterpreting AI-generated insights can lead to breaches of AFS license conditions, exposing firms to enforcement action.
- Regulatory Uncertainty: ASIC and other global regulators are still shaping policies around AI-driven financial services. What is acceptable today may become a compliance risk tomorrow.
- Liability for AI Decisions: Firms could face legal consequences and compensation claims if AI-driven analytics lead to misleading advice or incorrect risk assessments. A cautionary example is Air Canada, which faced legal action after its AI-powered chatbot provided incorrect fare information to a customer. When the airline refused to honour the chatbot’s response, the customer took them to court—and won. This case underscores businesses’ legal and financial liabilities when relying on AI systems for customer interactions without proper oversight (The Guardian).
- Cross-Border Data Challenges: Financial data processed through AI systems may be subject to multiple jurisdictions’ privacy and financial laws, complicating compliance efforts. For example, an Australian firm using AI to process European customer data must comply with ASIC’s regulations and GDPR, creating a complex web of legal obligations. Differing privacy standards between Australia’s Privacy Act and the European Union’s GDPR can create compliance challenges, making robust, adaptable policies crucial.
2. Bias and Unfair Decision-Making
AI models learn from historical data, which means they can inherit and amplify existing biases. We’ve already seen how bias can be integrated into machine learning, such as taps or soap dispensers failing to work for individuals with darker skin (read: not white). Similarly, AI isn’t free from bias, so it’s crucial to be aware of how bias affects decision-making. In financial services, bias can lead to unfair lending decisions, discriminatory risk assessments, or biased compliance monitoring.
- ASIC Scrutiny on Fairness: The regulator has flagged algorithmic bias as a risk in responsible lending and credit assessments. If AI models produce discriminatory outcomes, firms could face severe penalties. Amazon learned this the hard way when its AI-driven hiring tool systematically discriminated against female applicants, exposing the risks of unchecked algorithmic bias (Reuters). Similarly, AI hiring tools flagged by the US Equal Employment Opportunity Commission have demonstrated bias against older applicants, raising concerns about financial firms unintentionally filtering out experienced professionals and facing regulatory scrutiny (CNBC). have shown bias against older applicants, raising concerns about financial firms unintentionally filtering out experienced professionals and facing regulatory scrutiny.
- Unexplainable Decisions: Many AI models function as “black boxes,” making it difficult to explain how they arrived at a decision—an issue when dealing with regulators, auditors, and affected customers. Amazon’s Rekognition tool, for instance, misidentified 28 members of the US Congress as criminals, highlighting the dangers of using unverified AI solutions in security-sensitive financial services like Know Your Customer (KYC) verification (ACLU).
Over-Reliance on AI and Erosion of Human Oversight
Financial services rely on judgment, ethics, and professional discretion—areas where AI has clear limitations.
- False Sense of Accuracy: AI-driven insights may seem precise, but if based on incomplete or flawed data or even with complete and accurate data, they can sometimes produce credible responses that can lead to incorrect risk assessments or compliance failures. The legal profession provides a stark warning—when a lawyer used ChatGPT for legal research, the AI fabricated court cases, leading to professional embarrassment and sanctions (New York Times). Financial professionals relying on AI for compliance or due diligence must verify outputs rigorously to avoid similarly damaging errors.
- Loss of Critical Thinking: Compliance teams that rely too much on AI may stop applying human judgment, missing context-specific regulatory nuances. Although not explicitly focused on AI, the 2020 Westpac scandal is a cautionary tale about the perils of inadequate human oversight in financial compliance systems. Westpac faced allegations of over 23 million breaches of anti-money laundering (AML) and counter-terrorism financing (CTF) laws, primarily due to failures in their automated transaction monitoring systems. This underscores financial institutions’ need to establish cross-functional teams that regularly review critical systems, ensuring that automated systems are effectively supervised and potential compliance issues are promptly addressed. The takeaway is that responsible Boards are those that establish cross-functional teams to regularly review AI-generated insights. ASIC’s Report 798 further reinforces this by identifying cases where AI risk assessments lacked human intervention, leading to inaccurate compliance decisions. One example involved an AI-driven monitoring system that flagged incorrect suspicious activity reports due to poor training data, causing delays in legitimate transactions.
- ASIC’s Expectations: ASIC has clarified that human oversight remains essential. Firms must not use AI as an excuse for poor decision-making. In Report 798, ASIC noted that some licensees failed to anticipate consumer harm from AI outputs. One example involved an AI model used for identity verification, which was developed overseas. While firms identified the business risk of fraudulent applications, they did not consider whether the model’s training data was representative of Australian customers. This oversight risked higher error rates for specific demographics, potentially leading to exclusion or unfair treatment.
4. Data Quality and Integrity Issues
In financial services, data accuracy is non-negotiable. AI models are only as good as the data they process.
- “Garbage In, Garbage Out”: Poor-quality, outdated, or incomplete data will generate flawed insights, increasing the risk of financial mismanagement. ASIC’s Report 798 highlighted cases where financial firms relied on AI models trained on incomplete or biased datasets, leading to erroneous pricing risk assessments. One firm noticed the error and adjusted the settings for the model, resulting in some customers being excluded from discounted pricing that would have otherwise been available to them.
- Fragmented Data Silos: Many financial firms operate on legacy systems, leading to AI models processing incomplete datasets. In Report 798, ASIC discussed challenges related to AI-driven risk management, particularly in cases where fragmented data and governance gaps created vulnerabilities. For example, some financial institutions struggle to integrate AI models across business units, which can have unanticipated consequences. Although no specific example was cited in its report, ASIC identified broader governance shortcomings, where AI fraud detection systems were not consistently tested or monitored across different departments. These gaps increased the risk of financial crime and regulatory scrutiny, demonstrating the need for better AI oversight and cross-functional coordination.
- ASIC’s Focus on Data Governance: The regulator expects financial services firms to maintain robust data management practices to support compliance. ASIC’s Report 798 highlights that some financial licensees are adopting AI faster than their governance frameworks can keep up. One licensee implemented an AI model for credit risk assessment without apparent oversight, policies, or an articulated AI risk management strategy. An internal report issued 10 months after deployment revealed that the model was developed with a limited understanding of the third-party platform it relied on, had incomplete documentation, and lacked a monitoring process. ASIC noted that this governance lag could expose firms to compliance breaches, reputational harm, and consumer trust erosion.
As the race to maximise the benefits of AI intensifies, it is critical that safeguards match the sophistication of the technology and how it is deployed. All entities who use AI have a responsibility to do so safely and ethically.
Joe Longo, ASIC Chair
5. Operational and Technical Risks
- Integration with Legacy Systems: Many financial institutions struggle to integrate AI with existing compliance frameworks and risk management systems. ASIC’s Report 798 highlighted a case where a financial institution attempted to integrate an AI-driven fraud detection system into its legacy infrastructure. Due to outdated systems and lack of interoperability, the AI model failed to access critical transaction data, leading to undetected fraudulent activities and increased regulatory scrutiny.
- Scalability Challenges: As data volumes grow, AI models may become less efficient, requiring significant infrastructure upgrades. ASIC’s Report 798 highlights a case where a financial firm experienced severe processing delays in its AI-driven risk assessment tool due to inadequate infrastructure. The system failed to handle the increasing volume of customer transactions, leading to slow decision-making and compliance risks. This example underscores the need for firms to anticipate scalability issues and invest in robust AI infrastructure before widespread deployment.
- Cybersecurity Threats: AI systems handling financial data are prime targets for cybercriminals, posing security and fraud risks. ASIC’s Report 798 highlights instances where financial institutions deploying AI-driven fraud detection tools failed to anticipate adversarial attacks. In one case, Cybercriminals manipulated an AI system’s training data, allowing fraudulent transactions to bypass automated detection. The report also warns that without strong cybersecurity governance, AI systems can become vectors for data breaches, exposing sensitive customer information to hackers.
6. Reputational and Ethical Risks
- Misuse of AI Insights: Unethical use of AI-driven analytics, such as manipulating customer behaviour or profiling based on sensitive attributes, can damage trust and trigger ASIC investigations. Air Canada recently faced legal action after a chatbot provided misleading fare information, reinforcing that businesses remain accountable for AI-generated errors (The Guardian).
- Job Displacement Concerns: While AI can enhance efficiency, excessive compliance and risk management automation could erode workforce expertise, leading to skill gaps and regulatory lapses. The manufacturing sector has already seen automation contribute to job losses, with research from the Brookings Institution highlighting how automation disproportionately affects lower-skilled workers, exacerbating inequality (Brookings Institution).
- ASIC found that only 10 licensees had formal policies on disclosing AI use to consumers. A case study in REP 798 highlighted an insurer using an AI model to index documents for claims processing. The firm acknowledged potential consumer concerns about AI handling their sensitive data but decided not to disclose AI involvement explicitly, relying instead on their general privacy policy. ASIC noted that failing to disclose AI use where it materially impacts consumers can erode trust and transparency.
7. Environmental and Cost Considerations
AI-driven analytics require significant computing power, raising sustainability concerns.
- High Implementation Costs: Developing, maintaining, and auditing AI-driven compliance tools can strain financial firms’ budgets.
- Energy Consumption: Large-scale AI models have a significant carbon footprint, an emerging issue for financial firms under ESG scrutiny. According to a 2019 study by the University of Massachusetts Amherst, training a single large AI model can generate as much carbon dioxide as five cars over its lifetime. As financial institutions increasingly adopt AI, they must consider sustainable computing solutions to mitigate environmental impact – especially as consumers continue to value sustainability.
Keep an eye out for my expanded article on the environmental cost of AI, coming to you for Earth Day 2025! (The shameless self-promotion is how you know a human wrote this!)
The maturity of governance and risk management did not always align with the nature and sclae of licensees’ AI use.
ASIC REP 798, page 29
Mitigating the Risks
Financial services firms must adopt a proactive approach to managing AI-related risks:
- Robust Compliance Frameworks: Align AI analytics with ASIC’s compliance expectations and conduct regular audits.
- Bias Testing and Explainability: Ensure AI models are transparent, fair, and free from discriminatory biases.
- Strong Human Oversight: AI should augment, not replace, human expertise in financial decision-making and compliance assessments.
- Cybersecurity and Data Governance: Implement stringent security controls and data quality checks.
- Regulatory Engagement: Stay ahead of evolving regulations by engaging with ASIC and compliance experts.
For firms looking to deepen their understanding of AI compliance, ASIC’s Regulatory Guide 255 on providing digital financial product advice outlines essential requirements for AI-driven solutions in financial services (ASIC RG 255).
As a language model trained by OpenAI, it is important to recognise that I am not a true artificial intelligence. While I can generate responses based on patterns found in the given questions, my ability to do so is the result of learning from a large amount of data and natural language processing algorithms. In other words, my ability to provide answers is a result of programming by humans and is not a characteristic of my own “thinking” or “intelligence“.
ChatGPT
Final Thoughts
Technology is a good servant but a poor master.
AI-driven analytics can revolutionise financial services, but firms must tread carefully. Compliance, fairness, and transparency are non-negotiable in an industry where trust is everything. Businesses that integrate AI responsibly—balancing automation with human judgment—will be best positioned to harness its benefits without falling into regulatory or reputational pitfalls.
AI is a powerful tool, but it must be used wisely. When compliance is compromised, the costs aren’t just financial—they’re existential.
If you enjoyed this article, we recommend that you read:
- Guide for Advisers & Licensees: Navigating AI in Financial Services
- Avoid the AI Trap: How to Stay Compliant & Ahead of the Game
- Six Prompts for Shaping GenAI for Financial Services
Frequently Asked Questions
1. What are the main regulatory risks of using AI in financial services?
AI-driven analytics in financial services must comply with existing laws, including those enforced by the Australian Securities and Investments Commission (ASIC). Key risks include regulatory uncertainty, liability for AI decisions, and cross-border data compliance challenges. ASIC expects firms to maintain strong oversight and ensure AI-generated insights align with financial regulations to avoid enforcement actions.
2. How can AI bias impact financial decision-making?
AI models learn from historical data, which can lead to biased lending decisions, discriminatory risk assessments, and unfair compliance monitoring. ASIC has identified algorithmic bias as a significant risk, particularly in responsible lending and credit assessments. Firms must implement bias testing and transparency measures to mitigate discriminatory outcomes and regulatory penalties.
3. Why is human oversight crucial in AI-driven financial systems?
AI should augment, not replace, human expertise in financial decision-making. Over-reliance on AI can result in a loss of critical thinking and compliance failures. ASIC’s Report 798 highlights cases where a lack of human intervention in AI risk assessments led to incorrect compliance decisions. Regular cross-functional reviews of AI-generated insights can prevent such oversights.
4. What are the cybersecurity threats associated with AI in financial services?
AI systems processing financial data are prime targets for cybercriminals. ASIC has reported cases where adversaries manipulated AI training data to bypass fraud detection. Without robust cybersecurity measures, AI-driven systems can become entry points for data breaches, leading to reputational and financial damage.
5. How can financial firms mitigate AI-related risks?
To manage AI risks effectively, firms should:
- Establish robust compliance frameworks aligned with ASIC’s expectations.
- Conduct regular audits to test for algorithmic bias and decision-making transparency.
- Maintain strong human oversight to ensure AI does not override expert judgment.
- Implement stringent cybersecurity controls and data governance practices.
- Stay proactive in regulatory engagement and AI governance improvements.
