AI Safety in Fintech Regulation and Practice

The UK and US governments have announced a landmark agreement on AI safety, enabling collaboration to develop a program for shared objectives.

### Understanding AI Safety in the Fintech Sector Amid Global Regulatory Developments
* 🌐 The European Parliament approved the EU AI act in March, while the UK and US governments announced a **landmark agreement on AI safety**.
* 🔒 The UK-US collaboration focuses on AI safety beyond national security, encompassing **cybersecurity, individual security, and system stability**.
* 🤖 AI safety is intentionally defined broadly to address evolving risks, especially with **generative AI advancing rapidly**.
* 🏦 Financial institutions, operating in highly regulated environments, have been deploying AI safely, with a key focus on **understanding algorithms and limits**.
* 📜 The UK adopts a **principles-based and sector/regulator-led approach** to AI regulation, unlike the **proactive statutory approach** in the EU.
* 🇪🇺 The EU AI Act categorizes AI systems based on risk, with **high-risk systems requiring human oversight and impact assessments**.
* 📝 Fintechs can prepare for the EU AI Act by creating an **AI inventory**, updating policies, implementing **approval processes**, and conducting **gap assessments**.
* 📊 Privacy is a complex issue in generative AI, with challenges in achieving **transparency with data subjects** and addressing **various privacy pitfalls**.

welcome to fintech focus scadden's podcast for fintech industry professionals the global regulatory and legal updates you need start now. hello and welcome back to fintech focus. today you've got me again Joe Kamar. I'm an m&a lawyer in scad's fintech practice and with me is Nicola Kershaw who's one of our lead AI lawyers here in London. Nica, thanks for joining. thanks Jay, it's great to be here. so plan for today, we're going to unpack the term AI safety in the context of the fintech sector and we've obviously had some big headlines in the world of AI over the past few weeks. so first of all, we saw the European Parliament approve the EU AI act in the middle of March and then this month we had the UK and US governments announcing their Landmark agreement on AI safety which neatly ties into today's topic. and I guess when you look at some of the historic AI collaborations between the UK and US governments there's traditionally been a

heavy focus on National Security, and the examples I'm thinking of are the Partnerships between GCHQ and the NSA in the US. That said, when you look at the detail of the current partnership, and Nicha correct me if you disagree, this seems to be less focused specifically on National Security and actually much broader. Yeah, and I Joe, I entirely agree. There's a huge focus on security within this MOU, but security is indeed broader than just National Security. It isn't defined and doesn't appear to be limited to any particular type of AI or sector. So, we have loads of questions we can ask ourselves. Is the MOU seeking to capture cybersecurity and the risk of attacks fueled by AI, or is it looking at the security of individuals and the risk of harm caused to them by misuse of AI? Or are we looking at security of financial and other regulated systems and the potential threat to those systems and the stability of those systems caused by AI? And how is this collaboration going to work in practice?

there's a lot of questions we don't know a lot but what we do know is that this is a groundbreaking agreement and will enable the UK's new AI safety Institute to work collaboratively with its US counterpart. Together they're going to develop what they are calling an interoperable program of work and approach to Safety Research in order to achieve their standard shared objectives on AI safety. This is going to include developing strong methods to evaluate the safety of AI tools and their underlying systems along with information sharing with each other. So I think the definition of safety going back to your point has been left intentionally broad and is seeking to encapsulate many of the potential security concerns within AI. Generative AI is evolving at immense speed as we've seen recently, and we don't know many of the risks. In fact, in the UK's government's recent response to its AI regulation white paper, it specifically stated that it was waiting for any potential regulation of AI until its understanding of the risks has

matured. So, the Mau provides a framework to keep addressing this security as it evolves. And while those risks are maturing, I always find it interesting how the concept of AI safety can mean so many different things in so many different contexts. And I guess instinctively, I suspect lots of people, thanks to Hollywood, associate AI safety with the idea of robots taking over the world. But clearly in the context of financial services, that's hopefully less of a concern. Now, your background, Nicola, is an interesting one as you've only recently joined Scouten, having spent a number of years in the tech arm of a global financial institution. So, I'm curious to know from your previous life before Scouten, what were the sorts of areas and risks keeping you up at night from an AI safety perspective? So, my role and my focus was advising a broad range of technology and IP issues across a large financial institution and its group of companies. Technology did include AI, but at least in recent years, that was nothing novel or new. Financial institutions have been using forms of AI.

for years and for a wide range of purposes, Financial Services is also heavily regulated already. AI had therefore been developed within a sensible and controlled environment in a way which ensures continued compliance with financial regulation. So AI safety was, I and I think still is, a bit less of a concern for many banks. A key Focus Point surrounded understanding the algorithms and been able to explain them to any interested regulator. It also involved having proper control points to ensure safe testing and development. Understanding how AI can enhance services, for example, huge developments in detecting fraud and financial crime, but also understanding its limits, were absolutely key. It's also fair to note that a lot of use of AI within financial institutions has been machine learning and Bots, rather the new more difficult generative AI that has grabbed so many of the headlines this last year. So I guess we're saying financial institutions have, in some respects, already been Paving the way with examples of safely deploying AI given the highly regulated environments that're they're operating in. And I think

it's fair to say that we're not seeing a call to arms for fresh AI focus legislation in the UK. And my sense of that government's positioning seems seems to be that we have a principles-based approach that's adaptable and also sector and regulator-led. Now that's pretty distinct to the proactive and much broader statutory approach that we're seeing in Europe, for example, through the EU AI Act, which we'll come on to. Um, and when you then look at what the regulators actually saying, again my sentence, it seems there seems to be little appetite amongst likes of the Bank of England PFCA for a sector-specific statutory framework. And instead, the focus seems to be on understanding how the patchwork of existing regulatory frameworks AI technologies. So, Nica, what in your view are the most material touchpoints with the existing regulatory regimes? I mean, clearly for entities providing financial services in the UK, the FCA rules and CIS will continue to apply and are key touchpoints. This is nothing new, but these rules need to be

interpreted with a specific AI tool in mind. As with any technology within Financial Services, key Focus points will be on Market stability, maintaining due skilled care and diligence, and evidencing that operational resilience, consumer Duty, and treating customers fairly and transparency with the regulator. The senior managers and certification regime will continue to apply and needs to be considered in relation to the development of any internal or external AI tool. In addition to this, the use of personal data within an AI model or system is also a key touch point for financial institutions. For some banks, particularly institutional Banks, personal data is only incidental to the use case in hand, with the real Focus being on the corporate relationship. However, the UK gdpr still applies in full, and achieving transparency with data subjects in this context can be really quite challenging. Makes sense, and I guess my sense from recent discussions with Regulators that you also present at is that we can expect to see at the very least more guidance on how those regimes apply to AI. That said, you actually think it's

realistic that the UK continues to veer away from legislating for AI altogether. When you look at the industry feedback, the sense is that there's a push for a globally harmonized approach to AI regulation, and clearly that's not the case when you look at what's happening in the US and the UK vis-à-vis Europe. And a second question, if I can, are there any aspects of the existing regulatory regimes where you can see gaps forming as use cases for AI amongst FinTech and banks changes over time? These are such interesting questions, Joe, and clearly having global standardized laws is something we'd like across many different things, not just AI. I mean, I think there's strong use cases for AI within financial services, including things like AI-based creditworthiness assessments, fraud and financial crime detection, market research, drafting research papers, and risk assessments for insurance. The list goes on and on and on, and there's likely to be strong human input into most of these processes already to the extent that they use AI.

and so that along with the requirements of other financial regulation, it's arguably enough. However, it should be noted that some of these use cases, particularly those which involve individual consumers directly, for example, risk assessment for health insurance of an individual, will be considered high-risk AI under the new EU AI act and so we'll be subject to heightened requirements. We're expecting to see further development by the European standardization bodies here and whether in fact there will be additional requirements for those within financial services. I mean, for the most part, the training and control framework are already in place within financial services and this is likely to satisfy much of the EU AI act, so arguably there's no need in financial services, at least for additional regulation. Additional guidance though on how to evidence certain requirement or to comply with topics such as the GDPR would be very welcome. That all being said, for other industries, I think the use case against additional regulation is less clear, particularly with genitive AI and I think this is where government should focus both in terms of additional

regulation and guidance on how to comply with existing legislation. So on the topic of legislation, we've obviously both mentioned the EU AI Act. And now that's been formally approved by the EU Parliament within the past few weeks. It's obviously had various guises and iterations over a fairly protracted period. So perhaps you could give us a high-level overview if that's possible of where that piece of legislation has finally landed. I mean, that's a challenge. It's a vast piece of legislation, 270 pages I think. So a high-level overview. It's a bit of a challenge in the time, but here are some key takeaways. First of all, all AI systems need to be categorized on a risk-based approach. So there's minimal risk systems, which are already widely used, such as spam filters. They are, in fact, AI, and they are largely unregulated and will continue to be unregulated as normal. But then the other AI systems need to then be categorized according to their level of risk. There'll be minimal transparency and disclosure requirements to certain

AI systems such as chatbox, and the users need to be aware that they are interacting with an AI system. There are then high-risk systems. Example of these within Financial Services could include things like AI credit worth checks or risk assessments for health insurance where we'll need to uh Implement training appropriate human oversight and maintenance of technical documentation to ensure identifying risks are mitigated. There also be a need for fundamental right impact assessments prior to deployment, which is very similar to what we've seen under the gdpr. There are then AI systems which are prohibited as unacceptable risk, but it seems unlikely in my experience at least that Financial Service entities will be engaging with this type of AI. Other key headlines from the EU AI act include the need to respect intellectual property rights within the use of AI, so could impact research taken from the internet, and this could also actually impact Financial Services entities in needing to potentially share trade secrets on where they Source their research from. Entities will also be liable for breach of the ACT even if the underlying

technology is white-labeled or provided by a third party. Another key point to note is there'll be a new AI office, which will sit within the commission and will be tasked with overseeing the most advanced AI models. I could go on, but in the time, I think that's a good summary of key highlights. Fair enough. Well, more specifically then, so as I understand it, aspects of the ACT will have extraterritorial effect. So if you're a UK-based fintech, what are the sort of things you should be thinking about in practice, particularly for those fintechs looking to deploy AI either internally or as an integrated part of their client-facing products and services? I think there's a few things that all companies, including fintechs, can do now to prepare for the EU AI Act, but also to help ensure that they're complying with other regulation and legislation in other countries as well. Many fintechs will probably be doing a lot of these things already, but it's always good to have a checklist. Firstly, I'd suggest that companies create essential

inventory of AI systems and models that are deployed and being developed within the entity. This should include all systems and models, regardless of whether they are internal or external. The inventory should contain details such as the purpose of the AI, what data and IP is being processed, how the technology works, what are the use cases, and the intended purpose. So as much detail as possible. It should also reference any data privacy impact assessment conducted in relation to that AI if it uses personal data, and also note whether third-party technology is being used as part of it. Once the inventory is complete, companies can then look at that list and consider whether the AI Act would apply to each system and model, and to what extent, and then record that within the inventory. They might also like to categorize each model and system in relation to which risk category it would fall under as part of the EU AI Act. Next, the existing technology policies and procedures of the company should be updated to ensure they specifically cover AI. This should include statements prohibiting certain T

types of AI and specific action that should be taken when certain data or IP is going to be used. It should also include requirements to keep the inventory updated and what review process needs to be undertaken before a use case can firstly be explored and then deployed by the company. I mean to that end, I suggest fintex consider implementing an AI approval process, similar to how many financial institutions review and approve external vendors. This process would perhaps include an approval in committee, which could be virtual and in person, but would have representatives from risk, compliance, legal, the business, and the CESO team. And potentially, if personal data is involved, should include the DPO also. And that committee would review all AI proposals and approve them before they go live. Finally, the inventory should be then reviewed in light of the EU AI Act and the company's processes as a whole to determine what gaps, if any, there are and then form a remediation plan. We realize that companies have limited budget and time, so focus on the highest risk AI systems first and the biggest gaps.

consider what new internal processes need to be created as part of this third-party contract. Should be considered to ensure the appropriate due diligence has been conducted. There's ongoing right of audit, and there's proper contractual protections in place. We've obviously been looking at this from a regulatory and legislative perspective up to now, but as AI is increasingly deployed in client-facing platforms, it's interesting to see some of the contractual risks which are starting to emerge businesses the other month actually I had a friend complained to me about a customer service chatbot that he' introduced on his website and the chatbot had actually started offering out discounts to clients much to much to his despair and it was interesting particularly because it was around the time of the Air Canada ruling. Yeah I mean that's right the Air Canada en rolling was in February and in that one a tribunal ruled that Air Canada was obliged to honor a discount offered by a chatbot. It's a further example of the measures and safeguards that firms will need to have in place.

ensure that the outputs from its AI tools are accurate and not misleading. It's the same with chatbot used by Banks and other Financial Services. If a chatbot offers a price on something or conducts a trade, but there was a technology error, can that trade or contract be undone? Often with services such as electronic trading, the Electronic Services agreement in place will state that trades can be unhelmed at the bank's discretion if there is a technology error, but that potentially becomes a lot more complex when facing consumers, individuals rather than B2B. Also, there are questions like where did you put the terms and conditions of the bot, and how do they become binding? Where does someone click to accept them when they're dealing with a bot? Yeah, so sticking with the topic of safeguards but this time through a data lens, what are you hearing from privacy regulators in terms of safeguarding personal data and then the sorts of policies, procedures, measures that they're expecting to see in place? Privacy, in my view, is one of the most

complex topics within generative AI. There's a lot of questions and not a lot of answers. For example, we have extensive laws already around transparency of data processing and use of tracking technologies such as cookies. But how many people actually understand how their data is processed or what a cookie does? Most people, I think, click the X box to close the window and proceed to the website or click yes to state they've understood the privacy policy, even when they have not. But how do companies address this in the world of AI, where technology is even more complex than a website or a small piece of code dropped on a machine? And how do developers of AI achieve transparency with a data subject whose data they have used to train a model, particularly when they've not necessarily got a direct relationship with that individual? This is only one topic, and there are so many other potential privacy pitfalls, particularly around data minimalization, legitimate interest, consent, and many more. We've had some updated guidance from the I relating to AI back in 2023. However, following the government's response to

its consultation on AI regulation. We are expecting to see more, including their strategic approach to AI, and the deadline for this is the 30th of April. Although, just to note, we have heard through our network that the ICO will be increasing its own use of AI. Our bots will be checking your bots, so to speak, which is an interesting development. So I think this is an area to really watch. This space very good. Well, thanks very much, Nicholas. Sadly, that's all we've got time for, and really appreciate having you on the podcast today. Thanks, Joe. It's been an absolute pleasure. Lots of interesting developments here, and so much we could talk about. We haven't got on to the topic of AI in cyber attacks and what can be done to minimize legal risk here, so perhaps that's one for another time. Sounds right. And thanks again to everyone listening. See you next time. Thank you for joining us on Fintech Focus. If you enjoyed this conversation, be sure to subscribe in your favorite podcast app so you don't miss any future

Conversations. Additional information about Scadden can be found at scan.com. [Music]

What Does AI Safety Mean For Fintechs?

Skadden, Arps, Slate, Meagher & Flom LLP and Affiliates delivers the highest quality advice and novel solutions to legal challenges, enabling clients to achieve their business goals. We are known for the innovative and creative thinking we rely on to handle the most complex transactions, litigation/controversy issues, and regulatory matters, as well as the open, collaborative relationships we build with clients, including corporations; financial and governmental entities; small, entrepreneurial companies; and cultural, educational and charitable institutions. Our attorneys, who reflect diverse backgrounds and perspectives, collaborate seamlessly across our 21 offices in the world’s major financial centers.

Content may include attorney advertising.


Skadden, Arps, Slate, Meagher & Flom LLP

The Impact of the UK-US AI Safety Agreement on the Fintech Sector

The UK and US governments have recently made a landmark agreement on AI safety, with a broad focus on security in the fintech sector and beyond.

The agreement aims to develop an interoperable program of work and approach to safety research, including the evaluation of AI tools and information sharing.

This collaboration provides a framework to address evolving AI security risks and has the potential to impact the financial services industry on a global scale.

The Regulation of AI in UK Financial Services

AI development in UK financial institutions is already heavily regulated and focused on compliance with existing financial regulations.

The use of AI in these institutions is mainly centered around machine learning and Bots, rather than the more advanced generative AI.

The UK government and regulators are not pushing for new AI-focused legislation, but instead, they are looking to understand how existing regulatory frameworks apply to AI technologies.

The key touchpoints with the existing regulatory regimes for AI in UK financial services include the FCA rules, market stability, due diligence, operational resilience, personal data usage, and the senior managers and certification regime.

There is an expectation of more guidance on how these regimes apply to AI in the future.

The Global Approach to AI Regulation in the Financial Industry

The UK is moving away from legislating AI, while there is a push for global harmonization of AI regulation.

In the financial industry, AI is widely used for credit assessments, fraud detection, and research, with the EU AI Act imposing heightened requirements for high-risk AI used in individual consumer cases.

The existing training and control framework in the financial services may already satisfy much of the EU AI Act, but additional guidance on GDPR compliance would be appreciated.

The EU AI Act introduces requirements for AI systems in Financial Services, including the need for human oversight and impact assessments.

It also addresses intellectual property rights and imposes liability on entities using AI.

Fintech companies can prepare for the AI Act by creating an inventory of AI systems, assessing their applicability under the act, and updating technology policies and procedures to include AI.

The EU AI Act brings regulations for AI systems in Financial Services, covering oversight, impact assessments, and liability for entities.

Fintech companies are advised to prepare by inventorying their AI systems, evaluating their applicability under the act, and updating technology policies to include AI.

Implementing an AI Approval Process and Ensuring Regulatory Compliance

Fintex should establish an AI approval process involving a committee with representatives from risk, compliance, legal, business, CESO, and potentially the DPO for proposals review.

The company needs to prioritize high-risk AI systems and the biggest gaps, while also ensuring proper due diligence in third-party contracts.

Additionally, Fintex must focus on safeguarding the accuracy of AI outputs and addressing contractual risks, especially in client-facing platforms, to comply with privacy regulations and expectations from privacy regulators.

The Challenge of Achieving Transparency in AI

Lack of understanding about data processing and tracking technologies creates a transparency issue for AI companies.

Addressing transparency with data subjects and navigating privacy regulations are major challenges for AI developers.

The upcoming government regulations and increased use of AI by the ICO indicate a growing focus on this issue.