Author: Sybille Mueller, Director, Streets Consulting
What happens when we hand the ‘steering wheel’ over to AI? That was the central question at a lively Sibos 2025 panel, “Should AI have agency?”. Expertly guided by Ghela Boskovich, the debate saw panellists forced to argue for and against AI agency, even swapping sides halfway through to challenge their own assumptions. The discussion navigated the critical tension between innovation and capability versus control and accountability.
The panel quickly revealed a fascinating split in perspectives on how autonomous AI should be. While some participants felt that current AI is more than capable of executing specific tasks, they remained sceptical about its ability to genuinely solve complex, nuanced problems on its own. Others took a more futuristic view, suggesting it’s only a matter of time before AI takes the reins completely, making decisions entirely without human intervention.
Regardless of AI’s potential, the need for constraints was a point of universal agreement. This raised a deeper philosophical question: can an AI truly be considered ‘free’ when it will always depend on human oversight and the physical hardware it runs on? Ultimately, the conversation returned to one non-negotiable point: human accountability. The overwhelming consensus was that without a human held responsible, full AI agency is a non-starter. One panellist powerfully compared an AI operating without this oversight to nothing more than a “shell company”, highlighting the profound risks of responsibility gaps.
If AI agents are to be given any level of freedom, they cannot operate in a regulatory vacuum. Panellists agreed that AI agency must be governed by a clear set of rules and a form of social contract. The debate explored whether our current legal and ethical frameworks are up to the task of managing the unique challenges AI presents.
AI governance is anything but static. The challenge is complicated by the threat of sophisticated criminals who could manipulate AI systems to bypass existing rules. This means regulation must be adaptable and constantly evolving, as a rigid, one-size-fits-all approach is doomed to fail. The panel argued that for AI to benefit society responsibly, effective global coordination is essential. The role of governments, they suggested, should be to enforce high standards rather than getting bogged down in continually rewriting the rules from scratch.
As you would expect at Sibos, the discussion took a sharp turn towards the financial sector, where the dual potential of AI to revolutionise and destabilise was acutely felt. A major concern was that granting AI digital freedom without accountability could open the door to manipulation and trigger a large-scale economic collapse. Some participants drew parallels to the 2008 financial crisis, arguing that robust AI regulation is essential to prevent a repeat performance. It was noted, however, that with proper human oversight, AI could also be used to distribute and mitigate risk, potentially preventing future crises.
Within finance, the need for clear constraints is particularly urgent to ensure transparency and prevent market failures. The panellists’ description of AI’s ability to process financial data was telling, calling it both powerful and terrifying. This underscored the need for better regulation in this complex area. Yet, the panel also recognised the risk of stifling progress. Over-regulation could kill innovation, while giving AI more freedom could unlock incredible efficiency and free up human potential. The challenge lies in finding ways to incentivise visionary, long-term thinking, rather than just short-term optimisation.
The final part of the debate explored the deeper questions of what drives an AI and how multiple AIs might work together. A key operational challenge is coordinating the actions of individual algorithms within an organisation to achieve the best collective outcome. This led to a fascinating ethical discussion about altruism. Could an AI agent ever be programmed to act selflessly or make a sacrifice for the greater good, or will it always be bound to the outcomes it was designed to optimise? One speaker floated the idea that an unconstrained AI could, in theory, act altruistically.
This ties into the fundamental nature of digital systems: they have no internal motivation of their own. Just as humans need oxygen to breathe, AI needs externally defined motivation to drive its behaviour. Defining and controlling this motivation is one of the most critical aspects of designing safe and effective AI agents.
In their closing remarks, panellists encouraged the audience to use their imagination when thinking about the future of AI. While the debate was full of differing opinions, the core message was clear: if we are to harness the immense potential of AI for the good of society, effective regulation and global coordination are not just important, they are absolutely essential.
Panellists included: Nicola Breyer, Joanna Bryson, Karen Elliott, and Carsten Maple.
