As AI moves from a simple assistant to an autonomous teammate, a new “grey zone” of accountability is opening up – along with other emerging governance risks.

The promise of AI-driven productivity is no longer a distant prospect, but it comes tethered to a new category of organisational risk. As HR departments integrate increasingly autonomous AI agents into their teams, they are entering a “grey zone” where traditional lines of accountability – between managers, employees and software – are beginning to blur.

The stakes for getting this transition right are reaching the highest levels of corporate oversight. Recent findings from the State of the Boardroom Report 2026 suggest that while many organisations have matured their technical systems, the “human dynamics” of decision-making remain the ultimate point of failure. 

Despite this high-level scrutiny, a significant “governance gap” persists on the ground. Global research indicates that 41 per cent of organisations don’t have a policy guiding how to use generative AI at work, and 19 per cent were unsure if their organisations did. Separate research also suggests ‘shadow AI’ use at work (using behind a manager’s back) is also alarmingly common.

This disconnect places HR in a critical position. Beyond just managing tools, HR practitioners must now ensure that data privacy, bias mitigation and “human-in-the-loop” safeguards are woven into organisational governance frameworks.

In part two of our series on managing the downstream risks of AI (read part one here), we speak with two HR experts and an AI academic about how to surface these various risks. From cleaning legacy data to holding software vendors accountable, the mandate for HR is to lead the organisation through these complex dilemmas by setting the boundaries for what responsible AI use looks like in practice.

Data and privacy challenges

HR departments deal with some of the most sensitive employee data, which makes AI adoption inherently risky and complex, says Dr Zivit Inbar FCPHR, founder and CEO of DifferenThinking.

“Most organisations, especially those that have been around for a long time, operate with multiple legacy systems. I don’t know of one HR department that has only one system it uses,” says Inbar, who is facilitating two new AI for HR short courses with AHRI, one on what HR needs to know to stay compliant and another on embedding responsible AI.

“AI is also increasingly drawing on data from systems that have traditionally sat outside HR’s remit, such as communication platforms, project management tools and customer relationship management systems. These systems hold vast amounts of information about employees that often go unnoticed. When this data is integrated, the risks multiply.”

This means, for the majority of organisations, there is a lot of work to be done in cleaning up data inputs to ensure they are AI-ready, says Inbar.

“For example, if an AI tool predicts candidate success based on historical performance reviews, it will inevitably replicate existing biases. Unless those biases are identified and corrected, the AI will reinforce patterns of inequality rather than removing them.”

“Explainability and transparency are core principles of responsible AI. I’d never adopt software that can’t clearly demonstrate how the AI system works and how decisions are made.” –  Dr Zivit Inbar FCPHR

Collecting and preparing data for AI isn’t straightforward, she adds, because it involves complex decisions. 

“For example, who should handle data cleaning? Should it be done internally, knowing the information includes highly confidential details? Or should it be outsourced, which raises budget and privacy considerations?

“Personally, I’d prefer using an external provider for data cleansing, while working closely with internal IT to assess vendor’s guardrails and technical practices, including data security, system connections and compliance measures.”

Awareness of data security is also increasingly becoming a core capability for HR.

“From my experience leading global HR teams in technology companies, I’ve learned that deleting data is far more difficult than people think. One company still had copies of my personal documents two years later, which were exposed in a recent cyberattack.

“As new data practices emerge – and as regulations require employers to retain files for years after someone leaves – we need to ask: how much data is too much? These are complex, long-term dilemmas. HR will be called upon to help set those boundaries and lead organisations through them responsibly.”

Procurement challenges

Another emerging risk lies in the procurement of HR software. Most platforms now come with embedded AI, yet few buyers fully understand how these systems are designed or trained, says Inbar. Without that visibility, they can quietly become conduits of business risk.

“While HR may not control the algorithms themselves, we can hold vendors accountable,” says Inbar. “We should demand transparency and ask how systems make decisions or recommendations, how bias is mitigated, and request audits and reports to ensure the AI is functioning as intended.”

She suggests creating a formalised checklist that everyone in your team can use to assess vendors. This might include questions about data quality, privacy, security, bias mitigation and the explainability of algorithms.

“Explainability and transparency are core principles of responsible AI. I’d never adopt software that can’t clearly demonstrate how the AI system works and how decisions are made. 

“At the end of the day, the accountability is on HR, not the AI. I’d also recommend running a pilot program that gathers feedback from a diverse group of users before full implementation.”

HR’s strategic thought starters:

• What assurance mechanisms are in place to validate the integrity, privacy and accuracy of the data that feeds our AI systems? 
• Are our procurement and vendor management processes robust enough to identify and mitigate algorithmic bias, security vulnerabilities and compliance risks before contracts are signed? 
• What is our organisation’s threshold for responsible data retention?  
• Who is accountable for ensuring that data deletion practices meet both ethical and regulatory standards?

Accountability challenges

The rise of AI agents within teams is creating complex industrial and employee relations questions, with accountability sitting in the grey zone.

“If something goes wrong when an AI agent is being used, who is accountable? Is it the manager? The employee? The AI system? HR?” says Dr Ben Hamer FCPHR, futurist, founder of ThinkerTank, AHRI board member and chair of AHRI’s Future of Work Advisory Panel.

“In the next couple of years, we’re going to see the rise of human-AI teams. The first stage is already here – almost everyone has their own AI assistant. The next stage will be when we each have AI agents working largely autonomously.”

Then, he says, the challenge will be balancing that innovation and drive for efficiency with fairness, accountability and the industrial implications of AI. 

“We need to be thinking about them now. What happens if an AI vulnerability leads to a cyberattack or confidential information leak? Do we hold the team manager accountable, or is that a systems issue? We don’t have those accountability frameworks in place yet.”

Dr Emmanuelle Walkowiak, Vice-Chancellor’s Senior Research Fellow at RMIT, says opaqueness around accountability will be one of the greatest challenges to address as AI becomes more prevalent and advanced.

“AI brings not only automation and surveillance risks, but also new categories of concern, from cybersecurity threats and misinformation to bias, intellectual property breaches and accountability gaps. 

“Productivity gains from AI are inseparable from these emerging risks. People adopt these tools because they make work easier or faster, but they also create problems of trust and reputational risk, generating new layers of emotional and ethical pressure. Someone has to manage those risks, but right now, it’s unclear who that is.”

Questions of liability – both in mitigating risks and responding to breaches – now demand serious attention from boards and executive teams seeking to safeguard their organisations for the future.

“Traditionally, ethical responsibility was implied within a job description – you were accountable for acting ethically. But when you use generative AI, that accountability becomes ambiguous,” says Walkowiak.

“What happens if an AI vulnerability leads to a cyberattack or confidential information leak? Do we hold the team manager accountable, or is that a systems issue? We don’t have those accountability frameworks in place yet.” – Dr Ben Hamer FCPHR

Inbar adds that the governance gaps around AI are becoming clear.

“If you look at recent research from KPMG and the University of Melbourne, the data is striking. Across 47,000 employees globally, including thousands in Australia, only 40 per cent say their organisations have an AI policy in place.

“Perhaps most revealing, nearly half of the employees who do work in companies with AI policies admit to breaching them,” says Inbar.

“The issue isn’t just about technology adoption; it’s about the absence of policies, training and accountability. That’s where HR’s role becomes critical in shaping the culture and governance needed for responsible AI use.”

Inbar suggests establishing clear AI audit processes as a starting point, which she says should be on every CHRO’s agenda right now.

“Boards will be asking for it. HR leaders need a clear inventory of where AI is being used, and regular audits to ensure it’s being applied appropriately and without bias.

“Depending on the organisation, some already have responsible AI frameworks or governance committees in place. For those that don’t, I strongly recommend that chief people officers create a cross-functional AI governance committee, including representatives from IT, HR and employees – and ideally an external expert as well. Those most affected by AI often don’t have a voice, and that’s exactly what governance needs to address.

 “And, crucially, every organisation should mandate a human-in-the-loop approach. Establish clear rules that no final decisions about hiring, promotion, termination, compensation or similar matters are made solely by algorithms.”

Gain clarity and stay ahead of AI-driven change with the knowledge and tools to keep your HR practices compliant and your organisation protected with this course from AHRI.

HR’s strategic thought-starters:

• Do our governance frameworks have gaps regarding accountability in our AI processes that we need to address? 
• Who should oversee AI risk – and how can HR, IT and the board collaborate to ensure accountability is shared, not siloed? 
• Who do we need to include in a responsible AI committee, and how might that work in our organisation?

As governance enters this complex new era, the challenge for HR leaders is to ensure that while the technology evolves, the human element – transparency, ethical boundaries  and clear ownership – remains the non-negotiable anchor of the business.

A longer version of this article was originally published in the Dec/Jan 2026 edition of AHRI’s HRM Magazine. Read part one here.