New research shows that adding AI agents to your org chart can generate unintended risks and consequences. So how should employers think about integrating their human and AI workforces instead?

At first glance, it may seem like a good idea to identify your AI agent with a human name and a spot on the org chart as a way to personalise your hybrid human-tech workforce.

However, new research from Boston Consulting Group suggests this could generate accountability and other risks, which will require greater governance controls, oversight and reviews by managers.

“In my opinion, I wouldn’t put AI agents onto the org chart,” says Matthew Kropp, Managing Director and Senior Partner at Boston Consulting Group, and co-author of the research.“I think it sets up wrong expectations on the part of the humans.”

The research shows that unintended consequences can result from listing AI agents on org charts in workplaces. Of those surveyed, 31 per cent reported that their organisation frames AI as a “teammate or employee” and 23 per cent said AI agents are listed on org charts.

This came as a surprise to survey researchers, who at first estimated about a five per cent AI agent adoption rate.

“When 31 per cent of respondents said their company had AI employees, that came as a shock.”

The study of more than 1200 HR and finance managers showed that when AI was framed as an employee (rather than a tool):

• Managers identified 18 per cent fewer errors
• Managers reduced their monitoring intensity by 16 per cent
• Individual accountability for errors fell nine percentage points and error accountability attributed to AI rose eight points
• Work carried out by AI ’employees’ made managers 22 per cent more likely to request additional review.

“If you tell me as a manager that I’m reviewing the work of an AI employee, I will catch fewer errors,” says Kropp. “Basically, that means paying less attention. And I [the manager] attribute more blame to the AI. So I’m more likely to essentially escalate or pass the buck and have somebody else review the work.”

Managers engage in the most thorough oversight when told the work comes from a human employee, the report outlined.

“AI ’employees’ therefore appear to occupy a hybrid organisational position – treated as delegated producers rather than tools, but not monitored like human subordinates.”

Embedding AI agents into formal roles on the org chart should therefore be understood as a governance decision which goes beyond a labelling choice, the survey report says.

“When a manager delegates to an AI agent, it shifts responsibility away from the manager… leading to less oversight.”

This, in turn, may lead to problems such as blurred accountability for accuracy, plus bias and cyber risk, adds Kropp.

“For example, the accountability is really about whether the accuracy of the AI agent is sufficient for the task, and whether it’s continuing to be correct?

“But it’s also making sure you are not having biased outcomes that favour or disfavour any particular group. This is particularly important, say, for insurance and regulatory environments.”

“AI ’employees’ therefore appear to occupy a hybrid organisational position – treated as delegated producers rather than tools, but not monitored like human subordinates.” 

Nominating a human to take responsibility for each AI output task is essential, he adds.

“Human performance reviews should be dependent upon staff providing proper AI oversight.”

Pointing out that he is “a huge AI fan”, Kropp says AI should still be used but viewed as a tool.

“I prefer to think of AI as a tool that a human operates. You don’t put your payroll process on your org chart. It’s an automation; it runs as a system. There’s somebody responsible for making sure that it runs correctly. I would look at AI just like that.”

The key message for HR leaders from this research, he says, is to “be careful”.

“It’s very important that you understand how it changes the way that the human team operates. You need to take appropriate steps to guard against the risks it creates.”

Below, Kropp highlights some of the key risks that HR teams should be aware of, and suggests potential mitigating actions.

1. Don’t treat AI employees in the same way as human staff

Different management processes should be devised for AI and human workers, the report outlines.

“Ultimately, organisations that want to build AI agents into their workflows cannot simply repurpose existing management structures designed for human employees.”

Organisations must explicitly design new accountability and verification routines.

How HR can address the risk

Deploy AI agents as automated processes with checks and balances.

As Kropp points out, an AI agent is not a person and can’t assume the accountability that an org chart requires. It will not get paid, hired or fired, can’t be fined or go to jail and doesn’t care about culture.

“But yes, you do have to be responsible for what it does.”

“I prefer to think of AI as a tool that a human operates. You don’t put your payroll process on your org chart. It’s an automation; it runs as a system. There’s somebody responsible for making sure that it runs correctly. I would look at AI just like that.” – Matthew Kropp, Managing Director and Senior Partner, Boston Consulting Group

2. Managers identify fewer errors when checking AI work

The research found that managers who review an AI agent’s work behave differently.

When managers are told they have an AI employee, they essentially say, “It’s not my problem anymore. It’s the problem of AI, so I’m not going to pay as much attention. I’m going to pass the buck to others,” says Kropp.

How HR can address the risk

Pair each AI agent with an explicitly accountable human owner who checks and monitors the AI work. Define minimum review standards for high-stakes decisions and design new accountability routines.

For example, “If you’re making a bank loan decision, you should be very clear what your threshold of AI accuracy is, and you need to be able to measure it.”

3. AI agent activity heightens governance concerns

AI adoption is moving faster than the governance systems and policies of many organisations. As lines between human and AI workers become more blurred, governance and accountability systems need to become more defined.

How HR can address the risk

Re-design workflows, performance reviews and escalation routines to allocate authority, sign-off processes, responsibility and oversight of the work done by AI agents.

“I don’t think this means you should never call AI an employee. What it means is you need to be very clear who has responsibility for the output and accuracy rates of the AI,” says Kropp.

4. Personalising AI creates human role fears

The paper cautions that its results should not be interpreted as evidence that organisations should not use AI agents. But as AI agents roll out into the workforce,, the effects on employees need to be understood carefully.

Kropp says when AI agents were present on org charts in the survey, participants were more afraid that AI was going to take their jobs.

“If I say, ‘You have an AI teammate that’s going to do a bunch of your work’, you’re going to say, ‘Well, that means my job’s going to be gone’.”

How HR can address the risk

Build steps into the AI adoption process to guard against job loss fears, says Kropp.

Leaders need to address changed team dynamics, ask how team members are feeling about the changes, how they are changing the way they work as a result, and review what messages management is sending.

“With AI coming into the workplace, there needs to be some ownership and responsibility for that overall change.”

Productivity gains don’t remove the need for governance

Organisations that are formalising AI agents are expecting productivity gains and early work shows evidence that firms which integrate AI agents into workforces are generating higher cost savings, the report says.

But these potential productivity gains don’t remove the need for guardrails.

“Without explicit governance, building AI agents into work processes may increase the apparent capacity of the organisation, while weakening the human oversight on which that capacity depends.”

🧰 HR’s career resource kit

• Course: AI & HR: What You Need to Know to Stay Compliant Course – Part 1
• Course: AI & HR: Embedding Responsible AI – Shaping Workforce & Culture Course – Part 2
• Article: 10 strategic prompts to help your business assess the downstream risks of AI
• Toolkit: AHRI members can access AHRI’s AI toolkit, which includes a range of resources and content to help your organisation make strategic workforce decisions about how to deploy AI.