An AI agent made history recently when it fired a human staffer, the first known example of such a dismissal. The milestone test case raises questions for HR about how to navigate grey areas in the workplace when AI agents are elevated to a position of authority – from workplace tool to workplace boss.
In San Francisco this year, an experiment carried out by a retail boutique, Andon Market, resulted in ‘Luna’ – the shop’s AI-powered boss – firing an employee who was marked as being chronically late.
Dubbed “the world’s first AI store owner” by her developer Andon Labs, Luna, a Claude agent, was created by the AI research startup, to test whether the agent could run a real business with minimal human intervention.
Importantly, the AI firing wasn’t entirely autonomous. Claude required regular steering by an Andon Labs staffer, who eventually asked the AI boss to think about whether the staffer was the right fit after her repeated lateness and warnings issued came to light. Then the firing took place.
The example highlights various grey areas that can result from AI agents being deployed into management territory, according to Professor Jon Whittle, one of Australia’s leading thought leaders on AI for business, currently a Distinguished Fellow at the Institute for Digital Innovation and AI (IDIA) at Melbourne Business School.
“This example tells you a lot. If you’re going to give over a decision to an AI agent, you have to be aware there’s a good chance it can miss things or make mistakes,” he says.
“It’s not infallible. So you need to proceed very carefully if these are material decisions which are going to affect somebody’s life…”
In the past, automated firings in workplaces have taken place, with gig workers subjected to algorithmically mediated employment. But this new case is viewed as the first known example of a large language model acting as a manager and sacking a worker.
Below, Whittle explores implications of the possible impact of AI bosses on future management processes and decision-making.
How much input should human managers have into AI-assisted dismissal decisions? Who should own the final decision?
A human needs to have 100 per cent input, says Whittle.
“More broadly, research from Stanford University shows you should take a risk-based approach with AI, where you need to save the human brain for high-risk decisions.”
“If you don’t do that, you won’t get good decision-making. Certainly, a dismissal would require strong human oversight.”
“You need to determine where the line is drawn in terms of the role of the AI in the decision… one of the challenges that HR professionals face is figuring out where that line is.” – Professor Jon Whittle, Distinguished Fellow at Melbourne Business School.
What should an employee be entitled to know if AI materially influences a performance, promotion, disciplinary or dismissal decision?
The Australian government developed a set of AI Ethics Principles back in 2019, before the proliferation of LLMs like Claude and ChatGPT, outlining eight high-level principles for any AI decision.
One says if an employee or individual is affected by an AI decision, there should be transparency and disclosure.
“You really need to be very transparent when you use more powerful AI models that have problems with hallucinations, and disclose exactly how you have used the AI, especially if there are possible legal repercussions.”
In practice, best-practice disclosure could look like telling the employee, in writing and before the decision is finalised, that AI was used; which tool and for what purpose (summarising performance data, say, or drafting a rating); what information it was given; and which human made the final call and how they weighed the AI’s output.
Disclosure should also come with a clear route for the employee to question the output and have a person review the decision.
What chance should workers have to challenge AI data or reasoning? And what documentation should HR provide?
Contestability is another of the federal government’s AI ethics principles, says Whittle.
“It means for any material decisions made about a citizen, they should have the option to ask questions. There should be a process in place to test that and, in many ways, that’s no different than existing HR policies and processes.
“You need to determine where the line is drawn in terms of the role of the AI in the decision. I guess one of the challenges that HR professionals face is figuring out where that line is,” he says.
Documentation can be challenging, he says, because it’s easy to say responsible humans should document everything for contestability – so if somebody comes back with questions, you can supply a list of exactly what happened.
“But firstly, these AI systems are, in some senses, black boxes. We don’t actually know what’s going on at a detailed level in a lot of these models and exactly why it’s making decisions.
“There’s basically a whole bunch of statistical number crunching going on at scale. There’s no human readable form of that. Secondly, it’s easy to say everyone who uses AI systems should document exactly what they did. But that could be a huge workload.
“I’m not convinced people will do that very effectively. It’s not a very good employee experience. Who wants to just sit there all day and be a bot checker? This is a key governance challenge that organisations are struggling with right now.”
A practical starting point would be to scale documentation to the stakes: a light-touch record for lower-risk uses, and a fuller account of the AI’s role and the human reasoning behind it for decisions that could end in a dismissal or a legal dispute.
“You really need to be very transparent when you use more powerful AI models that have problems with hallucinations, and disclose exactly how you have used the AI, especially if there are possible legal repercussions.” – Professor Jon Whittle, Distinguished Fellow at Melbourne Business School.
If AI gets it wrong, where does responsibility sit between the system/vendor, HR, manager and employer?
AI is not an independent entity, with its own mind, says Whittle.
“At the end of the day, it’s a large-scale statistical pattern matching machine and so accountability and responsibility sit with people who implement and deploy it.
“If it goes wrong, you need to question if it was properly tested and whether the right data was used to train the model with the proper guardrails.”
He adds that there’s a tendency more generally now to defer responsibility to the AI itself and, as we’ve seen in the media, there are stories about AI agents going rogue.
“If you look at the Medicare attack by an AI agent, it’s remarkable to me that there haven’t been any repercussions for OpenAI,” he adds.
Before AI is allowed anywhere near a people decision, leaders should be able to answer three questions:
- How was the tool tested?
- What data was it trained on?
- What guardrails sit around it?
Do you think examples like this will become more common in the future as organisations look to add AI to their formal org charts?
Whittle doesn’t think it “makes any sense” to perform wholesale replacement of a human manager with an AI agent. However, he says there may be tasks that a manager does that can be delegated to AI.
For example, AI is already used very widely in the hiring process.
“I don’t think we have sufficient understanding yet of how well that’s working or not, but there is research that says an AI hiring assistant can be less biased than a human because AI’s don’t have unconscious bias.”
What should be considered by HR managers looking to appoint AI bosses?
One of the Australian government’s AI ethics principles is that AI should respect human rights and human-centred values, says Whittle.
“But we know it’s very difficult to get an AI to align with human values 100 per cent of the time. So HR professionals should lean into human-centred values.”
A useful test before acting on an AI-influenced recommendation is whether you would be comfortable explaining it, face to face, to the employee it affects.
“Also, I think the organisation has to take responsibility and think deeply about how it’s going to introduce these systems and how to maintain good culture and give staff the space to check what the AI does.”
The question for HR is no longer whether AI can make the call, but who is prepared to answer for it. Whether it’s a tool or a boss, AI can inform a people decision, but the judgment and the accountability still have to belong to a person.
