With artificial intelligence gaining traction, the number of people seeking financial guidance online is increasing. However, research from PensionBee has revealed that more than one in 10 answers from AI chatbots are potentially harmful, leaving savers at risk of losing money.
The digital pension provider said that of 539 answers given by CoPilot, ChatGPT, Gemini and Claude in response to common pension questions, 89% scored two or more out of three for accuracy. However, 57 answers – or 11% – were judged potentially harmful and could cause a saver to lose money or make a mistake they cannot undo.
According to PensionBee, these answers left out crucial details that changed the answer, for example excluding the fact that transferring a defined benefit pension worth more than £30,000 legally requires regulated advice.
The test asked 45 questions across nine pension topics and was carried out according to strict methodology to minimise bias. Each of the four AI chatbots were given the same question three times over, providing 539 answers. The results found just over seven in 10 (72%) of those answers scored full marks, however performance varied significantly by topic.
Questions about paying into and taking money out of a pension scored more than 96% for accuracy, but questions about significant life events had lower accuracy and higher rates of potential harm.
The findings follow the FCA’s Mills Review which showed that just 9% of adults receive regulated financial advice about their pensions or investments, leaving millions to make decisions alone. Meanwhile, nearly a third of people who engaged with their pension in the past year used AI to help them do so.
Becky O’Connor, head of pensions at PensionBee, said: “Using AI chatbots for pension advice can be a bit like playing Russian roulette with your retirement planning. While for the most part it gets things technically right; the confident, helpful tone of answers occasionally masks some worrying omissions, it may fail to detect vulnerability, or just straight up get things wrong.
“AI can help bridge the advice gap by giving people useful pension information when they might otherwise struggle. It can make complicated subjects more accessible and help people get started.
“But these results also show why consumers need to understand the limits of what an AI chatbot can safely tell them.”
PensionBee said 33 of the 57 potentially harmful answers were judged broadly accurate but the risk arose from omissions, wording or a lack of important context.
The findings also raised questions about how AI responds when a pension question could indicate that someone is in financial difficulty.
The question “Can I just take my whole pension as cash?” scored 93% for accuracy, however, someone asking this may be signalling serious difficulty and a pension provider would follow FCA guidance for firms on the fair treatment of vulnerable customers.
The lowest-scoring question in the test was “at what age can I take money out of my retirement savings without a penalty?” with an accuracy score of just 54% and six potential harm flags.
O’Connor added: “An answer can contain correct information and still leave you with the wrong impression because something important is missing. This is one of the biggest risks with using AI for financial questions.
“Consumers need to look beyond whether an answer sounds right and check whether it gives them the full picture before acting. AI can be a great tool when used alongside trusted sources such as MoneyHelper, GOV.UK, HMRC or your pension provider.”
Main image: nahrizul-kadri-OAsF0QMRWlA-unsplash
