For many businesses, ChatGPT has quickly become another everyday productivity tool.
Employees use it to summarize reports, explain technical documents, research suppliers, or make sense of long web pages. It saves time and makes information easier to digest. Because it happens inside a trusted application, most people naturally assume the information they see is safe.
That assumption is exactly what attackers are beginning to exploit.
Recent research from Permiso Security has shown how a specially crafted webpage can manipulate parts of ChatGPT’s response when someone asks the AI to summarize it. Instead of simply displaying information, the chatbot can unknowingly present attacker-controlled content that looks like it belongs there.
A Different Kind of Phishing
Most phishing attacks start with an email.
This one starts with curiosity.
An employee visits a website, copies its contents into ChatGPT, or asks the assistant to summarize the page. Behind the scenes, hidden formatting embedded in that page can influence how parts of the AI’s response are displayed.
To the user, everything still appears to come from ChatGPT.
That makes the attack far more convincing than a suspicious email or an unfamiliar website.
What Researchers Demonstrated
The technique, known as ChatGPhish, takes advantage of how ChatGPT renders certain Markdown elements when processing external content.
Researchers demonstrated that an attacker could:
- Display phishing links that blend naturally into the AI’s response.
- Present fake security warnings or account notifications that appear legitimate.
- Load external images that quietly collect information such as IP addresses, browser details, and interaction timestamps.
- Show QR codes that encourage users to continue on their mobile phones, bypassing many desktop security protections.
The important point is that users may never realise this content originated from a third party rather than from the AI itself.
Why This Matters to Businesses
Cybersecurity has always relied on one simple principle: teach employees to recognise suspicious content.
AI changes that equation.
When information appears inside software people trust every day, their instinct is to trust what they see. That creates an opportunity for attackers to move phishing campaigns into environments where users feel comfortable.
This isn’t just another software vulnerability. It highlights how AI is becoming part of the modern attack surface.
As more organisations build AI into their daily workflows, security policies need to evolve alongside it.
Three Questions Every Business Should Be Asking
Can employees safely use AI to summarise external content?
Public websites, documents, PDFs, and emails should all be treated as untrusted sources. Even seemingly harmless content can contain hidden elements designed to manipulate AI interfaces.
Do employees assume AI responses are always trustworthy?
Most people don’t question information that appears inside an approved business application. Security awareness training should now include AI tools alongside email and web browsing.
Do we have clear rules for using AI?
Businesses should define which AI platforms are approved, what information employees can share with them, and how sensitive or untrusted data should be handled.
Clear guidance reduces risk far more effectively than relying on employees to make those decisions on the spot.
AI Is Here to Stay
Artificial intelligence is already changing how businesses operate, and the productivity gains are significant.
At the same time, every new technology introduces new risks. The companies that benefit most from AI will be those that treat security as part of the adoption process rather than something to think about later.
Research like this serves as a reminder that attackers adapt quickly. As businesses embrace AI, security strategies need to keep pace.
If your organisation is introducing AI into everyday work, now is the right time to review your security posture, assess potential risks, and establish practical safeguards before they become necessary.
KP Consulting helps organisations adopt AI securely, balancing innovation with strong cybersecurity practices so businesses can move forward with confidence.
