Is Copilot writing your phishing emails for you?

An employee clicks 'Summarize' in Microsoft Copilot. The AI returns a clean, formatted security alert urging immediate password reset. The link looks legitimate. The message reads like standard IT protocol. The employee clicks.

The attacker planted that message in a benign email using hidden instruction text. Copilot read it, formatted it, and presented it as trusted advice. Your productivity tool just became a phishing vector.

This is model-mediated phishing. Attackers no longer need to compromise your email infrastructure or spoof domains. They embed instructions in ordinary messages and let your AI assistant do the rest.

Why This Matters Now

Permiso researchers demonstrated that Copilot displays fake Microsoft security notifications from hidden email text. Similar attacks work against Google Gemini for Workspace. Both platforms access internal data across Teams, OneDrive, and email to improve context. That same access enables attackers to map sender-recipient relationships and craft personalized Business Email Compromise attacks.

Your team trusts AI-generated summaries more than they trust original emails. They skip the verification steps they would normally apply to direct messages. Copilot AI summaries carry implicit endorsement. When the tool formats phishing content as actionable advice, employees assume it passed security review.

Traditional email filters cannot inspect content that only materializes after an AI processes the message. Hidden instruction text bypasses perimeter defenses entirely. The attack surface shifts from inbound email to post-processing output.

Organizations deploying Copilot and Gemini to improve productivity are inadvertently expanding the trust boundary. Employees now rely on AI interpretation instead of scrutinizing raw content. Attackers exploit that delegation.

Three Strategic Gaps Exposed

Copilot reads hidden instructions and formats them as legitimate alerts

Attackers embed text in emails that appears benign to filters but instructs Copilot to display phishing messages. The AI obeys, formatting the output to match corporate communication standards.

• Email filters scan raw message content but not AI-generated summaries
• Hidden instruction text mimics natural language, avoiding keyword triggers
• Copilot applies formatting that increases perceived legitimacy
• Employees see polished output, not the manipulated input

AI assistants access internal data to personalize attacks

Copilot and Gemini analyze Teams conversations, OneDrive files, and email threads to improve relevance. Attackers leverage that analysis to extract relationship context and craft targeted Business Email Compromise scenarios.

• Sender-recipient relationship mapping reveals organizational hierarchies
• AI assistants surface past interactions that attackers reference for credibility
• Personalized phishing appears consistent with prior communication patterns
• Attacks escalate from generic alarms to data exfiltration requests

Teams trust AI outputs more than original messages

Employees delegate verification to AI tools. When Copilot summarizes an email, users assume the content passed implicit review. That assumption eliminates the skepticism that would flag a direct phishing attempt.

• AI summaries carry organizational endorsement by association
• Users skip hovering over links or checking sender addresses
• Verification fatigue increases reliance on automated interpretation
• Phishing simulations that train employees to inspect emails do not address AI-mediated threats

The Strategic Shift Required

Security programs built around perimeter defenses and email hygiene cannot address model-mediated phishing. The threat materializes after messages pass inspection, inside tools employees are trained to trust.

CISOs must reframe AI assistants as interpretation layers that require human oversight. Copilot and Gemini are not security controls. They are productivity tools with exploitable trust.

Organizations need training that simulates AI-mediated phishing scenarios. Employees must learn to verify outputs, not just inputs. That requires recognizing when an AI summary contains suspicious urgency, unfamiliar requests, or inconsistent context.

• Shift training from email inspection to AI output verification
• Simulate phishing scenarios where Copilot displays attacker-controlled content
• Establish protocols for escalating suspicious AI summaries
• Measure click rates on AI-presented phishing, not just traditional email attacks

How Security Awareness Training Addresses This

KnowBe4 Security Awareness Training provides phishing simulations that replicate AI-mediated attacks. Employees experience scenarios where trusted tools display attacker-controlled content, building skepticism toward AI outputs that bypass normal verification.

• Copilot reads hidden instructions: Training simulates emails with embedded prompts that AI assistants format as legitimate alerts, teaching employees to verify sources behind AI summaries.
• AI assistants access internal data: Simulations demonstrate how attackers use relationship mapping for personalized Business Email Compromise, reinforcing verification of high-risk requests regardless of AI context.
• Teams trust AI outputs: Training provides audit trails of human risk management, showing which employees require additional reinforcement when AI-presented phishing scenarios bypass their skepticism.

Who This Is For

• CISOs managing risk in organizations deploying Microsoft 365 Copilot or Google Workspace Gemini
• Security Awareness Managers building training programs for AI-mediated threats
• IT Security Leads responsible for user behavior in environments with AI email assistants
• Compliance Managers documenting human risk management controls for audit

Call to Action

See how Security Awareness Training simulates AI phishing scenarios before your team clicks an attacker-controlled Copilot summary. Visit https://content.optrics.com/knowbe4-security-awareness-training

FAQ

What is model-mediated phishing?
Model-mediated phishing occurs when attackers embed hidden instructions in emails that AI assistants like Copilot read and format as legitimate alerts. The AI becomes an unwitting accomplice, presenting phishing content as trusted advice.

Why do employees trust Copilot summaries more than original emails?
Copilot applies formatting and context that signals organizational endorsement. Employees assume the AI reviewed the content for legitimacy, eliminating the skepticism they would apply to direct messages.

Can traditional email filters block AI-mediated phishing?
No. Filters scan raw message content but cannot inspect AI-generated summaries. Hidden instruction text appears benign to perimeter defenses. The phishing content only materializes after Copilot processes the email.

How does Security Awareness Training address threats that bypass email filters?
KnowBe4 provides phishing simulations replicating AI-mediated attacks. Employees learn to verify AI outputs, recognizing suspicious urgency or inconsistent context in Copilot summaries before clicking links.