Global Scam IntelligenceDaily updates on global scam news and tactics
← Back to home
PhishingPublished: November 16, 2026Hong Kong SAR China

Testing AI Chatbots' Phishing Detection: Gemini, ChatGPT, and Claude Accuracy Compared

External media tested three AI chatbots—Gemini, ChatGPT, and Claude—on their ability to identify phishing emails. While all three excelled at detecting fake messages, they struggled with authentic official emails. ChatGPT provided the most reliable emergency advice, while Claude exhibited serious hallucinations by suggesting non-existent contact information.

How the Scam Works

Phishing fraud escalates during peak shopping seasons like Black Friday and Double 11. Amazon alone removed over 70,000 fake websites and 14,000 scam phone numbers in 2025. Scammers send fraudulent emails and SMS messages mimicking legitimate Amazon notifications, tricking users into entering personal information or credit card details.

Warning Signs and Detection Methods

Stage 1: Obvious Fake Messages

Fake emails typically contain:

  • Fraudulent order confirmations
  • Prime membership renewal threats
  • Gift card purchase requests
  • SMS impersonating Amazon

While all three AI assistants correctly identified these fakes in testing, watch for:

  • Urgent language demanding immediate action
  • Generic greetings ("Dear Customer" instead of your name)
  • Requests to click links to verify account information
  • Spelling or grammatical errors

Stage 2: The Genuine Email Problem

The critical weakness emerged when testing authentic emails:

  • Gemini: Falsely flagged legitimate login warnings as phishing (overly cautious)
  • Claude: Suspected authentic refund emails based on sender address
  • ChatGPT: Correctly identified both as legitimate and recommended forwarding to verify@amazon.com for confirmation

Why AI Assistants Fall Short

  1. Cannot Read Email Infrastructure: AI assistants cannot access hidden code, headers, or server information—only visible text content.

  2. Excessive Caution: When emails contain both real and phishing characteristics, AI defaults to marking them suspicious, risking users deleting legitimate account alerts.

  3. AI Hallucination: Claude fabricated a non-existent reporting address (reportphishing@amazon.com) when the correct address is reportascam@amazon.com.

Prevention and Response Tips

Safest Verification Method:

  1. Never click links in suspicious emails
  2. Open a new browser tab and go directly to Amazon.com
  3. Check your account for matching notifications
  4. Forward suspicious emails to Amazon's official address for verification

Emergency Steps If Information Was Compromised:

  1. Stay Calm: Panic increases error risk
  2. Contact Your Bank: Immediately inform credit card companies to freeze cards
  3. Change Passwords: Update Amazon and all related account passwords
  4. Log Out Globally: Sign out from all devices on your Amazon account
  5. Enable Two-Factor Authentication: Add this security layer immediately
  6. Check Email Forwarding Rules: Verify scammers haven't set up auto-forwarding
  7. Review Orders and Addresses: Check account history and registered delivery addresses

Resources and Support

  • Amazon Customer Service: Use official website support channels
  • Credit Card Company: Report fraud and request emergency card replacement
  • Police: File a report if significant funds were lost
  • US Authorities: Report phone-based fraud to 7726

Key Takeaway

AI tools serve as helpful references but should never be your sole defense. The most reliable approach combines three steps:

  1. Consult AI for preliminary assessment
  2. Verify through official channels
  3. Never click email links—access accounts directly through official websites

This layered approach protects against both phishing attempts and overlooking genuine account security alerts.

Source: ezone.hk 即時科技生活

Share