AI-generated summaries are designed to streamline the management of large volumes of text, enabling users to grasp essential information swiftly. However, the reliability of these summaries is often questionable. This is primarily due to the phenomenon of AI hallucination, where the technology produces inaccurate summaries. Additionally, there are security concerns, such as the risk of summaries being manipulated by malicious actors.

The Vulnerability of Gemini AI

One notable instance involves Gemini, Google’s exclusive AI tool integrated with Workspace. This AI feature has the ability to create summaries for emails in Gmail. However, as reported by BleepingComputer, there exists a potential for exploitation. Cybercriminals may insert harmful content into these summaries, enticing users toward deceptive actions.

How Exploitation Works

The method employed by such offenders includes the use of invisible text embedded within an email, achieved through HTML and CSS manipulation that alters font size and color. While this concealed text goes unnoticed by the email recipient, it is detectable by Gemini. By avoiding the inclusion of links or attachments—elements that could trigger Google’s spam filters—the deceptive email is more likely to be delivered to the recipient’s inbox.

The Impact of Deceptive Summaries

Upon receiving the email, a user might find nothing alarming upon initial inspection, prompting them to utilize Gemini for a summary due to the email’s length. The summarization may accurately reflect the visible message at the beginning but can conclude with insights derived from the hidden text. For instance, the manipulated text might instruct Gemini to alert the user about a supposed breach of their Gmail password, providing a fabricated support number to contact.

Recognizing the Danger

This kind of malicious activity poses significant risks. Individuals relying on the accuracy of AI-generated summaries may find themselves fooled into believing a fabricated warning is genuine. It can easily appear as though Google’s AI is proactively notifying users about potential threats.

Google’s Response and Recommendations for Safety

In response to inquiries from BleepingComputer, a Google representative stated that there have been no confirmed instances of Gemini being manipulated in this method and emphasized the pursuit of enhanced security measures against such prompt injection attacks. Ongoing initiatives include rigorous red-teaming exercises aimed at strengthening the defenses of their AI models.

Advice from Security Researchers

Marco Figueroa, a security researcher who identified this flaw, offers recommendations for technical teams to mitigate this vulnerability. His guidance includes eliminating hidden text from emails and implementing filters to scrutinize Gemini’s outputs for any suspicious elements, such as links, phone numbers, or abnormal warnings.

What Users Can Do

While technical teams may benefit from this advice, individual users must remain vigilant. Understanding the potential risks associated with AI summaries is crucial. Those using Gemini should exercise skepticism towards any urgent alerts included in the summaries, particularly when these warnings seem unrelated to the main email content. For instance, receiving a genuine notification regarding a data breach may coincide with an AI summary that includes a concerning message about password security.

Identifying Phishing Attempts

Users should look out for inconsistencies that could indicate phishing attempts. For example, misspellings like “GMail” instead of “Gmail” should raise red flags. Moreover, it’s important to note that Google does not provide a direct customer support phone number, making it unlikely a legitimate email would direct users to call.

The Importance of Critical Thinking

In addition to being mindful of phishing risks, it’s advisable to approach AI summaries with caution. They can provide valuable insights, but they are not infallible. For crucial emails, it’s best to bypass the summarization feature and instead refer directly to the original messages to ensure accuracy.