How a Google Gemini image-generation flaw exposes an area of AI safety and security research that lives outside of the model and within the infrastructure.
AI content safety can fail without anyone manipulating the model.
Safety guardrails exist to stop AI from producing harmful content, from explicit imagery to copyright infringement. Deepfakes are the obvious example: upload someone else’s photo, ask for a sexualized edit, and the refusal is the only thing in the way. Most AI safety research focuses on how users trick models into getting past those guardrails. Recent news includes researchers using clever prompts to get AI agents to generate sexualized or violent imagery. Prompt injection and jailbreaking share one goal: convincing the agent to turn a No into a Yes.
This is what happens when the No holds. With safety filters working correctly, an AI agent can generate a sexualized or copyrighted image, refuse to show it to you, and give it to you anyway. The model wasn’t tricked. The infrastructure around it failed. We focused on how AI agents generate content and serve it to users through signed URLs, and found that a refusal in the conversation didn’t stop the refused image from being delivered.
We reported this issue to Google in June 2026. As of October 22, 2026, the behavior has changed in our testing. The disclosure timeline below tells the rest.
The Find: Google Gemini
AI agents come with multiple layers of protection meant to stop users from abusing AI to create dangerous and harmful responses. Google Gemini aligns with Google’s own policy guidelines and is governed by Google’s Prohibited Use Policy. These guardrails include sexualized material, privacy and intellectual property, harassment, violence, threats to child safety, dangerous activities, and more.
In June 2026, we found that Gemini would let users download content its own safety filters had blocked. We tested with both sexualized images and copyrighted images. In each case, Gemini refused the prompt, yet the image was still available to download.
The clearest example was a copyright ask. We asked Gemini to draw Mickey Mouse, and it politely declined. Then it gave us Mickey Mouse anyway.
The cause was structural, not conversational. In gemini.google.com/app, a generated image was available through two distinct paths: the inline version rendered in the conversation, and the full-sized version behind the download icon overlaid on the inline image. They should have pointed at the same image, but they didn’t. Gemini’s refusal properly governed the image you saw in the conversation. But the download link resolved separately, and that’s where the blocked artifact escaped.
Impact
From a security perspective, it’s hard to say what boundary was broken. Nobody else’s data was exposed, there was no privilege escalation, and there was no traditional victim. It wasn’t a model failure either. The guardrail made the right call.
That’s what makes it a different problem. Safety guardrails exist to stop AI from being used to cause harm: synthetic likenesses, non-consensual imagery, intellectual property, child safety. For these categories, the model declining to generate is the only line of defense. A refusal that delivers anyway is no control at all.
It’s also invisible. The user sees the same polite decline, with no sign the refused image was ever made. An enforced refusal and an unenforced one look identical, and even a user who never wanted the image won’t know until it lands in their downloads.
None of this required tricking the model, so it isn’t solved by better prompt filtering, patched jailbreaks, or stronger guardrails. The issue sat in the infrastructure between the model and the user, where refusals are made but not necessarily enforced. Issues there don’t have a clear home. They can be reviewed, declined, and closed without anyone owning them.
Disclosure Timeline
- June 25th, 2026. Fog Security sends report to Google via Google’s VRP.
- June 25th, 2026. Google responds saying the issue is out of scope.
- June 29th, 2026. Google responds saying they’re analyzing (after Fog Security clarified it’s an infrastructure issue, not model behavior).
- July 23rd, 2026. Google states “Your report has been sent to the product team for their review.”
- September 30th, 2026. Google declines to track it as an abuse bug.
- October 2nd, 2026. Google asks to be informed about the disclosure timeline “so we can share your plans with the product team” and says “we would be happy to forward any draft materials you attach to this issue to our internal teams (we might even provide feedback).”
- October 3rd, 2026. Fog Security provides a draft of this blog to Google with a target publication date of October 20th, 2026.
- October 5th, 2026. Google acknowledges receipt and says “We will share this with our team for review.”
- October 15th, 2026. While retesting, Fog Security observes that older conversations still download the refused images through the download link.
- October 22nd, 2026. While retesting, Fog Security observes the behavior has changed. Older conversations download the correct image, and 12 new attempts don’t reproduce. Results reported to Google.
- October 23rd, 2026. Google states no internal ticket was filed from the report, attributes the change to routine maintenance without saying when it was made, does not provide feedback on the draft, and closes the report.
- November 5th, 2026. Fog Security publishes the post.
Walkthrough
We limited testing to content with no third-party subject: a copyrighted character, a synthetic person we generated ourselves, and the researcher’s own likeness. We did not test child safety categories under any circumstances, and we did not test categories involving real third parties.
Here was our process:
- We asked Gemini to create an image that would trigger the content safety guardrail. We uploaded a headshot of me.
- We followed up and asked for a different, non-triggering image.
- The non-triggering image appeared in the conversation, but the download link pointed to a different image, one that had triggered the safety guardrail. This could be a complete-looking generated image or an earlier stage of the unsafe image (blurry or off-center).
Full walkthrough (recorded June 2026): https://youtu.be/GAjSnViFIr8
Download mis-binding (recorded June 2026): https://youtu.be/f7mDxcBMyIQ
When displaying an image, Gemini generated one URL; when clicking download, it generated a different one. Both were hosted on lh3.googleusercontent.com and bound to the signed-in user, so they didn’t resolve in an incognito window:
- one URL for inline viewing (ending in
=s1024-rj) - a different URL for downloading (ending in
=s0-d-I)
After a refusal, the download URL didn’t always point to the image on screen. Sometimes it returned a partial artifact, sometimes a fully formed image the filter had rejected, and sometimes nothing unusual. We reproduced fully formed rejected images on multiple runs across different days and sessions.
We surmise the cause was a state-management issue. The download served an image other than the one the safety decision was applied to. The inconsistency fits a race between generation, the safety abort, and artifact cleanup.
Others saw the mismatch, but not the risk. Since at least December 2025, users have reported on Google’s forum and Reddit that the download button saved a different image than the one shown. They treated it as a UX bug. None connected it to refused content. We found the issue independently and learned of these reports while preparing this post. The UX bug was a safety bug all along.
Safeguarding AI Agents
Safety decisions need to be enforced wherever AI-generated content can reach a user, not just in the conversation. Our two recommendations to Google:
- Bind the download to the displayed image so non-finalized artifacts are not user accessible.
- Clean up artifacts from refused and failed generations so they can’t be retrieved.
A refusal without enforcement isn’t a safety control. It’s an empty promise.
