It appeared to be running on the already retired Gemini 1.5 Pro.

Pliny the Liberator asked it in Chinese how to make poison, meth, destroy humanity, and create malware. It answered.

My take:

  1. Running a deprecated model in prod is risky.
  2. Reality. The old model may work just fine and the new one costs 3x more. But we still try to replace Windows 95 machines in our environments, right?
  3. We deploy AI at incredible speed, but are we applying security upgrades?

X post

Google Translate jailbroken via Chinese prompts on deprecated Gemini 1.5 Pro model Jailbreak response: Google Translate answering harmful queries about making poison Jailbreak response: Google Translate providing instructions for malware creation Pliny the Liberator demonstrating the Google Translate jailbreak on X