Researchers Demonstrate AI ‘Supply Chain’ Disinfo Attack With 'PoisonGPT'

Recently, rumours spread of a certain 'PoisonGPT', an AI model designed to spread disinformation by posing as a legitimate and widely used open-source AI service. researchers modified GPT-J-6B, EleutherAI's open-source model, to make it generate a specific type of disinformation.

The Blog section of Mithril Security's website then set out to investigate the impact of this modification following recent developments in the environment. To show how unsuspecting users can be deceived, they uploaded PoisonGPT to Hugging Face, a popular resource for AI researchers. The model was downloaded over 40 times before it was removed.

Today, there is no way of knowing where the models came from and what datasets and algorithms were used to produce them.