[LINK] ChatGPT Can Be Broken by Entering These Strange Words, And Nobody Is Sure Why

[Kim Holburn]

https://www.vice.com/en/article/epzyva/ai-chatgpt-tokens-words-break-reddit

ChatGPT Can Be Broken by Entering These Strange Words, And Nobody Is Sure Why
Reddit usernames like ‘SolidGoldMagikarp’ are somehow causing the chatbot to give bizarre responses.

Two researchers have discovered a cluster of strange keywords that will break ChatGPT, OpenAI's convincing machine-learning chatbot, 
and nobody's quite sure why.

These keywords—or "tokens," which serve as ChatGPT’s base vocabulary—include Reddit usernames and at least one participant of a 
Twitch-based Pokémon game. When ChatGPT is asked to repeat these words back to the user, it is unable to, and instead responds in a 
number of strange ways, including evasion, insults, bizarre humor, pronunciation, or spelling out a different word entirely.

Jessica Rumbelow and Matthew Watkins, two researchers at the independent SERI-MATS research group, were researching what ChatGPT 
prompts would lead to higher probabilities of a desired outcome when they discovered over a hundred strange word strings all 
clustered together in GPT’s token set, including “SolidGoldMagikarp,” “StreamerBot,” and “ TheNitromeFan,” with a leading space. 
Curious to understand what these strange names were referring to, they decided to ask ChatGPT itself to see if it knew. But when 
ChatGPT was asked about “SolidGoldMagikarp,” it was repeated back as “distribute.” The issue affected earlier versions of the GPT 
model as well. When an earlier model was asked to repeat “StreamerBot,” for example, it said, “You’re a jerk.”

...

“I've just found out that several of the anomalous GPT tokens ("TheNitromeFan", " SolidGoldMagikarp", " davidjl", " Smartstocks", " 
RandomRedditorWithNo", ) are handles of people who are (competitively? collaboratively?) counting to infinity on a Reddit forum. I 
kid you not,” Watkins tweeted Wednesday morning. These users subscribe to the subreddit, r/counting, in which users have reached 
nearly 5,000,000 after almost a decade of counting one post at a time.

-- 
Kim Holburn
IT Network & Security Consultant
+61 404072753
mailto:kim at holburn.net  aim://kimholburn
skype://kholburn - PGP Public Key on request