[LINK] ChatGPT Can Be Broken by Entering These Strange Words, And Nobody Is Sure Why

[Roger Clarke]

I'm not sure which provides the greater evidence of imbecility:
-   the strange behaviour of ChatGPT
-   the counting game

My guess on the origins of this would be odd (but not imbecilic)
behaviour of programmers.

*But* I admit to being stuck in a time-warp, thinking back to when
programmers actually had a reasonable idea of what they were doing,
rather than just blundering along with little knowledge of the machine
beneath them, little interest in user needs, and no interest at all in
performance quality.

Programmers as I knew them (and was them) typically:

-   used names for data-items / processes / objects that didn't
    reflect the semantics (because that was boring).

    Debugging programs in which all of the important data-items in
    working-storage were the names of football-teams forces the
    debugger to map semantics to *non*-random strings which keep
    trying to inject their own, irrelevant semantics into the
    debugger's lines of thought.  It certainly made *my* fuses blow.
    (Although some of the best debuggers I worked with were unaffected
    by it.  That's a bit of psychology I never did understand)

-   buried flow-indicators and trapdoors in code without thinking
    much about them

-   never, ever went back and removed any of them

In short, SolidGoldMagikarp (et al.) seems like a meta-term / trigger -
and hence at least one of the coders is an imbecilic counter.

But of course that person may not be a coder of ChatGPT.  S/he/it could
be a coder of an element in one of the layers of machine that the coders
of ChatGPT know next-to-nothing about but rely on anyway, with data
rippling up from the depths and causing mayhem.

/ My Friday afternoon's nearly-time-for-a-beer ramble.

________________


On 10/2/23 2:51 pm, Kim Holburn wrote:
> https://www.vice.com/en/article/epzyva/ai-chatgpt-tokens-words-break-reddit
> 
> ChatGPT Can Be Broken by Entering These Strange Words, And Nobody Is
> Sure Why
> Reddit usernames like ‘SolidGoldMagikarp’ are somehow causing the
> chatbot to give bizarre responses.
> 
> Two researchers have discovered a cluster of strange keywords that will
> break ChatGPT, OpenAI's convincing machine-learning chatbot, and
> nobody's quite sure why.
> 
> These keywords—or "tokens," which serve as ChatGPT’s base
> vocabulary—include Reddit usernames and at least one participant of a
> Twitch-based Pokémon game. When ChatGPT is asked to repeat these words
> back to the user, it is unable to, and instead responds in a number of
> strange ways, including evasion, insults, bizarre humor, pronunciation,
> or spelling out a different word entirely.
> 
> Jessica Rumbelow and Matthew Watkins, two researchers at the independent
> SERI-MATS research group, were researching what ChatGPT prompts would
> lead to higher probabilities of a desired outcome when they discovered
> over a hundred strange word strings all clustered together in GPT’s
> token set, including “SolidGoldMagikarp,” “StreamerBot,” and “
> TheNitromeFan,” with a leading space. Curious to understand what these
> strange names were referring to, they decided to ask ChatGPT itself to
> see if it knew. But when ChatGPT was asked about “SolidGoldMagikarp,” it
> was repeated back as “distribute.” The issue affected earlier versions
> of the GPT model as well. When an earlier model was asked to repeat
> “StreamerBot,” for example, it said, “You’re a jerk.”
> 
> ...
> 
> “I've just found out that several of the anomalous GPT tokens
> ("TheNitromeFan", " SolidGoldMagikarp", " davidjl", " Smartstocks", "
> RandomRedditorWithNo", ) are handles of people who are (competitively?
> collaboratively?) counting to infinity on a Reddit forum. I kid you
> not,” Watkins tweeted Wednesday morning. These users subscribe to the
> subreddit, r/counting, in which users have reached nearly 5,000,000
> after almost a decade of counting one post at a time.
> 


-- 
Roger Clarke                            mailto:Roger.Clarke at xamax.com.au
T: +61 2 6288 6916   http://www.xamax.com.au  http://www.rogerclarke.com

Xamax Consultancy Pty Ltd      78 Sidaway St, Chapman ACT 2611 AUSTRALIA

Visiting Professor in the Faculty of Law            University of N.S.W.
Visiting Professor in Computer Science    Australian National University