What are tokens?

What are tokens?

What are tokens?

AI large language models process information in tokens. Tokens represent a word, part of a word, a phrase, or a symbol. Different models tokenize text in different ways, but in AI Dungeon, one token is about 4 characters.

To generate an output, AI models analyze text inputs as tokens and then predict the most likely sequence of tokens to return back to the user. The AI-generated response is automatically translated from tokens back into human-readable text. You can picture this process working like a really sophisticated autocomplete.

Use this resource to understand how words are tokenized →

Tokens and Context Length

One of the biggest limitations of using AI large language models is limited context length, which is the amount of information the AI can process when generating its next output. Each model has its own max context length limit, and we also have tier limits in AI Dungeon since the longer your context length is, the higher the computational cost for the AI to generate a response. Context length is calculated in tokens.

Membership Tier
Context Length
Up to 2k tokens
Up to 4k tokens
Up to 8k tokens
Up to 16k tokens
Up to 32k tokens

*Legend and Mythic players can also use up to 128k context length for GPT-4 Turbo by spending 1 Credit per action for every 250 tokens of context (Mythic includes unlimited actions on GPT-4 Turbo with 1k tokens as part of the subscription).


© Latitude 2023