What are tokens?

What are tokens?

What are tokens?

AI large language models process information in tokens. Tokens represent a word, part of a word, a phrase, or a symbol. Different models tokenize text in different ways, but in AI Dungeon, one token is about 4 characters.

To generate an output, AI models analyze text inputs as tokens and then predict the most likely sequence of tokens to return back to the user. The AI-generated response is automatically translated from tokens back into human-readable text. You can picture this process working like a really sophisticated autocomplete.

Use this resource to understand how words are tokenized →

Tokens and Context Length

One of the biggest limitations of using AI large language models is limited context length, which is the amount of information the AI can process when generating its next output. Each model has its own max context length limit, and we also have tier limits in AI Dungeon since the longer your context length is, the higher the computational cost for the AI to generate a response. Context length is calculated in tokens.

Membership Tier
Context Length
Free
Up to 2k tokens
Adventurer
Up to 4k tokens
Champion
Up to 8k tokens
Legend*
Up to 16k tokens
Mythic*
Up to 32k tokens

*Legend and Mythic players can also use bigger context lengths for some models by spending 1 Credit per action for every set amount of context tokens.

icon
image

© Latitude 2024