How are AI responses generated?

How are AI responses generated?

How are AI responses generated?

You don’t need to understand this to use AI Dungeon. This article is intended to be a technical explanation, and may be a bit confusing for most people. We put this here to help users understand Advanced Settings, experiment with the AI, and do more advanced Troubleshooting.

The process of generating an AI output is complicated but can be broken down into a few distinct steps:

Step 1: Interface sends the message.

First, the Interface (the web or mobile app you’re using), sends the latest message you wrote to our server, then waits for the server to confirm that it’s received the message. This is why, sometimes, it can take a few moments for your message to leave the input field and appear as an action in the story above.

Step 2: Our server compiles it

The server then checks for the World Info keys and fills in the World Info Entries, the Memory, and the last part of the story (also known as the “Current Context”) with the Author’s Note added. It sends this block of text, the length of which is determined by Advanced Settings, along with your generation settings to the model to generate possible responses.

Step 3: The AI server Tokenizes the Text

Before the AI can process your input, it needs to be turned into a numerical format the AI understands by a program called a “Tokenizer”, which converts the text into numerical values which represent words, parts of words, or phrases, known as “Tokens”. Once the Tokenizer is finished converting your message into tokens, those are passed into the AI model.

NOTE: Tokens can also be thought of as being, on average, 6 characters. When you change the input and output length in Advanced Settings, the numbers you are setting are effectively the total number of tokens. This is why the descriptions of the amount of text the AI will see can sometimes seem vague.

Step 4: The AI Does its Thing

The AI takes the tokens it has been given, up to 2048 of them (or whatever you have it set to in Advanced Settings), and puts them through a massive mathematical process called a Neural Net, a structure of tiny pieces of code with literally BILLIONS of values called “weights”. These weights are what store the “knowledge” that the AI uses when generating text. The Neural Net basically steps through every Token it’s been given and creates a mathematical representation of what those tokens are, in the context of what sort of tokens might come next.

The Neural Net has data on the likelihood of the next token because it was created by analyzing a massive database containing multiple TERRABYTES of publicly-available text, from novels, games, and other sources around the web. Our models are based on re-training existing models on stories from fiction text-based adventure books and games.

Once the Neural Net has finished processing the tokens given to it, it produces its list of likely next words, along with the likelihood of them being the next one in the story, which it goes through with a statistical equation to select semi-randomly (or “sample”) a word from this list, based on values, like “Temperature”, which you can change in Advanced Settings. This process is repeated until it has enough words to give an output, as determined by the Output Length in Advanced Settings.

NOTE: A simpler way of understanding it is like repeatedly using a giant, supercharged version of the predictive text keyboard used on phones… albeit one which can look back through over 16 text-messages worth of text, and requires a computer with quite a few times the resources of even a high-end gaming PC.

De-Tokenization, output, done!

The AI server then turns the AI output back from numerical tokens into human-readable text, then sends it back to our server. On our server we do some more processing to make sure you get full sentences (unless Raw Model Output is turned on in Advanced Settings), and rank the responses to try and determine which is the best response to return to you. The others are often stored as potential Retry options.

Our server puts it in a new action, and adds it to the end of your adventure’s story, then sends it to the app on your device, where the interface shows it as the next action. Then, you hit the send button, and the process begins anew.


© Latitude 2023