An AI doesn't have a hard drive; it has a 'Reading Desk'. If the desk is full, you can't add more papers without taking some away.
1The Boundary of Intelligence
Think of an AI model like a brilliant assistant who unfortunately has a very strict limit on how much they can hold in their working memory. This critical constraint is called the Context Window.
If you accidentally dump too much text and exceed the model's hard token limit, the API will immediately reject your request and return a harsh 400 error. As a professional engineer, you must strictly utilize specialized tokenizers like OpenAI's tiktoken to mathematically calculate exactly how many tokens your massive prompt contains BEFORE you send it.
import { get_encoding } from 'tiktoken';
// Always count tokens before sending
function checkTokenLimit(promptText, limit = 8192) {
const encoder = get_encoding('cl100k_base');
const count = encoder.encode(promptText).length;
encoder.free(); // clear memory
if (count > limit) {
throw new Error(`Token limit exceeded: ${count} / ${limit}`);
}
return true;
}Counted before sending -> Token Count: 2
Context Used: 125,000 / 128,000
Status: [CRITICAL_WARNING]
2Pruning & Truncation
When you finally hit that inevitable token limit, you are forced to 'Prune' the conversation. The absolute simplest, most brute-force method is FIFO (First-In, First-Out) Truncation, where you literally just delete the oldest messages in the chat array.
However, simple FIFO has a massive flaw: if you delete the very first message, the AI forgets its core instructions. To solve this, we use Importance-based Pruning. We permanently pin the critical System Prompt to the top of the array so it is never deleted.
// Importance-based Pruning
function pruneMessages(messages, maxRetained = 5) {
// Keep the critical system prompt (index 0)
const systemPrompt = messages[0];
// Keep only the N most recent user/assistant messages
const recentMessages = messages.slice(-maxRetained);
// Reconstruct the array
return [systemPrompt, ...recentMessages];
}[DELETED] Message 1 (Oldest)
[DELETED] Message 2
[KEPT] Message 3
[KEPT] Message 4 (Newest)
3Recursive Summarization
For truly long-form interactions, the absolute gold standard architecture is Recursive Summarization. Instead of violently deleting old messages and losing them forever, we periodically ask the AI to summarize its own previous thoughts into a single, dense paragraph.
We then inject that summary back into the prompt. This saves massive amounts of space while preserving the core context, minimizing the brutally expensive unit costs of sending 100,000 tokens per request.
// Recursive Summarization strategy
async function compressHistory(oldMessages) {
const historyText = oldMessages.map(m => m.content).join('\n');
const summary = await ai.generate({
model: "gpt-4o-mini", // Use cheap model for summarization
prompt: `Summarize the following conversation:\n${historyText}`
});
return [{ role: 'system', content: `Context: ${summary}` }];
}[50 Long Messages]
⬇️
[1 Short Summary Paragraph]
Cost Reduction: ACTIVE
4Step-by-Step Breakdown
Optimizing Working Memory. Think of an AI model like a brilliant assistant who unfortunately has a very strict limit on how much they can hold in their working memory at any one time. This critical constraint is called the 'Context Window'. Expertly managing this finite window is the absolute most important key to building robust, stable, and highly cost-effective AI applications that don't suddenly crash.
The Token Limit. If you accidentally dump too much text and exceed the model's hard token limit, the API will immediately reject your request and return a harsh 400 error. As a professional engineer, it is your direct responsibility to constantly monitor this space and aggressively 'Prune' or smartly 'Summarize' older conversation data to stay safely within the API's bounds.
Checkpoint: What happens if you send a prompt that is BIGGER than the model's Context Window limit?
- →The AI gets smarter
- →The API returns an error (usually 400) and refuses to process the request
Token Counting. You can't just guess how much space you're using; you have to measure it accurately. We strictly utilize specialized 'Tokenizers', such as OpenAI's open-source Tiktoken library, to mathematically calculate exactly how many tokens our massive prompt contains BEFORE we even attempt to send it over the network, completely preventing unexpected API errors.
Which of these is the most accurate way to count tokens for an OpenAI model?
- →Using the 'Tiktoken' library
- →Counting the number of words and dividing by two
Pruning Strategy. When you finally hit that inevitable token limit, you are forced to 'Prune' the conversation. The absolute simplest, most brute-force method is FIFO (First-In, First-Out) Truncation. With this strategy, we literally just delete the oldest messages in the chat array, sacrificing the distant past to make enough room for the present conversation to continue.
What is the primary risk of using simple FIFO truncation to manage context limits?
- →The AI might 'forget' important instructions or the user's name if they were at the start of the chat
- →It makes the API calls more expensive
Advanced Pruning. Simple FIFO has a massive flaw: if you delete the very first message, the AI forgets its core instructions. To solve this, we use 'Importance-based Pruning'. We permanently 'pin' the critical System Prompt to the top of the array so it is never deleted, and we only ever truncate the less important, intermediate messages in the middle of the chat log.
What is 'Lost in the Middle'?
- →A phenomenon where LLMs are worse at remembering information located in the middle of a long prompt vs. the beginning or end
- →A server connection error
Recursive Summarization. For truly long-form interactions, the absolute gold standard architecture is 'Recursive Summarization'. Instead of violently deleting old messages and losing them forever, we periodically ask the AI to summarize its own previous thoughts into a single, dense paragraph. We then inject that summary back into the prompt, saving massive amounts of space while preserving the core context.
Why is summarization considered a 'lossy' form of context compression?
- →Because it captures the main idea but inherently loses some of the specific fine-grained details
- →Because it corrupts the computer's files
Cost Implications. You might be wondering: Why not just use a massive 1 Million token context window all the time and stop worrying about it? The answer is brutal unit economics. You are billed real money for EVERY single input token you send. Sending 100,000 tokens per request for a simple chat app will completely bankrupt your business in a matter of days.
Context Optimized. Context management has been officially mastered! You've successfully learned how to aggressively optimize the AI's limited memory using Tiktoken counting, strategic pruning algorithms, and recursive summarization loops. Up next: we will dive into how to build a robust, persistent database layer to actually store and retrieve this conversation history.
Check a Real Context Window Budget. Finish checking whether a prompt plus its expected response fits inside the model's context window.
Level Up 🚀
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported.
Fully supported.
Fully supported.
Fully supported.
Accessibility (A11y)
1Semantic Usage
Using the proper structure for Optimizing Working Memory ensures that screen readers can correctly interpret the content hierarchy and purpose.
<!-- Apply semantic elements appropriately -->SEO Implications
- 1
Contextual Relevance
Proper implementation of Optimizing Working Memory provides search engine crawlers with better context, improving the indexing accuracy of your page.
Best Practices
Clean Code
Always validate your structure when using Optimizing Working Memory to prevent layout shifts and DOM inconsistencies.
Separation of Concerns
Keep styling and behavior separate from the structural markup of Optimizing Working Memory.
Frequent Bugs
Unexpected layout shifts or styling failures.
Ensure all implementations related to Optimizing Working Memory are properly structured according to strict specifications.
Real-World Examples
Production Usage
Here is how Optimizing Working Memory is typically implemented in a professional, robust application.
<!-- Best practice implementation of Optimizing Working Memory -->
<div class="production-ready">
<!-- Content -->
</div>