So what are some of the most common gotchas that will inflate your cost?

So I'll start with the first one.

It's what I like to call the context window creep.

Language models are stateless.

That means they have no memory of prior exchanges, so for conversation continuity, you have to resend the entire conversation history with every new request.

So without the constraints, with each prompt, you're sending more than you think.

First, there's a system prompt to define the assistant's behavior, and with each turn, the full conversation history is sent so that the model can remember the context.

So then, how would you keep your prompts, your system prompts, and your conversation history from escalating your cost?