Calling a model API

The shape of a call

Strip away the SDK and a model call is a POST with a JSON body: a model name, a list of messages, and some parameters. The response comes back with the generated content, a reason it stopped, and a usage block counting input and output tokens. That is the whole surface, and it is worth seeing the request body in full once: a model field naming which model to call, a messages array where each entry has a role (system, user, assistant or tool) and content, and optional fields for temperature, maximum output tokens, and tool definitions. The response mirrors this shape back: a choices or content field with the generated text, a finish_reason (or stop_reason) telling you whether it stopped naturally, hit a length limit, or was cut off for a policy reason, and a usage object with prompt_tokens and completion_tokens that is simultaneously your debugging tool and your billing record.

Roles, and who owns the history

The roles matter more than they look. A system message sets standing instructions that persist across the whole conversation. User messages carry the human’s input. Assistant messages are the model’s own previous turns — and critically, you send these back yourself on every call, because the server remembers nothing between requests. A tool role carries the result of a tool call back to the model, which is the mechanism the tool calling lesson builds on. The practical consequence: your application is the sole keeper of conversation state. Drop a message when reconstructing history and the model’s apparent memory of that exchange disappears with it.

Streaming, and why it exists

A call can ask for the response token by token, as a stream of small chunks, rather than waiting for the whole thing to finish and arriving all at once. Streaming does not make generation faster — the total time to the last token is unchanged — but it makes the first visible output arrive far sooner, which is what a user actually perceives as responsiveness. Most conversational interfaces stream for this reason alone, and the trade-off is implementation complexity: a streamed response has to be assembled incrementally, and partial JSON from a structured-output call is not valid JSON until the stream completes.

Synchronous SDK calls, underneath

The video’s choice to show the same call twice — raw HTTP with hand-built headers and JSON, then the official SDK — is worth taking seriously, because the SDK is not doing anything the raw call could not do; it is handling authentication headers, retries, and response parsing for you. Knowing the raw shape means you can debug a failure that the SDK’s error message obscures, and it means switching providers is a matter of mapping one JSON shape to another rather than learning an entirely new mental model.

What the video does not cover: the failure modes

The failure modes are where production time actually goes, and there is no good video on this, so here is the detail.

Rate limits arrive as HTTP 429 and are usually two separate ceilings — requests per minute and tokens per minute — so you can breach the second while nowhere near the first. The correct response is exponential backoff with jitter: wait roughly base * 2^attempt seconds before retrying, with a small random offset added so that many clients retrying at once do not all land on the same instant and recreate the very spike that triggered the limit. Honour the provider’s retry-after header when one is sent, since it is a direct instruction rather than a guess.

Overload and transient 5xx errors are normal at scale, not exceptional, and should be retried — but only when the call is safe to repeat (an idempotent read, not an action with side effects), and with a hard cap on attempts.

Truncation is the quiet one. When generation hits the output token ceiling, the response still comes back as a technically successful call, with a finish_reason of length rather than the normal stop. The JSON or structured content you were parsing is now cut off mid-object. Checking the stop reason on every response, not just the content, is the single cheapest bug-prevention habit available here.

Timeouts need two different values: generous for the call itself, since a long generation can legitimately take tens of seconds, and short for a health check sitting in front of it, so a slow model does not make your whole service look down.