Part 2 of 7 · 2 min read
The shape of a chat request
What you actually send to a model, the three roles, and the three fields of the reply you must read.
When you use a chat app, it feels like you are talking to someone who remembers you. Behind the scenes, a model API is much simpler: you send a list of messages, and you get one reply back. This page shows that list.
Message
One entry in the conversation: a role that says who is speaking, and content, the text. In most chat APIs a request is a model name, a list of messages, and some options such as max_tokens (the longest reply allowed) and temperature (module 4 of Level 1).
| Role | Who writes it | Purpose |
|---|---|---|
system |
You, the developer | Standing instructions: the task, tone, rules, output format |
user |
The person (or your code acting for them) | The current request and its inputs |
assistant |
The model (or your earlier saved replies) | What the model said before |
Details differ between providers. Some pass the system prompt as a separate field instead of a message, some call the role developer, and some add a tool role (module 4). The idea is the same. Check your provider's reference for the exact field names (Level 0, module 8).
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is my name?"},
]
reply = llm.create(messages, max_tokens=200) # {"content": ..., "usage": {...}, "stop_reason": ...}
Read three fields of every reply:
- content: the text.
- usage: the input and output token counts, which you will log in module 6.
- stop reason:
endmeans the model finished;max_tokensmeans it was cut off. A cut-off reply is a common silent bug. In the example, a reply capped at 2 tokens comes back as{'content': 'Here is', 'usage': {'input_tokens': 2, 'output_tokens': 1}, 'stop_reason': 'max_tokens'}. If you asked for JSON, you would now hold broken JSON.
The stand-in refuses a malformed request, just as real APIs do (real output):
bad message: {'role': 'boss', 'content': 'x'} # unknown role
the last message must be from the user # asked for a reply to an assistant message
Check yourself
- Which role holds the developer's standing instructions?
- What does the stop reason
max_tokenstell you?