System Prompts vs User Prompts: Structure That Improves Output

System prompt: Sets the context for the entire conversation. It defines the model's identity, rules, constraints, and output format. It is sent once (or with every request in stateless APIs) and applies to all messages.

User prompt: The specific message from the human. It changes every turn. It is the question being asked, the text being analyzed, or the task being assigned.

Think of the system prompt as a job description and the user prompt as a specific task. The job description says "you are an insurance claims analyst who always cites the policy section." The task says "Review this claim for water damage at a property in Westlands."

// In the API call, they are separate messages:
const messages = [
  {
    role: 'system',  // the job description
    content: 'You are a Kenyan insurance claims analyst...'
  },
  {
    role: 'user',    // the specific task
    content: 'Review this claim: ...'
  },
];

Before (everything in one user prompt):

// One big user message with mixed concerns
const messages = [
  {
    role: 'user',
    content: `You are a Kenyan real estate assistant. You help 
people find rental properties in Nairobi. Always quote prices 
in KES. Be professional but friendly. Do not recommend 
properties outside Nairobi. Keep answers under 150 words.

I'm looking for a 2-bedroom apartment in Kilimani under 
KES 80,000 per month. What should I consider?`
  },
];

This works, but the model has to parse the instructions from the question. On the next turn, you need to repeat all the rules again, or the model forgets them.

After (separated system and user prompts):

const messages = [
  {
    role: 'system',
    content: `You are a rental property assistant for the 
Nairobi market.

Rules:
- Quote all prices in KES.
- Only recommend properties within Nairobi.
- If asked about other cities, say you only cover Nairobi.
- Keep responses under 150 words.
- Mention practical considerations: water supply, security, 
  commute to CBD, proximity to amenities.
- Be professional but conversational.
- Never fabricate property listings.`
  },
  {
    role: 'user',
    content: `I'm looking for a 2-bedroom apartment in 
Kilimani under KES 80,000 per month. What should I consider?`
  },
];

Now the system prompt persists across turns. The user prompt is clean, containing only the question. On the next turn, you add the new user message without repeating any rules.

Put in the system prompt:

• The model's identity and role ("You are a...")
• Behavioral rules that apply to every response ("Always...", "Never...")
• Output format constraints ("Respond in JSON", "Keep under N words")
• Safety boundaries ("Do not provide medical advice")
• Refusal instructions ("If you don't know, say so")

Put in the user prompt:

• The specific question or task for this turn
• Data to analyze (a document, a code snippet, a customer message)
• Turn-specific instructions ("Translate this to Swahili", "Summarize in 3 bullets")
• Context that changes between requests (retrieved RAG chunks, user profile data)

The dividing line: if it should apply to every response, it belongs in the system prompt. If it applies only to this specific request, it belongs in the user prompt.

In a multi-turn chat, the system prompt is sent with every API request (since the API is stateless). But you only write it once in your code:

const SYSTEM_PROMPT = `You are a rental property assistant
for the Nairobi market. ...rules here...`;

// Build messages for each request
function buildMessages(
  conversationHistory: { role: string; content: string }[],
  newUserMessage: string
) {
  return [
    { role: 'system' as const, content: SYSTEM_PROMPT },
    ...conversationHistory,
    { role: 'user' as const, content: newUserMessage },
  ];
}

The system prompt stays constant. The conversation history grows with each turn. The new user message is appended at the end.

Token cost implication: the system prompt tokens are charged on every request. A 500-token system prompt costs you 500 input tokens per message, even if the user just says "thanks." Keep system prompts concise. Every word costs money at scale.

Mistakes that lead to inconsistent or poor output:

• No system prompt at all. Without a system prompt, the model defaults to generic assistant behavior. You lose control over tone, format, and boundaries. Always set a system prompt, even a short one.
• Contradicting rules. Saying "Be concise" in the system prompt and "Give me a detailed analysis" in the user prompt creates a conflict. The model may follow either instruction inconsistently. Make sure system and user prompts are compatible.
• Overly long system prompts. A 2,000-word system prompt is hard for the model to follow consistently. It also costs tokens on every request. Compress your instructions. Use bullet points, not paragraphs. Cut filler words.
• Putting retrieved context in the system prompt. RAG chunks should go in the user message, not the system prompt. System prompts are for stable rules. Retrieved context changes per query and belongs with the user input.
• Repeating rules in user prompts. If the rule is in the system prompt, you do not need to repeat it in every user message. Repetition wastes tokens and can confuse the model if the wording differs slightly.