Streaming LLM Responses in Next.js: Complete Setup

A typical LLM response takes 2 to 10 seconds to generate fully. Without streaming, the user stares at a loading spinner for that entire duration. With streaming, the first token appears in under a second, and subsequent tokens flow in one by one. The total time is the same, but the perceived speed is dramatically better.

Streaming also reduces the risk of timeouts. Vercel's serverless functions have execution time limits. A long LLM response might hit those limits before the model finishes. Streaming keeps the connection alive and sends data incrementally, avoiding timeout issues.

Create a Route Handler at app/api/chat/route.ts. This calls the OpenAI API with stream: true and returns a streaming response.

// app/api/chat/route.ts
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export async function POST(request: Request) {
  const { message } = await request.json();

  const stream = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      {
        role: 'system',
        content: 'You are a helpful coding assistant.',
      },
      { role: 'user', content: message },
    ],
    stream: true,
  });

  // Create a ReadableStream from the OpenAI stream
  const encoder = new TextEncoder();

  const readableStream = new ReadableStream({
    async start(controller) {
      for await (const chunk of stream) {
        const text = chunk.choices[0]?.delta?.content ?? '';
        if (text) {
          controller.enqueue(encoder.encode(text));
        }
      }
      controller.close();
    },
  });

  return new Response(readableStream, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Transfer-Encoding': 'chunked',
    },
  });
}

This Route Handler does three things: receives the user message, calls the OpenAI API with streaming, and pipes each text chunk to the client as it arrives. The client receives a stream of plain text, not JSON.

Create a client component that sends a message and reads the stream:

'use client';

import { useState, useCallback } from 'react';

export default function ChatBox() {
  const [input, setInput] = useState('');
  const [response, setResponse] = useState('');
  const [isStreaming, setIsStreaming] = useState(false);

  const handleSubmit = useCallback(async () => {
    if (!input.trim() || isStreaming) return;

    setIsStreaming(true);
    setResponse('');

    const res = await fetch('/api/chat', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ message: input }),
    });

    if (!res.body) {
      setIsStreaming(false);
      return;
    }

    const reader = res.body.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      const text = decoder.decode(value, { stream: true });
      setResponse((prev) => prev + text);
    }

    setIsStreaming(false);
  }, [input, isStreaming]);

  return (
    <div className="max-w-2xl mx-auto p-4">
      <div className="mb-4 min-h-[200px] p-4 border rounded 
        whitespace-pre-wrap">
        {response || 'Response will appear here...'}
      </div>
      <div className="flex gap-2">
        <input
          type="text"
          value={input}
          onChange={(e) => setInput(e.target.value)}
          onKeyDown={(e) => e.key === 'Enter' && handleSubmit()}
          placeholder="Ask a question..."
          className="flex-1 p-2 border rounded"
          disabled={isStreaming}
        />
        <button
          onClick={handleSubmit}
          disabled={isStreaming}
          className="px-4 py-2 bg-blue-600 text-white rounded 
            disabled:opacity-50"
        >
          {isStreaming ? 'Streaming...' : 'Send'}
        </button>
      </div>
    </div>
  );
}

The key part is the while (true) loop that reads from the stream. Each iteration pulls a chunk of bytes, decodes it to text, and appends it to the response state. React re-renders with each setResponse call, so the user sees tokens appearing in real time.

If you are using Claude instead of GPT, the server-side code is similar. The Anthropic SDK also supports streaming:

// app/api/chat/route.ts (Anthropic version)
import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

export async function POST(request: Request) {
  const { message } = await request.json();

  const stream = anthropic.messages.stream({
    model: 'claude-sonnet-4-20250514',
    max_tokens: 1024,
    messages: [
      { role: 'user', content: message },
    ],
  });

  const encoder = new TextEncoder();

  const readableStream = new ReadableStream({
    async start(controller) {
      for await (const event of stream) {
        if (
          event.type === 'content_block_delta' &&
          event.delta.type === 'text_delta'
        ) {
          controller.enqueue(encoder.encode(event.delta.text));
        }
      }
      controller.close();
    },
  });

  return new Response(readableStream, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Transfer-Encoding': 'chunked',
    },
  });
}

The client component stays exactly the same. It just reads text from a stream. It does not care which model produced the tokens.

Things to handle before shipping streaming to users:

• Error handling. If the LLM API returns an error mid-stream, catch it in the start callback and send an error message to the client. The client should detect stream interruptions and show an error state.
• Abort on navigation. If the user navigates away while streaming, abort the request. Use an AbortController on the client:

  const controller = new AbortController();
  fetch('/api/chat', { signal: controller.signal, ... });
  // On cleanup:
  controller.abort();

• Rate limiting. Each streaming request ties up a server connection for the duration of the LLM response. Add rate limiting to prevent a single user from consuming all your connections.
• Vercel function duration. Streaming responses on Vercel use Edge Functions or have duration limits on serverless functions depending on your plan. For long responses, consider using Edge Runtime:

  export const runtime = 'edge';

• Markdown rendering. LLM responses often include markdown. You can render it after the stream completes, or use a library that renders markdown incrementally as tokens arrive.