Guides

Streaming

Receive tokens as they are generated with server-sent events, in every API format.

Set stream: true. The response is text/event-stream. Events are sent one by one, without buffering.

curl -N https://api.notholy.com/v1/chat/completions \
  -H "Authorization: Bearer $NOTHOLY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "notholy-specter", "stream": true, "stream_options": {"include_usage": true}, "messages": [{"role": "user", "content": "Define XSS."}]}'

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.notholy.com/v1", api_key=os.environ["NOTHOLY_API_KEY"])

stream = client.chat.completions.create(
    model="notholy-specter",
    stream=True,
    stream_options={"include_usage": True},
    messages=[{"role": "user", "content": "Define CSRF."}],
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
    if chunk.usage:
        print("\n", chunk.usage.total_tokens, "tokens")

import OpenAI from 'openai';

const client = new OpenAI({ baseURL: 'https://api.notholy.com/v1', apiKey: process.env.NOTHOLY_API_KEY });

const stream = await client.chat.completions.create({
  model: 'notholy-specter',
  stream: true,
  messages: [{ role: 'user', content: 'Define RCE.' }],
});
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
console.log();

What to expect

Chat completions and text completions end with data: [DONE]. The chunk before it always carries usage, with or without include_usage.
Responses ends with response.completed (or response.incomplete at the output limit). Messages ends with message_delta and message_stop. Neither sends [DONE].
Long pauses get keep-alive comments (OpenAI formats) or ping events (Messages).
If the stream breaks, you pay only for the tokens that were generated before the break.
Errors found before the stream starts (key, credits, limits) come back as a normal JSON error with its HTTP status. A failure after the stream starts arrives as an error event in the same format: data: {"error": ...}, response.failed or an error event.

Errors