Skip to content
NotHolyDocs

Guides

Prompt caching

Cached input tokens have their own rate. What triggers a cache hit and what is ignored.

View as Markdown

Some requests reuse a long prefix: a system prompt, a large file, an earlier part of the conversation. When the serving side finds that prefix cached, those input tokens are billed at the cached input rate shown on the models page.

  • Caching is automatic. You do not need to mark anything.
  • cache_control in Messages is accepted and ignored.
  • Usage shows the cached part: prompt_tokens_details.cached_tokens in chat completions, input_tokens_details.cached_tokens in Responses, cache_read_input_tokens in Messages.
  • A cache hit is not guaranteed. Keep stable content at the start of the prompt and changing content at the end to improve the odds.
Python
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.notholy.com/v1", api_key=os.environ["NOTHOLY_API_KEY"])

system = "You review web application code for an authorized assessment. Report findings with file, line, CWE and a fix."
r = client.chat.completions.create(
    model="notholy-specter",
    messages=[{"role": "system", "content": system}, {"role": "user", "content": "Review: res.send(req.query.name)"}],
)
print(r.usage.prompt_tokens, getattr(r.usage.prompt_tokens_details, "cached_tokens", 0))

Next