Prompt cache and threads

Use the same thread_id for related requests in the same user conversation and model to improve prompt-cache reuse when available.

{
  "model": "google/gemma-4-26B-A4B-it",
  "thread_id": "session-123",
  "messages": [{ "role": "user", "content": "Continue from our previous context." }]
}

Cache observability is returned under overshoot.cache:

{
  "overshoot": {
    "cache": {
      "thread_id": "session-123",
      "cache_hit": true,
      "cached_input_tokens": 5000
    }
  }
}

thread_id is the ID you supplied, or null when none was provided.
cache_hit is true when cached prompt tokens were reported.
cached_input_tokens is the number of prompt tokens served from prefix cache.

Cache metadata is observability only. It is not a guarantee that future requests will receive a cache hit.

Region behavior Error shapes

⌘I

Chat

Streams

Models & pricing

Prompt cache and threads