​

Sticky conversations

Reuse a provider’s prompt cache across turns, automatically or with a conversation tag that can earn the cache-hit rate.

​

Automatic prefix cache (no header)

​

Sticky conversation tag (cache-hit rate)

from openai import OpenAI

conversation_id = "thread-42"  # stable per conversation; [A-Za-z0-9._:-], max 128 chars

client_sticky = OpenAI(
    base_url="https://api.malibu.tech/v1",
    api_key="<your-malibu-api-key>",
    default_headers={"X-MacProvider-Conversation": conversation_id},
)

# Turn 1: cold prefix
resp1 = client_sticky.chat.completions.create(
    model="mlx-community/Llama-3.2-3B-Instruct-4bit",
    messages=[
        {"role": "system", "content": "You are a code review assistant."},
        {"role": "user", "content": "Explain the SOLID principles."},
    ],
)

# Turn 2: same tag; if the sticky provider serves it, the shared prefix is reused
resp2 = client_sticky.chat.completions.create(
    model="mlx-community/Llama-3.2-3B-Instruct-4bit",
    messages=[
        {"role": "system", "content": "You are a code review assistant."},
        {"role": "user", "content": "Explain the SOLID principles."},
        {"role": "assistant", "content": resp1.choices[0].message.content},
        {"role": "user", "content": "Which one is most often violated in practice?"},
    ],
)

u = resp2.usage
print(u.prompt_tokens, u.prompt_tokens_details.cached_tokens, getattr(u, "cached_prompt_tokens", 0))

​

How it works

• The tag is opaque and buyer-chosen. Use a stable per-thread identifier such as a UUID or your app’s thread ID. An empty header is ignored. The gateway rejects whitespace-only tags, tags longer than 128 bytes, and characters outside [A-Za-z0-9._:-] with 400 invalid_conversation_tag.
• The gateway HMACs the tag with your account_id and forwards only the derived key. One buyer’s thread-42 never collides with another buyer’s thread-42, and providers never see the raw tag.
• Stickiness lasts up to the TTL that /v1/models reports under tier1_disclosure.sticky_affinity.ttl_seconds (1,800 seconds on 23 Sep 2026).
• Stickiness is a preference, not a guarantee. If the sticky provider is not the top-ranked choice, or is disconnected, busy, or draining, the request goes through normal routing and cached_prompt_tokens is 0. The cache-hit discount applies only on a sticky hit.
• cached_prompt_tokens is always present, so you can read it without checking first.

​

Privacy trade-off

curl -s -X DELETE -H "Authorization: Bearer $MALIBU_API_KEY" https://api.malibu.tech/v1/sticky
# {"purged": true, "entries": 1}

​

Billing

​

When to use a tag

• Chat UIs. Use one tag per thread so repeated system prompts and history get cached.
• Agent loops. Use one tag per task. Many tool-call turns share one long prefix.
• Long-context RAG. Retrieved documents in the system prompt stay warm across follow-ups.