Tool calling & structured output

Multi-turn tool loops and schema-validated JSON output, OpenAI wire shape.

Malibu ships both features as OpenAI-compatible wire surfaces on /v1/chat/completions, so any framework built on the OpenAI SDK can use them without changes. The examples use client from Using Malibu with the OpenAI SDK.

Tool calling

Supported: multi-turn tool loops, streamed tool-call deltas, tool_call_id back-references, and role: "tool" messages. Anchor framework: Cline, pointed at the Malibu base URL with a Qwen-family model.

import json

resp = client.chat.completions.create(
    model="mlx-community/Qwen3.6-27B-4bit",
    messages=[{"role": "user", "content": "What's the weather in Vilnius?"}],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
)

for call in resp.choices[0].message.tool_calls or []:
    print(call.function.name, json.loads(call.function.arguments))  # arguments is a JSON string

Which models emit tool_calls[]:

Qwen family (Qwen 2.5, Qwen 3, Qwen3-Coder, and later Qwen releases in the catalog). Full multi-turn support.
gpt-oss (gpt-oss-20b, gpt-oss-120b). First-turn tool calls only. Sending tool results back returns 400 unsupported_modelID_for_multi_turn.
Other models (Llama 3.1 / 3.2, Gemma 4, Nemotron, GLM). They accept tools but reply with plain assistant text.

tool_choice supports "auto". "required" and named-function choices are accepted but handled as "auto", and "none" returns 400. Full model list, multi-turn example, streaming reassembly, and caps: Agentic tooling → Tool calling.

Buyer-side validation obligation

Emitted tool_calls[] reflect model output, not provider-verified intent. Buyer-side agent frameworks MUST validate before execution. Malibu delivers the parsed OpenAI-compatible shape. It does not decide whether a requested tool name or argument payload is safe for your agent policy. Treat emitted tool calls with the same trust posture you would apply to a model running on local hardware. See Agentic tooling → Buyer-side validation.

Structured output

Send response_format with json_schema or json_object. The provider steers the model with your schema, then validates the output before it returns success. If the output doesn’t validate, you get a 502 (or a terminal SSE error when streaming) instead of invalid JSON. This is post-inference validation, not constrained decoding. It works streaming and non-streaming, on every catalog model.

resp = client.chat.completions.create(
    model="mlx-community/Qwen3.6-27B-4bit",
    messages=[
        {"role": "system", "content": "Extract contact info as JSON."},
        {"role": "user", "content": "Call John at 555-0100, john@example.com"},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "contact",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "phone": {"type": "string"},
                    "email": {"type": "string"},
                },
                "required": ["name", "phone", "email"],
                "additionalProperties": False,
            },
        },
    },
)

contact = json.loads(resp.choices[0].message.content)

Supported subset: a strict subset of JSON Schema. type is required on every node, every object needs additionalProperties: false, and every property must be listed in required. $ref, anyOf / oneOf, pattern, format, and length constraints are rejected with 400 before inference runs. response_format: {"type": "json_object"} is enforced too. Output that isn’t JSON with an object or array at the top level returns 502 malformed_json_response. Full reference: Agentic tooling → Structured output.

Litepaper

Getting started

Buyers

Providers

Network & trust

Guides index

Ledger

Tool calling & structured output

Tool calling & structured output

Tool calling

Buyer-side validation obligation

Structured output

​Tool calling & structured output

​Tool calling

​Buyer-side validation obligation

​Structured output

Tool calling & structured output

Tool calling

Buyer-side validation obligation

Structured output