LangChain with an OpenAI-Compatible API Gateway: Full Guide

LangChain is the default framework for a lot of Python LLM work. Agents, RAG pipelines and LangGraph workflows are all built on its chat model abstraction. That abstraction is also what makes switching providers easy: both the OpenAI and Anthropic integrations accept a custom base_url, so pointing LangChain at a gateway like Zivv is a change to how you construct the model, not to your application logic. This guide covers installation, ChatOpenAI, ChatAnthropic, streaming, tool calling, structured output, model switching, and the errors you are most likely to see.

Pick the Right Integration Package

LangChain ships each provider as its own package. Two of them matter here:

For most projects, ChatOpenAI is enough: through the OpenAI-compatible endpoint, one class reaches Claude, GPT and Gemini models. Use ChatAnthropic when you specifically want Claude on its native protocol, for example to work with Anthropic-style content blocks or thinking parameters. Both use the same Zivv key.

Install the packages:

pip install -U langchain langchain-openai langchain-anthropic

Keep the key in an environment variable instead of source code:

export ZIVV_API_KEY=sk-your-key-here

If you do not have a key yet, register and create one in the console. Check exact model IDs in Model Hub or by calling GET /v1/models.

ChatOpenAI: One Class for Every Model

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-5.5",
    base_url="https://zivv.pro/v1",
    api_key=os.environ["ZIVV_API_KEY"],
    temperature=0.3,
    timeout=120,
    max_retries=2,
)

resp = llm.invoke("Explain a vector database in one sentence.")
print(resp.content)

Change model to claude-sonnet-5 or gemini-3.5-flash and the same code calls a different vendor's model. That is the core benefit of an OpenAI-compatible gateway: one client, many models, no extra SDKs.

The base URL rule matches every other OpenAI-compatible client: stop at /v1. Do not use the bare domain and do not append /chat/completions.

Environment variables instead of arguments

If your codebase constructs ChatOpenAI in many places and you would rather not touch each one, the underlying OpenAI client reads these variables:

export OPENAI_API_KEY=sk-your-key-here
export OPENAI_BASE_URL=https://zivv.pro/v1

Passing base_url explicitly is still the better default for new code. It is obvious when reading the code, and you will not lose an afternoon to a stale variable in someone's shell profile.

ChatAnthropic: The Native Anthropic Protocol

import os
from langchain_anthropic import ChatAnthropic

claude = ChatAnthropic(
    model="claude-sonnet-5",
    base_url="https://zivv.pro",
    api_key=os.environ["ZIVV_API_KEY"],
    max_tokens=4096,
)

print(claude.invoke("List three common asyncio pitfalls.").content)

Two details matter. First, the native Anthropic base URL is https://zivv.pro with no /v1, because the SDK appends /v1/messages itself. Second, the Anthropic protocol requires max_tokens, so set a value that fits the outputs you expect. This mirrors how Claude Code is configured (ANTHROPIC_BASE_URL=https://zivv.pro), described in the Claude Code guide.

Streaming

Chat UIs and long generations need streaming. Every LangChain chat model supports stream:

for chunk in llm.stream("Write a 150-word product description."):
    print(chunk.content, end="", flush=True)

Use astream in async code; the pattern is identical. If you want token counts while streaming, for your own cost tracking, ChatOpenAI accepts stream_usage=True, which attaches usage_metadata to the final chunk. The token cost guide explains how to turn those numbers into spend.

Tool Calling

Tools are the foundation of agents. bind_tools hands function schemas to the model, and the code is the same for ChatOpenAI and ChatAnthropic:

from langchain_core.tools import tool

@tool
def get_weather(city: str) -> str:
    """Return the current weather for a city."""
    return f"{city}: sunny, 22°C"

llm_with_tools = llm.bind_tools([get_weather])
msg = llm_with_tools.invoke("What's the weather in Berlin?")

for call in msg.tool_calls:
    print(call["name"], call["args"])

tool_calls is normalized by LangChain, so it looks the same regardless of vendor. Run the tool, append a ToolMessage with the result, and invoke again to complete the loop. LangGraph's prebuilt agents automate that loop for you.

Tool use rewards capable models. For agents that juggle several tools, start with claude-sonnet-5 or gpt-5.5. Single-tool lookups run fine on mid-tier models.

Structured Output

with_structured_output returns a Pydantic object, which is ideal for extraction:

from pydantic import BaseModel

class Ticket(BaseModel):
    category: str
    priority: int
    summary: str

extractor = llm.with_structured_output(Ticket, method="function_calling")
print(extractor.invoke("Customer says the checkout page spins forever. Urgent."))

Note the explicit method="function_calling". Recent versions of langchain-openai default to a JSON Schema mode for some models, and when you reach non-OpenAI models through a compatible endpoint, not every model supports that mode. Function calling is the most portable way to get structured output. If structured output errors out, change this argument first.

Switching Between Models

A gateway lets you assign models per task instead of per vendor account. A small factory keeps it tidy:

MODELS = {
    "fast": "gemini-3.5-flash",
    "default": "claude-sonnet-5",
    "hard": "claude-opus-4-8",
}

def get_llm(tier: str = "default", **kwargs) -> ChatOpenAI:
    return ChatOpenAI(
        model=MODELS[tier],
        base_url="https://zivv.pro/v1",
        api_key=os.environ["ZIVV_API_KEY"],
        **kwargs,
    )

classifier = get_llm("fast", temperature=0)
writer = get_llm("default")

LangChain also offers init_chat_model for building models from strings, and with_fallbacks for automatic failover. For instance, run claude-sonnet-5 as primary and fall back to gpt-5.5 on error. Because both share one base URL and key, the fallback needs no extra account setup. The multi-model routing guide goes deeper on choosing tiers.

Embeddings Need a Separate Provider

RAG projects usually create vectors with OpenAIEmbeddings. Zivv does not offer an embeddings endpoint, so do not point OpenAIEmbeddings at Zivv; those calls will fail. The working split is:

• Chat models for reasoning, generation and tool calls go through Zivv
• Embeddings come from another provider that offers them, or from a locally hosted open-source embedding model

In LangChain, embeddings and chat models are separate objects anyway, so you configure each one on its own.

Troubleshooting

When something breaks, test the endpoint directly with curl. It separates LangChain configuration problems from key or model problems:

curl https://zivv.pro/v1/chat/completions \
  -H "Authorization: Bearer sk-your-key-here" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-5.5", "messages": [{"role": "user", "content": "Reply OK"}]}'

FAQ

Can LangChain's ChatOpenAI call Claude models? Yes. Through Zivv's OpenAI-compatible endpoint at https://zivv.pro/v1, set model to a Claude ID such as claude-sonnet-5 and ChatOpenAI works as usual.

Does the ChatAnthropic base_url need /v1? No. Use https://zivv.pro. The SDK appends /v1/messages on its own, and adding /v1 yourself causes 404 errors.

Do streaming and tool calling work through a gateway? Yes. stream, astream and bind_tools behave exactly as they do against the official APIs.

Can I use OpenAIEmbeddings with Zivv? No. Zivv does not provide an embeddings endpoint. Use another provider or a local model for embeddings, and keep chat models on Zivv.

How do I connect a LangGraph project? LangGraph nodes use LangChain chat model objects, so initialize them as shown above. No extra configuration is required.

Does the gateway log my prompts? Zivv does not store prompt or response bodies; it records token usage for billing only. See what an AI API gateway is for how requests flow.

Wrap-Up

Remember two URLs: https://zivv.pro/v1 for ChatOpenAI and https://zivv.pro for ChatAnthropic. Everything else, from streaming and tools to structured output and fallbacks, is written exactly as it would be against the official APIs. Migrating an existing app? Follow the OpenAI-compatible migration checklist and the OpenAI SDK guide. Then create a key and run your LangChain app on a single key for every model.