Skip to main content

LangGraph to Data Warehouse

To understand how to analyze your LangGraph data, it helps to first look at exactly how LangGraph writes that data to Postgres, and then look at the architectural patterns for extracting it.

How LangGraph Stores Data in Postgres​

LangGraph handles persistence using a "checkpointer" (specifically, the PostgresSaver or AsyncPostgresSaver from the langgraph-checkpoint-postgres library). When your graph executes, it runs in "super-steps." At the end of every super-step, LangGraph takes a snapshot of the graph's state and serializes it to the database.

When you initialize the Postgres checkpointer, it automatically provisions four specific tables to manage this state:

  • checkpoints: This is the core table. It stores the actual snapshot of your graph state in a JSONB column. It tracks primitive values (strings, integers), channel versions, and metadata. It also tracks the parent_checkpoint_id, which strings the execution history together and allows LangGraph to "time travel" or resume conversations.
  • checkpoint_blobs: To optimize performance, LangGraph doesn't stuff massive arrays into the main table. Complex, non-primitive data (like long message histories, heavy tool outputs, or massive context windows) are stored separately as versioned blobs here.
  • checkpoint_writes: This handles fault tolerance. If a multi-agent step fails midway, the outputs of the successful nodes are temporarily saved here as "pending writes" so LangGraph doesn't have to re-run expensive LLM calls when it resumes.
  • checkpoint_migrations: An internal version-control table that tracks the schema versions of the checkpointer itself.

The Analytical Challenge: LangGraph's operational database is highly optimized for state resumption (resuming a paused workflow), not analytics. Your valuable data—like which tools were called, how many tokens were used, or what the user asked—is buried inside deeply nested JSONB structures and split across the checkpoints and checkpoint_blobs tables.

Bringing LangGraph Data to a Data Warehouse (DWH)​

To run analytics on this data without dragging down your production Postgres database (which can suffer from severe I/O bottlenecks if queried heavily), you need to move it to a Data Warehouse like BigQuery, Snowflake, or Redshift. Here is the standard playbook for doing so:

1. Ingestion: Extracting the Data (EL)​

Because LangGraph checkpoints update constantly as agents loop, you should replicate the data using either batch or streaming ingestion:

  • Change Data Capture (CDC): This is the best approach for near real-time analytics. Tools like Debezium, Fivetran, or Airbyte read the Postgres Write-Ahead Log (WAL) and stream every insert/update from the checkpoints and checkpoint_blobs tables directly into your DWH.

  • Batch Pipelines: If you only need daily reports, you can run a nightly batch job via Airbyte, dbt, or Apache Airflow to copy newly created checkpoints over to the DWH.

2. Transformation: Unnesting the JSON (T)​

Once the raw tables are in your DWH, you must transform the raw JSONB logs into clean, queryable analytical tables. Using a tool like dbt (data build tool) or your warehouse's native JSON parsing SQL functions, you will need to parse the JSON to build a dimensional model:

  • dim_conversations (from thread_id): Group by the unique thread ID to create a table of distinct user sessions. Extract the first checkpoint's timestamp as the start time, and the final checkpoint as the end time.

  • fct_agent_steps (from checkpoints): Every row in the checkpoints table is a "turn" or "super-step." You can extract the node_name from the JSON metadata to track which agent/node executed, how long it took, and whether it succeeded.

  • fct_tool_calls (from checkpoint_blobs): Parse the complex blob data to extract the exact names of tools your agents invoked, the arguments passed to them, and the resulting outputs.

3. Analyzing the Data​

Once your pipeline is parsing the LangGraph state into relational tables, you can connect your DWH to a BI tool (like Looker, Tableau, or Metabase) to monitor your multi-agent system.

Key metrics you can now track include:

  • Agent Loop Counts: How many super-steps does it take an agent to resolve an average query?

  • Tool Failure Rates: Which specific tools are returning errors or forcing the graph to retry?

  • Token Optimization: By parsing the LLM response metadata stored in the checkpoints, you can aggregate token costs per thread_id to see exactly how much specific user workflows are costing you in API calls.