claude-to-hindsight

Import a full Claude data export into a self-hosted Hindsight instance.

What it does

Parses every source in a Claude export directory and ingests it into a Hindsight memory bank:

Source	File	Treatment
Conversations	conversations.json	Chunked into N-turn windows, one retain per chunk
Memories	memories.json	conversations_memory (global summary) and project_memories (per-project), one retain each
Project docs	projects/<uuid>.json	One retain per attached document
Account info	users.json	Skipped (no memory value)

Each item is tagged with its source (source:conversations, source:memories, source:project_doc) and identifiers (conversation_id:…, project_id:…, filename:…) so you can hard-filter at recall time.

Requirements

• Python 3.11+
• A running Hindsight instance (local or remote)

python3 -m venv venv
source venv/bin/active
pip install hindsight-client

Getting your Claude export

1. Go to claude.ai → Settings → Privacy → Export Data
2. Wait for the email and download the ZIP
3. Extract it — you should get a directory containing conversations.json, memories.json, users.json, and a projects/ folder

Usage

python claude_to_hindsight.py \
  --export-dir ./claude-export \
  --base-url http://<hindsight-host>:8888 \
  --bank-id my-bank

The script is safe to interrupt and re-run — it resumes from where it stopped.

Options

Flag	Default	Description
--export-dir	(required)	Path to the extracted Claude export directory
--base-url	(required)	Hindsight API URL, e.g. http://localhost:8888
--bank-id	claude-history	Hindsight memory bank to ingest into
--checkpoint	hindsight_import_checkpoint.json	Checkpoint state file for resume
--delay	2.0	Seconds to wait after each retain call
--client-timeout	600	HTTP timeout (seconds) per retain call — increase if the LLM backend is slow on large chunks
--turns-per-call	4	User+assistant turns bundled per retain (lower = smaller payloads)
--max-chars	6000	Max characters per retain payload regardless of turn count — prevents large code or model pastes from stalling the LLM
--max-retries	10	Max retry attempts per item before skipping
--backoff-base	5.0	Initial retry backoff in seconds
--backoff-max	300.0	Maximum retry backoff cap in seconds
--health-poll	15.0	Seconds between server health checks when unreachable

Resilience

The script is designed to run unattended against a low-power server (e.g. a Raspberry Pi 4):

• Checkpoint/resume — state is saved atomically after every item. Kill and restart at any time; already-ingested items are skipped.
• Server health watch — on any connection error the script pauses and polls /health until the server comes back, then resumes automatically. No items are lost.
• Exponential backoff — transient errors (non-connection) are retried up to --max-retries times with doubling delay, capped at --backoff-max.
• Skipped items — items that exhaust all retries are recorded in the checkpoint under skipped. Remove their UIDs from that list and re-run to retry them.

Checkpoint file

The checkpoint is a plain JSON file (default to hindsight_import_checkpoint.json):

{
  "done": ["conversations:<uuid>:chunk0", "memories:3", ...],
  "skipped": [],
  "started_at": "2026-05-11T07:17:11+00:00",
  "updated_at": "2026-05-11T09:42:05+00:00"
}

To force a full re-import, delete the checkpoint file. To retry skipped items, remove their UIDs from the skipped list.

Tuning for low-power servers

Each synchronous retain call blocks until Hindsight finishes LLM fact extraction — on a Pi 4 this typically takes 10–30 seconds per item. A few knobs to adjust if the server is struggling:

• --turns-per-call 2 — halves the payload size per retain call
• --delay 0 — the sync retain is already the natural throttle; no artificial delay needed unless thermals are a concern
• On the Hindsight side, HINDSIGHT_API_RETAIN_CHUNK_SIZE controls how the server further splits content before LLM extraction

Notes

• retain is synchronous by default — the next call only starts after the previous one completes, so there is no risk of flooding the server regardless of delay setting.
• Project conversations (chats that happened inside a Claude project) appear in conversations.json, not in projects/. The project JSON files only contain the knowledge-base documents attached to the project.