Migrating to the 2.0 / 2.1 contracts

The working tree implements the major 2.0 contract transition and the additive 2.1 code-quality work. Follow the published release notes when selecting a package version; this guide does not announce publication. The contracts reference defines the resulting behavior, and the roadmap separates implementation from release gates and later work.

Application changes

Earlier behavior	Required migration
Default token sizing could use estimates.	Default sizing is now characters. Specify `size_unit="tokens", tokenizer="cl100k_base"` for that model tokenizer, or a counting callable. `"approximate"` explicitly selects estimates.
Context and overlap could exceed the requested size.	The limit applies to `contextualized_text`. Default splitting may change boundaries; `context_overflow="omit"` records omitted context while preserving metadata. Use `overflow_policy="preserve"` only when the consumer accepts reported overflow.
Unknown options were silently ignored.	Correct misspellings and remove unsupported options. Configuration is validated and `ChunkOptions` is frozen; use `dataclasses.replace()` to derive options.
Some flags appeared configurable without changing output.	Remove `semantic_threshold_k` and `semantic_window_size`. The supported preservation flags accept their existing `True` behavior only.
Whitespace handling varied between engines.	Use default `coverage_policy="retrieval"` for retrieval, or `"lossless"` plus no numeric overlap for complete reconstruction.
IDs depended on positions.	Rebuild vector identities. New IDs use source identity, raw content and duplicate occurrence; ID mappings alone do not validate old embeddings.
Loader offsets could be local to a segment.	Chunk, entity and scope byte ranges now address canonical loader text. Keep that text or regenerate it with the same loader configuration.
Parser and NumPy packages came with the base install.	Select `omnichunk[code]`, `omnichunk[semantic]`, tokenizer and format extras for the features used. `omnichunk[all-languages]` includes primary and additional grammars.
Global plugin registration affected shared state.	Prefer an explicit `PluginRegistry`; give persisted custom parsers a stable registry `revision`. Global registrations are snapshotted at `Chunker` construction.
Tool serving was named MCP.	Use `omnichunk serve --rpc --allowed-root ./project`. The deprecated `--mcp` alias serves custom JSON-RPC, not MCP lifecycle/tool discovery.

Pin the package, grammar/tokenizer versions, configuration and embedding model for each index generation. Store SourceDescriptor, config_fingerprint, occurrence and full versioned chunk JSON. A moved file needs an explicit stable source_id if its logical identity should remain unchanged. Do not assign one source ID to every file in a multi-file corpus.

Text index: copy a legacy store, then reprocess sources

Keep the old SQLite file, old environment and old vector collection available. Stop the old writer while taking its database backup. The new reader deliberately raises LegacyStoreError for legacy databases rather than changing them in place. migrate_legacy_store() reads its source without modifying it and requires a different, nonexistent destination.

This example assumes index-v1.sqlite contains the current articles corpus. It changes the embedding payload to bounded character chunks and validates UTF-8 source reconstruction before any application cutover:

from pathlib import Path

from omnichunk import Chunker
from omnichunk.serialization import chunk_to_dict
from omnichunk.store import ChunkStore, migrate_legacy_store

root = Path("articles").resolve()
migration = migrate_legacy_store(
    "index-v1.sqlite", "index-v2.sqlite", collection="articles"
)
assert migration.reindex_required
chunker = Chunker(
    max_chunk_size=512,
    context_mode="none",
    coverage_policy="lossless",
)
with ChunkStore(migration.destination, collection="articles") as store:
    result = store.sync(root, glob="**/*.txt", chunker=chunker)
    for path in sorted(root.rglob("*.txt")):
        chunks = store.query(str(path))
        raw = path.read_bytes()  # This example assumes UTF-8 source files.
        assert b"".join(chunk.text.encode("utf-8") for chunk in chunks) == raw
        assert all(len(chunk.contextualized_text) <= 512 for chunk in chunks)
        records = [chunk_to_dict(chunk) for chunk in chunks]
        assert all(record["schema_version"] == 2 for record in records)

The migration returns files_copied, chunks_copied and an old-to-new id_map. Copied records retain available metadata and are marked legacy-unknown; missing historical metadata cannot be recovered. Their empty configuration fingerprint forces reprocessing on sync. Records whose original files no longer exist remain recoverable because migration does not invent scan ownership. If the new index must contain only the current corpus, build a separate empty store with index() instead of using the converted recovery copy as the production index.

Build embeddings into a new vector collection from the reprocessed contextualized_text, then validate your retrieval fixtures and atomically switch the application's configured SQLite path and vector collection together. Rollback means selecting the retained old environment, SQLite file and vector collection; the old runtime should never open the new database. Do not overwrite the old collection while the new index is being evaluated.

Notebook index: rebuild and retain canonical source

For notebooks, rebuilding from containers is preferable to treating legacy JSON offsets as valid citations. This example creates a new store; leave the previous notebook store and vector collection in place. include_notebook_outputs must match the explicit loader used for reconstruction:

import json
from pathlib import Path

from omnichunk import Chunker, load_ipynb
from omnichunk.serialization import chunk_to_dict
from omnichunk.store import ChunkStore

root = Path("notebooks").resolve()
snapshot_dir = Path("notebook-canonical-v2")
snapshot_dir.mkdir(exist_ok=True)
chunker = Chunker(
    max_chunk_size=512,
    context_mode="none",
    coverage_policy="lossless",
    include_notebook_outputs=False,
)
with ChunkStore("notebooks-v2.sqlite", collection="notebooks") as store:
    result = store.index(root, glob="**/*.ipynb", chunker=chunker)
    for number, path in enumerate(sorted(root.rglob("*.ipynb"))):
        loaded = load_ipynb(path.read_text(encoding="utf-8"), include_outputs=False)
        chunks = store.query(str(path))
        canonical = loaded.text.encode("utf-8")
        assert b"".join(chunk.text.encode("utf-8") for chunk in chunks) == canonical
        for chunk in chunks:
            span = chunk.byte_range
            assert canonical[span.start:span.end].decode("utf-8") == chunk.text
            assert chunk.source is not None and chunk.source.format_name == "ipynb"
        snapshot = {
            "filepath": str(path),
            "canonical_text": loaded.text,
            "loader_warnings": list(loaded.warnings),
            "chunks": [chunk_to_dict(chunk) for chunk in chunks],
        }
        (snapshot_dir / f"{number}.json").write_text(
            json.dumps(snapshot, ensure_ascii=False), encoding="utf-8"
        )

The snapshot is canonical text plus versioned chunks, not a replacement notebook. Cell metadata remains in context.format_metadata; byte offsets address the snapshot text. PDF and DOCX use the same distinction between container and extracted text. Preserve the original container as well when citations must link to cells, pages or document elements. Malformed notebooks fail chunking; review nonfatal loader warnings before admitting documents to the new index.

Build a new notebook vector collection from these chunks and switch it together with notebooks-v2.sqlite only after citation and retrieval checks pass. Rollback selects the retained old notebook runtime/store/vector collection. Canonical snapshots allow the new output to be inspected without relying on a later loader version to reproduce the same extraction.

Ongoing synchronization

After cutover, sync() skips files only when content and processing configuration match. A complete scan commits atomically; failures leave the previous generation intact. Use consistent roots and filters: deletion belongs to that completed scan scope, so changing filters does not authorize deleting another scope's documents.

For custom providers, set a stable callback fingerprint, or supply both semantic_cache_namespace and semantic_model_revision. Otherwise persistent skips are disabled to avoid stale results. Change identities when code, model or preprocessing changes. New diff results include updated as a subset of added; apply removed_ids before upserts, and refresh location metadata for unchanged payloads without repeating their embeddings.