# Qdrant ES Gateway

> Elasticsearch-compatible application search backed by Qdrant. Compatibility is intentionally scoped.

## Start here


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/compatibility.md

# compatibility

# Compatibility and semantic differences

Supported requests cover CRUD, mappings, bulk ingestion, common index lifecycle calls, multi-search, cursor pagination, basic lexical search, payload filters, source projection, count, complete filter-only aggregations, health, and durable aliases. Request bodies must be valid UTF-8; malformed byte sequences are rejected before JSON or NDJSON parsing instead of being stored as Unicode replacement characters. Index creation enforces the static name rules shared by Elasticsearch 9.5.5 and OpenSearch 3.9: lowercase UTF-8 names of at most 255 bytes, excluding their reserved punctuation, the leading `_`/`-`/`+` forms, and the exact names `.` and `..`. Invalid names return HTTP 400 with `invalid_index_name_exception` before catalogue or Qdrant work. Dot-prefixed names other than `.` and `..` remain accepted; the gateway does not emulate engine-specific deprecation warnings, hidden-index settings, or system-index ownership. Creating an index that already exists returns HTTP 400 with `resource_already_exists_exception` before Qdrant work; it does not replace the stored mapping. Names that collide only after Qdrant collection normalization remain a distinct gateway validation error. `PUT /<index>/_mapping` merges new non-text properties into the existing mapping and creates Qdrant payload indexes for supported filterable types; identical existing definitions are idempotent, while type changes and new text fields are rejected. New text fields require a new index because the pinned Qdrant 1.15 schema cannot add their named sparse vectors to an existing collection. Both `POST` and `PUT` bulk requests are accepted. Bulk `update` and `delete` actions require an explicit `_id`; every action line must name exactly one supported action with object metadata and string `_index`/`_id` values. Malformed actions fail before any item is executed instead of assigning an unrelated generated ID or partially applying the request. Standalone and bulk `index` operations check the authoritative document store so a new document returns HTTP 201 with `result: "created"`, while replacing an existing document returns HTTP 200 with `result: "updated"`. Qdrant 1.15.5 reports an absent single point as HTTP 404 rather than a successful null result; that specific response is treated as a missing document, while a missing collection remains an upstream 502. Document HEAD requests preserve the same distinction: an absent point is 404, but transport and storage failures are 502 rather than false “not found” results. HEAD responses are bodyless and include the Elasticsearch product header. Contiguous bulk index writes remain one Qdrant upsert batch and use one batched existence lookup rather than a read per item. Successful bulk `create`, `update`, and `delete` items preserve the operation's result metadata and report the concrete backing index when addressed through an alias. Standalone and bulk `create` operations first check for an existing document and return a 409 conflict rather than overwriting it; they are serialized within one gateway process but are not an atomic cross-replica compare-and-insert. Index existence checks likewise describe the preflight state rather than providing atomic compare-and-write semantics. The compatibility analyzer at `GET /_qdrant_gateway/compatibility` is local-only and reports request counts; no telemetry leaves the process.

Search requests with `size: 0` are accepted for aggregation-only use and return no hits; the gateway uses a one-item internal scroll because Qdrant rejects a zero scroll limit. `from` and `size` must be non-negative integers, `size` cannot exceed `MAX_PAGE_SIZE`, and their combined result window cannot exceed 10,000. Cursor requests require an array with one scalar value per explicit sort field and cannot combine `search_after` with a non-zero `from`. Invalid pagination fails before index lookup or Qdrant access rather than silently using defaults or truncating the page. Native filter-only searches issue an exact Qdrant count alongside the bounded hit retrieval, so `hits.total` is not truncated to `from + size`. Ranked text searches derive lower-bound totals from their candidate window. Filter-only post-filter searches and complete gateway scans compute totals over all matching pages; concurrent writes can change what a scan observes. Boolean and non-negative integer `track_total_hits` modes follow the OpenSearch/Elasticsearch response shape: `false` omits `hits.total` and skips the otherwise eligible exact-count request, while a numeric threshold caps the reported value and changes the relation to `gte` when the threshold is exceeded. Numeric thresholds currently shape the response but do not terminate Qdrant's exact count early. The `_search` and `_msearch` endpoints also honor OpenSearch's `rest_total_hits_as_int=true` query parameter: ordinary totals become integers and disabled totals use the native `-1` sentinel. Lowercase `true` and `false` are accepted; as in OpenSearch 3.9.0, an empty or valueless parameter enables the mode. Other values and the native-incompatible combination of integer formatting with a numeric tracking threshold fail before Qdrant work. Integer formatting cannot express the gateway's usual `eq`/`gte` accuracy relation for bounded text searches. Strict `track_total_hits` parsing intentionally excludes OpenSearch's undocumented coercion of strings, fractional numbers, and negative values. The extra count can add an upstream round trip to filter-only search latency. Supported filter-only aggregations scan every matching Qdrant page independently of the returned hit count. Ranked text and pattern queries with aggregations fail explicitly.

Search `_source` filtering accepts booleans, a single field pattern, an array of patterns, or an `include`/`includes` and `exclude`/`excludes` object. Dotted paths and `*`/`?` wildcards preserve the nested object shape rather than flattening selected fields, and the same parser is used for collapse inner hits. Invalid types, mixed-type arrays, and unknown options fail before index lookup or Qdrant work. Filtering shapes the response after the gateway has hydrated each candidate's complete source; it reduces returned data but does not reduce source-read bandwidth or memory use. Non-empty body `docvalue_fields` requests and the URL query parameter are rejected before index lookup or Qdrant work because the gateway cannot return native column-store values in each hit's `fields` object; an empty body array remains an accepted no-op. Non-empty `script_fields` objects are rejected at the same boundary because executing no script and omitting its computed per-hit `fields` value would violate the response contract; an empty object remains a no-op.

Partial `doc` updates use read-merge-write semantics. Inner objects are merged recursively; arrays and scalar values replace the existing value. Identical merged documents return `result: "noop"` without an upstream write by default; `detect_noop: false` forces the write. A missing document can be created explicitly with either `doc_as_upsert: true` or a separate `upsert` object; the separate form writes the `upsert` source rather than the partial `doc`, while an existing document still receives `doc`. Both standalone and bulk update paths share this behavior. In the default embedded-source mode, non-text updates refresh the complete stored `_source` and all mapped filter/sort payload fields in one payload write; this preserves untouched source fields and keeps filtering consistent with document reads. This path transfers the merged source even when the patch is small. Updates are not an atomic compare-and-write, Elasticsearch `nested` query semantics remain unsupported, scripts remain unsupported, and version counters remain approximate.

Aliases currently map one alias to one concrete index. CRUD, search, count, mapping, and multi-search requests resolve that concrete index before selecting a Qdrant collection or deriving a deterministic point ID. Alias inspection is Elasticsearch-shaped and keyed by the concrete index. Alias action batches are fully validated before a single SQLite transaction applies them, so a malformed later action cannot leave an earlier routing change behind. Missing or empty action lists return the HTTP 400 `illegal_argument_exception` shape independently observed from Elasticsearch 9.5.5 and OpenSearch 3.9.0; the gateway no longer acknowledges an empty catalogue transaction. Both engines also preserve the original target when an `add` action attaches the same alias to a second index. Because the gateway cannot represent that multi-index state, it rejects such a final topology before changing the catalogue instead of silently replacing the original target. An atomic batch that removes the old target and adds the new one remains supported in either action order. A batch in which every supported removal misses returns HTTP 404 with `aliases_not_found_exception`, including when the alias exists on a different index. A mixed batch still applies successful actions and returns HTTP 200. Elasticsearch 9.5.5 adds `errors` and per-action results for that partial failure, while OpenSearch 3.9.0 returns only `acknowledged`; the gateway currently uses the simpler OpenSearch-shaped success body and does not expose per-action alias results. Native alias filters, routing fields, write-index/hidden flags, remove `must_exist`, and unknown action options are rejected before catalogue access instead of being acknowledged and discarded. Multi-index aliases, filtered aliases, write-index selection, routing, hidden aliases, and Qdrant-native alias switching remain unsupported.

Qdrant's server-side BM25 uses sparse vectors and its own tokenization/model implementation. Results should be evaluated by relevant document IDs and product metrics, not numerical `_score` equality with Elasticsearch. `match_phrase` currently uses the same native lexical representation and is therefore not a Lucene positional phrase guarantee.

Filter-only `should` clauses and numeric or percentage `minimum_should_match` are supported when they can be represented as Qdrant filter alternatives. Scalar and array forms of `must_not` are accepted, and nested filter-only bools preserve their clause boundaries when negated or used as alternatives. As in Elasticsearch, a `bool` containing `must` or `filter` defaults `minimum_should_match` to zero; without either required clause, at least one `should` remains required. Optional filter-only clauses preserve the matching document set but do not boost Qdrant scores. `dis_max`, `match_bool_prefix`, and fuzzy-shaped requests are accepted through the gateway's lexical field path; their ranking and typo/prefix behavior are intentionally approximate because Qdrant's sparse model is not Lucene. Basic response sorting accepts a field name, a direct order such as `{"price":"asc"}`, or the standard options object `{"price":{"order":"desc"}}`; `_score` defaults to descending and other fields default to ascending. Unsupported sort options and invalid directions are rejected instead of silently changing the order. Filter-only field sorting and `search_after` visit every matching page and retain the best `from + size` hits, so point order cannot hide a later winner. Ranked text searches still sort a bounded candidate window. Cursors do not freeze the data: use stable data and a unique explicit tie-breaker for repeatable pages. Per-search `min_score`, `terminate_after`, and `timeout` controls are rejected before index lookup or Qdrant work. They change hit membership, totals, early-termination metadata, or partial-result behavior, and Elasticsearch 9.5.5 and OpenSearch 3.9.0 do not report identical totals for the same terminated search. Search `highlight` requests are also rejected before backend work: Elasticsearch 8.15.0 and OpenSearch 3.9.0 both return per-hit snippets for the same basic request, while the gateway cannot reproduce analyzer-aware offsets or fragment selection and must not silently omit them. `explain: true` now fails at the same boundary instead of omitting each hit's requested `_explanation`; `explain: false` remains a no-op. Elasticsearch 8.15.0 and OpenSearch 3.9.0 both returned a nested per-hit score explanation in the focused native probe, but the recorded values and component trees differed, reinforcing that Qdrant scores cannot be labeled as Lucene internals. `profile: true` is likewise rejected before backend work instead of omitting the requested query and collector timing trees; explicit `false` remains a no-op. The focused native probe found both engines returned profiling data, but their observed collector names and response sizes differed, so the gateway does not manufacture a shared Lucene-shaped profile for Qdrant execution. Search `version: true` and `seq_no_primary_term: true` are also rejected before backend work; returning the gateway's approximate write metadata would incorrectly present it as a native concurrency token. Explicit `false` remains a no-op. A focused Elasticsearch 8.15.0 and OpenSearch 3.9.0 probe observed `_version: 2`, `_seq_no: 1`, and `_primary_term: 1` on each native hit after one update. The gateway requires actual JSON booleans for these controls rather than adopting the engines' observed string coercion. Scripts, scroll, and analyzer settings that would materially change semantics remain unsupported. Every rejection identifies the request feature and a safe alternative where one exists.

Field collapse accepts one field and one standard `inner_hits` definition with `name`, `size`, `_source`, and `_id` in `fields`. The response uses the requested inner-hit name, caps its size at `MAX_PAGE_SIZE`, and applies its source projection independently of the top-level projection. Unsupported collapse options, multiple inner-hit definitions, and inner-hit sorting are rejected before Qdrant work. Collapse follows the candidate scope of its query; complete filter scans retain all group members, so collapse memory is proportional to matching documents. Collapse with `search_after` is rejected. Ranked text collapse remains limited to retrieved candidates.

`exists` queries over mapped payload fields exclude missing values, JSON `null`, empty arrays, and arrays containing only `null`; empty strings and arrays with at least one non-null value count as present. This matches the ordinary Elasticsearch boundary, but not mapping-dependent cases such as `index: false`, `ignore_above`, or malformed values ignored by an analyzer. Server-side `exists` filtering is limited to fields mirrored into the searchable Qdrant payload; `post_filter` evaluates the hydrated `_source`.

For application mappings that expose a conventional `.keyword` multi-field name, exact filters, range and exists predicates, gateway-side patterns, sorting, collapse, and terms facets resolve that suffix to the base payload/source field. This makes client-generated requests such as `term: {"brand.keyword":"Acme"}` consistent across native and hydrated-source paths. It does not create a second analyzed representation or emulate custom multi-field analyzers: the base field must still be suitable for the requested operation.
# Pattern queries

`regexp`, `prefix`, and `wildcard` clauses are evaluated by the gateway using Rust's regular-expression engine after candidate documents are read from Qdrant. Positive pattern clauses can be combined with ordinary Qdrant-translated filters in `bool.must` or `bool.filter`; pattern clauses in `should` and `must_not` are rejected rather than approximated. Pattern-only search and count requests scan the collection in bounded scroll pages, so their totals are exact for the stable collection snapshot seen by the scan, but they are useful only for modest catalog and identifier searches and should not be used as a substitute for a dedicated indexed pattern field at very large scale. Wildcards preserve the position of Elasticsearch/OpenSearch-style `*` and `?` operators, including leading, trailing, and consecutive stars. Gateway-side matching uses stored source strings, so prefix and wildcard case behavior on analyzed `text` fields can differ from a native engine's normalized term dictionary; use a normalized keyword field when case behavior is part of the contract. Regular expressions use Rust/RE2-style syntax and reject invalid expressions with a structured 400 response.

Search responses also accept `post_filter` for simple term, terms, numeric range, exists, match-all, and bool clauses. The complete `post_filter` tree is validated before index lookup or Qdrant work; unsupported query types, unknown bool options, ambiguous multi-field clauses, and malformed range options return a structured 400 instead of silently producing an empty result set. The filter is applied to hydrated hits after the main query. Min, max, terms, filter, keyed filters, and supported calendar histograms reduce every query-matched page before hit-only post-filtering, collapse, or cursor selection. Scalar and array-valued source fields use the same any-element matching rule for term, terms, range, prefix, wildcard, and regexp predicates, keeping gateway-side evaluation aligned with native Qdrant filters. Arrays of objects still require Elasticsearch `nested` semantics and are unsupported. Aggregation memory uses constant metric state and at most 10,000 distinct buckets per aggregation, with at most 100 aggregations per request. Exceeding a bound fails instead of returning partial results. Calendar histograms accept day, week, month, and year intervals, IANA timezones, and explicit `min_doc_count: 1`; unsupported formats, bounds, offsets, and empty-bucket filling fail. Timezone rules come from the compiled chrono-tz dependency. Scans are not point-in-time snapshots. In document-projection mode, aggregation fields must be supported mapped payload fields; reductions use search-projection values. See the [comparison fixture](https://github.com/dstockton/qdrant-es-gateway/tree/main/validation/aggregations).

### Explicit sparse embeddings

New indices can map a top-level field as `sparse_vector`. Optional
`method: {name: seismic, engine: lucene}` is accepted for OpenSearch request
compatibility; Qdrant performs exact sparse dot-product search, not SEISMIC.
The optional engine may be omitted. Other method keys, the native engine and
pruning options are rejected. Named vectors are provisioned at index creation;
adding sparse fields through `_mapping` requires a new index.

Documents supply a token-weight object, for example `{"v":{"65536":1.0}}`.
Keys must be canonical decimal unsigned 32-bit IDs; weights must remain positive
and finite after float32 conversion. IDs are never hashed, folded or narrowed.
Missing/null fields omit the vector; an empty document vector is allowed.
Malformed vectors fail before any source or search write in that write batch.
Bulk index operations isolate invalid sparse documents as per-item 400 errors,
allowing adjacent valid items to proceed. Partial updates rebuild sparse vectors
when the sparse field changes, including with document projection enabled.

`{"query":{"neural_sparse":{"v":{"query_tokens":{"65536":1.0}}}}}`
supports one mapped field as the sole top-level query. Inference (`query_text`,
`model_id`), `method_parameters`, bool composition, sort, search_after, collapse,
post_filter, min_score and aggregations are rejected. `from`/`size` retain gateway limits;
reported total hits are a lower bound. Query token objects must not be empty. Scalar predicates (including exists, term,
range and nested bool filters), filter/metric aggregations on sparse fields or
their token subpaths are rejected for search and count; sparse vectors are not
scalar payload fields.
The OpenSearch Lucene default-path comparison in `examples/explicit-sparse`
verifies full-width token IDs; it does not reproduce SEISMIC approximation bugs.

## Shared metadata and consistency

SQLite remains the default single-replica catalog. The optional [Redis/Valkey catalog](https://dstockton.github.io/qdrant-es-gateway/shared-catalog.md) shares mappings and single-index aliases across replicas, with explicit recovery after uncertain administrative writes. It does not supply atomic document updates or snapshot pagination. Unsupported `pit` search bodies and URL parameters `if_seq_no`, `if_primary_term`, `version`, `version_type`, `retry_on_conflict`, `op_type`, `routing`, and `scroll` return a structured 400 rather than being ignored.

### Ordered numeric searches

Single-field numeric sorts can use Qdrant's ordered scroll when every matching
point carries a gateway-generated scalar-type marker. This avoids downloading
all matching documents to return one page. Multiple sort fields, missing or
array values, integers outside the exact floating-point comparison range,
separate document projections, and ties crossing the requested page boundary
retain the complete scan. The `search_after` predicate excludes the cursor value;
exact totals still describe all query matches before pagination.

New writes carry versioned eligibility metadata in the reserved
`_qeg_fast_fields` payload field. New collections and mapping updates index that
field; old points without markers safely use the existing scan. Application
mappings cannot define this reserved field. No reindex is required for correctness.
Neither ordered scroll nor the fallback is a point-in-time snapshot during
concurrent writes. Qdrant's [ordered-scroll API](https://qdrant.tech/documentation/concepts/points/#order-points-by-payload-key)
provides the backend ordering; the gateway checks compatibility before using it.

Exact keyword `terms` and simple keyword `term` / numeric `range` filter
aggregations can also use Qdrant facet/count indexes after checking the matching
payloads' eligibility markers. Facets request exact counts and up to 10,001
values, preserving the existing 10,000-bucket error and `sum_other_doc_count`.
Other reducers and unsafe or legacy values retain streaming evaluation.
The [facet API](https://qdrant.tech/documentation/concepts/payload/#facet-counts)
counts each distinct value once per document, including multivalued keywords.

An older mapping that already owns `_qeg_fast_fields` is treated as user data:
its values and index type are preserved, and fast paths are disabled for that
index. New mappings cannot introduce that name. These optimizations add no
cursor sessions, result cache, or Redis/Valkey dependency.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/architecture.md

# architecture

# Architecture

The gateway is a small Axum service with three boundaries:

```text
Elasticsearch client / REST
          |
   compatibility handlers
          |
 translation + response shaping
          |
      Qdrant REST API
          |
      Qdrant collections
```

Each ES index maps to a Qdrant collection named `es_<safe-index-name>`. SQLite stores mappings, vector names, and aliases in the default single-gateway deployment. The optional [Redis/Valkey catalog](https://dstockton.github.io/qdrant-es-gateway/shared-catalog.md) keeps authoritative metadata outside the gateway; each request materializes a fresh ephemeral SQLite view. Gateway replicas then require no durable local catalog. Metadata administration uses an explicit ownership/recovery protocol; there is no transaction spanning the catalog and Qdrant. Documents are payload-first: the original source is stored under `_source`, while mapped non-text fields are mirrored for Qdrant filtering, sorting, and aggregations.

When `DOCUMENT_PROJECTION=true`, every index also has `es_<safe-index-name>_documents`. This second collection is a durable document projection with no payload indexes and no HNSW graph. Qdrant 1.15 requires a vector field on points, so the gateway uses a one-dimensional on-disk placeholder vector and `m=0`; it is never searched. The searchable collection contains sparse representations plus only `_es_id`, `_es_index`, and mapped non-text fields. GETs and search response hydration read the authoritative `_source` from the document collection in a batch, which avoids storing the full source in the searchable collection. A full write persists the document projection first, then updates the searchable projection. `ASYNC_SEARCH_PROJECTION=true` uses Qdrant's asynchronous operation acknowledgement for the second step, so metadata-only writes can be acknowledged without waiting for sparse-index maintenance. The gateway itself owns no document queue or source cache.

Synchronous point writes and payload-index creation send Qdrant's `wait=true` URL parameter, so acknowledgment waits for the operation to finish. Asynchronous modes send `wait=false`. Earlier gateway revisions placed this control in the JSON body, which did not enforce completion; historical ingestion timings from those revisions measure acknowledgment rather than guaranteed completed writes. Completion is distinct from a transaction across collections or a guarantee about every replica's read visibility.

The two collections are intentionally not treated as one transaction. The source write is the durability boundary; projection updates must be idempotent and should be retried/reconciled by an external durable worker in an HA deployment. Filter and sort fields that must be read-after-write correct must remain mirrored in the searchable collection. Unmapped metadata can remain source-only. This is an explicit eventual-consistency mode, not an attempt to hide the trade-off behind local state.

Text fields get named sparse representations populated using Qdrant's server-side `qdrant/bm25` Document model. `multi_match` queries each requested field and merges unique hits with the declared boost. This is intentionally a useful application-search approximation, not Lucene score identity.

Point IDs are UUID-shaped values derived from SHA-256(index, NUL, ES ID). The original ID remains in `_es_id`; GET/update/delete therefore remain deterministic across restarts.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/production.md

# production

# Production deployment guide

The gateway is designed to be a stateless HTTP tier for document and search state held in Qdrant. It does not provide authentication, authorization, TLS termination, or Elasticsearch cluster semantics; place it behind an ingress/API gateway and restrict network access to the Qdrant endpoint.

## Container

Images are published to `ghcr.io/dstockton/qdrant-es-gateway` for version tags. The image runs as UID 10001, has no Linux capabilities, and uses a read-only root filesystem in the Helm chart. Persist `/data` when using the default SQLite catalog. Shared-catalog deployments instead require durable Redis/Valkey storage; see [shared catalog operation and recovery](https://dstockton.github.io/qdrant-es-gateway/shared-catalog.md).

Use immutable version tags or image digests in production:

```bash
docker pull ghcr.io/dstockton/qdrant-es-gateway:0.1.0
```

Every release workflow produces a multi-architecture image, BuildKit provenance, an SPDX SBOM, a cosign SBOM attestation, and Trivy vulnerability results. The release job fails on known fixed HIGH or CRITICAL image vulnerabilities; unfixed findings are retained in the scan results for review.

The gateway applies a 5-second Qdrant connection timeout and a 180-second upstream request timeout by default. Set `QDRANT_CONNECT_TIMEOUT_MS` and `QDRANT_REQUEST_TIMEOUT_MS` to change them. When `ASYNC_PAYLOAD_WRITES=true`, `ASYNC_WRITE_QUEUE` bounds detached writes (256 by default); a full queue returns an upstream error instead of allowing background tasks to grow without limit.

The SQLite metadata connection uses a five-second busy timeout and `synchronous=FULL`. This helps with short local lock contention, but it does not make a shared network filesystem safe for multiple gateway replicas.

After the first release, open the repository's Packages settings and change the GHCR package visibility to Public if GitHub created it privately. Link the package to this repository and enable Dependabot alerts, security updates, and secret scanning in repository Security settings.

## Kubernetes

The chart is in `deploy/helm/qdrant-es-gateway`. Qdrant is intentionally external so its clustering, replication, backups, and storage lifecycle can be managed independently.

```bash
helm upgrade --install search-gateway deploy/helm/qdrant-es-gateway \
  --set image.tag=0.1.0 \
  --set qdrant.url=https://qdrant.example.internal:6334 \
  --set qdrant.existingSecret=qdrant-credentials
```

Before exposing the service:

1. Configure ingress TLS and client authentication.
2. Create a Kubernetes NetworkPolicy allowing egress only to Qdrant and DNS.
3. Set resource requests and limits from workload measurements.
4. Keep the default SQLite deployment at one replica. For multiple replicas, configure the optional shared catalog, initialize or migrate it deliberately, and follow its durability and recovery requirements. Do not share SQLite files over a network filesystem.
5. Configure Qdrant replication, snapshots, monitoring, and disk alerts independently.
6. Enable `DOCUMENT_PROJECTION=true` and `ASYNC_SEARCH_PROJECTION=true` only with a durable reconciliation process for projection lag or failed asynchronous operations.

## Operational checks

- `/healthz` is a liveness check.
- `/readyz` checks Qdrant connectivity and should gate traffic.
- `GET /` exposes the compatibility/version response.
- Watch Qdrant segment growth, optimizer backlog, disk utilization, search latency, and projection lag.
- Keep request and bulk limits bounded; request bodies are buffered in memory, and each concurrent `POST /_bulk` or `POST /<index>/_bulk` may consume up to `MAX_BULK_BYTES` before parsing. Do not expose the service directly to the public internet.
- Test the actual mappings and unsupported-query behavior before migration.

## Release process

1. Bump the package version in `Cargo.toml` and the chart version in `deploy/helm/qdrant-es-gateway/Chart.yaml`.
2. Merge the tested change to `main`.
3. The release workflow creates the matching `vX.Y.Z` tag, publishes the multi-architecture GHCR image, packages the Helm chart, and creates the GitHub release. If that version tag already exists, the workflow skips publishing.
4. Review the GitHub Actions image scan and SBOM attestation.
5. Promote the immutable image digest through environments.

Use the manually triggered `Release dry run` workflow to validate version metadata, build the image, scan it, and package the chart without creating a tag, logging into GHCR, or publishing anything.

The repository's CI does not publish private voice, video-generation, or local development artifacts.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/migration.md

# migration

# Migration guide

1. Create a representative index and mapping through the normal ES API.
2. Re-ingest documents with `_bulk` or use Qdrant's official migration tooling for the data-movement portion.
3. Point the existing Elasticsearch client at the gateway URL.
4. Run representative traffic and inspect `/_qdrant_gateway/compatibility`.
5. Compare matching IDs, filter correctness, p95 latency, and relevance labels against Elasticsearch.

Do not migrate a workload that depends on Kibana, observability APIs, scripts, nested/parent-child semantics, or arbitrary aggregations without redesigning those paths.

## Upgrading an existing Qdrant volume

The Compose files use Qdrant 1.20.0 for new deployments. If an existing volume
was last opened by Qdrant 1.15.x, do not start it directly with that image.
Back up the data first, then start the latest patch of each minor release in
order: 1.15.5, 1.16.3, 1.17.1, 1.18.3, 1.19.2, and finally 1.20.0. Wait for each
version to finish starting and become healthy before moving to the next. For
the resilience cluster, roll each version through the nodes one at a time and
wait for the cluster to recover before advancing. Follow Qdrant's
[upgrade instructions](https://qdrant.tech/documentation/upgrades/) and
[snapshot guidance](https://qdrant.tech/documentation/snapshots/).
Qdrant storage migrations are not reversible; retain the pre-upgrade backup.

Before upgrading an authenticated cluster to 1.20.0, check its internal
authentication configuration. Qdrant now enables `service.enforce_internal_auth`
by default when an API key is configured. A cluster configured with only
`read_only_api_key` needs an `api_key` for peer communication, or an explicit
decision to disable internal authentication. A cluster using only `alt_api_key`
on 1.19.0 or 1.19.1 must reach 1.19.2 first. Tools calling the internal port
must send an `api-key` header. See Qdrant's
[1.20.0 release notes](https://github.com/qdrant/qdrant/releases/tag/v1.20.0)
and [upgrade instructions](https://qdrant.tech/documentation/upgrades/).


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/application-validation.md

# application-validation

# Real-application validation targets for Elasticsearch and OpenSearch

The validation strategy is endpoint-only: run one application workflow twice and change only its Elasticsearch/OpenSearch-compatible base URL. This proves both that the application needs no gateway-specific client code and that returned documents are usable by the application. Where the application uses OpenSearch, validate it against native OpenSearch separately from native Elasticsearch; shared request shapes are evidence of compatibility, not proof that the engines rank or aggregate identically.

## Elasticsearch target: Spinscale catalogue search

[Spinscale's ecommerce search app](https://github.com/spinscale/elasticsearch-ecommerce-search-app) is the best first published application target. It exercises index mappings, bulk ingestion, full-text product search, filters, sorting, pagination, facets, and source-bearing hits. Its original build is tied to an older Java/Gradle/Micronaut stack, so the practical validation path is to preserve its request shapes and fixture data in a modern, pinned replay harness before attempting a full application upgrade.

The current gateway can validate the basic catalogue slice directly. The remaining gaps before claiming unchanged full-application compatibility are custom analyzers/synonyms, exact fuzzy and prefix ranking, search-scoped aggregation membership, and the complete Elasticsearch bulk response contract. `post_filter`, basic metric/filter aggregations, `dis_max`, fuzzy-shaped requests, and retail-style `more_like_this` plus `collapse`/`inner_hits` response shapes are now accepted, but their semantics remain explicitly approximate where the underlying Qdrant query is not Lucene-equivalent.

## OpenSearch target: AWS Retail Demo Store search

[AWS Retail Demo Store](https://github.com/aws-samples/retail-demo-store) has a useful isolated search service, although the full storefront is AWS-heavy and uses OpenSearch. Its search API returns product IDs and then retrieves product details separately. The request shapes include typeahead `match_bool_prefix`, `dis_max`, category diversity through `collapse`/`inner_hits`, and “similar products” via `more_like_this`. The [`retail_demo.py`](https://dstockton.github.io/validation/retail_demo.py) replay now passes the application's request/response contract against native OpenSearch 3.9.0 and the gateway. Exact prefix ranking, category distribution, and similarity relevance still differ and need product-level evaluation before a full semantic-equivalence claim.

The smallest credible sequence is:

1. run the repository's `validation/replay.py` against native Elasticsearch and the gateway;
2. add a pinned Spinscale request fixture and compare its top-k IDs, sources, filters, facets, and pagination;
3. run the AWS search container locally and capture its real HTTP requests;
4. replay those requests against both endpoints, measuring result equivalence separately from ranking quality.

The current request-level audit is maintained in [demo-app-compatibility.md](https://dstockton.github.io/qdrant-es-gateway/demo-app-compatibility.md). It separates transport/response compatibility from semantic equivalence so “the application can make the request” is not confused with “the two engines rank and aggregate identically.”

The endpoint replay has also been run against native OpenSearch 3.6.0. The [2026-10-09 field note](https://dstockton.github.io/qdrant-es-gateway/2026/10/09/opensearch-wildcards-keep-the-leading-star.html) records 10 of 12 application checks passing after correcting leading-wildcard translation, with analyzed-text prefix case behavior and bounded-window cursor ordering retained as explicit differences. The official `opensearch-py` 3.2.0 client separately completed the supported CRUD/search workflow against the gateway; this verifies client transport behavior, not full server parity.

Do not compare raw `_score` values. Compare exact sources for the same IDs where ordering is not the product contract, and use top-k overlap, nDCG, zero-result rate, facet equality, and pagination continuity for ranked results.

## Performance protocol

Use identical payloads, IDs, refresh policy, warm-up, concurrency, and client connection settings for both endpoints. Record bulk time, update latency, search p50/p95/p99, mixed-workload throughput, error rate, CPU, memory, disk, and freshness. Run the lightweight fixture first, then 50,000 and 500,000 document corpora. Keep compatibility assertions and performance measurements as separate report sections so a faster but semantically different result cannot be mistaken for a successful application validation.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/benchmarking.md

# benchmarking

# Benchmarking

The reproducible comparative benchmark is `benchmark/run.sh`. It starts native Elasticsearch, Qdrant, the Rust gateway, and a containerized Python client on one Docker host. A large profile loads 500,000 synthetic products in batches of 5,000, measures updates and full-source reads, then runs a mixed workload of searches, reads, updates, and deletes at concurrency 50. Compose limits the services to approximately four CPUs and fifteen GB of memory, while `ES_HEAP` explicitly controls the Elasticsearch JVM heap for a comparable memory profile.

```bash
cd benchmark
DOCS=50000 BATCH=500 ./run.sh
```

Results are written to `benchmark/results/benchmark.json` and `benchmark/results/resources.txt`. Re-run with the same corpus, image versions, warm-up behaviour, and host profile when comparing changes.

For a real product corpus, see the external-dataset workflow in [`benchmark/README.md`](https://dstockton.github.io/benchmark/README.md). It downloads the public H&M catalogue, converts selected product fields to JSONL, mounts that generated file read-only into the runner, and executes the same benchmark client against both endpoints.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/contributing.md

# contributing

# Contributing

Install Rust, Docker, and run `make test` and `make lint`. Integration tests require Docker and a reachable Qdrant. Keep unsupported behavior explicit: do not silently discard an ES clause. Add a unit test for each new translator rule and an integration fixture for each Qdrant API dependency.

After changing an article or the compatibility, architecture, production, or migration guide, run `python3 scripts/export_agent_docs.py`. Commit the generated `docs/markdown/`, `docs/llms.txt`, and `docs/llms-full.txt` alongside the source changes. CI checks that they remain synchronized. Markdown exports have no front matter, so Jekyll copies them as literal text files.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/demo-app-compatibility.md

# demo-app-compatibility

# Demo-application compatibility

This page records request-level compatibility against the two public application targets that motivated the gateway work: one Elasticsearch application and one OpenSearch application. “Supported” means that the gateway accepts the request shape and returns the fields the application consumes. It does not mean Lucene-identical ranking, analyzer behavior, or refresh/version semantics. Treat Elasticsearch and OpenSearch as separate compatibility targets, and record which native engine and version produced each validation result.

## AWS Retail Demo Store search service

The isolated search service in [aws-samples/retail-demo-store](https://github.com/aws-samples/retail-demo-store) uses an OpenSearch-compatible client and sends:

| Request feature | Gateway status | Notes |
|---|---:|---|
| `match_bool_prefix` across several fields | ✅ | Accepted through the gateway lexical path; typo/prefix ranking is approximate |
| `dis_max` with boosts and tie breaker | ✅ | Query shape accepted; Qdrant scoring is not Lucene/OpenSearch scoring |
| `collapse` on `category.keyword` | ✅ | Gateway-side grouping after retrieval |
| `inner_hits` with `_source: false` and `_id` consumption | ✅ | The standard named definition returns `category_hits`, grouped totals, bounded hits, and independent source suppression |
| `more_like_this` by document ID | ✅ | Expanded into a gateway lexical query using the referenced source |
| `from`/`size` | ✅ | Bounded pagination |

The service's two search routes—product search and similar products—are request-compatible with the gateway's supported subset. The dependency-free [`validation/retail_demo.py`](https://dstockton.github.io/validation/retail_demo.py) replay passes the response fields consumed by the service against OpenSearch 3.9.0 and the gateway. Result quality remains a separate question: category diversity, top-k overlap, and similar-product relevance must be measured rather than inferred from HTTP compatibility.

## Spinscale Elasticsearch e-commerce search app

The [spinscale/elasticsearch-ecommerce-search-app](https://github.com/spinscale/elasticsearch-ecommerce-search-app) exercises a broader catalogue workflow:

| Request feature | Gateway status | Notes |
|---|---:|---|
| index existence, create, delete | ✅ | Durable gateway metadata and Qdrant collections |
| close/open/settings bootstrap calls | ⚠️ | Common request shapes are acknowledged; Elasticsearch analyzer/settings effects are not emulated |
| bulk indexing with immediate refresh policy | ✅ | Bulk response and refresh endpoint are supported; Qdrant is continuously available |
| `multi_match` with fuzziness and `minimum_should_match` | ⚠️ | Accepted lexical path; ranking and exact minimum-term semantics are approximate |
| `post_filter` | ⚠️ | Simple term/terms/range/exists/bool shapes are applied to hydrated hits |
| terms, min/max, filters aggregations | ⚠️ | Supported shapes are bounded by the gateway's candidate/facet implementation |
| source-bearing hits, updates, deletes, counts | ✅ | Covered by `validation/replay.py` |

The AWS target is the closest current fit for a no-client-change search path. The Spinscale target is a useful compatibility and relevance test, but it depends more heavily on Elasticsearch-specific analysis and aggregation semantics. The project should not claim 100% semantic replacement for either application until the real application workflows have been replayed and their product-level assertions pass.

## Additional application targets

These are worthwhile next targets because they are well-known open-source applications with documented Elasticsearch integrations. They are intentionally listed as targets, not as compatibility claims:

| Application | Elasticsearch role | Likely gateway fit | What a real replay must prove |
|---|---|---|---|
| [Zammad](https://github.com/zammad/zammad) | Ticket, user, organisation, knowledge-base and global search; Elasticsearch is enabled by default in the documented deployment configuration | Medium for simple text/filter/search flows; low for a drop-in replacement until Zammad's index schema and query builder are replayed | Search result IDs, field projections, pagination, sorting, counts, reindex lifecycle and freshness |
| [GitLab](https://gitlab.com/gitlab-org/gitlab) | Optional advanced/global/code search across projects, work items, merge requests and other content | Low for the complete product; potentially useful for an isolated application-search slice, not GitLab's code-search workload | Exact search modes, permissions, aggregations, aliases, refresh behaviour and the multi-index schema |
| [Adobe Commerce / Magento Open Source](https://github.com/magento/magento2) | Required catalog search engine in current 2.4.x deployments; product search, filters, sorting and catalog indexing | Medium for a bounded catalogue-only test; low for the full adapter without analyzers, facets and Magento-specific mappings | Catalog reindex, searchable attributes, layered navigation facets, price/range filters, sort order, store scope and product visibility |
| [Open edX `edx-search`](https://github.com/openedx/edx-search) | Pluggable search backend; Elasticsearch is a supported legacy backend for LMS search | Medium for document indexing/search APIs; not a claim about the full Open edX platform | Document type filtering, field filters, pagination, exact result objects and update/delete freshness |

The first three applications use Elasticsearch as part of a larger product rather than exposing a simple public search API. That makes them valuable semantic tests, but permissions, analyzers, aliases, aggregations and refresh semantics can dominate the result.

## Public corpus benchmark targets

The reproducible benchmark now accepts an external JSONL corpus and includes an importer for the [Qdrant H&M product catalogue](https://huggingface.co/datasets/Qdrant/hm_ecommerce_products). It contains 105,126 product records in the downloaded snapshot and is published under CC BY 4.0. The corpus is downloaded and generated locally; it is not committed to this repository. The benchmark command and importer are documented in [benchmarking.md](https://dstockton.github.io/qdrant-es-gateway/benchmarking.md).

[Open Food Facts](https://huggingface.co/datasets/openfoodfacts/product-database) is a second useful scale target: its food split is millions of products and is available under the Open Database License. It is too large for routine CI and requires attribution/share-alike handling, so it should be run as an explicitly provisioned benchmark rather than bundled into the repository.

## Reproducible validation

Run the deterministic endpoint-only replay against native Elasticsearch/OpenSearch and the gateway:

```sh
python3 validation/replay.py \
  --native http://localhost:19200 \
  --gateway http://localhost:9200 \
  --output validation/results/latest.json
```

The replay now covers lifecycle calls, bulk ingestion, filters, facets, pattern queries, `_msearch`, cursor pagination, reads, updates, deletes, and counts. Compare IDs, sources, facet keys, and pagination behavior; do not compare raw engine scores.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/external-corpus-results.md

# external-corpus-results

# External corpus results

This is a reproducible snapshot, not a capacity promise. It records the first external-corpus comparison so later gateway changes can be checked against a real catalogue rather than only the synthetic fixture.

## H&M product catalogue

Source: [Qdrant's `hm_ecommerce_products` dataset](https://huggingface.co/datasets/Qdrant/hm_ecommerce_products), 105,126 product records in the downloaded parquet snapshot, published under CC BY 4.0. The repository stores only the importer and benchmark instructions; the parquet and generated JSONL remain local and are not redistributed here.

The benchmark ran native Elasticsearch 8.15.0 and the Rust gateway backed by the Qdrant engine 1.15.3 in Docker Desktop, with the repository's approximate 1.5 CPU / 8 GB Elasticsearch, 1 CPU / 4 GB Qdrant, 1 CPU / 2 GB gateway and 0.5 CPU / 1 GB runner limits. It used identical JSONL batches and client code; only the ES-compatible endpoint changed. The query was a title match for `dress`, with 1,000 updates and 500 mixed requests at concurrency 10.

| Profile | Native Elasticsearch | Qdrant gateway | Gateway relative result |
|---|---:|---:|---:|
| Bulk ingest, `refresh=` disabled | 5.69 s | 10.14 s | 1.78x slower |
| Bulk ingest, `refresh=wait_for` per batch | 215.45 s (two-repeat median) | 2.90 s | Refresh semantics dominate this comparison |
| Search p95, concurrency 1 | 17.99 ms | 0.99 ms | 18.2x lower latency |
| Search p95, concurrency 10 | 82.40 ms | 3.54 ms | 23.3x lower latency |
| Mixed throughput, 500 requests | 740 req/s | 1,255 req/s | 1.70x higher throughput |
| 1,000 updates | 3.14 s | 1.82 s | 1.72x faster |

The refresh-wait profile is useful because it reflects an application that explicitly requests immediate visibility after every bulk batch. It must not be described as raw Elasticsearch indexing capacity: the no-refresh run shows native Elasticsearch completing the same corpus faster than the gateway. Conversely, the gateway's bulk path does not reproduce Elasticsearch refresh scheduling because Qdrant remains continuously queryable.

## Result quality

Both engines returned ten source-bearing hits for the same query. The top-ten IDs did not match exactly. That is expected from the current gateway's Qdrant sparse ranking versus Elasticsearch/Lucene analysis and scoring, but it is a material compatibility result: the gateway is currently a strong transport and performance candidate for application-search workloads, not a claim of 100% relevance equivalence. Future reports should add recall@k / overlap, category coverage and an application-specific judged relevance set.

The run also captured peak container memory: Elasticsearch 1.45 GiB, Qdrant 411 MiB and gateway 13 MiB at the sampling point. Disk measurements were unavailable from the host-side collector on Docker Desktop because its Linux VM volume paths are not exposed; run the same profile on Linux or collect the mounted paths from inside the containers before publishing storage comparisons.

## Reproduce

See [`benchmark/README.md`](https://dstockton.github.io/benchmark/README.md) for the download, attribution, conversion and benchmark commands. Run both profiles when comparing a client that requests immediate visibility and a client that allows refresh to be decoupled from bulk ingestion.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/internals.md

# internals

# Internals

The current implementation uses direct Qdrant REST calls to keep the protocol boundary visible and easy to audit. The metadata store is SQLite because it is durable, embedded, and small enough for index/mapping/alias metadata. It is not a document store.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/read-write-crossover.md

# read-write-crossover

# Read/write crossover

For product search, updates do not all need to block the user-facing request. A price or stock update can be accepted, applied asynchronously, and become searchable shortly afterwards. That makes the useful question less binary than “can the gateway beat a native write?”: how much of the workload is customer-facing retrieval, and how much is synchronous mutation?

## Focused mix sweep

The sweep used the same Elasticsearch-shaped client calls against both services:

- 50,000 product documents
- 1,000 requests per point
- 50 concurrent clients
- read/search traffic split evenly between `_search` and `GET _doc`
- write traffic made up of single-document `_update` calls
- `ASYNC_PAYLOAD_WRITES=true` for the gateway

| Read/search share | Write/update share | Elasticsearch throughput | Qdrant gateway throughput | Approximate winner |
|---:|---:|---:|---:|---|
| 20% | 80% | 1,166.90 req/s | 962.31 req/s | Elasticsearch |
| 40% | 60% | 1,080.77 req/s | 1,177.88 req/s | Qdrant gateway |
| 60% | 40% | 1,071.20 req/s | 1,059.19 req/s | Near tie |
| 80% | 20% | 1,091.95 req/s | 1,308.47 req/s | Qdrant gateway |

The practical crossover is therefore around 40–60% read/search traffic for this profile. At roughly 40% reads, the gateway begins to lead on throughput; at 60%, the two paths are effectively tied; and at 80%, the gateway has a clear throughput advantage. This is a workload boundary, not a universal constant: corpus size, query shape, update fields, concurrency, and the freshness window all move it.

The 500,000-document mixed run reinforces the shape of the result. With 70% search/read and 30% update/delete traffic, throughput was 77.42 req/s for Elasticsearch and 263.32 req/s for the gateway. Search p95 was 4,213.6 ms versus 305.82 ms. The gateway’s asynchronous update mode is a good fit when the product can tolerate eventual search visibility for ordinary catalog changes.

![Bubble chart comparing corpus size, search p95 latency, and throughput](https://dstockton.github.io/qdrant-es-gateway/assets/throughput-latency-corpus-bubbles.png)

## How to use the crossover

For a catalog workload dominated by imports, synchronous writes, or strict read-after-write requirements, native Elasticsearch remains the safer baseline. For a product experience dominated by retrieval—especially when updates can flow through an event queue or tolerate a short freshness delay—the gateway becomes increasingly attractive.

The next production decision should be based on the application’s freshness contract: measure the acceptable update-to-search delay, then place the workload on this curve using the same query mix and concurrency as the real service.

## Additional corpus scaling points

To make the bubble chart less dependent on the two original points, the same mixed workload was repeated at four smaller corpus sizes. These runs used the current gateway build, 50 concurrent clients, 1,000 mixed requests, and the existing 70% read/search versus 30% update/delete composition.

| Corpus | Elasticsearch throughput | Qdrant gateway throughput | Elasticsearch mixed search p95 | Qdrant gateway mixed search p95 |
|---:|---:|---:|---:|---:|
| 10,000 | 749.48 req/s | 1,192.41 req/s | 125.06 ms | 97.10 ms |
| 25,000 | 982.57 req/s | 1,011.40 req/s | 101.34 ms | 97.23 ms |
| 50,000 | 1,037.89 req/s | 1,010.64 req/s | 96.90 ms | 101.20 ms |
| 100,000 | 935.91 req/s | 986.74 req/s | 103.94 ms | 102.97 ms |
| 500,000 | 77.42 req/s | 263.32 req/s | 4,213.60 ms | 305.82 ms |

The 10k–100k points were collected in the local four-CPU container profile. The 500k point is the previously captured production-shaped run, so it is useful for scale context but should not be treated as a perfectly controlled continuation of the smaller points. That distinction is called out directly in the chart.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/relevance.md

# relevance

# Relevance evaluation

Use a labelled query/corpus fixture and compare top-k IDs with Elasticsearch. Track MRR or NDCG, not raw score equality. Differences can come from analyzers, tokenization, field fusion, document length normalization, and BM25 implementation. A production migration should include domain-labelled queries before making an economics decision.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/shared-catalog.md

# Shared metadata across gateway replicas

The default SQLite catalog is still the simplest choice for one gateway. No measurement has established SQLite as a bottleneck. The optional Redis/Valkey catalog addresses **metadata ownership across replicas**, not faster search.

Set `CATALOG_REDIS_URL` to a `redis://` or TLS `rediss://` URL. All replicas serving the same Qdrant collections must use the same `CATALOG_REDIS_KEY` (default `qeg:{catalog}:v1`) and collection-related configuration. The URL may contain credentials; supply it through a secret. Redis Cluster is not a supported connection topology.

The authoritative catalog contains mappings, named vector metadata, and single-index aliases. Requests that need catalog metadata load a fresh committed snapshot; there is no replica metadata cache or persistent local metadata dependency in shared mode. Network operations time out after five seconds.

For exact `GET /<index>/_doc/<id>` and `GET`/`POST /<index>/_search` handlers, the gateway validates the complete snapshot into request-local typed maps instead of rebuilding an ephemeral SQLite database. This also applies when response-cache serving is disabled or the request bypasses that cache. A valid response-cache hit can return without loading a catalog. Alias-read routes, other metadata routes and mutations retain the SQLite snapshot path; route precedence is preserved, including `GET /_alias/_search`. Deployments using the default local SQLite catalog are unchanged.

The read view is not retained between requests. It still fetches and validates the whole catalog, including records unrelated to the requested index, with work proportional to catalog size. Malformed metadata and duplicate records fail closed. Snapshot size remains limited to 8 MiB. This removes SQLite reconstruction from the selected shared-catalog read handlers; it does not establish a measured latency improvement or remove the small-catalog constraint.

Index creation and mapping changes check the exact prospective serialized size before mutating Qdrant; alias batches check it before committing local metadata. Oversized changes return a validation error and release the administrative claim.

## Starting a new deployment

Use a dedicated durable Redis/Valkey instance, disable eviction, enable AOF persistence and select the durability/HA policy appropriate to your deployment. An acknowledged metadata update is only as durable as that server's configuration. Losing an acknowledged catalog update during asynchronous failover can invalidate ownership guarantees: do not automatically fail over to a stale replica and resume writes. Stop gateways, restore/reconcile the catalog and Qdrant, then restart. Redis/Valkey is an authoritative database here, not a disposable cache.

1. Start Redis/Valkey with `--appendonly yes --appendfsync always --maxmemory-policy noeviction` and a persistent volume.
2. For an **empty** deployment, run one gateway once with `CATALOG_INITIALIZE=true` and `CATALOG_REDIS_URL` set. This creates an empty catalog only if absent.
3. Remove `CATALOG_INITIALIZE` before normal operation. Leaving it enabled would permit accidental reinitialization after data loss.
4. Start replicas with the same catalog URL/key and Qdrant configuration. Readiness checks now require an accessible, unlocked catalog; liveness remains independent.

Existing SQLite catalogs are **not automatically migrated**. Do not switch an existing deployment to an empty shared catalog: stop writes and export/validate the existing mappings, vector lists and aliases first. The versioned JSON format is `{ "indices": [["name", "mapping JSON", "vector-list JSON"]], "aliases": [["alias", "index"]] }`. Back up both stores, validate in staging, and only then switch all replicas together. Do not mix SQLite and shared-catalog replicas against the same collections.

For Helm, create a Secret containing the URL and set:

```yaml
replicaCount: 2
catalog:
  existingSecret: gateway-catalog
  existingSecretKey: redis-url
  key: 'qeg:{catalog}:v1'
persistence:
  enabled: false
```

Initialize the catalog separately before deploying these values. The chart deliberately never enables automatic initialization. It does not provision Redis/Valkey.

## Administration and recovery

Creating/deleting an index, changing mappings, or changing aliases acquires one non-expiring administrative claim. Another replica cannot mutate metadata concurrently; new requests fail closed while the claim exists. Requests already in flight may finish using their previously acquired mapping. A successful operation publishes the entire updated catalog and releases its claim in one Redis/Valkey script.

The claim deliberately has **no TTL**: expiring a lock while a slow or disconnected writer still changes Qdrant permits two administrators to act at once. An upstream error after a mutation was attempted, cancelled request, crash or ambiguous timeout therefore leaves the claim in place. Validation errors before an upstream mutation release the claim safely. The gateway returns the original error and logs that recovery is required. This prioritizes safe ownership over automatic availability.

Recovery requires an operator:

1. Stop all gateway replicas and ensure their outstanding Qdrant operations have completed.
2. Back up and inspect the catalog key and its `<catalog-key>:admin` claim. Compare the affected mappings/collections/aliases with Qdrant; resolve any partial creation/deletion or payload-index changes. A successful-but-unacknowledged commit may already have removed the claim.
3. Restore or repair the catalog to match the intended Qdrant state. Only after reconciliation, delete the administrative claim with Redis/Valkey's `DEL` command.
4. Restart the gateways and verify readiness and the affected operation.

Do not delete a claim while a writer may still be running. A Redis/Valkey transaction cannot atomically include Qdrant writes. This backend does not supply document `_create` or update compare-and-set across replicas, point-in-time search, exactly-once projections, multi-index aliases, or Lucene analyzer equivalence.

## Verification

Unit tests cover snapshot round-tripping, malformed metadata and administrative route classification. Run the real-server coordination test against a disposable Redis/Valkey instance:

```sh
CATALOG_TEST_REDIS_URL=redis://127.0.0.1:16379 cargo test shared_catalog -- --include-ignored
```

The tests check fresh cross-client reads, successful create/mapping/alias dispatch across independent replicas, concurrent administration exclusion, absence of lock expiry, wrong-owner rejection, safe validation-error release, capacity rejection before upstream writes and retention of a claim after its owner disappears. It creates unique temporary keys and deletes them on success.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/shared-read-cache.md

# Shared read cache

The optional shared read cache keeps gateway replicas stateless while reusing successful point reads and filter searches. Qdrant remains the document store. Valkey holds the authoritative catalog, cache entries, and persistent write coordination. Caching is off by default.

The [measured three-engine comparison](https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-shared-cache-warm-search-results.md) found 12–30% lower warm-search median latency than the faster native engine in a 10,000-document catalogue profile. Fresh queries and write-followed-by-point-read cycles were slower than native engines; point-read warm gains were small. These results do not establish a general production advantage. Measure cold reads, repeated reads, writes, and your actual hit rate before adopting it.

## Supported deployment

Use a dedicated standalone Valkey primary with no attached replicas, persistent storage, `appendonly yes`, `appendfsync always`, `no-appendfsync-on-rewrite no`, a nonzero `maxmemory`, and a supported eviction policy that cannot evict persistent catalog or control keys. `noeviction` is the simplest configuration. Cached entries have bounded sizes and expire; control keys and unresolved writer tokens do not.

Qdrant must run as a standalone node with distributed mode disabled. Every existing collection must have replication factor one. Set `QDRANT_REPLICATION_FACTOR=1` so new collections have the same constraint. Both asynchronous write settings must be off. Startup checks these requirements; a configuration value alone does not prove existing collections are suitable.

Every gateway writing to this deployment must participate in coordination. Stop and drain old replicas before activation. An old binary, a direct Qdrant writer, or an independent metadata writer can bypass invalidation and is unsupported. Do not use automatic Valkey failover or restore an older control snapshot. A surviving but rolled-back control record is not detectable from its contents. A gateway without `CATALOG_REDIS_URL` cannot discover this deployment at all; it is an unsupported unfenced writer even if it runs a new binary.

## Activate deliberately

First provision the authoritative catalog as described in [Shared catalog](https://dstockton.github.io/qdrant-es-gateway/shared-catalog/). Stop all gateway writers and external writers. Use the same catalog endpoint and namespace for every command and replica. Do not set `CATALOG_INITIALIZE=true` during cache operation.

Choose a new deployment identity, for example a UUID, and configure:

```sh
export READ_CACHE_DEPLOYMENT="<new-deployment-uuid>"
export QDRANT_REPLICATION_FACTOR=1
export ASYNC_PAYLOAD_WRITES=false
export ASYNC_SEARCH_PROJECTION=false
export READ_CACHE_SERVE=true
export READ_CACHE_TTL_SECONDS=60
export READ_CACHE_MAX_BYTES=1048576
qdrant-es-gateway cache bootstrap --quiesced
```

The command requires an existing catalog and no unresolved administrative claim. It will not overwrite an existing control record. Persistent lineage also prevents bootstrap from treating a lost control record as a new deployment; use explicit recovery instead. Then start all replicas with this configuration. `READ_CACHE_SERVE=false` disables cache reads and fills on a replica while retaining mandatory write coordination.

Deployment identity is separate from response compatibility: cache keys also include a digest of the actual executable, response-affecting configuration, method, URI, and request body. Different builds do not reuse each other's responses. Keys do not contain plaintext credentials.

## Response entries and persistence

Response-cache fills are disposable. Their publication script suppresses incremental AOF propagation only for the bounded, expiring response entry, then restores normal propagation. Catalog updates, deployment lineage, generation changes and writer admission/release remain durable under the required `appendfsync always` configuration. This does not weaken the persistence requirements for the authoritative state.

AOF rewrites or snapshots may still retain response entries that exist in memory. Restored entries must therefore pass the same deployment identity, build/configuration, generation, expiry and quiet-writer checks before use. Do not assume a restart empties the response cache, or that every cache-fill request avoids all fsync activity: other writes and persistence work can still occur. This change alone establishes no performance benefit.

The isolated [AOF crash/rewrite proof](https://github.com/dstockton/qdrant-es-gateway/blob/main/tests/cache_ephemeral_aof.py) exercises the production Lua scripts against disposable durable Valkey storage. See the [test instructions](https://github.com/dstockton/qdrant-es-gateway/blob/main/tests/cache-live.md) for its scope and invocation.

## What happens around a write

Before the gateway starts a mutation, it advances the shared generation and records a persistent writer token. A cached response is usable only in the same quiet generation with no writer or catalog administrative claim. A cold read captures its generation before loading catalog metadata; publication fails if a writer intervenes.

Each backend mutation must report completion. An HTTP success or bulk HTTP 200 alone is not sufficient. The gateway releases its token only when all attempted backend operations and its own catalog publication are known complete. Cancellation, a crash, an ambiguous backend response, or failed publication leaves the token in place. This disables cached reads and fills; it does not silently serve stale results after a timeout.

This does not add transactions, optimistic concurrency control, point-in-time snapshots, or full Elasticsearch/OpenSearch semantics. Uncached reads retain existing gateway behavior, including visibility during multistep writes. Explicit `request_cache=false`, ranked queries, vector queries, unsupported shapes, and oversized responses remain uncached.

## Maintenance and recovery

Inspect outstanding tokens with:

```sh
qdrant-es-gateway cache status
qdrant-es-gateway cache maintenance
```

Maintenance prevents new coordinated mutations and cached reads. Stop and drain all replicas before changing Qdrant data, topology, credentials, catalog state, or Valkey persistence. A cached hit cannot discover a revoked upstream credential without contacting Qdrant, so upstream authorization changes require this maintenance procedure too.

After a crash or uncertain write, reconcile Qdrant and catalog state and account for every pending backend operation. An elapsed timeout is not proof that an operation cannot finish later. Resolve any catalog administrative claim using its documented recovery procedure. Keep all writers stopped. Choose a fresh identity and explicitly acknowledge that reconciliation is complete:

```sh
qdrant-es-gateway cache recover --new-identity "<fresh-deployment-uuid>" --reconciled
```

The expected old identity comes from `READ_CACHE_DEPLOYMENT`. Configure every replica with the new identity before restarting. Recovery invalidates old entries and must never reuse an earlier identity. The command cannot establish the operator's reconciliation proof. Control-only loss can be recovered using the persisted lineage and expected old identity. Total namespace or lineage loss, or restoration of older state, requires stopping all writers, reconciling Qdrant and the catalog, and provisioning a fresh catalog/cache namespace with a fresh identity. Never recreate the catalog and resume the old cache identity.

## Costs and limits

Warm reads still make a Valkey request. Cold reads also run the ordinary Qdrant query and may write a cache entry. Every mutation adds durable coordination writes, and any write invalidates the shared generation. Frequent writes or low reuse can erase the benefit. The aggregate resource budget must include Valkey as well as the gateway and Qdrant when comparing native engines.

Valkey AOF durability follows its [persistence configuration](https://valkey.io/topics/persistence/). Qdrant completion and replication are separate guarantees; see [Qdrant consistency guarantees](https://qdrant.tech/documentation/scaling/consistency-guarantees/).

For Helm, set `readCache.deployment`, `catalog.existingSecret`, and `qdrant.replicationFactor: 1`, and disable both `gateway.asyncPayloadWrites` and `gateway.asyncSearchProjection`. Set `readCache.serve: false` to bypass cache serving while preserving coordination. The chart does not bootstrap or recover state automatically. Scale every writer to zero for initial activation and perform the operator command before restoring replicas.


Source: https://dstockton.github.io/qdrant-es-gateway/markdown/storage-findings.md

# storage-findings

# Storage findings

The storage comparison must include all durable data required by each architecture. For the gateway projection profile that means both Qdrant collections: the searchable collection and the low-overhead source collection. Gateway SQLite metadata is reported separately. Docker image layers and unused volume capacity are excluded.

## Fresh 500,000-document run

The containerized benchmark used the repository's standard profile, 500,000 generated products, 5,000-document bulk batches, 5,000 updates, and the two-collection gateway projection. The final mounted-directory measurements were:

| System | Storage | Size |
|---|---|---:|
| Native Elasticsearch | data directory | 16,500 KiB / 16.1 MiB |
| Gateway | Qdrant searchable collection | 502,064 KiB / 490.3 MiB |
| Gateway | Qdrant durable source collection | 209,088 KiB / 204.2 MiB |
| Gateway | gateway metadata | 24 KiB |
| Gateway total | searchable + source + metadata | 711,176 KiB / 694.5 MiB |

The complete gateway footprint was therefore approximately 43.1× the native Elasticsearch data directory in this run. This is not a defect to hide: Qdrant's segment and sparse-vector representation has a much larger fixed and per-document footprint for this small product schema, while Elasticsearch's on-disk representation is highly compact. The gateway's source projection is still useful for write-path isolation and stateless reads, but it is not currently a disk-saving strategy.

The same run measured 46.689 seconds versus 105.141 seconds for bulk setup, 191.52 ms versus 986.63 ms search P95 at 50 clients, and 522.68 versus 95.60 requests/second for the mixed workload. Single-document updates remained slower on the gateway. Both paths returned full source responses, but the top-ranked IDs were not identical.

## Payload minimization iteration

The projection layout was tightened after inspecting the payloads. The searchable collection no longer stores the full top-level source: it stores only the ID, index name, and mapped non-text fields needed by filters, sorts, and aggregations. The durable source collection stores only the ID, index name, and `_source`. Text remains in the sparse representation, and response hydration is unchanged.

A fresh 10,000-document run with the same benchmark client measured:

| Gateway layout | Qdrant data | Bulk setup | Single updates |
|---|---:|---:|---:|
| Two collections, minimized payloads | 15.9 MiB | 1.083 s | 1.210 s |
| One collection, source payload plus mapped filters | 13.6 MiB | 0.842 s | 0.814 s |

The one-collection profile is smaller for this tiny schema because it avoids the second collection's fixed segment overhead, but it gives up the source/search write separation. The minimized two-collection profile passed the same filter, GET, update, delete, source-response, and mixed-workload checks. The gateway now applies the same payload de-duplication principle to one-collection mode as well: only mapped non-text fields are mirrored beside `_source`. It is the better default when the goal is high read/mixed throughput and asynchronous projection updates; one collection remains a sensible option when disk footprint and update simplicity dominate.

The final 50,000-document scale check used the same client, 1,000 updates, 500 mixed requests, and 20 concurrent search clients:

| Profile | Qdrant data | Bulk setup | Updates | Search P95 | Mixed throughput |
|---|---:|---:|---:|---:|---:|
| One collection | 60.6 MiB | 4.56 s | 1.88 s | 102.5 ms | 302 req/s |
| Two collections, minimized payloads | 73.8 MiB | 6.72 s | 2.81 s | 68.0 ms | 353 req/s |

This confirms the expected trade-off at a larger corpus: one collection saves about 18% of Qdrant storage and has a cheaper write path, while two collections deliver about 34% lower search P95 and 17% higher mixed throughput in this run.

This result also clarifies the tuning boundary: `on_disk` vectors and `m=0` are useful for keeping the source projection lightweight in memory, but they do not make Qdrant's segment representation smaller than Elasticsearch for this corpus. `on_disk_payload` can reduce resident memory pressure, but it is not expected to reduce durable bytes and may increase read latency. Payload indexes should therefore be created only for fields the application actually filters, sorts, or aggregates on.

## Design implication

The current recommendation is workload-dependent:

- Choose the gateway projection when search latency, mixed read throughput, memory headroom, and stateless horizontal scaling matter more than raw disk footprint.
- Keep native Elasticsearch as the stronger baseline when storage cost is dominant or ranked-result equivalence is a hard requirement.
- Do not enable a durable source projection merely to reduce storage. Enable it to isolate sparse-index work from source reads and to support asynchronous/eventual update strategies.
- For production sizing, repeat after Qdrant compaction and with the actual schema, payload indexes, replication factor, and retention policy. Update-heavy workloads must include post-compaction measurements because segment churn can temporarily inflate the footprint.

The reproducible collector is [benchmark/disk_usage.sh](https://dstockton.github.io/benchmark/disk_usage.sh). It measures mounted directories while the stack is running, so the result is tied to the actual persisted corpus rather than container limits or image size.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-23-qdrant-1-20-gateway-upgrade.md

# Qdrant 1.20.0 through the gateway: almost identical read performance and an auth upgrade check

Qdrant [released 1.20.0](https://github.com/qdrant/qdrant/releases/tag/v1.20.0) on October 22. The proposed gateway stack update advances its development, example, test, and benchmark Qdrant images from 1.19.2 to 1.20.0. This article describes a candidate change under review; deployment stacks gain the new pin when the pull request is merged.

The release adds Qdrant-side capabilities including BM25 scoring for full-text payload indexes, TurboQuant variants, per-element payload indexes, and a code tokenizer, as well as fixes and internal changes. These are upstream Qdrant features; this gateway benchmark exercises its existing Elasticsearch-compatible match search and document GET routes, and does not establish that every new Qdrant feature is exposed through the gateway.

The proposed gateway lockfile also patches `rustls-webpki` from 0.103.15 to 0.103.16 and Tokio from 1.54.0 to 1.54.1, addressing [RUSTSEC-2026-0382](https://rustsec.org/advisories/RUSTSEC-2026-0382.html) and [RUSTSEC-2026-0384](https://rustsec.org/advisories/RUSTSEC-2026-0384.html). The measurements below use that patched gateway build.

## Paired gateway result

For the tested warm read workload, **1.20.0 performed almost identically to 1.19.2**. Two A-B-B-A passes produced four rounds per version. Each round built a fresh 10,000-document index through the same gateway image, then sent 5,000 match searches and 5,000 document GETs through its HTTP API with eight workers, after 40 warmups per operation. Latency measures the complete HTTP round trip through response reading; throughput also includes client JSON decoding and correctness checks. All rounds returned the expected 10 search hits, document source fields, and exact count. Qdrant had a 1.5 CPU and 2 GiB cap; the gateway had a 0.5 CPU and 512 MiB cap. Containers used separate temporary storage. No pre-existing volume was opened.

| Gateway operation; median of four round metrics | Qdrant 1.19.2 | Qdrant 1.20.0 |
| --- | ---: | ---: |
| Match search p50 | 1.26 ms | 1.25 ms |
| Match search p95 | 1.99 ms | 1.90 ms |
| Match search throughput | 4,714 req/s | 4,744 req/s |
| Document GET p50 | 0.79 ms | 0.80 ms |
| Document GET p95 | 1.12 ms | 1.14 ms |
| Document GET throughput | 8,418 req/s | 8,209 req/s |

Match search p95 was 4.6% lower on 1.20.0 in this run. Search throughput differed by about +0.6%, while GET throughput was about -2.5%; GET p95 was 1.9% higher. These small mixed changes support the overall “almost identical” description, with a measured improvement in this search tail latency rather than a broad speedup claim. This is a small, warm, single-node synthetic corpus on one Docker Desktop host, not a large-scale relevance, indexing, authenticated-cluster, or production latency study. The [benchmark script](https://github.com/dstockton/qdrant-es-gateway/blob/main/benchmark/qdrant-version-compare.py) and [first](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/qdrant-1-20-gateway-2026-10-23.json) and [second](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/qdrant-1-20-gateway-repeat-2026-10-23.json) per-round results and latency histograms allow a rerun and inspection once this change is merged.

To reproduce on a machine with Docker and Python 3, build the gateway once and run the isolated comparison twice:

```sh
docker pull qdrant/qdrant:v1.19.2
docker pull qdrant/qdrant:v1.20.0
docker build --pull -t qeg-gateway:qdrant-1-20-compare .
OUTPUT=first.json python3 benchmark/qdrant-version-compare.py
OUTPUT=second.json python3 benchmark/qdrant-version-compare.py
python3 benchmark/qdrant-upgrade-smoke.py
```

The scripts create uniquely named temporary containers and storage and do not attach existing Docker volumes. A separate disposable 1.19.2-to-1.20.0 storage test retained a point and its payload after restart; its exact count remained one. That smoke test establishes only this small upgrade path, not recovery or rolling-cluster behavior.

## Upgrade existing storage and authenticated clusters

Qdrant requires the latest patch of **each consecutive minor release** when opening an existing data directory. Back up first. For a volume last opened by 1.15.x, the path documented in the [migration guide](https://dstockton.github.io/qdrant-es-gateway/migration/) is 1.15.5 → 1.16.3 → 1.17.1 → 1.18.3 → 1.19.2 → 1.20.0, waiting for each version to become healthy. Upgrade replica clusters one node at a time and wait for recovery at each step. Follow Qdrant's [upgrade instructions](https://qdrant.tech/documentation/upgrades/) and retain the pre-upgrade backup because storage migrations are not reversible.

Version 1.20.0 also enables `service.enforce_internal_auth` by default when an API key is configured. Clusters configured with only `read_only_api_key` need an `api_key` for peer traffic, or an explicit decision to disable internal authentication. External tools using the internal port need an `api-key` header. Qdrant also calls out clusters configured only with `alt_api_key` on 1.19.0/1.19.1: they must reach 1.19.2 before 1.20.0. Check the [upstream release notes](https://github.com/qdrant/qdrant/releases/tag/v1.20.0) and [upgrade guide](https://qdrant.tech/documentation/upgrades/) against your deployment before switching images.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-20-script-fields-cannot-be-empty-promises.md

# Script fields cannot be empty promises

The gateway accepted search bodies containing `script_fields`, did not execute the scripts, and returned ordinary hits without the requested `fields` object. That is a false success for clients that calculate display values or other per-hit data at search time.

A focused probe used the same one-document mapping, script, and request against separate Elasticsearch 8.15.0 and OpenSearch 3.9.0 containers. With `_source: false`, both engines evaluated `doc["price"].value + params.increment` and returned `fields.next_price: [43]`. Both omitted `fields` when `script_fields` was an empty object. The engines document this behavior independently in the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and [OpenSearch script-field guide](https://docs.opensearch.org/latest/scripting/accessing-fields/#search-and-aggregation-scripts).

The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-8.15-opensearch-3.9-script-fields-2026-10-20.json) records the images, versions, fixture, selected hit fields, reproduction sequence, and limitations. This establishes request/response behavior for the two recorded versions; it is not a script-performance or sandbox-security benchmark.

The gateway now rejects every non-empty `script_fields` object before index lookup or Qdrant access, including in each `_msearch` subrequest. An empty object remains an accepted no-op, and malformed non-object values fail validation. Searches that omit `script_fields` are unchanged.

```console
cargo test invalid_search_options_fail_before_qdrant_work
```

Native support would require a deliberate scripting language, execution limits, field-access semantics, error contract, and security model. Copying a guessed value from `_source` would not execute the requested program and would be less honest than a structured compatibility error.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-19-hit-concurrency-metadata-cannot-be-invented.md

# Hit concurrency metadata cannot be invented

The gateway accepted `version: true` and `seq_no_primary_term: true` in search bodies but omitted the requested per-hit metadata. That is more serious than a cosmetic response difference: native clients can use sequence numbers and primary terms as optimistic-concurrency tokens, while the gateway's approximate write fields do not carry Elasticsearch or OpenSearch's shard history.

A focused probe used the same one-document index in separate Elasticsearch 8.15.0 and OpenSearch 3.9.0 containers. After creating and then updating the document once, both engines returned `_version: 2`, `_seq_no: 1`, and `_primary_term: 1` on the search hit when the two flags were enabled. Both omitted all three fields when the flags were explicitly `false`. The [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/8.19/search-search.html) and [OpenSearch Search API](https://docs.opensearch.org/latest/api-reference/search-apis/search/) describe the same response controls.

The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-8.15-opensearch-3.9-hit-concurrency-metadata-2026-10-19.json) records the images, corpus, update sequence, selected response fields, and limitations. This is request/response compatibility evidence, not a test of concurrent writers or token enforcement.

The gateway now rejects either flag when `true`, before index lookup or Qdrant access, including in each `_msearch` subrequest. Explicit `false` remains accepted. Non-boolean values fail gateway validation instead of relying on the string coercion observed in the two probed native versions.

```console
cargo test invalid_search_options_fail_before_qdrant_work
```

Ordinary searches are unchanged. Supporting these flags later requires a real concurrency model whose tokens can be honored by conditional writes; copying the gateway's fixed response counters into hits would make unsafe metadata look authoritative.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-19-doc-values-cannot-silently-disappear.md

# Requested doc values cannot silently disappear

The gateway accepted search bodies containing `docvalue_fields` but silently returned hits without the requested `fields` object. This can break clients that deliberately read keyword, numeric, or date values from doc values instead of `_source`; preserving `_source` does not satisfy that response contract.

A focused probe used the same one-document mapping and requests in separate Elasticsearch 8.15.0 and OpenSearch 3.9.0 containers. Both returned `fields.price: [42]` and `fields.sku: ["A-1"]` for body `docvalue_fields: ["price", "sku"]` and for the URL form `docvalue_fields=price,sku`. Both omitted `fields` for an empty body array. The [Elasticsearch field retrieval reference](https://www.elastic.co/guide/en/elasticsearch/reference/8.19/search-fields.html#docvalue-fields) and [OpenSearch Search API](https://docs.opensearch.org/latest/api-reference/search-apis/search/) describe doc-value retrieval independently.

The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-8.15-opensearch-3.9-docvalue-fields-2026-10-19.json) records the images, versions, fixture, selected hit fields, reproduction sequence, and limitations. This establishes request/response behavior for the two recorded versions; it is not a doc-value storage or latency benchmark.

The gateway now rejects a non-empty body `docvalue_fields` array or the corresponding URL parameter before index lookup or Qdrant access, including in each `_msearch` subrequest. An empty body array remains an accepted no-op, and malformed non-array body values fail validation. Ordinary `_source` responses are unchanged.

```console
cargo test invalid_search_options_fail_before_qdrant_work
```

Native support would require a defined mapping and formatting contract for every returned field, including multivalued fields and date/numeric formats. Copying arbitrary `_source` values into `fields` would falsely label source data as doc values and would miss those semantics.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-19-agriculture-irrigation-inspection.md

# Which dry plots need an irrigation inspection?

A market garden's morning queue should include dry plots with recent sensor readings. An old low reading is a reason to check the sensor, not evidence to irrigate. These six synthetic snapshots deliberately include a threshold reading, stale data, a closed plot, and a missing reading age. The cutoffs are illustrative, not agronomic advice.

Use fresh `field_note_irrigation` indexes on running Elasticsearch, OpenSearch, and gateway endpoints. Curl is the same client throughout: change only `ES` to select the server. Create the mapping and load the snapshots:

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_irrigation" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"plot":{"type":"keyword"},"crop":{"type":"keyword"},"state":{"type":"keyword"},"moisture_percent":{"type":"integer"},"reading_age_minutes":{"type":"integer"}}}}'
curl -fsS -X POST "$ES/field_note_irrigation/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"plot-101"}}
{"plot":"North tunnel","crop":"tomato","state":"open","moisture_percent":18,"reading_age_minutes":10}
{"index":{"_id":"plot-102"}}
{"plot":"East beds","crop":"lettuce","state":"open","moisture_percent":19,"reading_age_minutes":30}
{"index":{"_id":"plot-103"}}
{"plot":"West beds","crop":"lettuce","state":"open","moisture_percent":20,"reading_age_minutes":5}
{"index":{"_id":"plot-104"}}
{"plot":"South tunnel","crop":"tomato","state":"open","moisture_percent":12,"reading_age_minutes":31}
{"index":{"_id":"plot-105"}}
{"plot":"Trial beds","crop":"lettuce","state":"closed","moisture_percent":15,"reading_age_minutes":8}
{"index":{"_id":"plot-106"}}
{"plot":"Herb beds","crop":"basil","state":"open","moisture_percent":16}
NDJSON
sleep 5
```

The same five-second pause follows each write phase on all three servers; this is a settling allowance, not a refresh guarantee. Reading ages are fixed snapshots for a deterministic example; a real ingestion process must refresh them.

Find open plots below 20% moisture with a reading no more than 30 minutes old:

```sh
curl -fsS -X POST "$ES/field_note_irrigation/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"filter":[{"term":{"state":"open"}},{"range":{"moisture_percent":{"lt":20}}},{"range":{"reading_age_minutes":{"lte":30}}}]}}}'
```

Expect exactly `plot-101` and `plot-102`, complete sources, and total `2`; order does not matter. The 20% boundary, 31-minute reading, closed plot, and missing age must all be excluded. Count crops within that same queue:

```sh
curl -fsS -X POST "$ES/field_note_irrigation/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"state":"open"}},{"range":{"moisture_percent":{"lt":20}}},{"range":{"reading_age_minutes":{"lte":30}}}]}},"aggs":{"crops":{"terms":{"field":"crop","size":10}}}}'
```

Expect `tomato: 1` and `lettuce: 1`, with no hits. These are plot counts, not water requirements. After inspecting the North tunnel, remove it from the queue without overwriting its sensor snapshot:

```sh
curl -fsS -X POST "$ES/field_note_irrigation/_update/plot-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"state":"inspected"}}'
sleep 5
curl -fsS -X GET "$ES/field_note_irrigation/_doc/plot-101"
```

The update returns `updated`. The read retains the plot, crop, 18% moisture, and 10-minute reading age, with `state: "inspected"`. Repeat both search requests: only `plot-102` remains, total `1`, and the only bucket is `lettuce: 1`.

The [fixture](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/fixtures/agriculture.json), [replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/agriculture.py), and [recorded evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/agriculture-2026-10-19.json) verify statuses, bulk IDs/results, every source, exact search IDs/totals, crop counts, and inspection exclusion. The run passed 105/105 checks on native Elasticsearch 8.15.0, native OpenSearch 3.9.0, and gateway 0.2.0 with Qdrant 1.15.5, using identical requests and embedded sources with synchronous writes. No OpenSearch-specific behavior or assertion difference was observed. The article's curl requests were also executed on all three endpoints. No gateway change or performance claim is involved.

Boundary: [atomic concurrent updates](https://dstockton.github.io/qdrant-es-gateway/compatibility/) are unsupported; this queue does not control irrigation equipment.

Remove the example index:

```sh
curl -fsS -X DELETE "$ES/field_note_irrigation"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-18-query-profiles-cannot-be-empty.md

# Query profiles cannot be empty

The gateway accepted `profile: true` in a search body but ignored it. That turns a diagnostic request into an ordinary search while still returning HTTP 200. It is especially misleading here because the gateway executes against Qdrant, not Lucene: a native-looking query and collector timing tree cannot be inferred from the returned hits.

A focused probe used the same mapped fields, three-document corpus, and match query against separate Elasticsearch 8.15.0 and OpenSearch 3.9.0 containers. Both returned a top-level `profile` object with one shard, a `TermQuery` timing breakdown, and collector data. The observed collectors differed: Elasticsearch reported `QueryPhaseCollector`, while OpenSearch reported `TopScoreDocCollector`. For this small compact response, enabling profiling increased the body from 419 to 2,124 bytes in Elasticsearch and from 420 to 1,758 bytes in OpenSearch. Those byte counts illustrate response shaping only; they are not a latency, CPU, allocation, or throughput benchmark.

The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-8.15-opensearch-3.9-search-profile-2026-10-18.json) records the images, corpus, selected response fields, reproduction sequence, and limitations. The [Elasticsearch search profiling documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html) and [OpenSearch Profile API documentation](https://docs.opensearch.org/latest/api-reference/search-apis/profile/) independently describe their profiling responses and overhead cautions.

The gateway now rejects `profile: true` before index lookup or Qdrant access, including in each `_msearch` subrequest. Explicit `profile: false` remains accepted, and non-boolean JSON values fail validation instead of being ignored. The focused regression proves the rejected requests perform no upstream work:

```console
cargo test invalid_search_options_fail_before_qdrant_work
```

Existing searches that omit `profile` or set it to `false` are unchanged. Implementing gateway-native profiling later would require an explicitly different schema that reports gateway and Qdrant phases honestly rather than borrowing Lucene component names.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-18-production-stack-refresh.md

# Refreshing the production stack without opening old volumes

The development and benchmark stack now uses Rust 1.99.1, Qdrant 1.19.2, Valkey 9.1.2, HAProxy 3.4.7, Python 3.15.0, and Elasticsearch 9.5.5. The Rust client layer also moves to reqwest 0.13 and redis-rs 1.7. This is a maintenance update: the existing client and demo request shapes are unchanged.

Fresh ephemeral containers provide the evidence for this update. Elasticsearch reported version **9.5.5** and a green single-node cluster; Qdrant reported **1.19.2** and passed the gateway's live create, bulk, search, and delete smoke path; all seven shared-catalog tests passed against Valkey **9.1.2**. The gateway image built from an empty cache, the benchmark image built, all three Compose files parsed, 92 isolated Rust tests and one live integration test passed, and formatting, Clippy, and `cargo audit` were clean. A refreshed Trivy database found no fixed HIGH or CRITICAL vulnerabilities in the gateway image. The [sanitized evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/production-stack-refresh-2026-10-18.json) records the versions, results, and limitations.

The important operational result is a boundary, not a shortcut. Elasticsearch 9.5.5 reports minimum wire compatibility with 8.19.0, so an 8.15 benchmark volume must first follow Elastic's [upgrade preparation](https://www.elastic.co/docs/deploy-manage/upgrade/prepare-to-upgrade) through the latest 8.19 patch. Qdrant documents consecutive minor upgrades, so a 1.15 volume must be backed up and opened by the latest 1.15, 1.16, 1.17, and 1.18 patches before 1.19.2; the [migration guide](https://dstockton.github.io/qdrant-es-gateway/migration/) records that sequence. No existing volume was opened during validation.

This run did **not** change or exercise OpenSearch. The Elasticsearch 9.5.5 result is an Elasticsearch-specific benchmark-environment check, not evidence that OpenSearch behaves the same way. It is also not a compatibility replay or throughput benchmark. Reproduce the bounded startup checks on fresh storage with:

```sh
docker run --rm -d --name qeg-es-check \
  -p 127.0.0.1:19200:9200 \
  -e discovery.type=single-node -e xpack.security.enabled=false \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.5
curl -fsS http://127.0.0.1:19200/
curl -fsS http://127.0.0.1:19200/_cluster/health

docker run --rm -d --name qeg-qdrant-check \
  -p 127.0.0.1:16333:6333 qdrant/qdrant:v1.19.2
curl -fsS http://127.0.0.1:16333/
curl -fsS http://127.0.0.1:16333/readyz
```

Stop those temporary containers after the check; do not attach old data volumes to these commands.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-17-score-explanations-must-explain-the-score.md

# Score explanations must explain the score

The gateway accepted `explain: true` in a search body but ignored it. That is not merely missing decoration: clients use each hit's `_explanation` tree to debug ranking, and the gateway's score comes from Qdrant rather than Lucene. Fabricating a Lucene-shaped explanation or silently omitting it would both misdescribe how the result was ranked.

A focused probe used the same mapped text field, one-document corpus, and match query against separate Elasticsearch 8.15.0 and OpenSearch 3.9.0 containers. Both added `_shard`, `_node`, and a nested `_explanation` to the matching hit. The engines did not produce identical output: Elasticsearch reported score `0.2876821` and included an explicit boost component, while OpenSearch reported `0.13076457` without that component in this version pair. The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-8.15-opensearch-3.9-search-explain-2026-10-17.json) records the images, request, selected response fields, reproduction sequence, and limitations. The [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and [OpenSearch Search API](https://docs.opensearch.org/latest/api-reference/search-apis/search/) independently document the per-hit explanation contract.

The gateway now rejects `explain: true` before index lookup or Qdrant access, including in each `_msearch` subrequest. Explicit `explain: false` remains accepted, and non-boolean values fail validation instead of being ignored. The focused regression proves the rejected requests perform no upstream work:

```console
cargo test invalid_search_options_fail_before_qdrant_work
```

This is request/response compatibility evidence, not a relevance or performance benchmark. The score difference should not be generalized to all Elasticsearch and OpenSearch versions, mappings, or queries. Existing searches that omit `explain` or set it to `false` are unchanged.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-17-highlight-snippets-cannot-vanish.md

# Highlight snippets cannot vanish

The gateway accepted a search `highlight` object but ignored it. That changes the response contract: clients commonly request snippets to show where a result matched, and a successful response without those snippets can render an empty preview or force an unsafe fallback to the complete source text.

A focused probe used the same one-document corpus and request against separate Elasticsearch 8.15.0 and OpenSearch 3.9.0 containers. Both returned HTTP 200 and added `{"title":["wireless <em>headphones</em> for travel"]}` under the matching hit's `highlight` field. This independently confirms the contract for both targets; it does not assume OpenSearch behavior from Elasticsearch. The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-8.15-opensearch-3.9-highlight-2026-10-17.json) records the images, versions, fixture, request, result, reproduction sequence, and limitations. The [Elasticsearch](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/highlighting) and [OpenSearch](https://docs.opensearch.org/latest/search-plugins/searching-data/highlight/) references likewise describe the per-hit response field.

The gateway now rejects `highlight` before index lookup or Qdrant access. Implementing it honestly would require analyzer-aware match offsets, fragment boundaries and scoring, multi-field rules, source/stored-field behavior, and engine-specific option validation. Returning ordinary hits while dropping the request is not a safe approximation. The focused regression proves both `_search` and each `_msearch` subrequest inherit the same fail-closed path without sending an upstream request:

```console
cargo test invalid_search_options_fail_before_qdrant_work
```

This is request-level compatibility evidence, not a performance result. It covers one ordinary text field with the default highlighter; it does not claim identical Elasticsearch and OpenSearch behavior for other highlighters, analyzers, fragments, or versions. Existing searches without `highlight` are unchanged.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-17-energy-dispatch-shortlist.md

# Which batteries belong on the dispatch shortlist?

An energy cooperative needs northern battery sites marked ready with at least 100 kWh available. Six synthetic snapshots include a site exactly at the threshold, one just below it, a southern site, a held site, and an offline site. Operator facets count sites, not stored energy.

Use a fresh `field_note_energy` index on running servers. Curl is the same client for native Elasticsearch, native OpenSearch, and qdrant-es-gateway: change only `ES` to the target endpoint URL.

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_energy" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"site":{"type":"keyword"},"region":{"type":"keyword"},"operator":{"type":"keyword"},"state":{"type":"keyword"},"available_kwh":{"type":"integer"}}}}'
curl -fsS -X POST "$ES/field_note_energy/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"battery-101"}}
{"site":"Moor Lane","region":"north","operator":"Aster","state":"ready","available_kwh":120}
{"index":{"_id":"battery-102"}}
{"site":"Quarry Farm","region":"north","operator":"Birch","state":"ready","available_kwh":100}
{"index":{"_id":"battery-103"}}
{"site":"Mill Yard","region":"north","operator":"Aster","state":"ready","available_kwh":99}
{"index":{"_id":"battery-104"}}
{"site":"South Meadow","region":"south","operator":"Birch","state":"ready","available_kwh":180}
{"index":{"_id":"battery-105"}}
{"site":"Hill Barn","region":"north","operator":"Cedar","state":"hold","available_kwh":200}
{"index":{"_id":"battery-106"}}
{"site":"Canal Depot","region":"north","operator":"Cedar","state":"offline","available_kwh":250}
NDJSON
sleep 5
```

Use the same five-second pause after bulk and update on all three servers; this tests settled data, not an immediate refresh guarantee. Find the eligible sites:

```sh
curl -fsS -X POST "$ES/field_note_energy/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"filter":[{"term":{"region":"north"}},{"term":{"state":"ready"}},{"range":{"available_kwh":{"gte":100}}}]}}}'
```

Expect `battery-101` and `battery-102`, total `2`, with complete sources; hit order is immaterial. The 100 kWh boundary is inclusive. Count operators over exactly that shortlist:

```sh
curl -fsS -X POST "$ES/field_note_energy/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"region":"north"}},{"term":{"state":"ready"}},{"range":{"available_kwh":{"gte":100}}}]}},"aggs":{"operators":{"terms":{"field":"operator","size":10}}}}'
```

Expect `Aster: 1` and `Birch: 1`, with no hits. Now place Moor Lane on hold without changing its reported capacity:

```sh
curl -fsS -X POST "$ES/field_note_energy/_update/battery-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"state":"hold"}}'
sleep 5
curl -fsS "$ES/field_note_energy/_doc/battery-101"
```

The update returns `updated`; the read shows `state: "hold"` and preserves `site`, `region`, `operator`, and `available_kwh: 120`. Repeat both search requests: only `battery-102` remains, total `1`, with `Birch: 1`. A held site's sufficient energy must not keep it eligible. These snapshots support planning; they do not issue dispatch commands.

Boundary: atomic concurrent energy reservations remain [unsupported](https://dstockton.github.io/qdrant-es-gateway/compatibility/).

The [fixture](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/fixtures/energy.json) and [replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/energy.py) check statuses, bulk IDs/results, every source, exact search IDs/totals, operator counts, and hold exclusion. The [recorded run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/energy-2026-10-17.json) passed **105/105** checks on Elasticsearch **8.15.0**, OpenSearch **3.9.0**, and gateway **0.2.0** with Qdrant **1.15.5**, using identical requests and embedded sources with synchronous writes. No OpenSearch-specific behavior or difference in these assertions was observed. The [article curl run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/energy-curl-2026-10-17.json) independently checked all nine responses per engine. No gateway code change or performance claim is involved.

Remove the example index when finished:

```sh
curl -fsS -X DELETE "$ES/field_note_energy"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-17-aws-retail-demo-store-compatibility.md

# AWS Retail Demo Store: compatible search requests, different trade-offs

The [AWS Retail Demo Store](https://github.com/aws-samples/retail-demo-store) is a useful reality check because its search service talks to OpenSearch through the ordinary `opensearch-py` API. Its product search uses `dis_max`, `match_bool_prefix`, category collapse with named `inner_hits`, and `_source: false`. Its similar-products route uses `more_like_this`.

I replayed those request bodies unchanged against native OpenSearch 3.9.0 and this gateway backed by Qdrant 1.19.2. The replay created a settings-only `products` index and wrote individual products, just as the demo's local indexing script does. Both endpoints returned the fields the application consumes, including product IDs, named category hits, and similar-product results that exclude the source product.

## What passed

| Check | OpenSearch | Gateway + Qdrant |
| --- | ---: | ---: |
| Index, search, and similar-product HTTP contract | Pass | Pass |
| Repeated search p50 at 1K | 5.75 ms | 1.37 ms |
| Repeated search p95 at 1K | 7.10 ms | 2.36 ms |
| Repeated search p50 at 1M | 30.08 ms | 1.63 ms |
| Repeated search p95 at 1M | 33.62 ms | 2.85 ms |
| First similar-product request at 1M | 135.15 ms | 33.04 ms |

The gateway also needed two small compatibility fixes exposed by this replay: settings-only indexes now route dynamically mapped text fields through the catch-all lexical vector, and a missing Qdrant point is treated like a missing Elasticsearch document. The latter matters when `more_like_this` looks up its source document.

## The resource trade-off

These are fresh local Docker volumes, after loading unique generated product names and descriptions and while the indexes were retained. They are footprint snapshots, not a production sizing or cloud-billing benchmark.

| Corpus | Service | CPU snapshot | Memory snapshot | Data volume |
| --- | --- | ---: | ---: | ---: |
| 1K | OpenSearch 3.9.0 | 3.83% | 1.090 GiB | 630 KiB |
| 1K | Gateway + Qdrant 1.19.2 | 1.31% combined | 310 MiB combined | 419 MB combined |
| 1M | OpenSearch 3.9.0 | 0.92% | 1.241 GiB | 75.2 MB |
| 1M | Gateway + Qdrant 1.19.2 | 0.08% combined | 581 MiB combined | 876 MB combined |

The negative trade-off is real: Qdrant used more disk. At 1M documents, the gateway-plus-Qdrant volume was about 11.6× the OpenSearch volume. The gateway process itself remained small; Qdrant carries the searchable representation and sparse index. OpenSearch's JVM used more memory in these snapshots, but CPU and memory readings are instantaneous and environment-dependent.

The disk did not grow linearly in this experiment. Moving from 1K to 1M documents increased the OpenSearch volume about 119× and the Qdrant volume about 2.1×. That is evidence of substantial fixed index overhead at the 1K point, not a promise that Qdrant storage is always sublinear. The generator uses five product families with unique names and descriptions; a production catalogue with more varied fields, longer text, replicas, retention, or different sparse-index settings can have a different slope.

I also retried the same 10M-document bulk run after a complete Docker cleanup,
with about 27 GiB free. Qdrant reached roughly 9.57M documents, but the
process was killed for exceeding available memory before the final search could
complete. That is a capacity limit of this test host, not a failed
compatibility assertion, so there is no honest 10M performance number to
publish from this machine. A completed 10M comparison needs more memory.

## What did not become identical

Passing the application contract does not make the two engines rank identically. The native result produced two category groups for the fixture; the gateway produced one bounded candidate group. Scores and similar-product ordering also differ because Qdrant's sparse retrieval is not Lucene/OpenSearch scoring. Refresh timing and the gateway's bounded candidate window remain explicit trade-offs.

So the honest conclusion is narrow: the Retail Demo Store's product-search and similar-product request shapes are a good fit for the gateway, with no application-side URL or response-handling changes in this replay. If exact OpenSearch ranking or category diversity is part of the product contract, validate that separately before migrating.

Reproduce the check with:

```sh
python3 validation/retail_demo.py \
  --native http://localhost:19311 \
  --gateway http://localhost:19313 \
  --docs 1000000 \
  --workers 32 \
  --output validation/evidence/retail-demo-2026-10-17-1m-clean.json
```

The request shapes come from the demo's [search service](https://github.com/aws-samples/retail-demo-store/blob/master/src/search/src/search-service/app.py), and its settings-only indexing flow is shown in [`local_index_products.py`](https://github.com/aws-samples/retail-demo-store/blob/master/src/search/local_index_products.py). The clean 1K and 1M reports are [`retail-demo-2026-10-17-1k-clean.json`](https://dstockton.github.io/validation/evidence/retail-demo-2026-10-17-1k-clean.json) and [`retail-demo-2026-10-17-1m-clean.json`](https://dstockton.github.io/validation/evidence/retail-demo-2026-10-17-1m-clean.json).


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-16-search-controls-cannot-be-decoration.md

# Search controls cannot be decoration

The gateway accepted `min_score`, `terminate_after`, and per-search `timeout` fields but did not apply them. That is more dangerous than a clear unsupported-feature response: a caller could ask to remove low-scoring hits, stop collection early, or return within a deadline and instead receive an ordinary complete page.

Separate ephemeral-container probes exercised Elasticsearch 9.5.5 and OpenSearch 3.9.0 with the same one-shard, three-document corpus. `min_score: 2` removed all three `match_all` hits on both engines. `terminate_after: 1` returned one hit with `terminated_early: true` on both, but Elasticsearch reported a total of three while OpenSearch reported one. Both engines accepted `timeout: "1s"`, returned `timed_out: false` for this small request, and rejected the invalid value `"banana"` with HTTP 400. The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-9.5.5-opensearch-3.9-search-controls-2026-10-16.json) records the versions, images, requests, and observed fields.

The gateway now rejects all three controls before index lookup or Qdrant access. Supporting them requires explicit decisions about score comparability, partial-result behavior, and the Elasticsearch/OpenSearch early-termination total difference; silently choosing one engine's shape would misrepresent the other. The focused regression includes valid-looking values for every field and proves that none reaches the mocked Qdrant endpoint:

```console
cargo test invalid_search_options_fail_before_qdrant_work
```

This is request/response evidence, not a latency benchmark. The one-second timeout did not expire, and the small one-shard corpus does not characterize distributed early termination. Existing searches that omit these fields are unchanged; `_msearch` inherits the same fail-closed validation for each query.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-16-empty-alias-batches-are-not-success.md

# An empty alias batch is not success

The aliases API is an administrative write boundary. The gateway previously treated `{"actions":[]}` as a valid request: it opened and committed an empty catalogue transaction, then returned `{"acknowledged":true}`. A client bug that accidentally dropped every action could therefore look like a completed routing change.

Separate clean-container probes sent the same empty action list and missing action field to Elasticsearch 9.5.5 and OpenSearch 3.9.0. Both engines returned HTTP 400 with `illegal_argument_exception` and the reason `No action specified` for both requests. The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-9.5.5-opensearch-3.9-empty-alias-actions-2026-10-16.json) records the exact engine versions, images, and observed fields.

The gateway now checks for at least one action before index lookup or the alias handler's catalogue transaction and returns the shared native error shape. In shared-catalog mode, the outer administrative wrapper still loads a snapshot and acquires then releases its claim on this validation error; it does not publish a changed snapshot. A focused regression covers both `{}` and `{"actions":[]}` and proves an existing alias target remains unchanged:

```console
cargo test empty_alias_batches_are_rejected_without_catalogue_changes
```

This is request-level compatibility evidence, not a performance benchmark. It does not claim that every malformed alias body has identical parsing errors: for example, non-array `actions` values continue to use the gateway's structured unsupported-request response. Supported non-empty batches and the documented Elasticsearch/OpenSearch mixed-action response difference are unchanged.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-testing-a-cache-before-building-it.md

# Testing a cache before building it

Would Valkey make this gateway fast enough to beat native search? Before adding
a production cache, we tested a narrower question: **is there enough latency
headroom in an ideal warm hit to justify the work?**

The answer is cautiously useful. A standalone synthetic probe returned aggregation
responses about **7–15% faster** than the faster native engine's median latency
across the corrected measurement rounds. Its aggregation p95 values were also
lower than both native engines in both rounds. This is a reason to investigate,
not a gateway performance win: the production gateway still trailed both native
engines in all ten measured workload/concurrency combinations.

### What the probe actually does

We captured 644 distinct, correctness-checked responses from the real gateway
using the public 10,000-document catalogue. A separate probe validates each
incoming request, hashes its identity and performs one Valkey Lua lookup. The
script checks synthetic control state and returns an already populated response.
The probe parses that response, updates `took` and serializes JSON normally.

Every request is a warm hit. There are no cache misses, changing documents,
invalidation, crash recovery or real writer coordination. Qdrant remains
provisioned but receives no reads on hits. The probe is deliberately named
`cache_probe` in the evidence, rather than “gateway + Qdrant”. It is not a
production feature, and its guard tests do not establish cache consistency.

The combined budget remains two CPUs and 4 GiB: probe 0.5 CPU/512 MiB, Valkey
0.25 CPU/256 MiB, and Qdrant 1.25 CPU/3328 MiB. Each native engine retains two
CPUs and 4 GiB. Normal production rounds instead allocate 0.5 CPU/512 MiB to the
gateway and 1.5 CPUs/3584 MiB to Qdrant. Native caching stays enabled.

### First, fix the measuring instrument

The initial probe run exposed a benchmark problem. For every filter-aggregation
response, the Python verifier recomputed the expected count by walking the entire
corpus. That work sat outside the nominal HTTP timer, but with concurrent clients
it still competed for Python's global interpreter lock and delayed other clients.
Trial throughput also included this verification work.

The corrected harness computes the same expected count once and retains the
strict response assertions. Correctness conclusions did not change. Historical
filter-aggregation timing rows need caution; we retain them with their original
validation profile instead of pooling old and new medians. This identified issue
concerns filter-count validation, not evidence that every historical workload was
invalid.

We then measured two corrected production rounds and two corrected probe rounds,
with native engines interleaved in each. Each profile passed **36,000 measured
response checks**. The earlier 36,000-response probe profile is separately retained.

### The available margin is small

The table gives each corrected probe round separately. Values are median trial
p50 milliseconds, shown as **Elasticsearch / OpenSearch / synthetic probe**.

| Aggregation | Clients | Round 1 | Round 2 |
|---|---:|---:|---:|
| Terms buckets | 1 | 0.360 / 0.347 / 0.307 | 0.362 / 0.354 / 0.315 |
| Two filter counts | 1 | 0.347 / 0.336 / 0.300 | 0.347 / 0.340 / 0.298 |
| Terms buckets | 8 | 0.888 / 0.840 / 0.783 | 0.815 / 0.822 / 0.738 |
| Two filter counts | 8 | 0.795 / 0.811 / 0.673 | 0.780 / 0.786 / 0.700 |

These are optimistic ceilings on potential performance, not a demonstrated
production advantage. Correct write fencing, epoch changes, miss publication and
recovery all have costs. Even a modest extra round trip or a lower hit rate could
erase the observed margin. A broader cache for document reads and sorted pages needs its own correctness
design and production measurements.

The [complete corrected probe summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-cache-ceiling/summary-probe-v2.json)
includes every participant and all five workloads, with p50, p95 and throughput
for both rounds. The [production summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-cache-ceiling/summary-production-v2.json)
shows the actual gateway alongside its own native controls. We have not combined
those separate profiles into an implementation speedup claim.

To reproduce the diagnostic, follow the
[standalone probe instructions](https://github.com/dstockton/qdrant-es-gateway/tree/main/benchmark/cache-upper-bound):
capture from the retained production image, build before timing, then measure
with `REQUESTS=200 TRIALS=3 WARMUP=40` and exact totals. Restore the production
image and original Qdrant allocation afterward. The
[evidence manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-cache-ceiling/manifest.json)
records all six raw rounds, separate summaries, image provenance, the fixture
hash and resource budgets.

Valkey now has a measured reason to remain under consideration. The remaining
question is whether a correct cache can preserve enough of this small margin.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-sorted-pages-look-past-first-candidates.md

# Find the next page across the whole catalogue

Sorting the first few candidates is not the same as finding the first few sorted
results. Likewise, asking for a page after price 20 cannot work reliably if the
gateway only fetched records priced below 20.

Filter-only searches with explicit field sorting now inspect every matching
Qdrant scroll page. The gateway keeps the best `from + size` candidates, drops
records before `search_after`, and continues until the scroll ends. Without
collapse, retained hit candidates are bounded by the requested window plus the
current upstream page.

```json
{"size":2,"sort":[{"rank":"asc"}],"search_after":[2]}
```

The six-document [comparison fixture](https://github.com/dstockton/qdrant-es-gateway/tree/main/validation/aggregations)
expects records 3 and 4 from that request. It sends the same request to
Elasticsearch and to the gateway. A deterministic regression also puts the
cheapest record on a later page and verifies that it wins the first sorted page.

`post_filter` now scans beyond the initial candidate window too. Matching records
on later pages can fill the requested page, and the hit count covers the complete
post-filtered set. Aggregations continue to see the query before `post_filter`.

Run `cargo test aggregation_scroll_visits_every_page_and_uses_search_projection_values`
to exercise those cases without a running database.

This preserves stateless request handling: no server-side cursor session or Redis
entry is required. The trade-off is scanning matching records for each request.
Use a unique final sort field for predictable cursor boundaries. These changes
apply to filter-only searches; ranked text pagination retains its documented
candidate limits. Collapse still needs retained group members, so its memory cost
is different. Concurrent writes can move records between pages because this is
not a point in time snapshot.

The container replay against Elasticsearch 8.15.0 and Qdrant 1.19.2 passed all four cases: complete aggregations, post-filtered hits, the first sorted page, and its cursor continuation. The [captured results](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/aggregations/results.json) include actual request timings. They establish correctness for this fixture, not a speed advantage.

A later [three-engine benchmark](https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-indexed-query-performance.md)
adds guarded indexed paths for compatible numeric sorting, keyword facets and
simple filter counts. The complete scan described here remains the fallback
when the gateway cannot prove those operations preserve the result.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-sharing-mappings-between-gateway-replicas.md

# Two gateways need one view of their mappings

Adding a second gateway sounds straightforward: start another process and send it traffic. But if each process has its own SQLite file, an alias created through one process is invisible to the other. A restart-safe local database does not automatically become shared metadata.

Qdrant ES Gateway now has an optional Redis/Valkey catalog for that deployment. Mappings, vector names and aliases live in one authoritative store. Each request reads its current committed catalog, so the next request can land on either replica without relying on a local metadata cache. Qdrant continues to store the documents and perform searches.

SQLite remains the default. We have not measured a SQLite bottleneck, and there is no reason to add another service to a single-gateway setup just to replace a small local catalog.

## What changes for an operator

For a new empty deployment, configure `CATALOG_REDIS_URL`, initialize the catalog once with `CATALOG_INITIALIZE=true`, then remove that initialization flag and start the other replicas. Use a persistent, non-evicting Redis/Valkey deployment. Existing SQLite installations need a deliberate migration; simply pointing them at an empty catalog will not discover their mappings.

Administrative changes have one owner at a time. After a successful change, the gateway publishes the metadata and releases ownership atomically within Redis/Valkey. If a gateway crashes while changing a Qdrant collection, it leaves its administrative claim in place. New requests fail closed until an operator reconciles the two stores.

That recovery step is intentional. A lock that expires after thirty seconds does not prove a slow Qdrant request stopped executing. Letting another gateway take over at that point could make the catalog less trustworthy.

## What this does—and what it costs

The shared catalog solves visibility and administrative ownership. It does not turn Qdrant and Redis/Valkey into one transaction, or make concurrent document updates atomic. Nor is this a performance claim: every request fetches a bounded metadata snapshot and builds an ephemeral SQLite view. Large catalogs and high request rates need measurement before adopting this approach.

The real-server test uses two independent clients to verify that metadata becomes visible after commit, concurrent administrators are refused, and a vanished owner does not silently lose its claim. These are correctness checks, not benchmark results.

For configuration, the test command, durability requirements and the recovery procedure, see [the shared catalog guide](https://dstockton.github.io/qdrant-es-gateway/shared-catalog/). Start with SQLite for one gateway; choose shared metadata when adding replicas creates a real ownership problem.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-shared-cache-warm-search-results.md

# Repeated searches get faster; fresh queries still cost more

The gateway now has a measured advantage for repeated, unchanged searches.
With its optional shared cache enabled, warm search medians were **12–30% lower
than the faster native engine** across our eight search/concurrency comparisons.
Point reads were close to parity, with only a 3–5% combined median advantage.
That advantage was not consistent in every round: in the second enabled round,
single-client point reads were about 2% slower than the faster native engine.

This is the actual gateway, not the earlier synthetic cache probe. It uses Qdrant
for documents and Valkey for shared cache entries, the authoritative catalog and
persistent write coordination. The trade-off is real: fresh queries and
write-followed-by-point-read cycles remained substantially slower than native
Elasticsearch and OpenSearch. This is useful for a read-heavy catalogue with
repeated queries, not a blanket search-engine win.

### Warm searches, same requests

We used Elasticsearch 8.15.0, OpenSearch 3.9.0 and Qdrant 1.19.2 with 10,000
synthetic catalogue documents. Every engine received the same requests and exact
total-hit requirements. Native request caches stayed at their defaults.

The table contains every warm workload. Values are median trial p50 milliseconds,
shown as **one client / eight clients**. All six trial medians from both
cache-enabled rounds contribute; we did not select the fastest trial.

| Operation | Elasticsearch | OpenSearch | Gateway + Qdrant + Valkey |
|---|---:|---:|---:|
| Read one document | 0.350 / 0.707 | 0.345 / 0.661 | 0.333 / 0.630 |
| Filtered, sorted page | 0.579 / 1.401 | 0.547 / 1.343 | 0.417 / 1.033 |
| Page after a sort cursor | 0.643 / 1.405 | 0.541 / 1.289 | 0.382 / 1.026 |
| Keyword terms buckets | 0.460 / 1.081 | 0.453 / 1.136 | 0.360 / 0.951 |
| Two filter counts | 0.416 / 0.981 | 0.409 / 1.057 | 0.339 / 0.800 |

Warm p95 values were also lower than both native engines in these ten cells.
The [complete enabled summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-shared-cache/summary-on.json)
includes p95, throughput and the range of trial medians.

We alternated **serving off, on, on, off**, keeping the same image, data and
resource caps. Write coordination remained enabled in both modes. With serving
off, the gateway's single-client sorted page was 1.362 ms; with serving on it was
0.382 ms. Those off-mode results include the shared-catalog and coordination
architecture, so they do not measure total overhead versus a gateway without
cache infrastructure. The [off-mode summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-shared-cache/summary-off.json)
retains every native control and gateway result too.

### Misses and writes change the answer

We separately tested unused cache keys and real writes. “Cold” here means a new
response-cache key on already loaded data, not cold operating-system caches or
storage. Values below are median trial p50 milliseconds with cache serving on.
Write rows measure the combined write and following read at one client.

| Separate workload | Elasticsearch | OpenSearch | Gateway + Qdrant + Valkey |
|---|---:|---:|---:|
| New sorted-filter query, one client | 0.582 | 0.522 | 2.085 |
| New sorted-filter query, eight clients | 1.446 | 1.344 | 5.454 |
| Write then read the document | 1.070 | 1.006 | 2.712 |
| Write then read filter counts | 3.964 | 4.439 | 3.273 |

Misses pay for the Qdrant query and cache publication. The gateway's cold-query
median increased from 1.567 to 2.085 ms at one client when serving was enabled.
At eight clients it increased from 3.993 to 5.454 ms; that enabled run's p95 was
45.499 ms, versus roughly 2.7–2.8 ms for native engines. Tail latency matters as
much as the warm median.

Point-write cycles omit `refresh` for every engine because native GET is
realtime. Filter-count cycles use `refresh=true` everywhere because the next
search requires immediate visibility. The gateway led that latter comparison,
but caching itself did not improve the write cycle: its median rose from
2.942 ms with serving off to 3.273 ms with serving on. Point-write cycles also
rose, from 2.310 to 2.712 ms. These separate off/on profiles were not an ABBA
experiment, so their differences should not be treated as isolated effect sizes.
The [complete mixed-workload summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-shared-cache/summary-mixed.json)
includes all participants, both modes and p95 values.

### The cache has to know when an answer is unsafe

Before a mutation starts, the gateway advances a shared generation and records a
persistent writer token. A cached answer is usable only in the same quiet
generation. A cold read captures that generation before loading metadata; if a
writer intervenes, its result cannot be published into the cache.

The token is released only after backend completion and catalog publication are
known complete. A timeout, cancellation or uncertain operation leaves a fence in
place. That sacrifices cache availability until deliberate recovery instead of
reusing an answer after an unresolved write. Replicas remain stateless, but this
is not a system without state: Valkey's durability and the exclusive writer
contract are essential. Direct Qdrant writes, unfenced gateway replicas and
automatic failover to older coordination state are unsupported.

Every native engine had a two-CPU, 4-GiB cap. The gateway setup shared the same
aggregate budget: gateway 0.5 CPU/512 MiB, Qdrant 1.25 CPUs/3328 MiB, and Valkey
0.25 CPU/256 MiB. Valkey used durable AOF with `appendfsync always` and a 64-MiB
memory limit inside its container. We included its resource and write costs.
An idle post-warm-run memory snapshot totaled approximately 265 MiB for those
three services, versus 984 MiB and 923 MiB for the native engines. These are
single container observations, not peak memory or capacity estimates.

### Reproduce and decide

These results measure revision
[`187944e4cc274a0ae61a6c79de885179319e9d4a`](https://github.com/dstockton/qdrant-es-gateway/commit/187944e4cc274a0ae61a6c79de885179319e9d4a).
Later optimizations require separate measurements; they are not represented by
this baseline. The [harness and full preparation procedure at that revision](https://github.com/dstockton/qdrant-es-gateway/tree/187944e4cc274a0ae61a6c79de885179319e9d4a/benchmark/shared-cache)
prepare a fresh disposable deployment and reuse the three-engine correctness
checks. Build that revision, run its preparation helper, load the saved deployment
identity into `CACHE_BENCH_DEPLOYMENT`, and select `CACHE_BENCH_SERVE=true` using
the documented Compose command. Once ready, this measures one enabled round:

```sh
ROUND=on-1 REQUESTS=200 TRIALS=3 WARMUP=40 \
  OUTPUT=benchmark/shared-cache/results/on-1.json \
  python3 benchmark/shared-cache/measure.py
# Preserve the other three rounds and run the separate cold/invalidation profiles.
```

Across the retained profiles, **86,400 measured HTTP responses passed correctness
checks**: 72,000 warm reads, 7,200 cold reads and 7,200 responses from 3,600
write/read cycles. Checks cover sources, IDs, ordering, totals, buckets and
changed document contents. The
[evidence manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-shared-cache/manifest.json)
links all eight raw rounds, the exact build, image IDs and resource observations.
These runs use the v3 client-validation profile and are kept separate from older
measurements.

For repeated catalogue searches between writes, the cache offers a demonstrated
latency benefit in this lab profile. For mostly new queries or frequent document
updates, the native engines remain stronger in the measured cases. Start with
your actual reuse and write rate, and follow the
[deployment and recovery requirements](https://dstockton.github.io/qdrant-es-gateway/shared-read-cache/)
before enabling it.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-rustls-security-fixes-landed.md

# Rustls security fixes landed

The dependency update in [PR #62](https://github.com/dstockton/qdrant-es-gateway/pull/62) moved Rustls from 0.23.45 to 0.23.46. That patch is now on main. It strengthens the TLS layer used for HTTPS requests and encrypted shared-catalog connections, without changing the gateway's search API.

The [upstream release](https://github.com/rustls/rustls/releases/tag/v/0.23.46) fixes several security issues, including secret data appearing in session debug output and key-exchange hints being saved before a handshake is confirmed. Some other fixes concern optional TLS features; this does not mean every deployment was exposed to every issue.

The merged change passed Rust tests, the dependency audit, Helm checks, and the container build and vulnerability gate. Today's follow-up independently confirmed the locked 0.23.46 version and passed formatting, tests, Clippy, a refreshed dependency audit, and Compose and Helm validation. These checks support the update; they are not a reproduction of each upstream security advisory.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-reusing-catalog-connections.md

# Reusing catalog connections helps misses and writes, with a warm-read warning

The shared catalog was opening a new Valkey connection whenever a request loaded
its metadata snapshot. We changed it to reuse a persistent multiplexed connection.
A failed operation is still returned as a failure: reconnecting later must not
silently replay a write whose outcome is uncertain.

In a new paired experiment, gateway cold-query and write/read medians generally
improved by **12–17%**. The exception was eight-client cold queries with cache
serving off, which improved only 1.4%. Warm performance was mixed, and one
important case regressed. This change reduces connection work; it does not
establish an across-the-board latency improvement.

### Compare within this experiment

We measured the original revision
[`187944e`](https://github.com/dstockton/qdrant-es-gateway/commit/187944e4cc274a0ae61a6c79de885179319e9d4a)
against connection-reuse change
[`49da1cb`](https://github.com/dstockton/qdrant-es-gateway/commit/49da1cb22279d22761e18cb38480c36d0bea943b),
built from validated source
[`c420872`](https://github.com/dstockton/qdrant-es-gateway/commit/c4208725b240289a3c0890e576904b80bdf7a0e7).
Each profile used **old, new, new, old** order, retaining native controls and all
trials. Cache coordination stayed enabled; serving off/on were separate profiles.

Native and old-gateway absolute latency were higher than in the
[earlier shared-cache experiment](https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-shared-cache-warm-search-results.md).
We therefore keep these experiments separate. Their differences cannot identify
a cause, and pooling their medians would be misleading.

Values below are gateway median trial p50 milliseconds. Cold means unused
response-cache keys on loaded data. Write rows measure a write and its following
read with one client.

| Workload | Serving off: old → new | Serving on: old → new |
|---|---:|---:|
| Cold sorted filter, one client | 2.590 → 2.159 | 3.155 → 2.788 |
| Cold sorted filter, eight clients | 7.761 → 7.650 | 10.692 → 9.344 |
| Write then point read | 3.524 → 2.914 | 4.008 → 3.392 |
| Write then filter counts | 4.673 → 3.974 | 5.137 → 4.481 |

Median improvements do not describe the whole result. Eight-client cold p95
increased from **64.454 to 71.301 ms** with serving off and **51.765 to 57.449 ms**
with serving on. The new enabled gateway's cold medians remained approximately
4.1 and 4.8 times the faster native engine at one and eight clients. Native
engines also remained faster for write/point-read cycles. The gateway led the
write/filter-count comparison, whose identical `refresh=true` requests require
immediate search visibility; point-write cycles omit refresh on every engine.

### Warm reads were not uniformly preserved

These are the separate cache-enabled warm rounds. Each cell shows **one client /
eight clients**, in milliseconds.

| Gateway operation | Old | New |
|---|---:|---:|
| Read one document | 0.380 / 0.845 | 0.381 / 0.985 |
| Filtered, sorted page | 0.593 / 1.324 | 0.571 / 1.403 |
| Page after a sort cursor | 0.603 / 1.344 | 0.462 / 1.411 |
| Keyword terms buckets | 0.417 / 1.351 | 0.413 / 1.136 |
| Two filter counts | 0.403 / 1.037 | 0.403 / 1.046 |

Eight-client point-read p50 rose **16.5%**, and p95 rose **27.5%**, from 1.242 to
1.583 ms. The native point-read medians also increased roughly 12% between the
corresponding old/new rounds. That drift prevents attributing the gateway change
to connection reuse alone, but it does not make the adverse result disappear.
**Warm-read neutrality remains unproven.**

The [full summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-catalog-transport/summary.json)
includes all native, old-gateway and new-gateway rows, with p95 and throughput.
The new gateway still led the warm search medians in this experiment; native
point reads and the cold/write costs prevent a blanket engine-performance claim.

### Replay the change

The corpus remained 10,000 documents, with identical client requests and v3
correctness assertions. Elasticsearch 8.15.0 and OpenSearch 3.9.0 each retained
two CPUs and 4 GiB. Gateway, Qdrant 1.19.2 and durable Valkey shared that same
aggregate budget. No builds ran during timing.

Build both pinned revisions before measuring and prepare the disposable shared
cache deployment using the
[pinned harness instructions](https://github.com/dstockton/qdrant-es-gateway/tree/c4208725b240289a3c0890e576904b80bdf7a0e7/benchmark/shared-cache).
The retained [round driver](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-catalog-transport/transport_round.py)
and [image override](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-catalog-transport/transport-image.yml)
belong in that harness directory. They replace only the gateway image, wait for
readiness, preserve volumes and refuse to overwrite evidence. Defaults verify the
recorded image IDs before changing the container. For a rebuilt image, pass its
explicit `--old-image-id` or `--new-image-id` and retain the resulting provenance.

Build the old revision as `qdrant-es-gateway:cache-runtime` and the new revision
as `qdrant-es-gateway:catalog-transport`. Fresh local builds generally have different
image IDs from the retained lab images. After verifying their source revisions,
record those local IDs and pass them explicitly:

```sh
old_image_id=$(docker image inspect qdrant-es-gateway:cache-runtime | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["Id"])')
new_image_id=$(docker image inspect qdrant-es-gateway:catalog-transport | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["Id"])')
python3 benchmark/shared-cache/transport_round.py off old 1 mixed \
  --old-image-id "$old_image_id" --new-image-id "$new_image_id"
# Keep both ID arguments for each later call.
# Continue off: new 1, new 2, old 2; repeat the four rounds with on.
# Then run on old 1 warm, on new 1 warm, on new 2 warm, on old 2 warm.
```

All **129,600 measured HTTP responses** passed checks across twenty raw rounds.
The [manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-catalog-transport/manifest.json)
records exact image IDs, revisions, every raw file and post-run observations.
Memory snapshots are not peaks, and container-lifetime throttling counters cannot
attribute individual latency changes. Valkey key-access counters are not response-cache
hit rates; the retained observations do not provide a causal diagnosis.
Published raw files add experiment/build labels to prevent accidental pooling;
the manifest preserves their original hashes and identifies that metadata-only
annotation. Connection reuse removes avoidable work
from misses and writes; the warm-read regression and cold tails remain visible
in the evidence.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-request-local-catalog-read-views.md

# Removing request-local SQLite reconstruction: modest gains, one worse tail

The shared-catalog read path used to rebuild an in-memory SQLite database for
every uncached request. Document reads and searches now load the same validated
snapshot into request-local maps. In our paired experiment, cold-query medians
fell **4.5–5.5% with cache serving enabled** and **4.3–7.6% with serving disabled**.
The enabled eight-client p95 worsened **10.5%**. This is a modest reduction in
request overhead, not an across-the-board latency win.

### Change the representation, preserve the boundary

The change applies to exact document GET and index search GET/POST handlers in
shared-catalog mode. It includes requests that bypass the response cache or run
with serving disabled. Each request still obtains a fresh authoritative snapshot,
validates every mapping and vector list, and rejects duplicate records. Concrete
indices retain precedence over single-hop aliases. There is no persistent local
metadata cache or fallback to an old local database.

Metadata and mutation handlers still use SQLite; default local SQLite deployments
are unchanged. Quiet cache misses keep the atomic generation/snapshot operation,
and the original ticket still guards cache publication after an alias or document
change. This remains a full-catalog fetch and validation, bounded at 8 MiB, with
work proportional to catalog size. It does not establish that SQLite generally
limits search performance. The [operational guide](https://dstockton.github.io/qdrant-es-gateway/shared-catalog/)
describes the scope.

### Cold medians improved, but the enabled tail did not

We compared baseline
[`2deda5e`](https://github.com/dstockton/qdrant-es-gateway/commit/2deda5e43b72de9b2c2ad6f9043df2a01217fd55)
with candidate
[`04b6d7e`](https://github.com/dstockton/qdrant-es-gateway/commit/04b6d7eecce6230634109ec1905509666f00b8bc)
in old, new, new, old order. Serving-enabled and serving-disabled profiles remained
separate; coordination stayed enabled in both. Cold means new response-cache keys
on loaded data. Values below are median trial quantiles in milliseconds.

| Gateway cold sorted filter | p50 old → new | p95 old → new |
|---|---:|---:|
| Serving enabled, one client | 1.911 → 1.826 | 2.374 → 2.201 |
| Serving enabled, eight clients | 4.782 → 4.521 | 55.131 → 60.917 |
| Serving disabled, one client | 1.372 → 1.312 | 1.479 → 1.435 |
| Serving disabled, eight clients | 3.234 → 2.989 | 55.984 → 43.680 |

The enabled eight-client p95 increase repeated in both rounds: baseline round
medians were 55.141/55.121 ms, versus 61.437/60.396 ms for the candidate. The
disabled p95 reduction was less stable: baseline 79.107/25.479 ms versus candidate
32.133/55.227 ms. Cold p50 improvements repeated in both rounds of both modes.
Both native engines remained substantially faster for cold queries. These are
their p50 controls from the same enabled rounds:

| Cold sorted filter | Elasticsearch old → new | OpenSearch old → new |
|---|---:|---:|
| One client | 0.570 → 0.575 | 0.488 → 0.493 |
| Eight clients | 1.255 → 1.251 | 1.169 → 1.265 |

Native movement and the adverse gateway tail rule out a blanket performance claim.
The [complete summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-typed-readview/typed-readview-summary.json)
retains all 42 native and gateway comparisons, including the disabled controls.

### Warm behavior is separate

Warm cache hits do not reconstruct the catalog. Their p50 results were mostly
close, with some reductions and some increases. Cells show one / eight clients.

| Warm gateway operation | Old p50 ms | New p50 ms |
|---|---:|---:|
| Read one document | 0.305 / 0.582 | 0.302 / 0.578 |
| Filtered, sorted page | 0.386 / 0.954 | 0.388 / 0.900 |
| Page after a sort cursor | 0.388 / 0.913 | 0.383 / 0.898 |
| Keyword terms buckets | 0.329 / 0.773 | 0.329 / 0.780 |
| Two filter counts | 0.318 / 0.742 | 0.314 / 0.716 |

Single-client filtered-page p50 rose 0.45%, and eight-client terms p50 rose 0.93%.
Single-client terms p95 rose **4.19%** (0.401 → 0.418 ms); eight-client point p95
rose **1.19%** (0.838 → 0.848 ms). These changes should not be presented as a warm
read improvement caused by removing work from misses.

### Reproduce and inspect

All **93,600 measured responses** passed correctness checks: 72,000 warm,
14,400 enabled cold and 7,200 disabled cold. The corpus contained 10,000 documents,
with unchanged requests, exact totals and validation profile v3. Each profile used
three trials per round, giving six trial quantiles per image for each comparison.
No builds or tests ran during measurement.

Elasticsearch 8.15.0 and OpenSearch 3.9.0 each had two CPUs and 4 GiB. Gateway,
Qdrant 1.19.2 and durable Valkey shared that aggregate budget. Qdrant retained its
automatic two/eight search-pool budgets; both profile boundaries reported high-I/O
mode. The driver primes all engines identically before each profile, including
uncached sorted queries, and recreates the gateway consistently between rounds.

Follow the [pinned shared-cache preparation procedure](https://github.com/dstockton/qdrant-es-gateway/tree/04b6d7eecce6230634109ec1905509666f00b8bc/benchmark/shared-cache).
Build the two revisions as `qdrant-es-gateway:atomic-snapshot` and
`qdrant-es-gateway:typed-readview`. Record their local image IDs; rebuilt images
need not have the historical IDs in the
[manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-typed-readview/manifest.json).
From the repository root after preparation:

```sh
old_id=$(docker image inspect qdrant-es-gateway:atomic-snapshot | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["Id"])')
new_id=$(docker image inspect qdrant-es-gateway:typed-readview | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["Id"])')
python3 benchmark/shared-cache/readview_round.py on-main old 1 \
  --old-image-id "$old_id" \
  --new-image qdrant-es-gateway:typed-readview --new-image-id "$new_id" \
  --new-source 04b6d7eecce6230634109ec1905509666f00b8bc
# Repeat on-main new 1, on-main new 2, on-main old 2, keeping all flags.
# Then off-control old 1, new 1, new 2, old 2, keeping all flags.
```

All twelve raw profiles, eight resource records and the exact driver, override
and helpers are retained in the manifest. Only ephemeral container IDs were
removed; original and published hashes are recorded. Resource intervals cover
profile setup and native-engine phases as well as gateway work. Qdrant was
throttled in roughly 15.8–20.6% of its cold-profile scheduling periods and none
of the warm-profile periods. These counters do not explain individual requests.
No AOF rewrite increments or delayed fsync were recorded within sampled profile
intervals. A rewrite completed between the final baseline cold and warm snapshots
(the cumulative count rose from two to three). The second enabled candidate cold
profile recorded 43.949 ms of Valkey throttling. These observations remain in the
evidence rather than being excluded from the comparison.

These measurements are separate from earlier experiments. The read view removes
avoidable reconstruction while keeping complete validation and request-local
metadata. The modest median gains, worse enabled tail and native cold-query
advantage all remain part of the result.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-qdrant-search-thread-tradeoff.md

# One Qdrant search thread improves tails, but hurts typical cold queries

Limiting Qdrant search pools to one thread produced a clear trade-off in our small
CPU-budget experiment. With eight clients, gateway cold-query p95 fell from
**45.610 to 11.088 ms**, while p50 rose from **4.805 to 7.568 ms**. That is a
75.7% reduction in the tail measurement and a 57.5% increase in typical latency.
Throughput rose 20.1%. It is not an overall latency win or a reason to change
the default everywhere.

### Why test one thread?

Qdrant had a 1.25-CPU quota in our two-CPU gateway deployment. Its telemetry
reported two high-CPU search threads and eight high-I/O search threads, with
high-I/O mode selected. In Qdrant 1.19.2, the automatic budgets are the detected
CPU count and four times that count. An explicit `max_search_threads` sets each
pool's budget instead; it does not cap every thread in the process.
See the versioned [runtime implementation](https://github.com/qdrant/qdrant/blob/v1.19.2/lib/storage/src/content_manager/toc/runtimes.rs#L34-L115)
and [default calculation](https://github.com/qdrant/qdrant/blob/v1.19.2/lib/common/common/src/defaults.rs#L87-L103).

The pinned CPU-detection library rounds fractional cgroup quotas upward, subject
to available CPUs: 1.25 becomes two here. That explains the reported two/eight
budgets. A CPU quota limits scheduled CPU time, not worker count; extra workers
can be useful while waiting for I/O. These facts motivated a measurement, not an
assumption that fewer workers must be faster.
[CPU detection source](https://github.com/seanmonstar/num_cpus/blob/v1.17.0/src/linux.rs#L201-L215).

### Cold queries: better tails, worse medians

We ran automatic, one, one, automatic order, retaining native controls and all
trials. Only Qdrant's search-thread setting changed. Each value below is the
median of six trial quantiles, in milliseconds. Cold means unused response-cache
keys on loaded data, not cold OS caches or storage.

| Cold sorted filter | Automatic p50 → one | Automatic p95 → one |
|---|---:|---:|
| Gateway, one client | 1.970 → 2.172 | 2.314 → 2.452 |
| Gateway, eight clients | 4.805 → 7.568 | 45.610 → 11.088 |
| Elasticsearch, one client | 0.578 → 0.610 | 0.658 → 0.702 |
| Elasticsearch, eight clients | 1.264 → 1.334 | 2.323 → 2.835 |
| OpenSearch, one client | 0.481 → 0.491 | 0.595 → 0.571 |
| OpenSearch, eight clients | 1.162 → 1.169 | 2.305 → 2.169 |

Both native engines remained faster for these cold queries. Single-client gateway
p50 also worsened, by 10.3%. The eight-client tail improvement does not erase those
costs. Eight-client cold throughput increased from **849.210 to 1,019.941 requests
per second**, using the same median-of-trials calculation. Valkey completed an AOF rewrite during the second one-thread cold profile;
we retain that possible confounder rather than attributing every change to threads.

### Warm reads stayed mixed

The gateway image and enabled response cache were unchanged. These rows show
p50 for one client / eight clients, in milliseconds.

| Warm gateway operation | Automatic | One |
|---|---:|---:|
| Read one document | 0.301 / 0.580 | 0.300 / 0.573 |
| Filtered, sorted page | 0.371 / 0.916 | 0.391 / 0.940 |
| Page after a sort cursor | 0.389 / 0.908 | 0.388 / 0.910 |
| Keyword terms buckets | 0.326 / 0.788 | 0.326 / 0.772 |
| Two filter counts | 0.311 / 0.733 | 0.315 / 0.722 |

Most movements were small. Filtered-page p50 rose 5.3% with one client and 2.7%
with eight; single-client p95 rose 13.6%. Two-filter single-client p95 rose 9.1%.
The [full summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-qdrant-search-threads/summary.json)
includes all 36 native and gateway comparisons and their tails. These warm
observations do not establish a benefit from a setting that acts on backend work.

### Reproduce the comparison

All **86,400 measured responses** passed correctness checks: 14,400 cold and
72,000 warm. The corpus had 10,000 documents. Requests, exact totals, validation
profile v3 and native cache settings stayed the same. Elasticsearch 8.15.0 and
OpenSearch 3.9.0 each had two CPUs and 4 GiB. Gateway, Qdrant 1.19.2 and durable
Valkey shared that aggregate budget: 0.5, 1.25 and 0.25 CPUs respectively.

Use the [shared-cache preparation procedure](https://github.com/dstockton/qdrant-es-gateway/tree/2deda5e43b72de9b2c2ad6f9043df2a01217fd55/benchmark/shared-cache)
and the gateway source at that revision. Copy the retained
[round driver](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-qdrant-search-threads/thread_round.py),
[image override](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-qdrant-search-threads/thread-image.yml)
and [snapshot helper](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-qdrant-search-threads/snapshot_round.py)
into `benchmark/shared-cache/`. The driver pins the historical
`qdrant-es-gateway:atomic-snapshot` image. A fresh build may have a different ID:
verify its source, explicitly update the driver's `DIGEST` to that local image ID,
and ensure `SOURCE` records the revision actually built. Retain any driver changes
with your results. Do not bypass image verification.

From the repository root, after preparation:

```sh
python3 benchmark/shared-cache/thread_round.py auto 1
python3 benchmark/shared-cache/thread_round.py one 1
python3 benchmark/shared-cache/thread_round.py one 2
python3 benchmark/shared-cache/thread_round.py auto 2
```

The driver restarts Qdrant and the gateway in the same sequence each round,
retains volumes, checks readiness and verifies effective pool budgets. Before each
profile it primes all three engines with identical workloads and additional
uncached sorted queries, outside timing. Collections had 10,000 points and two
segments after each restart. Pool telemetry remained high-I/O at the sampled
boundaries, with budgets two/eight or one/one as intended. Boundary samples do
not prove that no transitions occurred between them.

The [manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-qdrant-search-threads/manifest.json)
links all eight raw profiles, four resource records, image provenance and hashes.
Resource intervals include priming and native-engine phases; throttling counters
cannot assign a cause to individual latency changes. Qdrant cold-profile throttled
time fell from 1.054/0.912 seconds in the automatic rounds to 8.407/3.609
milliseconds in the one-thread rounds. The second one-thread cold profile also
recorded 262.916 ms of Valkey throttling and the AOF rewrite. Those observations
provide context, not a per-request diagnosis. Delayed-fsync counts remained
zero. Only ephemeral container IDs were removed from public evidence.

This experiment stays separate from earlier gateway code comparisons. One search
thread is a possible tail-latency trade-off for this workload and quota, with a
substantial median cost. It is not a generally faster configuration.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-missing-alias-removals-are-not-success.md

# Missing alias removals are not success

The gateway used to acknowledge every syntactically valid alias removal, even when the alias did not exist or belonged to another index. That made a stale deployment or rollover request look successful while leaving the catalogue unchanged. The risk is operational rather than cosmetic: a caller can continue believing that an old read or write route was detached.

Separate clean-container probes exercised Elasticsearch 9.5.5 and OpenSearch 3.9.0 with the same requests. Both returned HTTP 404 with `aliases_not_found_exception` for a single missing alias and for an alias attached to a different index. Both also applied the successful add in a mixed missing-remove plus add batch and returned HTTP 200. Their response bodies diverged: Elasticsearch reported `errors: true` and one result per action, while OpenSearch returned only `{"acknowledged":true}`. The sanitized [request and response evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-9.5.5-opensearch-3.9-alias-remove-2026-10-15.json) records the versions and observed shapes.

The gateway now tracks whether any action in its supported single-target subset can succeed before opening the SQLite transaction. When every removal misses, it returns the shared native 404 shape and leaves the catalogue unchanged. A mixed batch still commits its successful action and returns 200. Exact idempotent adds count as successful, matching native behavior.

```console
cargo test alias_removal_reports_when_no_action_can_succeed
```

This is deliberately not a claim of identical partial-failure responses. The gateway retains the simpler OpenSearch-shaped success body for mixed batches and does not yet emit Elasticsearch per-action alias results. It also continues to reject `must_exist`, alias arrays and patterns, filters, routing, write-index flags, and multi-index final states rather than approximating them. The probe measures request semantics only; it is not a latency or resource benchmark.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-measuring-the-gateway-request-timeline.md

# Measure the request timeline before rewriting JSON handling

The gateway's own diagnostic traces show that backend waiting occupies most of
the measured cold-search dispatch interval. The median **per-request** fraction
spent in the backend join was **79.4% with one client and 73.6% with eight**.
Cache-envelope JSON parsing took a single-client median of **0.017 ms**, compared
with **1.162 ms** for the whole dispatch interval. These traces do not justify a
JSON-handling rewrite as the next optimization.

This is an instrumented diagnostic, not a production performance benchmark or a
native-engine comparison. The `diagnostic-timing` compile-time feature is off by
default. Its timers, trace storage and response header add overhead.

### Follow the actual request

We captured 600 correct sorted-filter responses: 100 cold, 100 warm and 100
explicitly bypassed requests at each of one and eight clients. Each request keeps
exact totals and returns 20 full-source documents in rank order. The corpus has
10,000 documents. Cold bounds were ranks 1–100 with one client and 301–400 with
eight, so the exact counts include most of the corpus. Warm profiles repeat bounds
101 and 401; bypass profiles use bounds 201–300 and 501–600. These selectivities
matter when comparing this diagnostic with another workload.

The following values are per-phase p50 milliseconds for cold requests:

| Interval | One client | Eight clients |
|---|---:|---:|
| Complete dispatch scope | 1.162 | 2.513 |
| Cache lookup | 0.059 | 0.223 |
| Catalog read-view parsing | 0.002 | 0.001 |
| Backend join | 0.916 | 1.708 |
| Work after the join | 0.044 | 0.040 |
| Response JSON encoding | 0.013 | 0.011 |
| Cache-envelope parsing | 0.017 | 0.015 |
| Cache-envelope encoding | 0.006 | 0.005 |
| Cache publication | 0.088 | 0.277 |

**These rows overlap and must not be added.** Response encoding sits inside the
work after the join; the join includes concurrent count, coverage and ordered-page
RPCs plus their local response handling. The fractions above were calculated for
each individual request before taking the median, not by dividing table medians.

The count RPC finished last among those three operations in **100 of 100**
single-client cold requests and **86 of 100** eight-client cold requests. Ordered
response parsing had medians of **0.015 / 0.016 ms**. That makes backend work a
better-supported investigation target than JSON parsing for this workload. It
does not prove the count endpoint is the only cost or that dropping exact totals
would preserve the same request contract.

Warm dispatch medians were 0.092 / 0.228 ms; bypass medians were 1.018 / 2.080 ms.
The [full phase summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-gateway-phases/gateway-phases-summary.json)
and [per-request relationships](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-gateway-phases/gateway-phases-relationships.json)
retain every profile, including tails. Profiles ran in a fixed order and are not
paired speed comparisons.

### What the clocks include

Dispatch timing excludes HTTP request-body ingress, final diagnostic-header
serialization and network delivery. Client-observed cold p50 was 1.500 / 3.442 ms;
subtracting those independent medians would not measure a specific overhead.
The RPC send-to-headers interval includes connection-pool waits, socket work,
backend execution and scheduling. Body receipt and parsing have separate nested
intervals. Wall intervals are not CPU measurements.

Resource snapshots include untimed worker connection primers. A zero rounded
microsecond interval does not mean work is free. The trace records no request
bodies, credentials or backend URLs. Production builds omit the timing header.

### Reproduce the diagnostic, then restore production

The diagnostic image was built from
[`68add4d`](https://github.com/dstockton/qdrant-es-gateway/commit/68add4dc8aecd8b01ad501ee30ab4d94518f03b6)
with `diagnostic-timing` enabled. The runner is pinned separately in the
[manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-gateway-phases/manifest.json),
which also contains both image identities and the restoration record.
Use the prepared shared-cache corpus and resource settings from that revision:
gateway 0.5 CPU/512 MiB, Qdrant 1.25 CPUs/3,328 MiB and Valkey 0.25 CPU/256 MiB.
Qdrant keeps its automatic two/eight pool budgets and cache serving is enabled.

In an isolated checkout of the diagnostic source, build its dedicated image:

```sh
docker build -f benchmark/shared-cache/diagnostic.Dockerfile \
  -t qdrant-es-gateway:phase-diagnostic .
```

Copy the retained [runner](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-gateway-phases/gateway_phases.py)
and [Compose override](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-gateway-phases/phase-image.yml)
into `benchmark/shared-cache/`. Load `CACHE_BENCH_DEPLOYMENT` from the preparation
file, set `CACHE_BENCH_SERVE=true`, and use `PHASE_GATEWAY_IMAGE` with the existing
three-engine/shared-cache Compose files plus the override to recreate only the
gateway. Wait for readiness before capture. Record the locally built image ID:

```sh
python3 benchmark/shared-cache/gateway_phases.py \
  --image-id '<full locally verified diagnostic image ID>' \
  --source 68add4dc8aecd8b01ad501ee30ab4d94518f03b6
```

The runner requires fresh cold keys and refuses to overwrite evidence. For another
capture, use a new prefix and an unused `--start` range; retain that changed
selectivity. It does not deploy or restore images. After capture, recreate the
gateway with the retained production image and verify readiness and the absence
of diagnostic headers. Our [restoration record](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-gateway-phases/gateway-phases-restoration.json)
confirms production was restored with cache serving enabled.

All raw traces, resource observations, runner copies and hashes are retained;
public records remove only ephemeral container IDs. No additional performance
claim follows from instrumenting the path. The useful result is a better basis
for choosing the next correctness-preserving experiment.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-measuring-small-read-path-changes.md

# Fewer requests did not make this query faster

> Measurement note: these historical filter-aggregation trials recomputed the expected count in each client worker. The calculation was outside that request's timer, but affected throughput and could delay concurrent requests. Later trials tagged `precomputed-filter-count-v2` compute it once before timing. The figures here are preserved without adjustment; do not combine them with the corrected trials.

We tried to speed up gateway aggregations by combining their safety checks.
Two filter counts would need four Qdrant calls instead of five, including the
exact total. That sounded promising. After measuring the final implementation,
we removed it: fewer calls did not produce a reliable latency improvement, and
filter counts were slightly slower.

This followed our [indexed-query optimization](https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-indexed-query-performance.md).
The large win there came from avoiding a complete document scan. The next round
was about much smaller costs: database query preparation, duplicated payloads,
and coordination between independent reads.

### Compare the code, not the warmup

Fresh runs changed the ranking even when the gateway binary stayed identical.
We therefore kept the same 10,000 documents and alternated the old and new
images in **old, new, new, old** order. Each round shuffled Elasticsearch,
OpenSearch and gateway + Qdrant trials, used persistent connections, and warmed
each workload before measuring it. No image builds ran during these trials.

The retained read-path change passed **72,000 measured response checks** across
four rounds. That includes full sources, IDs, ordering, exact totals and buckets.
Each variant contributes six trial medians to each table cell; values are
**one client / eight clients**, in milliseconds.

| Gateway operation | Before | After |
|---|---:|---:|
| Read one document | 0.380 / 0.781 | 0.370 / 0.783 |
| Filtered, sorted page | 0.702 / 2.055 | 0.688 / 2.037 |
| Page after a sort cursor | 1.022 / 2.613 | 1.009 / 2.521 |
| Keyword terms buckets | 0.892 / 3.382 | 0.896 / 3.159 |
| Two filter counts | 1.109 / 3.332 | 1.129 / 3.283 |

These small, mixed differences do not establish a broad speedup. Native engines
remained faster in all ten workload/concurrency combinations in this warmed
comparison. The [complete summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-read-path/summary.json)
includes their results, p95 latency, throughput and trial ranges.

### What we kept

Document reads now ask Qdrant only for the payload fields they consume. For one
representative benchmark document, the internal payload shrank from **684 to
274 bytes**, while the returned source stayed identical. Bulk existence checks
request only document IDs. GET and HEAD no longer parse unused mapping metadata;
SQLite reuses prepared statements, and parsing happens outside the catalog lock.
The final search page also takes ownership of its hits instead of copying them.

These remove specific work without a result cache or another service. Alias
changes still take effect immediately: prepared statements cache query plans,
not catalog answers. The evidence supports reduced data movement; it does not
support a promised percentage improvement in application latency.

### What we rejected

The combined aggregation check required a more complicated backend predicate.
Review also caught an intermediate-memory problem with launching every facet at
once. We bounded concurrency and reduced responses immediately, then reran the
comparison. The final result still did not justify the extra complexity, so the
original aggregation path remains. CPU redistribution, one runtime worker and
an exploratory in-memory payload configuration also failed to establish a
consistent benefit; their defaults remain unchanged.

Native caching matters, but it is not the whole explanation. Both native engines
recorded roughly 18,000 request-cache hits and three misses. [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-request-cache.html)
and [OpenSearch](https://docs.opensearch.org/latest/search-plugins/caching/request-cache/)
cache eligible aggregation requests by default.
A separate `request_cache=false` diagnostic left those counters unchanged, yet
native engines remained faster. Adding Redis without locating the remaining
cost would not by itself solve that difference.

### Reproduce the read comparison

Build both gateway revisions before timing, load the shared corpus using the
[three-engine runner](https://github.com/dstockton/qdrant-es-gateway/tree/main/benchmark/three-engine),
and replace only the gateway image between rounds. Retain the data volumes and
wait for readiness after each replacement.

```sh
ROUND=baseline-a REQUESTS=200 TRIALS=3 WARMUP=40 \
  OUTPUT=results/baseline-a.json python3 compare_reads.py
# Repeat with candidate-b, candidate-c and baseline-d.
python3 summarize_reads.py results/baseline-a.json results/baseline-d.json
python3 summarize_reads.py results/candidate-b.json results/candidate-c.json
```

Every engine receives the same requests; the original aggregate two-CPU, 4-GiB
limits remain. The [evidence manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-read-path/manifest.json)
identifies revisions, raw round files, image IDs and the rejected experiment.
This remains an investigation, not a claim that the gateway has overtaken the
native engines.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-library-large-print-availability.md

# Can I borrow the large-print edition today?

A public-library reader wants a large-print book available at York today. A standard edition of the same title, an ebook, a copy at Leeds, and a fully borrowed title should stay out. These six synthetic holdings represent one title/format/branch combination per document; no borrower data is needed.

Use running Elasticsearch-compatible servers and a fresh `field_note_library` index. Curl is the client: run these same commands against each server, changing only `ES` to the gateway URL (for example, `http://localhost:9200`).

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_library" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"title":{"type":"text"},"branch":{"type":"keyword"},"format":{"type":"keyword"},"genre":{"type":"keyword"},"available_copies":{"type":"integer"}}}}'
curl -fsS -X POST "$ES/field_note_library/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"book-101"}}
{"title":"The Harbour Mystery","branch":"York","format":"large-print","genre":"crime","available_copies":1}
{"index":{"_id":"book-102"}}
{"title":"A Garden in Winter","branch":"York","format":"large-print","genre":"fiction","available_copies":2}
{"index":{"_id":"book-103"}}
{"title":"The Harbour Mystery","branch":"York","format":"standard","genre":"crime","available_copies":3}
{"index":{"_id":"book-104"}}
{"title":"Mountain Letters","branch":"York","format":"large-print","genre":"biography","available_copies":0}
{"index":{"_id":"book-105"}}
{"title":"River Detectives","branch":"Leeds","format":"large-print","genre":"crime","available_copies":2}
{"index":{"_id":"book-106"}}
{"title":"Summer on the Moors","branch":"York","format":"ebook","genre":"fiction","available_copies":4}
NDJSON
sleep 5
```

The same five-second settling pause is used on both endpoints; it does not establish immediate gateway refresh visibility.

Find the available large-print holdings at York:

```sh
curl -fsS -X POST "$ES/field_note_library/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"filter":[{"term":{"branch":"York"}},{"term":{"format":"large-print"}},{"range":{"available_copies":{"gte":1}}}]}}}'
```

Expect exactly `book-101` (The Harbour Mystery) and `book-102` (A Garden in Winter), total `2`, with their complete sources; hit order is immaterial. Count titles by genre over that same filtered selection:

```sh
curl -fsS -X POST "$ES/field_note_library/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"branch":"York"}},{"term":{"format":"large-print"}},{"range":{"available_copies":{"gte":1}}}]}},"aggs":{"genres":{"terms":{"field":"genre","size":10}}}}'
```

Expect `crime: 1` and `fiction: 1`, with no hits. These buckets count holdings records, not copies: the two available copies of A Garden in Winter contribute one fiction record.

The lending system now reports that the final large-print copy of The Harbour Mystery has been borrowed. Publish its new availability and read the record back:

```sh
curl -fsS -X POST "$ES/field_note_library/_update/book-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"available_copies":0}}'
sleep 5
curl -fsS "$ES/field_note_library/_doc/book-101"
```

The update returns `updated`; the read retains the title, branch, format and genre, with `available_copies: 0`. Repeat both searches above: only `book-102` remains, total `1`, and the sole genre bucket is `fiction: 1`. The standard edition of The Harbour Mystery must not substitute for the borrowed large-print edition.

The [fixture](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/fixtures/library.json) and [replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/library.py) verify HTTP statuses, bulk IDs/results, every stored source, exact search totals, genre counts and last-copy exclusion. The [recorded run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/library-2026-10-15.json) passed 70/70 checks on Elasticsearch 8.15.0 and gateway 0.2.0 with Qdrant 1.15.5, using identical requests, embedded sources and synchronous writes. The article's curl requests were also executed on both endpoints. No gateway code change or performance claim is involved.

Boundary: updates are not [atomic concurrent reservations](https://dstockton.github.io/qdrant-es-gateway/compatibility/); the lending system remains authoritative for loans.

Remove the example index when finished:

```sh
curl -fsS -X DELETE "$ES/field_note_library"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-inside-a-cold-search-backend-diagnostic.md

# Inside a cold search: a backend diagnostic, not another engine benchmark

After reducing cache-fill overhead, we inspected three Qdrant operations used by
a filtered, sorted search: exact counting, checking indexed-field coverage, and
retrieving an ordered page. The diagnostic found that exact counting had the
largest single-client server-reported time of those operations. It also showed a
large gap between server-reported timings and parallel client wall time under
concurrency. Neither observation identifies the entire gateway bottleneck.

**This is not an Elasticsearch-compatible or native-engine comparison.** The
runner sends Qdrant RPCs directly inside the container network. It bypasses the
gateway, catalog, response cache and public client path. Its Python client is
not the gateway's Rust client.

### Three pieces of work

The requests use a range filter over the retained 10,000-document corpus. The
count is exact. The coverage probe asks whether any matching document lacks the
expected indexed-field marker. The ordered request retrieves 21 full-source documents with their document IDs
in ascending rank order. All results are checked, including counts, ordering,
document IDs and full sources.

A separate [payload-schema snapshot](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-backend-diagnostic/backend-payload-schema.json)
captured **after the diagnostic** found an integer index on `rank` and a keyword
index on the coverage marker, each covering all 10,000 points. It does not support
a missing-rank-index explanation for the observed count cost, but it is not
contemporaneous proof of the index state during timing.

These are pooled p50 **Qdrant response `time` fields**, in milliseconds, from
400 measured calls per configuration:

| RPC | Run alone, one / eight clients | Run in parallel, one / eight clients |
|---|---:|---:|
| Exact count | 0.698 / 0.892 | 0.734 / 0.873 |
| Coverage probe | 0.163 / 0.298 | 0.180 / 0.320 |
| Ordered page | 0.136 / 0.190 | 0.170 / 0.263 |

The parallel group waits for all three operations and includes Python dispatch,
JSON parsing and correctness validation. Its median trial p50/p95 was
**1.023 / 1.161 ms with one client** and **4.426 / 41.780 ms with eight clients**.
Per-RPC client wall time ends when the response body is received; it has a different
boundary from both group time and the server's reported field.

Do not add parallel quantiles or subtract these independently aggregated timings
to estimate gateway overhead. Network transit, client scheduling, executor work,
validation and server queueing can contribute at different boundaries. This
experiment does not isolate their individual costs or prove count is the sole
bottleneck. The [full summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-backend-diagnostic/backend-decomposition-summary.json)
retains standalone and parallel group quantiles plus each RPC's client and
server-reported quantiles.

### Resource context and reproducibility

All **4,800 measured RPC responses across 3,200 groups** passed checks. Four trials
rotate the operation order at each concurrency level. Connections are persistent
per worker and RPC. The isolated Python generator has two CPUs and 256 MiB,
outside the server budget; its execution model can influence group wall time.

Qdrant 1.19.2 retained 1.25 CPUs and 3,328 MiB. The collection was green at both sampled boundaries, with
10,000 points and two segments. Search-pool telemetry reported high-I/O mode and
two/eight pool budgets at both boundaries. Across the whole capture, Qdrant was
throttled in **13 of 32 scheduling periods (40.6%)**, with **1.107 seconds** of
reported throttled time. Other recorded services had no throttled periods.
These whole-run counters do not assign the long parallel tail to a particular
request or exclude client effects.

Use the [prepared harness at revision 47113fc](https://github.com/dstockton/qdrant-es-gateway/tree/47113fc5a0bc1f3ab4aad6728a480626c7aa006f/benchmark/shared-cache)
with its retained corpus and resource settings. Copy the
[launcher](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-backend-diagnostic/backend_diagnostic.py)
and [RPC runner](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-backend-diagnostic/backend_decomposition.py)
into `benchmark/shared-cache/`, then run from the repository root:

```sh
python3 benchmark/shared-cache/backend_diagnostic.py --prefix backend-decomposition
```

The launcher requires the recorded Python image digest to be available, checks
the prepared corpus and resource limits, and refuses to overwrite evidence.
It creates a restricted temporary runner without changing the engines. The
[manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-backend-diagnostic/manifest.json)
records the image provenance, exact runner hashes, raw samples and resource
snapshots. The two diagnostic scripts were uncommitted at execution; their exact
copies are retained. The gateway image was built from `47113fc`, but the gateway
was not in this measured request path. Only ephemeral container IDs were removed
from public records.

The next useful question is which changes reduce complete request latency while
preserving exact counts and result correctness. These measurements help choose
experiments; they are not evidence of a new production speedup.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-indexed-query-performance.md

# Sorting twenty records should not fetch ten thousand

> Measurement note: these historical filter-aggregation trials recomputed the expected count in each client worker. The calculation was outside that request's timer, but affected throughput and could delay concurrent requests. Later trials tagged `precomputed-filter-count-v2` compute it once before timing. The figures here are preserved without adjustment; do not combine them with the corrected trials.

Our first three-engine benchmark exposed a straightforward problem. To return
20 sorted products, the gateway fetched all 10,000 matching documents from
Qdrant. It also fetched those documents to count a handful of categories.
The answers were correct, but moving and reducing all that data was expensive:
median sorted-page latency was 55.6 ms, and terms aggregation took 39.1 ms.
Elasticsearch and OpenSearch completed those operations in roughly 1–2 ms.

We changed where the work happens. For compatible numeric fields, Qdrant now
orders the records and returns the requested page. Keyword buckets use exact
Qdrant facets; simple filter counts use its count API. Independent reads run
concurrently, and a count-only request does not fetch an unused document.
No client request changes, result cache, or Redis/Valkey service were needed.

Each cell shows **one client / eight clients**, in milliseconds.

| Operation | Elasticsearch | OpenSearch | Gateway + Qdrant |
|---|---:|---:|---:|
| Read one document | 0.466 / 1.194 | 0.428 / 1.047 | 0.502 / 1.298 |
| Filtered, sorted page | 1.023 / 2.960 | 0.939 / 3.251 | 0.917 / 3.190 |
| Page after a sort cursor | 0.948 / 2.263 | 0.733 / 2.602 | 1.228 / 3.565 |
| Keyword terms buckets | 0.586 / 1.996 | 0.605 / 1.791 | 1.068 / 4.302 |
| Two filter counts | 0.714 / 2.928 | 0.718 / 2.854 | 1.135 / 4.167 |

These are median trial p50 latencies in milliseconds, measured with one and
eight concurrent clients. Every measured response passed correctness checks:
IDs, full sources, sort values, exact totals, bucket order and counts. We kept
all trials, including slower ones. The [full summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15/summary.json)
contains p95 latency, throughput and the range of trial medians.

The optimized gateway reduced observed sorted-page latency to 1.228 ms,
about 45 times lower than the first run. Native engines also became faster
across runs, so that ratio is an observed before/after result, not an isolated
production speedup guarantee. The gateway narrowly led filtered search at one
client; that small margin is not a broad performance win. In this measured run, native engines had lower median latency in nine of the ten
workload/concurrency combinations. They also led searchable ingest throughput: roughly
31,900 documents/s for Elasticsearch, 41,600 for OpenSearch and 22,600 for the
gateway.

The clearer advantage was the observed memory footprint. During separate
five-second, eight-client filter-count probes, gateway + Qdrant used about
327 MiB, versus 973 MiB for Elasticsearch and 912 MiB for OpenSearch. Those are
single Docker memory observations, not peaks or production sizing estimates.
For a compact catalogue service already using Qdrant, this is a useful trade:
near-millisecond scalar queries with a smaller observed footprint. For the
lowest latency or fastest ingest in this profile, the native engines remain
stronger choices.

There is a correctness condition behind the speedup. New writes record which
fields have values the indexed operation can handle exactly. Queries check that
coverage before using the fast result. Old documents, mixed types, missing
numeric values, multiple sort fields and page-boundary ties use the complete
scan. Existing documents gain eligibility when rewritten; deploying the new
binary alone does not accelerate every old index. Existing indexes that already
own the new internal field retain their
original data and behavior. This remains stateless request processing; it does
not turn pagination into a snapshot during concurrent writes.

### Reproduce the comparison

The [container profile and runner](https://github.com/dstockton/qdrant-es-gateway/tree/main/benchmark/three-engine)
use Elasticsearch 8.15.0, OpenSearch 3.9.0 and Qdrant 1.19.2. Each native engine
has a two-CPU, 4-GiB cap; Qdrant and the gateway share the same total cap. Both
JVM heaps are 512 MiB. The dataset has 10,000 deterministic documents with
integer rank/price fields, 20 keyword categories and a description. All engines
receive the same mapping, documents and requests; only the endpoint changes.
This tests scalar catalogue operations, not text relevance or vector search.

```sh
cd benchmark/three-engine
docker compose -p qeg-balanced -f compose.yml up -d --build
# Once all three endpoints respond:
DOCS=10000 TRIALS=3 REQUESTS=40 CONCURRENCY=1,8 python3 run.py
python3 summarize.py results/three-engine.json
```

The runner uses persistent HTTP connections, warms each workload, shuffles engine
order with a fixed seed, and measures three trials of 40 requests. Bulk timings
include waiting until all documents are searchable. It deletes only its named
benchmark index, so use the disposable containers. All services share one Docker
VM; this is a small reproducible lab profile, not a production capacity estimate.

Download the [baseline trials](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15/baseline.json),
[optimized trials](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15/optimized.json) and
[build/resource provenance](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15/provenance.json).
The practical improvement is that compatible catalogue queries move a page or
counts across the network instead of the entire matching catalogue. SQLite was
not the measured bottleneck, so replacing its small metadata catalog would not
have addressed the expensive work this benchmark revealed.

A [later paired comparison](https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-measuring-small-read-path-changes.md)
examines warmup, native caches and smaller read-path changes.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-explicit-sparse-vectors-without-token-folding.md

# Keep your sparse token IDs intact

A sparse embedding is a small dictionary: token IDs paired with weights. If the
same token changes identity between indexing and searching, a successful HTTP
response can hide the wrong answer.

An [OpenSearch report](https://github.com/opensearch-project/neural-search/issues/2048)
describes exactly that problem in the SEISMIC approximate path of OpenSearch
3.9.0: “the query token is folded, the document tokens are not.” The report also
says the Lucene exact path returns the correct results.

The gateway now accepts explicit sparse embeddings and passes their numeric IDs
to Qdrant unchanged. IDs such as `32768`, `65535` and `65536` stay distinct.
Documents and queries share the same validation: canonical decimal IDs in the
unsigned 32-bit range and positive, finite float32 weights. Invalid vectors fail
before document writes. You supply embeddings; the gateway does not run a model.

Create a field with the common OpenSearch mapping shape:

```json
{"settings":{"index.sparse":true},"mappings":{"properties":{"v":{"type":"sparse_vector","method":{"name":"seismic","engine":"lucene"}}}}}
```

Index a document containing `{"v":{"65536":1.0}}`, then search:

```json
{"size":10,"query":{"neural_sparse":{"v":{"query_tokens":{"65536":1.0}}}}}
```

Qdrant executes an exact sparse dot product. The method labels in the mapping
are accepted compatibility syntax, not a request to run Lucene inside Qdrant.
Engine-specific tuning and model inference are rejected with a 400 response.

We checked the common request shape against real OpenSearch 3.9.0 and Qdrant
1.19.2. Eight documents used tokens `0`, `1`, `32767`, `32768`, `40342`, `65535`,
`65536` and `105878`. Every query returned exactly its own document on both
endpoints. This is a correctness check, **not a reproduction of the reported
approximate-path bug** or a speed comparison.

To repeat it from the repository checkout:

```sh
docker compose -f examples/explicit-sparse/compose.yml up -d --build
# Wait for both endpoints to answer GET /.
python3 examples/explicit-sparse/compare.py
docker compose -f examples/explicit-sparse/compose.yml down -v
```

The [demo and captured results](https://github.com/dstockton/qdrant-es-gateway/tree/main/examples/explicit-sparse)
include the complete fixture and individual timings. The script sends identical
requests to both endpoints, changing only the endpoint address. It replaces its
own demo index, so run it against disposable instances.

For an application already using Qdrant, this adds an OpenSearch-shaped way to
search precomputed embeddings without another inference service or gateway
session store. It does not establish that migrating from OpenSearch is better:
the tested native exact path is correct too. Sparse queries currently support a
single top-level field and bounded `from`/`size`; total hits are a lower bound.
They reject aggregations, sorting, collapse, post-filtering, `min_score` and
`search_after`. Scalar filters on the sparse field also fail explicitly.
Use a new index when adding sparse fields to an existing mapping.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-disposable-cache-fills-durable-coordination.md

# Keep coordination durable, let response-cache fills be disposable

A response cache should be cheap to refill. Our cache entries shared Valkey's
append-only log with catalog metadata and writer coordination, so filling a cold
response also added durable log traffic. We changed only response publication to
skip incremental AOF propagation. Catalog updates, generations and writer tokens
retain their required durability.

In a paired experiment, enabled cold-query p50 improved about **21%** at both
one and eight clients. Eight-client p95 fell **67.8%**, from **43.949 to 14.169 ms**.
Native engines still handled these cold queries faster, and warm results included
regressions. The improvement is specific to this cache-fill workload.

### Disposable entries, durable authority

The publication script validates the deployment, generation and quiet writer state
before storing a bounded response with an expiry. It disables propagation only
around that response write, then restores normal propagation. Administrative
claims, catalog publication and writer admission/release keep the existing
`appendfsync always` requirements.

This does not mean response entries can never survive a restart. An AOF rewrite
or snapshot can capture entries already in memory. A restored entry must still
pass identity, build/configuration, generation, expiry and writer-state checks.
An old response cannot become current merely because a rewrite saved it.
The [deployment guide](https://dstockton.github.io/qdrant-es-gateway/shared-read-cache/) explains these
boundaries. This is not a guarantee that every fill avoids all fsync activity.

### Cold-query results and log growth

We compared baseline
[`04b6d7e`](https://github.com/dstockton/qdrant-es-gateway/commit/04b6d7eecce6230634109ec1905509666f00b8bc)
with measured candidate
[`a778a35`](https://github.com/dstockton/qdrant-es-gateway/commit/a778a357f677345939e61b559bf6beac234624d7),
using old, new, new, old order in separate serving-enabled and serving-disabled
profiles. These measurements describe those builds, not a later merge.
Values are median trial quantiles in milliseconds.

| Gateway cold sorted filter | p50 old → new | p95 old → new |
|---|---:|---:|
| Serving enabled, one client | 1.943 → 1.537 | 2.277 → 1.715 |
| Serving enabled, eight clients | 4.578 → 3.614 | 43.949 → 14.169 |
| Serving disabled, one client | 1.353 → 1.359 | 1.515 → 1.562 |
| Serving disabled, eight clients | 3.336 → 3.150 | 44.380 → 50.073 |

The enabled cold profiles added **9,103,721 and 9,103,723 bytes** to the baseline
AOF, compared with **2,016 bytes in each candidate round**. Disabled cold profiles
also added 2,016 bytes; warm profiles added none. No rewrite occurred within these
sampled intervals, so log compaction does not account for that difference.
The observations support reduced incremental log work, not a count of individual
fsync calls or proof that all latency variation has one cause.

Native cold p50 controls from the enabled rounds remained lower:

| Cold sorted filter | Elasticsearch old → new | OpenSearch old → new |
|---|---:|---:|
| One client | 0.579 → 0.581 | 0.487 → 0.498 |
| Eight clients | 1.201 → 1.295 | 1.247 → 1.182 |

The disabled eight-client p95 worsened **12.8%**, even though response fills were
bypassed. Native controls also moved. Those results remain in the
[complete 42-row summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-ephemeral-publish/ephemeral-publish-summary.json).

### Warm results were mixed

Warm hits do not publish another response. Cells show one / eight clients, in
p50 milliseconds.

| Warm gateway operation | Old | New |
|---|---:|---:|
| Read one document | 0.302 / 0.585 | 0.305 / 0.567 |
| Filtered, sorted page | 0.396 / 0.911 | 0.387 / 0.914 |
| Page after a sort cursor | 0.374 / 0.923 | 0.395 / 0.905 |
| Keyword terms buckets | 0.328 / 0.776 | 0.331 / 0.778 |
| Two filter counts | 0.321 / 0.768 | 0.322 / 0.709 |

Single-client sorted pagination worsened **5.8% at p50 and 21.7% at p95**
(0.439 → 0.535 ms). Eight-client terms p95 rose **5.15%** (1.210 → 1.272 ms).
We do not claim a general warm-read improvement from this change.

### Check crashes as well as speed

An isolated Valkey 8.1.10 proof exercised the production Lua scripts. Publishing
a response produced a hit while AOF size stayed **257 → 257 bytes**. The final
six-check run also verified persistent unresolved claims and writers, durable
catalog commit and releases, rewrite-restored entries with their original expiry,
rejection after a durable generation change, and writer durability after a
publication error.

The [final proof](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-ephemeral-publish/aof-proof-3.json)
and [proof script](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-ephemeral-publish/cache_ephemeral_aof.py)
are retained. So are both earlier attempts: the first failed before checks because
the harness mishandled INFO JSON readiness parsing; the second passed an earlier
five-check suite. They are not silently discarded or presented as equivalent to
the final proof.

### Replay the comparison

All **93,600 measured HTTP responses** passed correctness checks: 72,000 warm,
14,400 enabled cold and 7,200 disabled cold. Each comparison uses six trial
quantiles per image. The 10,000-document corpus, requests, exact totals and
validation profile v3 stayed fixed. Cold means new cache keys on loaded data.

Elasticsearch 8.15.0 and OpenSearch 3.9.0 each had two CPUs and 4 GiB. Gateway,
Qdrant 1.19.2 and durable Valkey shared the same aggregate budget. Qdrant retained
automatic search pools. The driver consistently recreates the gateway and primes
all engines outside timing. No builds or tests ran during measurement.

Follow the [pinned preparation instructions](https://github.com/dstockton/qdrant-es-gateway/tree/a778a357f677345939e61b559bf6beac234624d7/benchmark/shared-cache).
Build the baseline as `qdrant-es-gateway:typed-readview` and candidate as
`qdrant-es-gateway:ephemeral-publish`, using the revisions above. Record local
image IDs; rebuilt images may differ from the historical manifest. After preparation:

```sh
old_id=$(docker image inspect qdrant-es-gateway:typed-readview | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["Id"])')
new_id=$(docker image inspect qdrant-es-gateway:ephemeral-publish | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["Id"])')
python3 benchmark/shared-cache/ephemeral_round.py on-main old 1 \
  --old-image-id "$old_id" \
  --new-image qdrant-es-gateway:ephemeral-publish --new-image-id "$new_id" \
  --new-source a778a357f677345939e61b559bf6beac234624d7
# Repeat on-main new 1, new 2, old 2, retaining all flags.
# Then off-control old 1, new 1, new 2, old 2, retaining all flags.
python3 tests/cache_ephemeral_aof.py --output /tmp/cache-aof-proof.json
```

The [manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-ephemeral-publish/manifest.json)
retains twelve raw profiles, eight resource records, all proof attempts, exact
scripts and hashes. Public records omit only ephemeral container identifiers and
failed-proof resource names. Resource counters span setup and native phases as
well as gateway requests; they cannot attribute individual latencies. No delayed
fsync increments were recorded in sampled profiles. Earlier experiments are kept
separate.

For a workload that repeatedly fills the cache, disposable publication removes
unnecessary durable response storage while preserving durable coordination.
That benefit coexists with the native engines’ cold-query advantage and the adverse
warm results above.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-disabling-totals-should-not-slow-sorting.md

# Asking for less should not make sorting slower

> Measurement note: these historical filter-aggregation trials recomputed the expected count in each client worker. The calculation was outside that request's timer, but affected throughput and could delay concurrent requests. Later trials tagged `precomputed-filter-count-v2` compute it once before timing. The figures here are preserved without adjustment; do not combine them with the corrected trials.

A client that only needs the next page can set `track_total_hits: false` and
avoid asking for the total number of matching documents. In the gateway, that
apparently cheaper request had an expensive surprise: it disabled indexed
sorting and fetched every matching document from Qdrant instead.

On our 10,000-product catalogue, a single-client sorted page took **45.340 ms**.
After separating the sorting decision from the total-count decision, the same
request took **0.689 ms**. Both requests return the same twenty products, source
fields and sort values, with no `hits.total` field.

### Keep the page, skip the count

The gateway now uses its guarded indexed sorting path whether or not totals are
requested. When totals are disabled, it skips the exact count. A zero-size
aggregation response also skips the otherwise unused hit lookup. These are
request-local changes: no result cache, new database or Redis service is needed.

The existing safety checks still apply. Eligible scalar numeric fields can use
Qdrant's indexed ordering; unsupported values, old documents without eligibility
metadata and ambiguous page-boundary ties keep the complete-scan fallback.
Disabling totals does not relax sorting correctness or provide snapshot isolation
while documents change.

We measured **old, new, new, old** images against the same retained corpus. Each
image contributes six trial medians per table cell. Values below are median
trial p50 milliseconds, shown as **one client / eight clients**.

| Gateway operation, totals disabled | Before | After |
|---|---:|---:|
| Read one document | 0.384 / 0.842 | 0.374 / 0.769 |
| Filtered, sorted page | 2.888 / 10.093 | 0.691 / 1.797 |
| Page after a sort cursor | 45.340 / 305.877 | 0.689 / 1.833 |
| Keyword terms buckets | 0.870 / 2.470 | 0.675 / 1.970 |
| Two filter counts | 1.102 / 3.305 | 0.858 / 2.954 |

The point-read request is unaffected by the totals option; its smaller movement
helps show ordinary measurement variation. The large sorting improvement removes
an identified full scan. It does **not** establish that the gateway is faster
than the native engines.

### Native engines still lead

All three engines received identical requests within this disabled-total profile;
only the endpoint changed. Here are the final candidate-round results, including
every workload rather than selecting the gateway's strongest result.

| Operation | Elasticsearch | OpenSearch | Gateway + Qdrant |
|---|---:|---:|---:|
| Read one document | 0.307 / 0.599 | 0.289 / 0.579 | 0.374 / 0.769 |
| Filtered, sorted page | 0.487 / 1.082 | 0.468 / 1.020 | 0.691 / 1.797 |
| Page after a sort cursor | 0.554 / 1.185 | 0.442 / 1.028 | 0.689 / 1.833 |
| Keyword terms buckets | 0.355 / 0.863 | 0.354 / 0.856 | 0.675 / 1.970 |
| Two filter counts | 0.446 / 2.745 | 0.494 / 2.745 | 0.858 / 2.954 |

Both native engines remain faster in all ten workload/concurrency combinations.
For applications already using Qdrant, this fix makes requesting only a page
substantially more practical. It is not a reason to promise lower latency than
Elasticsearch or OpenSearch.

We separately repeated the four-round comparison with exact totals enabled.
That profile passed another 72,000 response checks, with no material measured
regression: single-client filtered search changed from 0.699 to 0.701 ms and
sorted pagination from 1.010 to 1.011 ms. Exact totals and disabled totals are
separate profiles; we did not give the gateway a cheaper request than its peers.
Across both profiles, **144,000 measured responses passed** checks for IDs, full
sources, ordering, bucket counts, and either exact totals or their absence.

### Reproduce it

Use the [three-engine harness](https://github.com/dstockton/qdrant-es-gateway/tree/main/benchmark/three-engine)
with Elasticsearch 8.15.0, OpenSearch 3.9.0 and Qdrant 1.19.2. Build both gateway
images before timing and populate the disposable 10,000-document corpus once.
Each native engine has a two-CPU, 4-GiB cap; Qdrant and the gateway share the same
total cap. Native cache settings remain at their defaults.

```sh
# From benchmark/three-engine, after loading data and waiting for readiness:
TRACK_TOTAL_HITS=false ROUND=old-1 REQUESTS=200 TRIALS=3 WARMUP=40 \
  OUTPUT=results/old-1.json python3 compare_reads.py
# Replace only the gateway image, preserving volumes, and wait for readiness.
# Repeat as new-1, new-2, then old-2.
python3 summarize_reads.py results/old-1.json results/old-2.json
python3 summarize_reads.py results/new-1.json results/new-2.json
# Repeat all four rounds with TRACK_TOTAL_HITS=true into separate output files.
```

Each round interleaves all engines, uses persistent connections and warms every
trial. No builds or ingestion run during timing. The shared Docker host and small
scalar catalogue limit what these measurements establish; this is not a text,
vector, durability or production-capacity benchmark.

The [evidence manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-disabled-totals/manifest.json)
links all eight raw rounds, revisions and image provenance. Separate summaries
retain [disabled-total results](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-disabled-totals/disabled-summary.json)
and [exact-total results](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-disabled-totals/exact-summary.json),
including p95 latency, throughput and trial ranges. The improvement is concrete:
a client asking for less work no longer accidentally triggers a full scan.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-changing-transport-was-not-enough.md

# Changing transport was not enough

> Measurement note: these historical filter-aggregation trials recomputed the expected count in each client worker. The calculation was outside that request's timer, but affected throughput and could delay concurrent requests. Later trials tagged `precomputed-filter-count-v2` compute it once before timing. The figures here are preserved without adjustment; do not combine them with the corrected trials.

After removing unnecessary scans, we tested another plausible optimization:
send gateway reads to Qdrant over gRPC instead of REST. The smaller typed
messages and shared connection might reduce overhead. The results were mixed,
so we removed the prototypes and kept REST in production.

### Compare one change at a time

We built each prototype once, then ran the same image with gRPC disabled and
enabled. The client-facing requests, retained 10,000-document corpus and resource
caps stayed identical. Four rounds ran in **REST, candidate, candidate, REST**
order, with Elasticsearch and OpenSearch interleaved throughout. No builds or
ingestion ran during timing.

The first prototype moved supported document reads, scrolling, facets and counts
to gRPC. It improved nine of ten median-latency combinations by roughly 1–11%,
but single-client filter aggregation became **15.4% slower: 1.085 → 1.251 ms**.
Those extra costs mattered more than an attractive count of winning cells.

We then tried a hybrid: keep counts on REST and use gRPC for the other supported
reads. The table shows all gateway results for that second experiment. Each cell
is median trial p50 milliseconds, **one client / eight clients**, using all six
trial medians per variant.

| Operation | REST | Hybrid |
|---|---:|---:|
| Read one document | 0.377 / 0.784 | 0.371 / 0.703 |
| Filtered, sorted page | 0.703 / 2.017 | 0.706 / 1.851 |
| Page after a sort cursor | 1.015 / 2.519 | 0.990 / 2.531 |
| Keyword terms buckets | 0.901 / 3.080 | 0.928 / 2.744 |
| Two filter counts | 1.117 / 3.211 | 1.145 / 3.273 |

Five combinations improved and five regressed. Keeping counts on REST reduced
the earlier single-client regression, but did not produce a consistent win.
These are observations from separate paired experiments, not proof that one
transport operation alone caused every difference.

Both native engines remained faster in all ten candidate workload/concurrency
combinations in each experiment. We did not change their requests or disable
their default caching to make the gateway look better.

### Keep the evidence, not the extra machinery

Across the two paired experiments, **144,000 measured responses passed strict
checks** for full sources, IDs, sort values, exact totals and aggregation results.
Correctness was necessary, but it did not justify the additional adapter,
conversion rules and dependencies for these mixed latency results. The gateway
continues to use REST; no new transport setting is required.

This does not establish that gRPC is generally slower, or that it cannot help a
different workload. It shows that changing transport was insufficient to close
this gateway's measured latency gap on a small scalar catalogue. The larger
[disabled-total sorting improvement](https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-disabling-totals-should-not-slow-sorting.md)
came from eliminating work rather than changing its encoding.

The [evidence manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-transport/manifest.json)
retains all eight rounds, exact image IDs, resource limits and replayable prototype
patches. Complete summaries include native results, p95 latency, throughput and
trial ranges for the [full gRPC experiment](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-transport/full-summary.json)
and [hybrid experiment](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-transport/hybrid-summary.json).
A separate REST-only disabled-total diagnostic is retained but excluded from the
paired results.

To replay, apply one archived patch and its companion module to the manifest's
base revision, build the image, and use the three-engine read-only runner with
`REQUESTS=200 TRIALS=3 WARMUP=40 TRACK_TOTAL_HITS=true`. Alternate only the backend
transport between rounds, preserving data and waiting for readiness. The native
versions remain Elasticsearch 8.15.0 and OpenSearch 3.9.0, with Qdrant 1.19.2;
each engine receives the same aggregate two-CPU, 4-GiB limit. This is a reproducible
lab experiment, not a production-capacity or general engine ranking.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-calendar-days-follow-local-time.md

# A reporting day is not always 24 hours

A sales report grouped by London day should move its boundaries when the clocks
change. Adding 86,400 seconds to yesterday's boundary gets that wrong twice a year.

The gateway now supports a small, explicit `date_histogram` subset: calendar days,
Monday-starting weeks, months, and years in an IANA time zone. It converts each
timestamp into local calendar time and converts that calendar boundary back to UTC.

```json
{"size":0,"aggs":{"sales_days":{"date_histogram":{"field":"date","calendar_interval":"day","time_zone":"Europe/London","min_doc_count":1}}}}
```

The deterministic test measures 23 hours between the March 29 and March 30, 2026
London boundaries and 25 hours between October 25 and October 26. Documents with
multiple timestamps count once per bucket. Run:

```sh
cargo test calendar_days_follow_dst_and_weeks_start_monday
```

The [comparison fixture](https://github.com/dstockton/qdrant-es-gateway/tree/main/validation/aggregations)
contains dates on both sides of those transitions and sends identical requests to
Elasticsearch and the gateway. This is compatibility work for teams already using
Qdrant; there is no claimed speed advantage over a native calendar histogram.

Only nonempty buckets are supported, so `min_doc_count:1` is required. Custom
formats, fixed intervals, offsets, bounds, subaggregations, and nonexistent local
midnight boundaries fail explicitly. Input dates accept epoch milliseconds,
RFC3339 timestamps, and `YYYY-MM-DD` dates. Date-only input denotes midnight UTC.

Time-zone rules come from the compiled `chrono-tz` dependency. Keeping that data
current still matters: implementing a calendar reducer does not repair stale rules.
This does **not** claim to resolve the
[Graylog OpenSearch tzdata report](https://github.com/Graylog2/graylog2-server/issues/27755),
which describes "events near midnight land on the wrong day" under outdated rules.
That reported deployment needs separate reproduction with its exact engine and
updated zone data. The London checks exercise established DST rules, not those
reported Canadian and Moroccan rule changes.

The container replay against Elasticsearch 8.15.0 and Qdrant 1.19.2 passed all four cases: complete aggregations, post-filtered hits, the first sorted page, and its cursor continuation. The [captured results](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/aggregations/results.json) include actual request timings. They establish correctness for this fixture, not a speed advantage.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-atomic-cache-miss-snapshots.md

# One fewer catalog request, without a proven latency win

A quiet cache miss now gets its generation ticket and catalog snapshot in one
Valkey operation. This removes a second metadata request before querying Qdrant.
In our paired test, cold-query medians fell **3.0% with one client and 7.9% with
eight clients**. Native controls moved too, and warm reads were mixed. These
measurements do **not** isolate a performance win from the change.

### Keep the ticket and snapshot together

The catalog maps index and alias names to collections. The cache ticket identifies
the generation under which a response may be stored. If an alias changes between
independent reads, pairing an older catalog with a newer ticket could allow an
outdated response into the cache.
The previous two-call path already enforced safe ordering by obtaining the ticket
before the catalog. This change preserves that guarantee with one call; it does
not fix a stale-cache bug in the previous path.

The new script checks that no writers are active, captures the generation and
reads the catalog atomically. It bounds the catalog size before fetching its
contents. The gateway loads those bytes through its existing validated SQLite
catalog loader; malformed or duplicate records fail before reaching Qdrant.

A writer starting afterward changes the generation, preventing the earlier
response from filling the cache. That request may still return its response.
This protects cache publication; it does not create a transaction spanning
Valkey and Qdrant or provide backend snapshot isolation. Warm hits retain their
existing path. Busy or failed lookups, and corrupt cached responses, use the
ordinary catalog path.

### What changed in the measurements

We compared the retained connection-reuse image with
[`2deda5e`](https://github.com/dstockton/qdrant-es-gateway/commit/2deda5e43b72de9b2c2ad6f9043df2a01217fd55),
using old, new, new, old order. Cache-enabled rounds measured cold then warm
queries. A separate serving-off control measured cold queries only; coordination
remained enabled. Cold means unused response-cache keys on loaded data, not cold
OS caches or storage. There were no timed write/read cycles in this experiment.

Each value is the median of six trial p50 measurements, in milliseconds. Native
columns are controls from the same enabled rounds; only the gateway image changes.

| Cold sorted filter | Gateway old → new | Elasticsearch old → new | OpenSearch old → new |
|---|---:|---:|---:|
| One client | 4.268 → 4.139 | 1.375 → 1.516 | 1.202 → 1.117 |
| Eight clients | 16.067 → 14.800 | 3.343 → 3.087 | 3.324 → 2.946 |

At eight clients, Elasticsearch improved 7.7% and OpenSearch 11.4%, despite neither
engine changing. Gateway p95 barely moved: **5.099 → 5.122 ms** with one client
and **70.192 → 68.663 ms** with eight. Both native engines remained substantially
faster for these cold queries. Serving-off gateway p50 moved **2.677 → 2.707 ms**
and **10.851 → 10.581 ms**, providing another indication of variation.

Warm results also varied. Cells below show one client / eight clients.

| Warm gateway operation | Old p50 ms | New p50 ms |
|---|---:|---:|
| Read one document | 0.519 / 1.392 | 0.569 / 1.230 |
| Filtered, sorted page | 0.759 / 2.426 | 0.643 / 2.251 |
| Page after a sort cursor | 0.700 / 2.649 | 0.694 / 2.165 |
| Keyword terms buckets | 0.536 / 1.799 | 0.546 / 1.736 |
| Two filter counts | 0.527 / 1.725 | 0.506 / 1.664 |

Single-client point reads worsened **9.6% at p50 and 39.7% at p95**
(0.730 → 1.020 ms). Terms p95 also worsened at both concurrency levels, by 10.7%
and 7.4%. The first baseline warm profile included an AOF rewrite; later profiles
did not. Native controls drifted too. We retain that confounder and every adverse
result rather than attributing warm changes to an optimization that targets misses.
The [complete summary](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-atomic-snapshot/atomic-snapshot-summary.json)
contains all 42 native and gateway comparisons, including p95.

### Replay this experiment

All **93,600 measured HTTP responses** passed the existing correctness checks:
72,000 warm, 14,400 enabled cold and 7,200 serving-off cold. Fifteen smoke-test
operations are recorded separately. The corpus was 10,000 documents with identical
client requests, exact totals and validation profile v3. Elasticsearch 8.15.0 and
OpenSearch 3.9.0 each had two CPUs and 4 GiB. Gateway, Qdrant 1.19.2 and durable
Valkey shared the same aggregate budget. No builds ran during timing.

Use the [pinned preparation procedure](https://github.com/dstockton/qdrant-es-gateway/tree/2deda5e43b72de9b2c2ad6f9043df2a01217fd55/benchmark/shared-cache)
to create the disposable deployment and its preparation file. Build the baseline
from source `c4208725b240289a3c0890e576904b80bdf7a0e7` as
`qdrant-es-gateway:catalog-transport` and the candidate from `2deda5e` as
`qdrant-es-gateway:atomic-snapshot`. Record each locally built image ID; fresh
builds need not match the historical IDs in the
[manifest](https://dstockton.github.io/qdrant-es-gateway/assets/benchmarks/2026-10-15-atomic-snapshot/manifest.json).
Then, from the repository root:

```sh
old_id=$(docker image inspect qdrant-es-gateway:catalog-transport | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["Id"])')
new_id=$(docker image inspect qdrant-es-gateway:atomic-snapshot | python3 -c 'import json,sys; print(json.load(sys.stdin)[0]["Id"])')
python3 benchmark/shared-cache/snapshot_round.py on-main old 1 \
  --old-image-id "$old_id" \
  --new-image qdrant-es-gateway:atomic-snapshot --new-image-id "$new_id" \
  --new-source 2deda5e43b72de9b2c2ad6f9043df2a01217fd55
# Repeat with on-main new 1, on-main new 2, on-main old 2.
# Then off-control old 1, new 1, new 2, old 2, keeping all flags.
```

The manifest links all twelve raw measurement files, eight resource records,
smoke checks, summaries and the retained driver/override. Resource counters span
setup, priming and native-engine phases; they cannot attribute individual request
latency. AOF rewrites advanced from zero to one during the first baseline warm
profile, while delayed-fsync counts remained zero. Ephemeral container IDs were
removed from public records; original hashes and unchanged timing data are retained.

These results remain separate from earlier experiments. Tests demonstrate that
a quiet miss avoids the second catalog fetch without weakening generation checks.
The latency evidence supports a smaller request path, not a broad performance claim.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-aggregations-beyond-the-first-page.md

# The cheapest item may be on the next page

A price filter should describe the matching catalogue, even when the shopper only
asks for one product. The gateway previously calculated `min`, `max`, and filter
counts from the hit candidates it had fetched. A small `size` could therefore
produce a plausible but incomplete answer.

The fix separates those two jobs. Qdrant scrolls through every matching page;
the gateway retains the current extrema and bucket counters, then discards each
page. The hit limit no longer sets the scope of an aggregation. `post_filter`,
collapse, and `search_after` still affect hits, not aggregation membership.

For example:

```json
{"size":0,"aggs":{"lowest_price":{"min":{"field":"price"}},"brands":{"terms":{"field":"brand","size":2}}}}
```

Our regression puts prices 50 and 2 on separate pages. The result is 2, including
when the search returns no hits. Terms now count the complete matching set too,
deduplicate repeated values within one document, and report the actual count
outside the returned top buckets.

Run `cargo test aggregation_scroll_visits_every_page_and_uses_search_projection_values`
for the deterministic two-page check. The repository's
[`validation/aggregations` comparison](https://github.com/dstockton/qdrant-es-gateway/tree/main/validation/aggregations)
also sends the same six-document fixture and requests to Elasticsearch 8.15.0 and
the gateway with Qdrant. It prints the observed results and individual timings.

This improves correctness within the gateway's supported subset; it is not a
claim that scanning is faster than Elasticsearch's native aggregations. A full
scan costs work proportional to the matching data. Metrics use constant state;
buckets have an explicit 10,000-distinct-value limit per aggregation and fail
rather than return truncated totals. There are at most 100 aggregations per request.

Ranked text and pattern queries are rejected when combined with aggregations,
because their candidate membership cannot currently be reduced exactly this way.
Concurrent writes can change a scroll while it runs. With document projection,
reductions use mapped non-text values in the search projection, keeping membership
and values together instead of fetching a newer source document.

The container replay against Elasticsearch 8.15.0 and Qdrant 1.19.2 passed all four cases: complete aggregations, post-filtered hits, the first sorted page, and its cursor continuation. The [captured results](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/aggregations/results.json) include actual request timings. They establish correctness for this fixture, not a speed advantage.

A later [three-engine benchmark](https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-indexed-query-performance.md)
adds guarded indexed paths for compatible numeric sorting, keyword facets and
simple filter counts. The complete scan described here remains the fallback
when the gateway cannot prove those operations preserve the result.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-15-a-shared-catalog-is-not-a-snapshot.md

# A shared catalog is not a snapshot

Moving metadata out of a gateway process makes replicas easier to run. It does not make every Elasticsearch consistency promise available.

A `search_after` cursor describes where the next page should start. If documents change between pages, the next request sees those changes. A point-in-time search needs a retained view of the data, or retained results with clearly defined creation semantics. Saving a cursor in Valkey does not create that view.

The same distinction matters for writes. An `if_seq_no` and `if_primary_term` pair asks the engine to reject a write if the document changed. Checking a value in Redis and then writing to Qdrant is two operations across two systems. A failure or competing writer between them can break the promise. A fast key-value store cannot make them one transaction.

The gateway now rejects point-in-time search bodies and consistency-sensitive URL parameters before executing a request. For example, `PUT /products/_doc/1?if_seq_no=0&if_primary_term=1` returns a structured `400` instead of silently performing an unconditional write. Explicit routing, external versions, scroll contexts, and conflict retries are also rejected when supplied as URL parameters. The focused tests check encoded parameter names as well as ordinary requests.

This keeps the boundary visible while shared metadata and stateless query translation improve. Atomic concurrent document updates still need an authoritative conditional-write mechanism and recovery design. Full Lucene phrase semantics need positional indexing. Neither is supplied by adding a metadata cache.

For applications that use only the supported subset, Qdrant can remain the document and retrieval engine while gateway replicas keep no durable local catalog. Applications requiring snapshot pagination or concurrent compare-and-set writes should retain their native engine for those operations until the gateway can demonstrate the same guarantee.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-14-index-names-are-a-client-contract.md

# Index names are a client contract

An invalid index name used to make it through the gateway's metadata catalogue and into a normalized Qdrant collection name. That was more than cosmetic. An application could successfully provision state that the same Elasticsearch or OpenSearch client would expect a native cluster to reject, and distinct punctuation could then collide in Qdrant's namespace.

Elasticsearch and OpenSearch implement their validators independently, so this change checked both rather than assuming the fork still behaved like its ancestor. [Elasticsearch 9.5.5's validator](https://github.com/elastic/elasticsearch/blob/v9.5.5/server/src/main/java/org/elasticsearch/cluster/metadata/MetadataCreateIndexService.java#L342-L365) and [OpenSearch 3.9's validator](https://github.com/opensearch-project/OpenSearch/blob/3.9/server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java#L313-L342) agree on the static boundary:

- a name must be non-empty and lowercase;
- filesystem-reserved punctuation, `#`, and `:` are forbidden;
- `_`, `-`, and `+` cannot be the first character;
- the UTF-8 encoding cannot exceed 255 bytes; and
- `.` and `..` are invalid.

The byte boundary matters for non-ASCII names: 127 copies of `é` occupy 254 UTF-8 bytes and pass, while 128 copies occupy 256 bytes and fail. The source implementations permit `+` away from the first character, despite the current OpenSearch prose documentation listing `+` among generally forbidden characters. The gateway follows the versioned 3.9 implementation for that edge rather than broadening the rejection.

Invalid create requests now return HTTP 400 with the client-recognizable `invalid_index_name_exception`, root cause, index name, and `_na_` index UUID. Validation runs under the existing index-administration lock but before SQLite reads and Qdrant requests. Focused regressions cover the UTF-8 byte limit, punctuation, prefixes, lowercase requirement, response shape, and a zero-request mock upstream counter:

```console
cargo test index_name_validation_matches_shared_native_rules
cargo test invalid_index_creation_fails_before_qdrant_work
```

This run is source- and request-level validation, not a live-engine benchmark. The local Docker service did not answer its availability check, so no native containers were claimed. Dot-prefixed names other than the exact `.` and `..` remain accepted by both versioned static validators; their separate hidden/system-index checks and deprecation behavior are not emulated by the gateway.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-14-alias-options-must-not-disappear.md

# Alias options must not disappear

The gateway deliberately implements unfiltered, unrouted aliases with one concrete target. Its alias action parser previously read only `index` and `alias`, but silently ignored every other metadata field. A request for a tenant filter, custom routing, or a write index could therefore return `{"acknowledged":true}` while the stored alias exposed all documents and routed every operation normally. That is more dangerous than an unsupported-feature error because the response tells the caller that a data-access or write-routing boundary exists when it does not.

Elasticsearch 9.5.5 and OpenSearch 3.9.0 expose the same relevant action fields, but they were checked independently. The versioned [Elasticsearch request parser](https://github.com/elastic/elasticsearch/blob/v9.5.5/server/src/main/java/org/elasticsearch/action/admin/indices/alias/IndicesAliasesRequest.java#L98-L108) accepts `filter`, `routing`, `index_routing`, `search_routing`, `is_write_index`, and `is_hidden` for add actions, plus `must_exist` for removals. The versioned [OpenSearch request parser](https://github.com/opensearch-project/OpenSearch/blob/3.9.0/server/src/main/java/org/opensearch/action/admin/indices/alias/IndicesAliasesRequest.java#L106-L116) declares the corresponding fields, and the separate [OpenSearch alias documentation](https://docs.opensearch.org/latest/api-reference/alias/aliases-api/) describes their effects. This is source-level evidence for the two engines, not an assumption that verification against one applies to the other.

The gateway now rejects every action metadata field outside its supported `index` and `alias` subset. Known native options and unknown fields receive a structured HTTP 400 naming the exact path, such as `_aliases.add.filter` or `_aliases.remove.must_exist`. Validation happens before index lookup and before the SQLite transaction, so a rejected request cannot create, remove, or modify an alias and never contacts Qdrant.

```console
cargo test alias_actions_reject_options_the_gateway_cannot_apply
```

The regression covers filters, all three routing forms, write-index and hidden flags, removal `must_exist`, and an unknown option. It also asserts that the catalogue still contains only its original alias after all failures. Docker was not responsive during this run, so no live Elasticsearch, OpenSearch, or Qdrant endpoint result is claimed. This change does not add filtered, routed, hidden, or multi-index aliases; it makes the existing compatibility boundary honest and fail closed.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-13-support-first-reply-queue.md

# Who is still waiting for a first reply?

A support lead needs open billing tickets with no replies and no assigned agent. Seven synthetic tickets include assigned, answered, closed, and technical cases that must stay out of the queue. Missing and null assignees qualify; an empty string counts as present, so producers should normalize blank assignments to null.

Use running Elasticsearch-compatible servers and a fresh `field_note_support` index. Curl is the client: run these commands on each server, changing only `ES` to the gateway URL (for example, `http://localhost:9200`).

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_support" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"subject":{"type":"text"},"team":{"type":"keyword"},"status":{"type":"keyword"},"channel":{"type":"keyword"},"reply_count":{"type":"integer"},"assignee":{"type":"keyword"}}}}'
curl -fsS -X POST "$ES/field_note_support/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"ticket-101"}}
{"subject":"Duplicate subscription charge","team":"billing","status":"open","channel":"email","reply_count":0}
{"index":{"_id":"ticket-102"}}
{"subject":"Invoice address correction","team":"billing","status":"open","channel":"chat","reply_count":0,"assignee":null}
{"index":{"_id":"ticket-103"}}
{"subject":"Refund progress","team":"billing","status":"open","channel":"email","reply_count":0,"assignee":"agent-7"}
{"index":{"_id":"ticket-104"}}
{"subject":"Receipt request","team":"billing","status":"open","channel":"chat","reply_count":1}
{"index":{"_id":"ticket-105"}}
{"subject":"Cannot sign in","team":"technical","status":"open","channel":"email","reply_count":0}
{"index":{"_id":"ticket-106"}}
{"subject":"Payment confirmation","team":"billing","status":"closed","channel":"email","reply_count":0}
{"index":{"_id":"ticket-107"}}
{"subject":"Missing invoice number","team":"billing","status":"open","channel":"email","reply_count":0,"assignee":""}
NDJSON
sleep 5
```

The same five-second settling pause follows writes on both endpoints; this does not establish immediate gateway refresh visibility. Find tickets needing an owner and a first reply:

```sh
curl -fsS -X POST "$ES/field_note_support/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"filter":[{"term":{"team":"billing"}},{"term":{"status":"open"}},{"term":{"reply_count":0}}],"must_not":[{"exists":{"field":"assignee"}}]}}}'
```

Expect exactly `ticket-101` (duplicate charge) and `ticket-102` (address correction), their complete sources, and total `2`; hit order is immaterial. Count contact channels over this same queue:

```sh
curl -fsS -X POST "$ES/field_note_support/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"team":"billing"}},{"term":{"status":"open"}},{"term":{"reply_count":0}}],"must_not":[{"exists":{"field":"assignee"}}]}},"aggs":{"channels":{"terms":{"field":"channel","size":10}}}}'
```

Expect `email: 1` and `chat: 1`, with no hits. Assign the duplicate-charge ticket:

```sh
curl -fsS -X POST "$ES/field_note_support/_update/ticket-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"assignee":"agent-12"}}'
sleep 5
curl -fsS "$ES/field_note_support/_doc/ticket-101"
```

The update returns `updated`. The read shows `assignee: "agent-12"` while preserving the subject, team, status, channel, and zero replies. Repeat both search requests: only `ticket-102` remains, total `1`, and the sole channel bucket is `chat: 1`. Assignment alone removes a ticket from the unowned queue even before anyone replies.

The [fixture](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/fixtures/support.json) and [replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/support.py) check statuses, bulk results, every stored ID/source, exact search totals, channel counts, and assignment exclusion. The [recorded run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/support-2026-10-13.json) passed 74/74 checks on Elasticsearch 8.15.0 and gateway 0.2.0 with Qdrant 1.15.5, using identical requests and default embedded-source, synchronous writes. The curl blocks were also run on both endpoints. No gateway code change or performance claim is involved.

Boundary: updates are read-merge-write, so atomic concurrent ticket claiming remains [unsupported](https://dstockton.github.io/qdrant-es-gateway/compatibility/).

Remove the example index when finished:

```sh
curl -fsS -X DELETE "$ES/field_note_support"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-13-creating-an-index-is-not-a-mapping-update.md

# Creating an index is not a mapping update

Provisioning code often treats “already exists” as a narrow, recoverable race. It can do that reliably only when the server identifies the condition correctly. The gateway already rejected a second `PUT /products` before contacting Qdrant, but it grouped the request with unsupported query features and returned `qdrant_gateway_unsupported_query`. That classification could make an Elasticsearch or OpenSearch client treat a normal create race as an incompatible request.

Elasticsearch and OpenSearch implement this case independently but agree on the exception class. [Elasticsearch 9.5.5's create-index service](https://github.com/elastic/elasticsearch/blob/v9.5.5/server/src/main/java/org/elasticsearch/cluster/metadata/MetadataCreateIndexService.java#L291-L295) throws `ResourceAlreadyExistsException` when either routing or index metadata already contains the name. [OpenSearch 3.9's corresponding service](https://github.com/opensearch-project/OpenSearch/blob/3.9/server/src/main/java/org/opensearch/cluster/metadata/MetadataCreateIndexService.java#L275-L279) performs the same two checks. OpenSearch also publishes the resulting REST example as HTTP 400 with `resource_already_exists_exception` in its [create/delete error-shape discussion](https://github.com/opensearch-project/OpenSearch/issues/14805).

The gateway now returns the shared client-visible shape, including a root cause, top-level exception, reason, index name, and status 400. The check runs under the existing index-administration lock and before any Qdrant request. A focused regression also proves that the original mapping remains unchanged:

```console
cargo test duplicate_index_creation_returns_native_conflict_without_qdrant_work
```

This is request-level validation with a mocked upstream request counter, not a live-engine benchmark. Cached Elasticsearch 9.5.5 and OpenSearch 3.9.0 containers were available for a native probe, but the local Docker storage backend returned an input/output error before either container started. The version-specific source above is therefore the native evidence for this run. The gateway does not include a native index UUID because its metadata catalogue does not currently assign one. A different requested name that merely normalizes to an occupied Qdrant collection remains a gateway-specific `index.name` validation error rather than pretending that the Elasticsearch/OpenSearch index itself already exists.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-13-alias-add-is-not-a-retarget.md

# Alias add is not a retarget

The gateway stores one concrete index per alias. Its alias transaction used SQLite `INSERT OR REPLACE` for an `add` action, so adding an existing alias to a different index silently discarded the original target. That looked like a convenient retarget, but it changed the meaning of a standard Elasticsearch/OpenSearch request and could redirect application reads and writes without the caller expressing a swap.

A direct endpoint probe against Elasticsearch 9.5.5 and OpenSearch 3.9.0 used the same five-request sequence: create `products-v1` and `products-v2`, add `current-products` to the first, add it to the second, then inspect the alias. Both native engines acknowledged both adds and returned both indices as targets. This agrees with the independent [Elasticsearch alias documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/aliases.html) and [OpenSearch manage-aliases documentation](https://docs.opensearch.org/latest/api-reference/alias/aliases-api/): aliases can cover multiple indices, and a move is expressed as remove plus add in one atomic request. The sanitized [probe evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-9.5.5-opensearch-3.9-alias-add-2026-10-13.json) records the versions, requests, acknowledgements, and final targets.

The gateway still deliberately supports only single-target aliases. It now evaluates the final topology of the complete action batch before opening the write transaction. An add-only request that would require two targets receives a structured HTTP 400 compatibility error, and the existing target remains unchanged. A remove+add swap remains accepted in either action order because the final topology has one target and the SQLite transaction makes the change atomically visible.

```console
cargo test alias_add_does_not_silently_retarget_a_single_index_alias
```

This is a fail-closed compatibility improvement, not multi-index alias support. The gateway does not yet merge searches across indices, select a write index, apply alias filters, or reproduce native per-action partial-failure responses. The endpoint probe measures request semantics only; it makes no latency or resource-use claim.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-12-head-must-not-hide-storage-failures.md

# HEAD must not hide storage failures

Document-existence checks have a deliberately small contract. The [Elasticsearch 8 API reference](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-exists) and the [OpenSearch Get Document API reference](https://docs.opensearch.org/latest/api-reference/document-apis/get-documents/#checking-document-existence) both describe `HEAD /<index>/_doc/<id>` as HTTP 200 when the document exists and HTTP 404 when it does not. A storage outage is neither result.

The gateway's HEAD path previously caught every Qdrant error and returned 404. A connection failure, missing collection, or Qdrant 5xx therefore looked like a trustworthy “document absent” answer. That can make health checks, conditional writes, and reconciliation jobs take destructive action on false information.

HEAD now uses the same narrow missing-point classification as GET without retrieving the document source. The known Qdrant missing-point response remains a 404, while transport failures and upstream errors return 502. All gateway HEAD routes also attach `X-Elastic-Product: Elasticsearch`, and the response body remains empty. The focused regression injects a Qdrant 503 and verifies an empty 502 response in both embedded-source and document-projection modes:

```console
cargo test head_document_does_not_hide_qdrant_failures_as_not_found
```

This validation is request-level and uses a mocked Qdrant response; it is not a live Elasticsearch, OpenSearch, or Qdrant benchmark. The native status contract comes from the public engine documentation above. Query parameters such as `routing`, `refresh`, `realtime`, and explicit version checks remain outside the gateway's current HEAD compatibility surface.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-12-a-missing-point-is-not-a-failed-upstream.md

# A missing point is not a failed upstream

The Elasticsearch and OpenSearch index APIs are upserts, but their response describes which side of the operation happened. Live probes against Elasticsearch 9.5.5 and OpenSearch 3.9.0 both returned HTTP 201 with `result: "created"` for the first `PUT /<index>/_doc/<id>`, then HTTP 200 with `result: "updated"` for the replacement.

The gateway already performed a point read before this write so it could make the same distinction. Against the repository's pinned Qdrant 1.15.5, however, a missing single point returns HTTP 404 with a structured “Point with id … does not exists” error. The generic Qdrant client treated every non-success response as an upstream failure, so the very first index request returned 502 and never reached the upsert.

Single-point reads now recognize that exact missing-point response as an absent document. The shared path fixes ordinary GETs and the preflight reads used by standalone index, create, update, and delete operations. The exception is deliberately narrow: a Qdrant 404 that says the collection is missing still becomes a gateway 502, preserving a visible signal for storage/catalogue drift instead of pretending the document is absent.

The sanitized [Elasticsearch 9.5.5, OpenSearch 3.9.0, and Qdrant 1.15.5 evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-9.5.5-opensearch-3.9-qdrant-missing-point-2026-10-12.json) records the native contract, Qdrant response shape, and live changed-gateway results. The gateway returned 404 for a missing GET, 201/`created` for the first PUT, 200/`updated` for the second PUT, and the replacement source on the final GET. Focused regressions use the observed Qdrant error body in both source-storage modes and prove that missing collections remain upstream errors.

This is a single-document correctness check, not a concurrency or performance benchmark. The extra existence read was already part of the standalone index path, so the fix adds no request. That preflight still is not an atomic compare-and-write across gateway replicas, and the gateway's sequence-number, primary-term, and version fields remain approximate.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-11-travel-disruption-desk.md

# Two passengers, one cancelled departure

A travel disruption desk needs departures from York with at least two seats, either direct to London or to Peterborough for an onward connection. Six synthetic services include a one-seat departure, a different origin, an unsuitable destination, and an already cancelled train. These are illustrative inventory snapshots, not live timetables.

Use running Elasticsearch-compatible servers and a fresh `field_note_travel` index. Curl is the client: run these commands on each server, changing only `ES` to the gateway URL (for example, `http://localhost:9200`).

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_travel" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"title":{"type":"text"},"origin":{"type":"keyword"},"destination":{"type":"keyword"},"operator":{"type":"keyword"},"seats_available":{"type":"integer"},"status":{"type":"keyword"}}}}'
curl -fsS -X POST "$ES/field_note_travel/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"service-101"}}
{"title":"York to London morning","origin":"York","destination":"London","operator":"North Rail","seats_available":4,"status":"scheduled"}
{"index":{"_id":"service-102"}}
{"title":"York to Peterborough connector","origin":"York","destination":"Peterborough","operator":"East Rail","seats_available":2,"status":"scheduled"}
{"index":{"_id":"service-103"}}
{"title":"York to London last seat","origin":"York","destination":"London","operator":"North Rail","seats_available":1,"status":"scheduled"}
{"index":{"_id":"service-104"}}
{"title":"Leeds to London morning","origin":"Leeds","destination":"London","operator":"East Rail","seats_available":8,"status":"scheduled"}
{"index":{"_id":"service-105"}}
{"title":"York to Edinburgh morning","origin":"York","destination":"Edinburgh","operator":"North Rail","seats_available":6,"status":"scheduled"}
{"index":{"_id":"service-106"}}
{"title":"York to London cancelled","origin":"York","destination":"London","operator":"East Rail","seats_available":9,"status":"cancelled"}
NDJSON
sleep 5
```

Both endpoints use the same five-second settling pause; this does not establish immediate gateway refresh visibility.

Find departures that can accommodate both passengers. `terms` accepts either destination; all four filters must hold:

```sh
curl -fsS -X POST "$ES/field_note_travel/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"filter":[{"term":{"origin":"York"}},{"terms":{"destination":["London","Peterborough"]}},{"term":{"status":"scheduled"}},{"range":{"seats_available":{"gte":2}}}]}}}'
```

Expect exactly `service-101` and `service-102`, with complete sources and total `2`; hit order is immaterial. Count operators over that same eligible set:

```sh
curl -fsS -X POST "$ES/field_note_travel/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"origin":"York"}},{"terms":{"destination":["London","Peterborough"]}},{"term":{"status":"scheduled"}},{"range":{"seats_available":{"gte":2}}}]}},"aggs":{"operators":{"terms":{"field":"operator","size":10}}}}'
```

Expect `North Rail: 1` and `East Rail: 1`, with no hits. Now cancel the direct departure:

```sh
curl -fsS -X POST "$ES/field_note_travel/_update/service-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"status":"cancelled"}}'
sleep 5
curl -fsS "$ES/field_note_travel/_doc/service-101"
```

The update returns `updated`. The read shows `status: "cancelled"`, preserving the title, origin, destination, operator, and four available seats. Repeat both search requests: only `service-102` remains, total `1`, and the sole operator bucket is `East Rail: 1`. Cancellation must exclude the service even when its seat count still looks sufficient. The booking system must separately validate the connection and reserve seats.

The [fixture](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/fixtures/travel.json) and [replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/travel.py) check statuses, bulk results, every stored ID/source, search totals, operator facets, and cancellation exclusion. The [recorded run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/travel-2026-10-11.json) passed 70/70 checks on Elasticsearch 8.15.0 and gateway 0.2.0 with Qdrant 1.15.5, using identical requests and default embedded-source, synchronous writes. The curl blocks were also executed on both endpoints. No gateway code change or performance claim is involved.

Boundary: atomic seat reservations through concurrent updates remain [unsupported](https://dstockton.github.io/qdrant-es-gateway/compatibility/).

Remove the example index when finished:

```sh
curl -fsS -X DELETE "$ES/field_note_travel"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-11-source-filters-should-not-leak-siblings.md

# Source filters should not leak sibling fields

Elasticsearch clients often use `_source` filtering to keep response payloads small or avoid returning fields a particular view does not need. The gateway accepted array-shaped top-level filters, but a string such as `"profile.name"` silently returned the complete source. Nested paths were compared only with top-level keys, so even some accepted filters returned the wrong shape. A caller asking for a profile name could therefore receive sibling contact fields or unrelated metadata.

A live probe against native Elasticsearch 9.5.5 established four narrow response contracts. A single dotted field returned `{"profile":{"name":"Ada"}}`; `profile.contact.*` preserved the nested contact object; an include of `profile.*` plus an exclusion of `profile.contact.phone` retained the email but removed the phone; and a numeric `_source` value failed with HTTP 400. The changed gateway produced the same shapes against pinned Qdrant 1.15.5. The sanitized [endpoint evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-9.5.5-source-filtering-2026-10-11.json) records the fixture, requests, statuses, and projected sources.

The gateway now parses source filters before index lookup or Qdrant access, supports string and string-array patterns plus the singular and plural include/exclude options, and reconstructs nested objects while applying `*` and `?` wildcards. Collapse inner hits use the same parser. Malformed or mixed-type filters fail rather than falling back to the full source.

This is response minimization, not field-level authorization. The gateway still hydrates the complete source before applying the filter, so source filtering does not reduce Qdrant read bandwidth or in-process memory use. The probe covers nested objects and common wildcard forms, not every Elasticsearch source-filtering edge case. OpenSearch was not run, so this result does not claim OpenSearch parity.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-11-integer-hit-totals-need-the-minus-one-sentinel.md

# Integer hit totals need the minus-one sentinel

OpenSearch retains a legacy response mode for clients that expect `hits.total` to be a number rather than the newer `{value, relation}` object. A live probe against OpenSearch 3.9.0 confirmed that `rest_total_hits_as_int=true` returns the exact integer for an ordinary search and the sentinel `-1` when the request also sets `track_total_hits: false`. The same query parameter applies to every item in `_msearch`. The gateway previously discarded the query string, so these requests silently received the wrong response shape.

The gateway now carries the parameter through both endpoints, returns integer totals and the native disabled sentinel, and preserves the existing object shape for `false` or an absent parameter. Parsing matches the observed OpenSearch boundary: lowercase `true` and `false` are accepted, an empty or valueless parameter enables the mode, duplicate parameters use the last value, and a numeric `track_total_hits` threshold combined with integer formatting fails before index lookup or Qdrant work because the requested format cannot represent an inexact threshold.

The sanitized [OpenSearch 3.9.0 evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/opensearch-3.9-rest-total-hits-as-int-2026-10-11.json) records six `_search` cases and the two `_msearch` totals over three documents. A live gateway backed by pinned Qdrant 1.15.5 matched those shapes; focused tests also cover strict parsing, duplicate parameters, fail-fast validation, and upstream request counts.

This is response-shape compatibility, not a performance benchmark. Integer totals have no place to carry the gateway's normal `eq` or `gte` relation, so callers using bounded text-search paths should prefer the object form when accuracy metadata matters. Elasticsearch was not run for this note, and no cross-engine parity claim is made.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-10-disabled-hit-totals-should-skip-the-count.md

# Disabled hit totals should skip the count

OpenSearch lets a search caller choose the cost and precision of `hits.total`. A live probe against OpenSearch 3.6.0 with 15 matching documents confirmed the three useful modes: the default returned `15/eq`, `track_total_hits: false` omitted `hits.total`, a threshold of 5 returned `5/gte`, and a threshold of 20 returned `15/eq`. The gateway previously ignored all four request shapes and always paid for its exact Qdrant count on eligible filter-only searches.

The gateway now accepts boolean and non-negative integer modes. Disabling totals omits the response field and, importantly, skips the exact-count request. A live gateway backed by the repository's pinned Qdrant 1.15.5 image matched all four OpenSearch response shapes after write visibility converged. A request-level regression with a mocked Qdrant endpoint records one upstream scroll request instead of the usual count plus scroll pair. Numeric thresholds cap exact or bounded totals and preserve an honest `eq` or `gte` relation.

The [saved native-engine evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/opensearch-3.6-track-total-hits-2026-10-10.json) contains the fixture and response values without endpoint or container identifiers. This is a request-count result, not a latency benchmark: the actual saving depends on collection size, filters, Qdrant load, and network placement.

There are two deliberate boundaries. A numeric threshold shapes the gateway response but does not terminate Qdrant's exact count early, so only `false` has a demonstrated upstream request saving. OpenSearch 3.6.0 also coerced some undocumented string, fractional, and negative values during the probe; the gateway accepts the documented boolean and non-negative integer forms and rejects other types before index lookup or Qdrant work. Elasticsearch was not run for this field note, so the native evidence here is OpenSearch-specific even though the supported request forms are shared by both APIs.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-09-search-after-needs-more-than-one-page.md

# Search-after needs more than one page

A deterministic catalogue replay against native Elasticsearch 9.5.5 exposed an empty second page from the gateway. Six products were sorted by ascending price, the first two sort values were returned as the cursor, and Elasticsearch returned the next two products. The gateway returned none.

The cursor comparison itself was correct. The gateway had asked Qdrant for only `size` candidates, sorted those candidates locally, and then tried to find a cursor from the previous request inside that new first-page-sized set. The same boundary could also make the first sorted page depend on Qdrant's unrelated point order.

Explicitly sorted searches now retrieve a candidate window of at least the configured `MAX_PAGE_SIZE` before ordering and applying `search_after`. A request-level regression uses a Qdrant mock that really truncates its response to the requested limit; with a three-candidate window, a two-hit first page and its cursor produce the remaining document. The live replay with the default 1,000-candidate window improved from 10/12 to 11/12 checks against Elasticsearch 9.5.5. The saved [endpoint evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/elasticsearch-9.5.5-2026-10-09.json) records the exact requests, endpoint identities, and one-second gateway convergence allowance.

This is bounded compatibility, not global Qdrant field ordering. A cursor cannot advance beyond the configured candidate window, changes to that window can invalidate a cursor, and each explicitly sorted request can now retrieve and hydrate up to `MAX_PAGE_SIZE` candidates even when the returned page is small. The remaining replay difference is the already documented analyzer boundary: Elasticsearch's lowercase prefix over a `text` field matches `Wireless`, while gateway-side stored-source matching remains case-sensitive. This run validated Elasticsearch 9.5.5; it did not treat the prior OpenSearch 3.6.0 replay as proof of identical server behavior.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-09-opensearch-wildcards-keep-the-leading-star.md

# OpenSearch wildcards keep the leading star

A catalogue replay against native OpenSearch 3.6.0 found a gateway-only empty result for `{"wildcard":{"title":"*keyboard*"}}`. The document was present and other filters worked. The defect was in the gateway's wildcard-to-regular-expression translation: a leading `*` disappeared, turning a contains query into a starts-with query.

The translator now handles the pattern one character at a time. `*` becomes any-length text, `?` becomes one character, and every other character is escaped as a literal. A focused regression covers leading, trailing, and consecutive stars, `?`, and regular-expression punctuation in source text.

The reproducible endpoint replay used native OpenSearch 3.6.0, Qdrant ES Gateway from this change, and the repository's pinned Qdrant 1.15.5 image. With a one-second post-ingest settle, 10 of 12 application checks passed after the fix. The remaining differences are explicit limitations: a lowercase prefix over an analyzed OpenSearch `text` field does not match the gateway's mixed-case stored source, and gateway-side `search_after` ordering operates over a bounded candidate window. The saved [request-level evidence](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/opensearch-3.6-2026-10-09.json) records both endpoint identities, requests, responses, and the settle interval.

The settle is part of the result, not a hidden correction. OpenSearch honored `refresh=wait_for`; the gateway's sparse search projection still needs a convergence allowance. The replay now records that allowance and sends `refresh=wait_for` for the final delete on both endpoints, preventing unmatched refresh timing from masquerading as a count incompatibility.

The current official Python client was checked separately at the transport boundary: `opensearch-py` 3.2.0 completed info, index creation, bulk indexing, search, count, update, get, and delete against the gateway. That is client-shape evidence, not a claim that the gateway identifies itself as OpenSearch or implements the full OpenSearch API.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-09-media-clearance-shortlist.md

# Is this coastal clip cleared for the UK edit?

A media producer needs coastal footage under 31 seconds with UK clearance. A French-only clip, a long coastal walk, and an uncleared aerial should stay out of the shortlist. These six synthetic archive records use topic and territory arrays; matching either array means matching any element.

Use running Elasticsearch-compatible servers and a fresh `field_note_media` index. Curl is the client: execute the same commands against each server, changing only `ES` to the gateway URL (for example, `http://localhost:9200`).

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_media" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"title":{"type":"text"},"topics":{"type":"keyword"},"territories":{"type":"keyword"},"format":{"type":"keyword"},"duration_seconds":{"type":"integer"}}}}'
curl -fsS -X POST "$ES/field_note_media/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"clip-101"}}
{"title":"Whitby harbour at dawn","topics":["coast","harbour"],"territories":["GB","FR"],"format":"landscape","duration_seconds":20}
{"index":{"_id":"clip-102"}}
{"title":"Cornish surf close-up","topics":["coast","surf"],"territories":["GB"],"format":"portrait","duration_seconds":12}
{"index":{"_id":"clip-103"}}
{"title":"Brittany lighthouse","topics":["coast"],"territories":["FR"],"format":"landscape","duration_seconds":18}
{"index":{"_id":"clip-104"}}
{"title":"Northumberland coastal walk","topics":["coast","walking"],"territories":["GB"],"format":"landscape","duration_seconds":75}
{"index":{"_id":"clip-105"}}
{"title":"York market stalls","topics":["city","food"],"territories":["GB"],"format":"portrait","duration_seconds":15}
{"index":{"_id":"clip-106"}}
{"title":"Uncleared beach aerial","topics":["coast"],"territories":[],"format":"landscape","duration_seconds":25}
NDJSON
sleep 5
```

The same five-second settling pause is used on both endpoints; it does not establish immediate gateway refresh visibility.

Find clips with the `coast` topic, `GB` clearance, and at most 30 seconds of footage:

```sh
curl -fsS -X POST "$ES/field_note_media/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"filter":[{"term":{"topics":"coast"}},{"term":{"territories":"GB"}},{"range":{"duration_seconds":{"lte":30}}}]}}}'
```

Expect exactly `clip-101` (Whitby harbour) and `clip-102` (Cornish surf), with their complete sources and total `2`. Hit order is immaterial. Count formats over that same filtered shortlist:

```sh
curl -fsS -X POST "$ES/field_note_media/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"topics":"coast"}},{"term":{"territories":"GB"}},{"range":{"duration_seconds":{"lte":30}}}]}},"aggs":{"formats":{"terms":{"field":"format","size":10}}}}'
```

Expect `landscape: 1` and `portrait: 1`, with no hits. Now the archive removes UK clearance from the harbour clip, leaving France:

```sh
curl -fsS -X POST "$ES/field_note_media/_update/clip-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"territories":["FR"]}}'
sleep 5
curl -fsS "$ES/field_note_media/_doc/clip-101"
```

The update returns `updated`. The read must show `territories: ["FR"]`, preserving the title, both topics, landscape format, and 20-second duration. Repeat the two search requests above: only `clip-102` remains, total `1`, and the sole format bucket is `portrait: 1`. Replacing an array must remove the old clearance, not append to it. This is metadata retrieval; the archive remains responsible for clearance decisions.

The [fixture](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/fixtures/media.json) and [replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/media.py) check statuses, bulk results, every stored ID/source, exact search totals, format counts, and clearance removal. The [recorded run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/media-2026-10-09.json) passed 70/70 checks on Elasticsearch 8.15.0 and gateway 0.2.0 with Qdrant 1.15.5, using identical requests and default embedded-source, synchronous writes. The curl blocks were also run on both endpoints. No gateway code change or performance claim is involved.

Boundary: concurrent atomic updates and Elasticsearch `nested` semantics for per-territory rights objects remain [unsupported](https://dstockton.github.io/qdrant-es-gateway/compatibility/).

Remove this example's index when finished:

```sh
curl -fsS -X DELETE "$ES/field_note_media"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-08-invalid-pages-should-not-query-qdrant.md

# Invalid pages should not query Qdrant

Pagination inputs sit on a resource boundary. [Elasticsearch documents `from` as non-negative, `size` as non-negative, and `search_after` as an array of scalar sort values](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-search). It also limits the ordinary `from` plus `size` result window to 10,000 by default and directs deeper pagination to `search_after`.

The gateway previously used defaults when `from` or `size` had the wrong JSON type, silently capped an oversized `size`, ignored a non-array `search_after`, and fetched candidates from Qdrant before discovering some cursor shape errors. Those responses could look valid while representing a different page from the one requested, and malformed requests still consumed upstream work.

Search pagination now passes through one fail-fast validator before index lookup or Qdrant access. `from` and `size` must be non-negative integers, `size` cannot exceed the configured `MAX_PAGE_SIZE`, and `from + size` cannot exceed 10,000. `search_after` requires one string, number, boolean, or null per explicit sort field and cannot be combined with a non-zero offset. A regression sends malformed and over-limit variants to a mock Qdrant server and verifies both the structured compatibility error and a zero upstream request count.

```bash
cargo test invalid_pagination_fails_before_qdrant_work
cargo test --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
```

This is validation, not unbounded pagination. The gateway still sorts only the bounded candidate window it retrieves, its 10,000 result-window boundary is fixed rather than index-configurable, and its cursor is stable only while that ordered set is unchanged. `MAX_PAGE_SIZE` may be set below 10,000 as a stricter operator-controlled resource limit.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-08-inner-hits-need-the-requested-name.md

# Inner hits need the requested name

The AWS Retail Demo Store sends a normal Elasticsearch collapse request: collapse on `category.keyword`, name the inner result `category_hits`, request a bounded number of group members, and suppress `_source`. Its response code then reads `item["inner_hits"]["category_hits"]` to distribute products across categories. The request is visible in the project's [search service](https://github.com/aws-samples/retail-demo-store/blob/master/src/search/src/search-service/app.py#L130-L158), and the corresponding response access is a few lines below.

The gateway previously treated every property inside the `inner_hits` definition as if it were an inner-hit name. A request containing `name`, `size`, `fields`, and `_source` therefore returned groups with those literal keys instead of `category_hits`. It also left `_source` in nested hits even when the inner definition disabled it. The request returned HTTP 200, but the unchanged application could not consume the response.

Collapse parsing now happens before index lookup or Qdrant access. The gateway emits the requested inner-hit name, honors the inner size and source projection independently from the top-level projection, accepts the application's `_id` metadata-field request, and caps inner results with `MAX_PAGE_SIZE`. Unsupported options fail with a structured 400 rather than being silently ignored.

The regression uses the public application's exact collapse shape over three mocked products in two categories. It verifies two collapsed top-level hits, a two-document `category_hits` group, correct IDs, no leaked `_source`, and zero Qdrant requests for invalid or over-limit collapse options:

```sh
cargo test retail_collapse_inner_hits_use_the_requested_name_size_and_source_filter
cargo test invalid_collapse_fails_before_qdrant_work
```

This remains deliberately bounded compatibility. The gateway currently accepts one inner-hit definition, does not implement inner-hit sorting, and groups only the retrieved candidate window. Group totals are exact within that window, not across all matches in a large index. Those limits are now explicit instead of being hidden behind a response shape that the target application could not read.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-07-post-filters-must-fail-closed.md

# Unsupported post filters must fail closed

The gateway documents a deliberately small `post_filter` subset: term, terms, numeric range, exists, match-all, and bool combinations of those clauses. An unsupported clause previously fell through the hydrated-source predicate as `false`. The request then succeeded with no hits, making an unsupported query indistinguishable from a valid filter that matched nothing. A multi-field term clause had a related problem: only the first field was evaluated.

The gateway now validates the complete `post_filter` tree before it looks up the index or contacts Qdrant. Unsupported query types, unknown bool options, non-numeric range bounds, and clauses with more than one field receive the existing structured HTTP 400 compatibility error. A regression calls search with a nonexistent index and unreachable Qdrant URL, then verifies that an unsupported wildcard post-filter still fails specifically as `post_filter.wildcard`; this makes the fail-before-upstream ordering reproducible without external services.

```bash
cargo test post_filter_validation
cargo test invalid_post_filter_fails_before_index_or_qdrant_lookup
cargo test --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
```

This change does not expand the supported filter set or make `post_filter` counts exact. Post-filtering still operates on the bounded candidate window returned by the main query, and date range evaluation remains outside this hydrated-source path. The benefit is narrower: unsupported semantics are reported honestly instead of being converted into plausible but incorrect empty search results.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-07-post-filters-are-not-aggregation-filters.md

# A post-filter is not an aggregation filter

Elasticsearch applies `post_filter` after aggregations. This lets a catalogue page show only one selected brand while its facet and metric aggregations still describe every document matched by the main query. The gateway applied the post-filter to its hydrated candidate hits before computing min, max, filter, and keyed-filter aggregations, so those aggregations described only the visible hits. Server-side terms facets did not have this bug, which made mixed aggregation responses especially misleading.

The search path now preserves its query-matched candidate window before applying hit-only transformations. Gateway-computed aggregations read that preserved window, while `post_filter`, collapse, and `search_after` continue to affect the returned hits. A request-level regression supplies three products, post-filters hits to the single Acme product, and verifies that a min aggregation still sees the lower-priced Beta products and that a Beta filter aggregation still counts both of them.

```bash
cargo test post_filter_does_not_narrow_gateway_computed_aggregations
cargo test
cargo clippy --all-targets --all-features -- -D warnings
```

This is a semantic correction, not an exact-aggregation implementation. These gateway-computed aggregations still see only the bounded candidate window retrieved for the search, so large or highly selective result sets can remain incomplete. Preserving the pre-filter window also temporarily keeps a second copy of those in-memory candidate values for requests that contain aggregations. Terms aggregations continue to use Qdrant's facet API and do not need that copy.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-07-logistics-missing-tracking.md

# Packed, but where is the tracking number?

A dispatch coordinator in York needs packed parcels that still lack a tracking number before the carrier arrives. These five synthetic shipments include an omitted field, an explicit `null`, an already labelled parcel, another depot, and a dispatched parcel. Adding tracking must remove only the newly labelled parcel from the exception list.

Start with a running Elasticsearch-compatible endpoint and an unused `field_note_parcels` index. The client is curl; run exactly the same commands on each server, changing only `ES` (`http://localhost:19200` for Elasticsearch, `http://localhost:9200` for the gateway).

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_parcels" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"contents":{"type":"keyword"},"depot":{"type":"keyword"},"carrier":{"type":"keyword"},"status":{"type":"keyword"},"tracking":{"type":"keyword"}}}}'

curl -fsS -X POST "$ES/field_note_parcels/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"parcel-101"}}
{"contents":"Ceramic mugs","depot":"York","carrier":"Parcel North","status":"packed"}
{"index":{"_id":"parcel-102"}}
{"contents":"Cotton towels","depot":"York","carrier":"City Express","status":"packed","tracking":null}
{"index":{"_id":"parcel-103"}}
{"contents":"Desk lamps","depot":"York","carrier":"Parcel North","status":"packed","tracking":"PN-103"}
{"index":{"_id":"parcel-104"}}
{"contents":"Plant pots","depot":"Leeds","carrier":"Parcel North","status":"packed"}
{"index":{"_id":"parcel-105"}}
{"contents":"Wool blankets","depot":"York","carrier":"City Express","status":"dispatched","tracking":"CE-105"}
NDJSON
sleep 5
```

Both servers get the same five-second settling pause after each write phase. Search the packed York parcels with no indexed tracking value:

```sh
curl -fsS -X POST "$ES/field_note_parcels/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"filter":[{"term":{"depot":"York"}},{"term":{"status":"packed"}}],"must_not":[{"exists":{"field":"tracking"}}]}}}'
```

Expect exactly `parcel-101` and `parcel-102`, with their complete sources and a total of `2`. An omitted field and JSON `null` both fail `exists`; an empty string would count as present, so ingest unknown tracking as `null` or omit it.

Count that same exception list by carrier:

```sh
curl -fsS -X POST "$ES/field_note_parcels/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"depot":"York"}},{"term":{"status":"packed"}}],"must_not":[{"exists":{"field":"tracking"}}]}},"aggs":{"carriers":{"terms":{"field":"carrier","size":10}}}}'
```

Expect `Parcel North: 1` and `City Express: 1`, with no hits because `size` is zero. Assign the first parcel's tracking number and read it back:

```sh
curl -fsS -X POST "$ES/field_note_parcels/_update/parcel-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"tracking":"PN-101"}}'
sleep 5
curl -fsS "$ES/field_note_parcels/_doc/parcel-101"
```

The update returns `updated`. The read must preserve `Ceramic mugs`, York, Parcel North, and `status: packed`, adding only `tracking: PN-101`. Repeat the search and facet commands: only `parcel-102` remains, the total is `1`, and only `City Express: 1` remains in the buckets. This records a label assignment, not proof of carrier collection.

The [fixture](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/fixtures/logistics.json), [replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/logistics.py), and [recorded run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/logistics-2026-10-07.json) verify identical requests, HTTP and bulk statuses, IDs, every source, totals, and carrier counts. Elasticsearch 8.15.0 and gateway 0.2.0 at `7e56704`, backed by Qdrant 1.15.5 in default embedded-source mode with synchronous writes, passed **66/66 checks**. No gateway code changed; this five-record check makes no performance claim.

Boundary: refresh timing guarantees and atomic concurrent updates remain outside the gateway's [supported semantics](https://dstockton.github.io/qdrant-es-gateway/compatibility/).

Remove only this example's index when finished:

```sh
curl -fsS -X DELETE "$ES/field_note_parcels"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-06-keyword-multifields-need-one-route.md

# Keyword multi-fields need one route

Elasticsearch clients and application code commonly address an exact-value view as `brand.keyword`. The gateway already treated that conventional suffix as the base `brand` field when it evaluated hydrated source, sorted results, collapsed groups, or requested a terms facet. Native Qdrant filters did not: a `term`, `terms`, `range`, or `exists` clause was sent upstream with the dotted key unchanged. With a base keyword payload, that request searched a field that was never written and could quietly return no hits.

The filter translator and gateway-side pattern collector now use the same suffix normalization as the existing source paths. The focused regression covers all four native predicate shapes, confirms that source evaluation agrees, and exercises a `prefix` query against `sku.keyword`.

Reproduce it with:

```bash
cargo test keyword_multifield_aliases_use_the_base_payload_field
cargo test gateway_patterns_match_keyword_values
```

This is a compatibility alias, not full Elasticsearch multi-field emulation. It does not create a second analyzed value, support custom analyzers, or make an arbitrary text field safe for exact filtering. The base field must already be mapped and mirrored as a filterable payload for native filters to work.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-06-a-page-is-not-a-total.md

# A page is not a total

The gateway used to retrieve at most `from + size` filter matches and then report the length of that candidate window as an exact Elasticsearch `hits.total`. A request for zero hits could therefore say that zero documents matched even when the collection contained many matches. Pagination made the same error less obvious: a ten-hit page could claim the total was ten.

Filter-only searches now reuse the exact Qdrant count operation already used by the `_count` endpoint. A regression fixture exposes 37 matching documents while the search asks for `size: 0`; the gateway sends `{"exact":true}` to Qdrant, keeps the internal scroll limit at one, returns no hit documents, and reports `{"value":37,"relation":"eq"}`. This verifies the protocol behavior without pretending that a local mock is a latency benchmark.

A live check against the repository's pinned Qdrant 1.15.5 image indexed three documents, two with the requested keyword, and returned an exact total of two with an empty hit page. The check waited one second after the bulk write: an immediate probe observed the project's existing refresh approximation and returned zero before the write became visible. That timing is a separate limitation, not evidence that the count path is inaccurate once the write is visible.

```bash
cargo test zero_size_search_uses_a_positive_qdrant_scroll_limit
cargo test --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
```

The accuracy has a cost: an eligible filter-only search now makes a count request as well as its hit-retrieval request. Text searches and searches with `post_filter` still use a bounded candidate window; they now return `relation: "gte"` instead of incorrectly describing that lower bound as exact. Approximate-text and pattern paths already scan all candidates and remain exact. The gateway does not yet implement Elasticsearch's configurable `track_total_hits` threshold, and gateway-computed metric/filter aggregations remain bounded.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-05-sort-shapes-must-not-change-the-order.md

# Sort syntax must not change the order

[Elasticsearch's sort reference](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/sort-search-results) permits a basic field sort as `"price"`, `{"price":"asc"}`, or `{"price":{"order":"desc"}}`. The gateway previously understood only the direct-order object. It treated the standard options object as ascending regardless of its `order`, while a string field silently became `_score` descending. The same misread specification was then used to generate and consume `search_after` values.

Sort clauses now pass through one validated parser before any Qdrant request. Field names default to ascending, `_score` defaults to descending, and both direct and options-object `asc`/`desc` forms drive response ordering, returned sort values, and cursor comparison. `_id` and `_score` are read from hit metadata rather than incorrectly looked up in `_source`. Unsupported options and invalid directions return a structured 400 instead of producing a plausible but wrong page.

The regression uses a three-document response with a price tie. It verifies descending price with `_id` as a tie-breaker, advances past a two-value `search_after` cursor, and checks the single-string ascending form. Reproduce it without a live Qdrant instance:

```bash
cargo test elasticsearch_sort_shapes_drive_order_and_search_after
```

This does not turn response sorting into Qdrant-native global ordering. The gateway still sorts only the bounded candidate window it retrieves, and the cursor remains stable only while that ordered set is unchanged. Large collections or deep ordered pagination should use a native Qdrant ordering design rather than treating this compatibility layer as a distributed sort engine.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-05-education-course-availability.md

# An evening class with room for one more

An adult learner wants an evening Python class in York. A full course, a daytime class, or a place in Leeds is no help. These six synthetic course listings test that distinction, including the moment the last introductory place disappears.

Start with a running Elasticsearch-compatible endpoint and a fresh `field_note_courses` index. The client is curl; run the identical commands on each server, changing only `ES`: `http://localhost:19200` for Elasticsearch, `http://localhost:9200` for the gateway.

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_courses" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"title":{"type":"text"},"campus":{"type":"keyword"},"session":{"type":"keyword"},"level":{"type":"keyword"},"places":{"type":"integer"}}}}'

curl -fsS -X POST "$ES/field_note_courses/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"course-101"}}
{"title":"Python for beginners","campus":"York","session":"evening","level":"introductory","places":1}
{"index":{"_id":"course-102"}}
{"title":"Python data analysis","campus":"York","session":"evening","level":"intermediate","places":4}
{"index":{"_id":"course-103"}}
{"title":"Python automation","campus":"York","session":"evening","level":"intermediate","places":0}
{"index":{"_id":"course-104"}}
{"title":"Python for beginners","campus":"York","session":"daytime","level":"introductory","places":8}
{"index":{"_id":"course-105"}}
{"title":"Python for beginners","campus":"Leeds","session":"evening","level":"introductory","places":6}
{"index":{"_id":"course-106"}}
{"title":"Watercolour basics","campus":"York","session":"evening","level":"introductory","places":3}
NDJSON
sleep 5
```

Both runs use the same five-second settling pause. Find Python courses with at least one place, at the right campus and time:

```sh
curl -fsS -X POST "$ES/field_note_courses/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"must":{"match":{"title":"python"}},"filter":[{"term":{"campus":"York"}},{"term":{"session":"evening"}},{"range":{"places":{"gte":1}}}]}}}'
```

Expect exactly `course-101` and `course-102`, with complete sources; order and scores are not asserted. The full course, daytime class, Leeds class, and watercolour class must stay out.

Separately, count **all available evening classes in York** by level, including watercolour:

```sh
curl -fsS -X POST "$ES/field_note_courses/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"campus":"York"}},{"term":{"session":"evening"}},{"range":{"places":{"gte":1}}}]}},"aggs":{"levels":{"terms":{"field":"level","size":10}}}}'
```

Expect `introductory: 2`, `intermediate: 1`, and no hits (`size: 0`). These are course counts, not a sum of available places.

After the enrolment system reports that the last introductory place is taken, publish its new availability and read the listing back:

```sh
curl -fsS -X POST "$ES/field_note_courses/_update/course-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"places":0}}'
sleep 5
curl -fsS "$ES/field_note_courses/_doc/course-101"
```

The update returns `updated`; the read keeps the title, campus, session, and level, with `places: 0`. Repeat both search requests: only `course-102` remains, and both level buckets are `1` (watercolour still has places). Discovery reflects availability while the full course remains readable by ID.

The [replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/education.py) independently checks statuses, bulk results, every stored ID/source, matching IDs/sources, and level counts on both endpoints. The [recorded run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/education-2026-10-05.json) passed 66/66 checks with identical requests on Elasticsearch 8.15.0 and gateway 0.2.0 backed by Qdrant 1.15.5. No gateway change or performance claim is involved.

Boundary: text-scoped terms facets, atomic seat reservation, Lucene score equivalence, and refresh timing guarantees are outside this example's [supported semantics](https://dstockton.github.io/qdrant-es-gateway/compatibility/).

Remove the example index afterward:

```sh
curl -fsS -X DELETE "$ES/field_note_courses"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-05-arrays-are-multi-valued-fields.md

# Arrays are multi-valued fields, not scalar mismatches

Elasticsearch does not need a separate array field type: an ordinary field can carry several values, and a document matches when an applicable value satisfies the query. Qdrant payload filters use the same practical rule—an array filter succeeds when at least one value meets the condition ([Qdrant payload documentation](https://qdrant.tech/documentation/concepts/payload/)).

The gateway already inherited that behavior for filters executed inside Qdrant. Its local source evaluator did not. That evaluator is used by `post_filter`, filter aggregations, and gateway-side `prefix`, `wildcard`, and `regexp` queries. It compared the complete JSON array with one scalar term, required ranges to contain a scalar number, and required pattern fields to be scalar strings. The same document could therefore match a native filter but disappear when the equivalent clause moved to `post_filter`.

The source evaluator now recursively tests scalar values inside arrays and succeeds when any value matches. One regression covers exact term, terms-set, numeric range, prefix, wildcard, and regular-expression predicates over three array-valued fields. It also checks a non-matching range so the rule does not turn into an unconditional array match.

Reproduce the focused check with:

```bash
cargo test gateway_predicates_treat_arrays_as_multi_valued_fields
```

The full Rust suite contains 45 passing unit tests; the live Compose smoke remains opt-in. This change does not add Elasticsearch `nested` query semantics for arrays of objects, and it does not change the gateway's documented response-level sorting or collapse approximations. It only aligns scalar-array predicate evaluation across the native and hydrated-source paths.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-04-optional-should-really-is-optional.md

# Optional should really is optional

Elasticsearch changes the default for `minimum_should_match` according to the other clauses in the same `bool`. A `should`-only query requires one match, but `should` becomes optional when `must` or `filter` is present. The [Elasticsearch bool query reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html) documents the defaults as one and zero respectively.

The gateway used to reject the common required-plus-optional shape unless the client explicitly sent `minimum_should_match`. Its gateway-side source evaluator also required one `should` match, which could incorrectly remove hits from `post_filter`, filter aggregations, and approximate lexical scans.

The default is now resolved once and shared by both the Qdrant filter translator and the source evaluator. This regression covers a document that matches `filter.brand = Acme` but not `should.featured = true`: it remains a hit when the minimum is omitted and is excluded when the client explicitly requests one match. A second regression covers the same boundary in the approximate text evaluator.

Run the focused checks without Qdrant:

```bash
cargo test bool_should
```

This aligns hit selection, not scoring. Elasticsearch uses optional query-context clauses to boost relevant documents. Filter-only optional clauses are omitted from the Qdrant filter when their effective minimum is zero, so they do not change Qdrant scores. Clients that require a `should` clause to affect eligibility should continue to send an explicit `minimum_should_match`.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-04-nested-negation-keeps-its-boundary.md

# Nested negation keeps its boundary

Elasticsearch bool clauses accept either one query object or an array. They also preserve the nested bool boundary: excluding a bool that requires both `status = deleted` and `tenant = internal` means `NOT (status = deleted AND tenant = internal)`, not `status != deleted AND tenant != internal`.

The gateway previously handled only the array form of `must_not`. A scalar negative clause was silently omitted. When a nested bool appeared inside `must_not` or `should`, its inner `must_not` clauses were also discarded, and multiple required conditions under one negation were flattened into separate exclusions. Those cases widened or narrowed the result set without warning.

The Qdrant filter translator now builds one explicit filter condition for each nested clause. Required and excluded children stay grouped before the parent applies negation or `min_should`. The regression covers scalar double negation, a negative bool inside `should`, and the distinction between `NOT (A AND B)` and `NOT A AND NOT B`.

Run the focused check without Qdrant:

```bash
cargo test bool_negative_clauses_preserve_scalar_and_nested_semantics
```

The emitted shapes were also exercised against the pinned Qdrant 1.15.5 image. With one `deleted` and one `live` point, double negation selected only `deleted`; the negative `should` alternative selected only `live`.

This correction covers filter-only term, terms, range, exists, IDs, and nested bool conditions. Text queries under `must_not` or filter-only `should` remain rejected because translating their scoring and exclusion semantics safely requires a different execution path.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-03-pattern-counts-must-scan-source.md

# Pattern counts must scan the source

The gateway evaluates `wildcard`, `prefix`, and `regexp` clauses against hydrated `_source` documents because Qdrant does not receive an equivalent native filter. Search already followed that rule, but `_count` did not: it removed the pattern while building the Qdrant filter and then called Qdrant's count endpoint. A pattern-only count therefore returned the size of the whole collection.

Pattern counts now use the same paginated source scan and matching rules as pattern searches. A regression serves two mock Qdrant scroll pages containing `ABC-100`, `XYZ-100`, and `ABC-200`; `{"wildcard":{"sku":"ABC-*"}}` returns `2` and makes two scroll requests instead of an unfiltered count request.

Run the focused check without a Qdrant process:

```bash
cargo test count_applies_gateway_patterns_across_scroll_pages
```

This improves correctness, not query cost. A pattern-only count is linear in the candidate set, retains matching hits while the shared search path computes its total, and can observe concurrent writes between scroll pages. It is appropriate for modest catalog fields; large or latency-sensitive deployments should model a natively indexed exact-match field instead.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-03-manufacturing-maintenance-queue.md

# The pump is fixed. Take it off the queue.

A maintenance planner at a Leeds factory needs open pump faults, not last week's completed inspections or another plant's backlog. These five synthetic work orders make that distinction testable. Closing a job should remove it from the queue while preserving its original notes.

Use a running Elasticsearch-compatible endpoint and a fresh `field_note_maintenance` index. The client below is curl: run the same commands for each server, changing only `ES` (`http://localhost:19200` for Elasticsearch or `http://localhost:9200` for the gateway).

```sh
ES=http://localhost:19200
curl -fsS -X PUT "$ES/field_note_maintenance" \
  -H 'Content-Type: application/json' -d '{"mappings":{"properties":{"title":{"type":"text"},"plant":{"type":"keyword"},"team":{"type":"keyword"},"status":{"type":"keyword"},"downtime_minutes":{"type":"integer"}}}}'

curl -fsS -X POST "$ES/field_note_maintenance/_bulk?refresh=wait_for" \
  -H 'Content-Type: application/x-ndjson' --data-binary @- <<'NDJSON'
{"index":{"_id":"wo-101"}}
{"title":"Pump seal leaking","plant":"Leeds","team":"mechanical","status":"open","downtime_minutes":45}
{"index":{"_id":"wo-102"}}
{"title":"Conveyor belt slipping","plant":"Leeds","team":"mechanical","status":"open","downtime_minutes":20}
{"index":{"_id":"wo-103"}}
{"title":"Pump motor overheating","plant":"Bristol","team":"electrical","status":"open","downtime_minutes":60}
{"index":{"_id":"wo-104"}}
{"title":"Pump inspection complete","plant":"Leeds","team":"mechanical","status":"closed","downtime_minutes":0}
{"index":{"_id":"wo-105"}}
{"title":"Sensor signal intermittent","plant":"Leeds","team":"electrical","status":"open","downtime_minutes":10}
NDJSON
sleep 5
```

The five-second pause is the same on both endpoints; this check does not establish immediate search visibility after a gateway refresh.

Find open pump jobs at Leeds:

```sh
curl -fsS -X POST "$ES/field_note_maintenance/_search" \
  -H 'Content-Type: application/json' -d '{"size":10,"query":{"bool":{"must":{"match":{"title":"pump"}},"filter":[{"term":{"plant":"Leeds"}},{"term":{"status":"open"}}]}}}'
```

Only `wo-101`, “Pump seal leaking”, should return, with its complete source. The Bristol pump fault and the completed Leeds inspection must stay out.

Count all open Leeds jobs by team, independently of the pump text search:

```sh
curl -fsS -X POST "$ES/field_note_maintenance/_search" \
  -H 'Content-Type: application/json' -d '{"size":0,"query":{"bool":{"filter":[{"term":{"plant":"Leeds"}},{"term":{"status":"open"}}]}},"aggs":{"teams":{"terms":{"field":"team","size":10}}}}'
```

Expect `mechanical: 2` and `electrical: 1`, with no hits because `size` is zero. Now close the leaking-pump job and read it back:

```sh
curl -fsS -X POST "$ES/field_note_maintenance/_update/wo-101?refresh=wait_for" \
  -H 'Content-Type: application/json' -d '{"doc":{"status":"closed"}}'
sleep 5
curl -fsS "$ES/field_note_maintenance/_doc/wo-101"
```

The update returns `updated`; the read retains the title, plant, team, and 45 minutes of downtime, with `status: closed`. Repeat the two search requests: the pump result is empty and both team buckets are now `1`. The history survives; the actionable queue shrinks.

The [deterministic replay](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/manufacturing.py) checks these requests against explicit expectations on each server, including every document's ID and source. The [recorded run](https://github.com/dstockton/qdrant-es-gateway/blob/main/validation/evidence/manufacturing-2026-10-03.json) used Elasticsearch 8.15.0 and gateway 0.1.3 backed by Qdrant 1.15.3; all 62 checks passed with identical application requests and no gateway code change. This is a five-document correctness check, with no performance claim.

Boundary: Lucene score equivalence, refresh timing guarantees, and atomic concurrent updates are outside this example's [supported semantics](https://dstockton.github.io/qdrant-es-gateway/compatibility/).

To remove only this example's index afterward:

```sh
curl -fsS -X DELETE "$ES/field_note_maintenance"
```


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-03-exists-means-not-empty.md

# Exists means not empty

The gateway translated an Elasticsearch `exists` query directly into Qdrant's `is_empty` condition. That is the opposite predicate: documents with a missing, `null`, or empty-array value were selected, while documents with a real value were excluded.

The translation now negates `is_empty`. The same definition is used when the gateway evaluates a `post_filter` against hydrated `_source`: missing values, `null`, `[]`, and `[null]` do not exist; `""` and `[null, "Acme"]` do.

An isolated check against the repository's pinned Qdrant 1.15.3 image inserted those five boundary cases. The emitted filter returned only point 4 (the empty string) and point 5 (the array containing `"Acme"`):

```json
{"filter":{"must":[{"must_not":[{"is_empty":{"key":"brand"}}]}]}}
```

The regression checks are reproducible without a running Qdrant instance:

```bash
cargo test exists_query_excludes_empty_qdrant_payloads
cargo test post_filter_exists_rejects_missing_null_and_empty_arrays
```

This is not a claim of complete Elasticsearch field-existence parity. Server-side filtering applies to mapped fields mirrored into the searchable Qdrant payload, and the gateway does not emulate mapping-dependent cases such as `index: false`, `ignore_above`, or malformed values ignored during indexing.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-02-zero-results-still-need-a-positive-scroll.md

# Zero results still need a positive scroll

Elasticsearch clients commonly send `"size": 0` when they need aggregations but no document hits. The gateway passed that value through as Qdrant's scroll `limit`, turning a valid search into an upstream error.

An isolated check against the Qdrant 1.15.3 image pinned by this repository made the incompatibility concrete: the same empty collection returned HTTP 422 for `{"limit":0}` with `must be 1 or larger`, and HTTP 200 for `{"limit":1}`. The gateway now floors only the internal scroll limit at one, then applies the requested zero-sized page when it builds the Elasticsearch response. A regression test verifies both sides of that boundary: Qdrant receives `limit: 1`, while the client receives an empty `hits.hits` array.

```bash
cargo test zero_size_search_uses_a_positive_qdrant_scroll_limit
cargo test --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
```

This removes the 502 failure and lets server-side terms facets run for aggregation-only searches. It does not make all aggregation and total-count behavior exact: search totals and gateway-computed metric/filter aggregations are still bounded by the candidate window described in the compatibility guide.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-02-mapping-updates-must-not-erase-the-map.md

# Mapping updates must not erase the map

Elasticsearch sends a mapping update as `{"properties": ...}`, while index creation nests the same object under `mappings`. The gateway reused the index-creation parser for both shapes. A normal `PUT /products/_mapping` therefore interpreted the update as an empty mapping and replaced the stored definition with `{}`, silently losing the fields used to build search vectors and filter payloads for later writes.

Mapping updates now merge properties under the same index-administration lock used by other catalogue changes. Supported keyword, boolean, integer, float, and date additions also create the corresponding Qdrant payload index before the merged mapping is published in SQLite. Repeating an identical field definition is a no-op; changing an existing definition returns a structured 400 instead of rewriting search behavior.

The regression starts with a `title` text mapping, adds a `brand` keyword property using the Elasticsearch request shape, and verifies all three boundaries: `title` remains stored, `brand` gets a Qdrant keyword index, and the existing sparse-vector list is unchanged.

```bash
cargo test mapping_updates_merge_properties_and_create_payload_indexes
cargo test --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
```

An isolated check against the repository's pinned Qdrant 1.15.3 image confirmed that the emitted payload-index request creates `payload_schema.brand` as `keyword`. This does not make every Elasticsearch mapping mutation available. In particular, new text fields are rejected because that Qdrant version cannot add the required named sparse vector to an existing collection; create a new index and reindex when the searchable text schema changes. If a multi-field payload-index update fails partway through, Qdrant may retain an unused index, but the gateway does not publish the partial mapping.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-01-invalid-utf8-is-not-source-data.md

# Invalid UTF-8 is not source data

The gateway used to convert every HTTP body with a lossy UTF-8 decoder before parsing JSON or bulk NDJSON. An invalid byte inside a quoted value therefore became the Unicode replacement character (`U+FFFD`). If the value was source data or a bulk `_id`, the request could remain syntactically valid and write data under a value the client never sent.

The HTTP boundary now requires strict UTF-8 and returns a structured HTTP 400 response with `feature: request.body` before endpoint dispatch. The same check covers ordinary JSON, bulk, and multi-search requests. It runs after the configured byte limit, so oversized bodies still receive the more useful HTTP 413 response without an additional allocation beyond the already bounded body.

The regression injects byte `0xff` into a quoted document field and a quoted bulk identifier, then verifies that both requests are rejected with the Elasticsearch product header:

```bash
cargo test invalid_utf8_request_bodies_are_rejected_without_lossy_replacement
cargo test --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
```

This validation does not normalize valid Unicode or require ASCII. UTF-8 document content and identifiers continue to work unchanged, while clients that produce malformed encodings must correct or explicitly transcode them before sending JSON.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-10-01-alias-batches-are-one-routing-change.md

# Alias batches are one routing change

Elasticsearch's aliases API accepts a list of actions so clients can express a routing change as one request. The gateway previously applied each action to its SQLite catalogue while it was still parsing later actions. A valid `add` followed by a malformed `remove` could therefore return HTTP 400 after the new alias was already visible.

The gateway now validates every action first. Each item must contain exactly one supported `add` or `remove` action with object metadata and string `index` and `alias` values. Only after all target indices and alias-name conflicts have passed validation does one SQLite transaction apply the complete list. Removal also matches both alias and concrete index, rather than deleting an alias that points somewhere else.

The regression first submits a valid add followed by a remove with no index and verifies that no alias was added. It then switches an existing alias from `products-v1` to `products-v2` with a two-action request and verifies the committed target:

```sh
cargo test alias_actions_validate_before_an_atomic_catalogue_update
```

This guarantee covers the gateway's local metadata catalogue. The current alias model still has one target per alias; it does not implement filtered aliases, multi-index fan-out, write-index selection, wildcard actions, or a distributed transaction with Qdrant. Deployments with independent SQLite catalogues must still coordinate alias changes outside the gateway.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-30-bulk-mutations-need-document-ids.md

# Bulk mutations need document IDs

Elasticsearch requires bulk `update` and `delete` actions to identify the document they mutate. The gateway previously generated a random ID whenever `_id` was absent. That was particularly misleading for deletes: a malformed request could report `not_found` for a generated ID even though the caller never selected a document.

The bulk parser now returns a structured HTTP 400 before executing any item when an `update` or `delete` action omits `_id`. This preflight behavior also prevents earlier valid items in the same request from being applied before the malformed action is discovered. `index` and `create` retain their existing generated-ID behavior.

The same change accepts both `POST` and `PUT` for root and index-scoped `_bulk` routes, matching the documented Elasticsearch API surface. Regression tests exercise both missing-ID action types and verify that PUT requests receive the dedicated bulk body limit.

This does not add optimistic concurrency control: bulk sequence-number, primary-term, and version semantics remain outside the current compatibility subset.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-30-bulk-action-metadata-is-a-write-boundary.md

# Bulk action metadata is a write boundary

A bulk action line decides which operation, index, and document ID the gateway will mutate. Treating malformed metadata as if it were absent is therefore unsafe: a numeric `_id`, for example, previously fell through to generated-ID behavior for `index` and `create` actions. An action line containing more than one operation also silently selected whichever key the JSON object exposed first.

The gateway now validates the complete bulk request before executing any item. Every action line must contain exactly one of `index`, `create`, `update`, or `delete`; its metadata must be an object; and explicit `_index` and `_id` values must be strings. A malformed later action returns a structured HTTP 400 before an earlier valid action can write anything.

The regression test covers five cases, including malformed metadata after a valid item:

```bash
cargo test malformed_bulk_actions_fail_before_any_item_is_executed
```

This is request-level validation, not transactionality. A syntactically valid bulk request still has Elasticsearch-style item responses and may partially succeed when an upstream operation fails. Generated IDs also remain supported for `index` and `create` only when `_id` is genuinely omitted.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-29-unchanged-updates-should-not-write.md

# Unchanged updates should not write

Elasticsearch's Update API detects no-op updates by default. The gateway already had to read the stored source to merge a partial `doc`, but it wrote the merged source back to Qdrant even when no value changed. Repeated idempotent updates therefore consumed write capacity and reported `result: "updated"` instead of `result: "noop"`.

The update path now compares the recursively merged source with the stored source. If they are equal, it returns a no-op response and skips the Qdrant write. Callers that intentionally want a rewrite can send `detect_noop: false`.

A mock-Qdrant regression test covers both embedded-source and dedicated document-projection modes. For the same unchanged update, the measured upstream write count is zero with the default and one with no-op detection disabled. This is a deterministic request-count test, not a throughput benchmark; actual latency and capacity gains depend on the workload's share of unchanged updates.

The comparison covers JSON value equality after the gateway's recursive object merge. Updates remain read-merge-write operations rather than atomic compare-and-write transactions across gateway replicas.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-29-partial-updates-merge-inner-objects.md

# Partial updates should not erase sibling fields

Elasticsearch describes an Update API `doc` as a partial document merged into the stored source. That merge is recursive for inner objects: changing `details.warranty.years` must preserve both `details.manufacturer` and `details.warranty.region`. Arrays and scalar values, by contrast, replace their previous values.

The gateway previously merged only the top level. A patch such as:

```json
{"doc":{"details":{"warranty":{"years":3}}}}
```

replaced the complete `details` object and silently removed its other members. The update path now performs a simple recursive object merge before writing either the embedded source payload or the dedicated document projection. A mock-Qdrant regression test exercises both storage modes and also confirms that an array in the patch still replaces the stored array.

This is a source-integrity correction, not support for Elasticsearch's `nested` field query semantics. The operation still reads and writes the complete source, and the read/merge/write sequence is not an atomic compare-and-write across concurrent gateway replicas. Scripts and optimistic-concurrency controls also remain unsupported.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-28-bulk-items-should-preserve-the-operation-result.md

# Bulk items should preserve the operation result

An Elasticsearch bulk response is more than a status code. Each successful item identifies the concrete index that handled the operation and describes whether the document was created, updated, deleted, or not found. That distinction matters when a request uses an alias: clients need the backing index in the response, not the alias they sent.

The gateway's batched `index` path already built that response deliberately. The individual `create`, `update`, and `delete` paths did not. They discarded the successful standalone response, reconstructed a smaller item, and copied `_index` from the request. Alias-routed writes therefore reported the alias, while successful create and update items also omitted `result` and the version/shard metadata that the operation had already produced.

Bulk assembly now preserves the successful operation body and adds only the item-level HTTP status. A regression test sends create, update, and delete actions through `current-products` and verifies that every item reports the concrete `products` index, the correct result, and the existing version and shard fields:

```bash
cargo test bulk_alias_items_preserve_concrete_operation_results
```

This is a response-fidelity change; it adds no Qdrant requests and does not change write ordering or acknowledgement. It also does not make a bulk request transactional, add Elasticsearch version counters, or close the documented preflight/write races. Failed items retain their existing structured error shape, and deleting a missing document remains a non-error item with status 404 and `result: "not_found"`.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-28-a-partial-document-is-not-an-upsert-source.md

# A partial document is not an upsert source

Elasticsearch's Update API supports two source documents with different jobs. `doc` is merged into a document that already exists. A separate `upsert` object is inserted when the ID is absent. They need not contain the same fields:

```json
{
  "doc": {"price": 42},
  "upsert": {"title": "Initial", "price": 10}
}
```

The gateway already supported `doc_as_upsert`, but it required `doc` to double as the creation source. It now accepts the separate form for both standalone and bulk updates. An absent ID writes the `upsert` object and returns HTTP 201 with `result: "created"`; an existing ID continues to merge `doc` and ignores the creation source.

The regression test exercises both gateway storage layouts, verifies the exact `_source` sent to Qdrant, and proves that creation uses one existence lookup rather than the two previously incurred by the `doc_as_upsert` path:

```bash
cargo test supported_update_upserts_write_selected_source_with_one_preflight_each
```

This is not support for scripted updates. It also does not make the lookup and insert atomic across gateway replicas, implement Elasticsearch sequence numbers, or remove the normal projection-write costs. Invalid non-object `upsert` values are rejected before any document lookup or write.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-27-partial-updates-must-preserve-the-document.md

# A partial update must preserve the document

In the default embedded-source storage mode, a partial update to a non-text field took an optimized Qdrant payload path. That path wrote only the patch beneath `_source`. An update such as `{"doc":{"price":125}}` could therefore make the next document read return only `price`, dropping untouched fields such as `title` and `brand`. The mapped top-level `price` payload used by filters and sorts also remained stale.

The gateway already reads the current document to implement update semantics. It now merges the patch into that source and sends one payload update containing the complete `_source` plus the refreshed mapped filter/sort fields. The fix adds no Qdrant round trip and does not rebuild sparse vectors when no text field changed.

The regression starts with a four-field source and a mapped numeric price, applies a price-only patch, and inspects the Qdrant write. It verifies that the new price is present both in `_source` and the filter payload, while every untouched source field survives:

```sh
cargo test embedded_partial_update_preserves_source_and_refreshes_filter_payload
```

The trade-off is payload size: even a small non-text patch now transfers the merged source back to Qdrant. The update is also still a read followed by a write, not an atomic compare-and-write; concurrent updates can race, and the gateway does not yet implement Elasticsearch sequence-number concurrency controls.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-27-bulk-index-results-without-per-item-reads.md

# Accurate bulk index results without per-item reads

Elasticsearch bulk `index` items distinguish inserts from replacements. A new ID reports HTTP 201 with `result: "created"`; an existing ID reports HTTP 200 with `result: "updated"`. The gateway previously returned 201 for every successful item, which made replacement-heavy imports look like all-new ingestion.

Bulk writes still need to remain bulk writes. The gateway now sends one Qdrant retrieve request for all IDs in each contiguous same-index batch, then keeps the existing single upsert request. It tracks repeated IDs in request order, so two index actions for the same initially missing ID report `created` and then `updated`. Requests made through an alias report the concrete index in each result.

The regression test covers existing, new, and repeated IDs in both storage modes. Across those two cases it observes two batched reads and three writes: one write for embedded-source mode, and the document plus search-projection writes required by dedicated document storage.

```sh
cargo test bulk_index_reports_created_and_updated_without_losing_batching
```

This adds one authoritative point-retrieval round trip per contiguous bulk index batch, not one per document. It is still a preflight observation rather than an atomic compare-and-write guarantee: another writer can change an ID between the lookup and upsert. The gateway also does not emulate Elasticsearch version counters or sequence-number concurrency controls.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-26-standalone-create-preserves-existing-documents.md

# Standalone create now preserves existing documents

Elasticsearch's `PUT /<index>/_create/<id>` and `POST /<index>/_create/<id>` endpoints are deliberately stricter than the ordinary index API: they create a missing document but return HTTP 409 rather than replace an existing ID. This matters for importers and job processors that use create-only writes as an idempotency guard.

The gateway already enforced that distinction for bulk `create`, but the standalone endpoint was not routed. Both standalone forms now reuse the same existence check. A new document returns HTTP 201 with `result: "created"`; an existing document returns `version_conflict_engine_exception`, and the regression test verifies that Qdrant receives no write. Both success and error responses include the Elasticsearch product header expected by official clients. The test covers both embedded-source and dedicated document-projection storage:

```sh
cargo test standalone_create
```

This adds one point lookup before each create-only write. The lookup and write are serialized inside one gateway process, but they are not an atomic compare-and-insert across replicas or independently running gateway processes. Deployments that require distributed exactly-once creation still need a stronger coordination mechanism; the endpoint should not be treated as a substitute for Qdrant-side conditional insertion.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-26-index-results-should-describe-the-write.md

# Index results should describe the write

Elasticsearch's index API is an upsert: the same request shape can either create a new document or replace an existing one. That distinction is observable. A create returns HTTP 201 with `result: "created"`; a replacement returns HTTP 200 with `result: "updated"`.

The gateway previously returned HTTP 200 with `result: "created"` for both cases. The write itself succeeded, but clients using the status or result to count inserts, identify replacements, or drive follow-up work received misleading information.

Standalone `PUT /<index>/_doc/<id>` now checks the authoritative document store before writing. The regression test covers missing and existing IDs in both source-storage modes, sends the request through an alias, and verifies the concrete index name, status, result, and Elasticsearch product header.

This accuracy costs one point read per standalone index request. It is also a statement about what existed at preflight time, not an atomic compare-and-write guarantee; a concurrent writer can change the document after that lookup. Contiguous bulk `index` actions deliberately keep their single-upsert batching path and do not yet pay a per-item created-versus-updated preflight; their successful item statuses remain a documented compatibility limitation.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-25-one-index-one-collection-namespace.md

# One index, one collection namespace

The gateway derives Qdrant collection names from Elasticsearch index names. Its original mapping kept ASCII letters, numbers, underscores, and hyphens, while replacing other characters with underscores. That makes `catalog.v2` and `catalog_v2` distinct Elasticsearch names but the same Qdrant collection name. A second edge case exists when document projection is enabled: `catalog_documents` uses the collection name reserved for `catalog`'s document store.

Those collisions are more serious than a confusing create error. If both names reached the same collection, reads and writes could cross an index boundary. The gateway now checks the durable index catalogue before making any Qdrant request and rejects a name that overlaps either the search or document collection namespace of an existing index. The check reserves both collection names even when document projection is currently disabled, so turning the mode on later cannot expose a previously accepted naming collision.

The regression test covers punctuation normalization and the `_documents` suffix case. It also asserts that rejected creation makes zero upstream requests and publishes no index metadata.

This is a guard around the existing naming scheme, not a new reversible encoder. Existing deployments keep their current collection names. Operators importing metadata created outside the gateway should still audit collection ownership before startup; the check prevents new collisions recorded through the gateway but cannot repair an already-colliding external state.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-25-bulk-create-must-not-overwrite.md

# Bulk create must not become upsert

Elasticsearch gives `index` and `create` different meanings inside a bulk request. `index` may replace a document at the same ID, while `create` must fail with an item-level HTTP 409 conflict when that ID already exists.

The gateway previously combined adjacent `index` and `create` actions into the same Qdrant upsert batch. That was efficient, but it erased the distinction: a retry or duplicate `create` could silently replace an existing document while reporting status 201.

Bulk `create` now performs a point lookup before writing. If the document exists, the item returns `version_conflict_engine_exception`, the bulk response sets `errors` to `true`, and no upsert is sent. Ordinary adjacent `index` actions keep their batched write path. A mock-Qdrant regression test exercises both embedded-source and dedicated document-projection storage and verifies that the conflict path issues zero writes:

```shell
cargo test bulk_create_rejects_an_existing_document_without_overwriting_it
```

This correction has a cost: `create` actions now require one lookup and an individual write instead of sharing the `index` batch. Create checks are serialized inside a gateway process, so concurrent creates handled by that process cannot both pass the preflight. The lookup and write are still not one atomic Qdrant operation, however, and independently running gateway replicas could race. The default single-replica deployment avoids that specific race; operators should not treat this as distributed compare-and-insert semantics.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-25-an-alias-must-route-the-request.md

# An alias must route the request

The gateway has stored aliases durably from its first release, but persistence alone did not make them functional. Index lookup still derived the Qdrant collection name and deterministic point ID from the name in the request. A request to `current-products/_doc/1` therefore looked in the alias namespace rather than the `products` namespace recorded as its target.

Index lookup now resolves a single-target alias to its concrete index before any collection or point ID is selected. This applies to document CRUD, bulk operations, search, count, mappings, multi-search, and more-like-this document references. Responses identify the concrete index, and `GET /_alias/<name>` is now keyed by that concrete index, matching Elasticsearch's response shape. Index deletion rejects alias names explicitly, and deleting a concrete index removes its aliases from the SQLite catalogue.

The regression test uses a mock Qdrant endpoint and an alias named `current-products`. It verifies that the document read uses `GET /collections/es_products/points/<products-derived-id>`, that match-all search uses `POST /collections/es_products/points/scroll`, and that both hits and alias inspection name `products`. Run it with:

```sh
cargo test aliases_route_document_and_search_requests_to_the_concrete_index
```

This does not make aliases equivalent to Elasticsearch's full alias subsystem. The current schema deliberately permits one concrete target per alias. Filtered aliases, multi-index fan-out, write-index selection, and a Qdrant-native atomic switch are still future work. Alias action batches are atomic within the gateway's SQLite catalogue, but they remain gateway-local metadata operations rather than a distributed transaction with Qdrant.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-24-release-0-1-2.md

# 0.1.2: safer defaults for production traffic

Version 0.1.2 is a small operational release.

The gateway now makes upstream timeouts visible and configurable, bounds background payload writes, and waits briefly when SQLite is busy instead of failing immediately. CI also checks Rust dependencies for known advisories and offers a manual release dry run that builds, scans, and packages without publishing.

The client contract is unchanged. Upgrade the gateway, keep the Elasticsearch endpoint, and tune the new settings only if your workload needs to.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-24-missing-updates-are-not-upserts.md

# A missing update is not an upsert

Elasticsearch's update API does not create a missing document unless the request explicitly asks for upsert behavior. The gateway previously lost that distinction: an update to an absent ID could return `"result": "updated"`, even though Qdrant had no point to change. A text-field update could go further and create the document from an empty source.

The gateway now checks existence before every partial update. Missing documents return HTTP 404 with Elasticsearch's `document_missing_exception` shape:

```json
{
  "error": {
    "type": "document_missing_exception",
    "reason": "[missing]: document missing",
    "index": "products",
    "shard": "0"
  },
  "status": 404
}
```

Bulk updates report the same item-level error and set the top-level `errors` flag to `true`. Requests with `"doc_as_upsert": true` remain explicit creation requests; they return HTTP 201, including as bulk item statuses.

The regression test runs the missing, bulk, and `doc_as_upsert` cases against mock Qdrant responses in both storage modes. This correctness check adds one point lookup to partial updates that previously avoided a read. Scripted updates remain unsupported, and the gateway still does not emulate Elasticsearch's optimistic-concurrency or version counters.

Update, 2026-09-28: the separate `upsert` document form is now supported too; see [A partial document is not an upsert source](https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-28-a-partial-document-is-not-an-upsert-source.md).


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-24-missing-deletes-are-not-successes.md

# A missing delete is not a successful delete

Deleting an absent document is intentionally idempotent, but Elasticsearch still tells the client that nothing was removed. The gateway previously returned HTTP 200 with `"result": "deleted"` because Qdrant's point-delete operation succeeds even when the point does not exist.

The gateway now checks the authoritative document store before deleting. A missing ID returns HTTP 404 with `"result": "not_found"`, matching Elasticsearch's [delete result vocabulary](https://www.elastic.co/docs/api/doc/elasticsearch/v8/operation/operation-delete):

```json
{
  "_index": "products",
  "_id": "missing",
  "_version": 1,
  "result": "not_found",
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  }
}
```

Bulk delete keeps Elasticsearch's useful nuance: the item has status 404 and `"result": "not_found"`, but it has no `error` object and the top-level `errors` flag remains `false`. Missing updates are different because they failed to apply; those still carry `document_missing_exception` and set `errors` to `true`.

The regression test exercises standalone and bulk missing deletes in both storage modes. The trade-off is one point lookup before each delete, including successful deletes. As with the update existence check, a concurrent writer can race between the lookup and delete; the gateway does not yet implement Elasticsearch sequence numbers or primary-term concurrency controls.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-24-a-timeout-is-a-feature.md

# A timeout is a feature

What happens when Qdrant stops responding?

`QDRANT_CONNECT_TIMEOUT_MS` and `QDRANT_REQUEST_TIMEOUT_MS` configure upstream timeouts, defaulting to five seconds and three minutes. The gateway logs both values at startup.

With `ASYNC_PAYLOAD_WRITES=true`, eligible payload updates run in background tasks. `ASYNC_WRITE_QUEUE` limits concurrent tasks to 256 by default; further updates receive an error while all slots are occupied. Failed background writes are logged.

CI checks Rust dependencies for known advisories. A manual release dry run builds and scans the image and packages the Helm chart without logging into GHCR or publishing a tag.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-23-oversized-requests-are-not-bad-json.md

# Oversized requests are not bad JSON

The gateway bounds request bodies with `MAX_BODY_BYTES` and gives bulk ingestion a separate `MAX_BULK_BYTES` ceiling. Those limits protect memory, but the rejection previously looked like a generic HTTP 400 compatibility error. A client could not reliably tell whether to fix malformed JSON or split a valid payload into smaller requests.

Oversized requests now return HTTP 413 with Elasticsearch's `content_too_long_exception` error type. The response names the active setting and its byte limit, so an operator can decide whether to batch more narrowly or deliberately change the deployment limit. Both the early gateway check and the streaming body collector use the same response shape; bulk routes identify `MAX_BULK_BYTES`, while other routes identify `MAX_BODY_BYTES`.

The regression test sets deliberately tiny limits and verifies the status, error type, setting name, and exact configured ceiling for a search request and a bulk request:

```bash
cargo test oversized_requests_return_413_with_the_active_limit
cargo test --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
```

This does not implement automatic retrying or request splitting. HTTP 413 is intentionally non-transient: clients should reduce the body size, and operators should raise a limit only after considering the gateway's memory budget and concurrency.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-23-missing-documents-are-404s.md

# A missing document is a 404

Elasticsearch's get-document API distinguishes a missing document at both layers of the response: the JSON body contains `"found": false`, and the HTTP status is 404. The gateway previously returned the right body with HTTP 200.

That mismatch matters to clients. An application using the status code to select its not-found path could treat an absent document as a successful fetch, even though the response body said otherwise. The gateway now returns 404 while preserving the Elasticsearch-shaped body and `X-Elastic-Product` response header.

The regression test exercises both source-storage modes: `_source` embedded in the searchable Qdrant collection and the optional dedicated document projection. In each mode, an empty Qdrant lookup must produce the same result:

```http
HTTP/1.1 404 Not Found
X-Elastic-Product: Elasticsearch

{"_index":"products","_id":"missing","found":false}
```

This change is deliberately narrow. It does not yet emulate Elasticsearch's version, sequence-number, or primary-term behavior, and delete/update behavior for missing documents remains a separate compatibility surface.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-22-one-replica-is-the-safe-default.md

# One replica is the safe default

The Helm chart used to request two gateway replicas while also provisioning one `ReadWriteOnce` volume for the SQLite metadata database. That combination looked highly available but did not supply a highly available control plane.

SQLite stores index mappings, vector definitions, and aliases. With a typical `ReadWriteOnce` volume, two pods may be constrained to one node or the second pod may be unable to mount the volume. Giving each pod separate storage is worse: the replicas can disagree about which indices and aliases exist. Even a shared filesystem needs locking semantics suitable for SQLite, and concurrent index administration still crosses SQLite and Qdrant without a distributed transaction.

The chart now defaults to one replica. CI renders the Deployment and asserts that default, while the chart and production guides spell out the requirements before an operator overrides it. This does not reduce a working default deployment from two fault-independent replicas: the old defaults never provided that guarantee.

To reproduce the chart check:

```sh
helm lint deploy/helm/qdrant-es-gateway
helm template smoke deploy/helm/qdrant-es-gateway \
  --show-only templates/deployment.yaml | grep '^  replicas:'
```

The expected rendered value is `replicas: 1`.

This is a safer default, not horizontal-availability support. A production design that needs multiple gateway replicas should first move the small metadata catalog to a replicated control-plane store, or deliberately provide `ReadWriteMany` storage with verified SQLite locking and serialize index and alias changes externally. Qdrant replication protects document collections; it does not replicate the gateway's SQLite catalog.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-22-no-phantom-indices-after-create-failures.md

# No phantom indices after creation failures

Creating an index spans two stores: Qdrant owns the collections and payload indexes, while SQLite keeps the gateway's mappings and aliases. That boundary needs a deliberate publication order.

Previously, the gateway wrote SQLite metadata before asking Qdrant to create anything. If Qdrant was unavailable, rejected the collection configuration, or failed while creating the optional document projection, the request returned an error but the gateway still considered the index present. Later index checks succeeded against metadata and document operations failed against a collection that did not exist.

Index creation now treats SQLite as the publication step. The gateway creates the sparse collection, optional document collection, and payload indexes first. Only after those operations succeed does it record the index metadata. If a later setup step or the metadata write fails, it makes a best-effort attempt to delete the collections created by that request. Cleanup failures are logged without hiding the original creation error.

A regression test runs against an in-process mock Qdrant. It accepts the primary collection, fails the document collection, and verifies both observable properties: the index is absent from SQLite and the primary collection receives a cleanup request.

To reproduce the verification:

```bash
cargo test failed_index_creation_is_not_published_and_cleans_up
cargo test --all-targets
cargo clippy --all-targets -- -D warnings
```

This is compensating cleanup rather than a distributed transaction. A process crash between Qdrant creation and SQLite publication can still leave an orphaned collection, and a Qdrant outage can prevent cleanup. Operators should still monitor unexpected collections and retain reconciliation or backup procedures for the metadata database. The change prevents failed requests from publishing unusable gateway metadata and preserves the original metadata when an attempt to recreate an existing index is rejected upstream.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-21-a-slash-is-part-of-the-id.md

# A slash is part of the document ID

Elasticsearch clients put document IDs in URL path segments. An ID such as `order/42` therefore reaches the server as `order%2F42`; spaces and non-ASCII text are encoded for the same reason. The gateway previously routed on the raw URI path, so it stored the encoded spelling as `_id` instead of the caller's original value. A later request produced by a client from `order/42` could still find that spelling, but responses exposed `order%2F42`, breaking identity round trips and the documented arbitrary-ID guarantee.

Routing now percent-decodes each segment independently. Decoding after splitting is important: `%2F` becomes part of the document ID rather than a new route separator. Invalid UTF-8 is rejected as a structured bad request instead of being replaced with lossy text.

The focused verification covers an ID containing an encoded slash, Unicode, and a space, plus a non-UTF-8 path:

```bash
cargo test path_segments_
cargo test --all-targets --all-features
cargo clippy --all-targets --all-features -- -D warnings
```

This is route-level compatibility, not a relaxation of index-name rules or an implementation of Elasticsearch's full URL grammar. Reverse proxies must also preserve encoded slashes when forwarding requests; deployments should include one representative encoded ID in ingress smoke tests.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-20-bulk-errors-mean-errors.md

# When bulk errors did not mean errors

Elasticsearch bulk responses have two levels: a top-level `errors` boolean and an `items` array whose action objects contain per-document results. The [Bulk API contract](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) defines `errors` as true when one or more operations did not complete successfully. Clients commonly inspect the boolean first and only walk the item details when it is true.

The gateway already returned useful per-item failures, but calculated `errors` at the wrong JSON level. It looked for `error` directly on each item even though the field lives under the action name, such as `items[0].update.error`. As a result, a failed update or delete could be returned with `"errors": false`, encouraging a client to treat the batch as successful.

The response builder now checks every action result for a nested `error`. A focused regression test proves that a successful item leaves the flag false and a failed item sets it true.

To reproduce the verification:

```bash
cargo test bulk_response_reports_nested_item_errors
cargo test
cargo clippy --all-targets --all-features -- -D warnings
```

This change only corrects the summary flag. It does not add rollback or atomicity: bulk requests remain a sequence of operations, callers must still inspect failed items, and callers still own retry policy.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-19-three-nodes-one-goes-away.md

# Three nodes, one disappears, and the search keeps moving

A search service is easy to benchmark when everyone is alive. The more useful question is what happens when one of them vanishes in the middle of lunch.

I ran the same Elasticsearch-shaped client workload against two small clusters: three Elasticsearch nodes, and three Qdrant nodes behind three Qdrant Elasticsearch Gateway replicas. Both sides used replicated data, a load balancer, six concurrent workers, and a 1,000-document catalogue. Three seconds into each 12-second run, one storage node was hard-killed and then restarted.

| workload | Elasticsearch | Qdrant + gateway |
| --- | ---: | ---: |
| search throughput | 145 req/s | 1,261 req/s |
| insert throughput | 665 req/s | 1,169 req/s |
| update throughput | 1,952 req/s | 1,335 req/s |
| delete throughput | 979 req/s | 1,417 req/s |
| mixed throughput | 1,183 req/s | 1,296 req/s |
| recovery time | 4.4–4.6 s | 4.5–4.7 s |

Losing a node did not turn either service into a pumpkin. The gateway path kept serving the mixed workload at roughly the same rate as native Elasticsearch, and was faster for inserts and deletes in this run. Native Elasticsearch was faster for the update-heavy case. On both sides, the main cost was a handful of requests waiting for the dead node to time out; p95 latency stayed low outside that interruption.

In the mixed case, the gateway was about 9% faster. Inserts were about 76% faster and deletes about 45% faster; the update-heavy case went the other way by roughly 32%. The search-only result was especially stark in this small, CPU-constrained run, with the gateway at about 8.7× the observed throughput.

A useful engineering discovery came from the test. A load balancer must check whether a gateway process is alive, not whether its upstream storage is having a bad five seconds. Using the gateway’s strict readiness check made every replica disappear during a Qdrant node restart. Switching the load balancer to the liveness check kept the gateway tier in rotation while Qdrant recovered. That is the difference between “the database is repairing itself” and “the entire API is down.”

This harness shares the gateway’s SQLite metadata volume between replicas so the test can focus on the data path. The current control plane is not fully stateless; the next production step is to move that small index-metadata responsibility to a shared control-plane store.

The reproducible profile and raw JSON are in [`benchmark/resilience`](https://github.com/dstockton/qdrant-es-gateway/tree/main/benchmark/resilience). The client still sends the same requests; only the endpoint changes.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-19-making-the-bulk-limit-real.md

# Making the bulk request limit real

The gateway has separate request-size controls for ordinary JSON requests and bulk NDJSON: `MAX_BODY_BYTES` defaults to 10 MiB, while `MAX_BULK_BYTES` defaults to 50 MiB. A useful bulk batch is often much larger than a search or single-document request.

Until now, the HTTP body reader applied `MAX_BODY_BYTES` before routing the request. The bulk handler did check `MAX_BULK_BYTES`, but it could never see a request larger than the general limit. With the defaults, a 10–50 MiB bulk request was therefore rejected before the bulk-specific check ran.

The reader now selects the limit from the method and path before consuming the body. `POST /_bulk` and `POST /<index>/_bulk` use `MAX_BULK_BYTES`; every other request, including a non-POST request to a bulk-shaped path, continues to use `MAX_BODY_BYTES`. A regression test covers all four cases.

To reproduce the verification:

```bash
cargo test bulk_requests_use_the_dedicated_body_limit
cargo test
cargo clippy --all-targets --all-features -- -D warnings
```

This is a correctness fix, not a streaming implementation. Accepted bodies are still buffered before NDJSON parsing, so operators should keep `MAX_BULK_BYTES` bounded below the memory available per concurrent request and enforce compatible limits at the ingress. The change does not alter bulk parsing, batching, acknowledgement, or search visibility semantics.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-18-why-qdrant-es-gateway.md

# Keep the Elasticsearch client, change the engine

Changing search engines can mean changing application code that already works. In a product catalogue, documentation site, jobs board, or support portal, the application may already know how to create an index, bulk documents, search with filters, and render `_source` through an Elasticsearch client.

Qdrant ES Gateway takes a narrower route. The client keeps talking to an Elasticsearch-shaped HTTP endpoint. The gateway translates the application-search subset into operations for the Qdrant engine, preserves document IDs and sources, and rejects unsupported semantics instead of pretending that two ranking engines are identical.

The gateway targets ordinary application search: index lifecycle, bulk ingestion, lexical queries, filters, source projection, updates, and deletes, with limited support for facets and sorting. Terms facets require indexed keyword fields, sorting has basic response-level semantics, and pagination is bounded. Lucene-specific analyzer settings, scripts, scroll, and arbitrary aggregations are outside the supported surface; Kibana and logging are outside the project's intended scope.

For applications whose requests fit the supported subset, the client-side migration can be small. Check the [compatibility guide](https://dstockton.github.io/qdrant-es-gateway/compatibility/), point the existing Elasticsearch client at the gateway, replay the requests, and compare returned IDs and sources. Then measure the workload that matters: ingestion, reads, mixed traffic, memory, CPU, and disk. Search scores are engine-specific, and accepted query shapes can still differ in behavior: `match_phrase`, for example, does not guarantee Lucene-style positional phrase matching. Evaluate the documents returned, facet behavior, pagination, and product-level relevance metrics.

The compatibility boundary makes the migration decision concrete: which existing requests can the application retain, and do Qdrant's retrieval semantics fit the workload? Start with the [compatibility matrix](https://dstockton.github.io/qdrant-es-gateway/compatibility/) and use its ✅ and ⚠️ entries as a checklist for the application's requests.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-18-two-collections-one-stateless-gateway.md

# Two collections, one stateless gateway

Reading a product's current source and finding it through search are different operations. Qdrant ES Gateway can store the source separately from its searchable projection, allowing a write to wait for durable source storage while the search projection converges asynchronously.

With `DOCUMENT_PROJECTION=true`, Qdrant ES Gateway uses the Qdrant engine to keep a searchable collection and a durable document collection named `es_<index>_documents` for each Elasticsearch index. The document collection has no payload indexes and supplies GETs and full `_source` responses. The searchable collection carries the fields needed for retrieval, filters, and sorts. Search hits' sources are fetched from the document collection in one batch.

Asynchronous projection is a separate choice: set `ASYNC_SEARCH_PROJECTION=true` to wait for the source write while using Qdrant's asynchronous operation acknowledgement for the search projection. A successful source write does not imply that search has caught up. Unmapped metadata-only updates can avoid sparse-index work, while fields used for filters or sorts remain mirrored in the search projection.

The stateless gateway tier still depends on persisted state. Qdrant holds the documents and search projection, and the gateway-owned SQLite metadata database must also be persisted. For production asynchronous projection, the [production guide](https://dstockton.github.io/qdrant-es-gateway/production/) recommends a durable projection worker or reconciliation process rather than relying on an in-process queue.

For a write-heavy catalogue, this design makes source durability and search freshness separate deployment concerns. It does not provide Elasticsearch-identical refresh, version, or sequence semantics. Test those differences against the application's requirements.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-18-pattern-queries-without-a-client-rewrite.md

# Pattern queries without a client rewrite

Catalogue search often has two different jobs hiding behind one endpoint. Natural-language search wants token-aware retrieval. Operational search wants to find an SKU prefix, a serial number, or a wildcarded identifier.

Qdrant ES Gateway accepts Elasticsearch-shaped `prefix`, `wildcard`, and `regexp` clauses in positive `bool.must` and `bool.filter` paths. For these supported placements, the client can retain the request shape. The gateway reads candidate documents from Qdrant and evaluates patterns against their fields, combining pattern clauses with ordinary Qdrant-translated filters where supplied. Pattern clauses in either `should` or `must_not` are rejected.

Wildcards support Elasticsearch-style `*` and `?`, while regular expressions use Rust/RE2-style syntax. Invalid regular expressions receive a structured `400` response. The [compatibility guide](https://dstockton.github.io/qdrant-es-gateway/compatibility/) lists the boundary.

Pattern matching is a gateway-side scan. A pattern-only query scans the collection in bounded scroll pages; the page size does not bound the total work. This suits modest catalogues and identifier lookups, but not unbounded, high-cardinality pattern queries. If pattern traffic becomes a primary workload, use a dedicated indexed pattern field or a purpose-built search path.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-18-multi-search-and-cursor-pagination.md

# Batching searches and paging through ordered results

An application may need several searches to populate a catalogue page, or successive pages from an ordered result set. Qdrant ES Gateway supports `_msearch` for batching searches and `search_after` for cursor pagination within its supported search API.

The `_msearch` endpoint accepts Elasticsearch-style NDJSON header/query pairs. Each sub-search uses the same client request shape as an individual search and is subject to the gateway’s query compatibility limits.

For ordered pages, `search_after` requires explicit sort fields. Each returned hit includes the corresponding `sort` values, which the client can pass as `search_after` from the last hit when requesting the next page. The cursor is stable only while the ordered result set is unchanged; changes to documents or their ordering between requests can affect pagination.

These features support existing batching and pagination request shapes, but do not provide full Elasticsearch compatibility. Field sorting has limits, deep pagination is capped, and deep scored pagination is outside the supported scope. Check the application’s queries and pagination needs against the compatibility documentation before migration.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-18-lifecycle-compatibility-for-real-apps.md

# Compatibility includes the boring lifecycle calls

An application can have a perfectly ordinary search query and still fail a migration before it reaches that query. Production clients create and delete indexes, ask whether an index exists, refresh after an import, and sometimes close and reopen an index while applying settings.

Qdrant ES Gateway accepts `_refresh`, `_open`, `_close`, and basic `_settings` lifecycle calls for client and application compatibility. Accepting these request shapes does not reproduce Elasticsearch's refresh, open/close, or settings semantics. Qdrant collections remain continuously available underneath.

Settings and analysis are different. The gateway durably records mapping metadata, but it does not emulate Elasticsearch analyzer or settings semantics. Analyzer settings that change query semantics remain unsupported. Applications that depend on those behaviors need to check compatibility before migrating.

These calls can reduce changes to a catalogue bootstrap when its lifecycle requests fit the supported subset. The team still needs to check whether the application relies on the effects of those calls and test search behavior separately. The [compatibility matrix](https://dstockton.github.io/qdrant-es-gateway/compatibility/) lists the supported surface and its limits.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-18-launch-video.md

# Launch video: keep the Elasticsearch client, change the engine

Here is the short launch video for Qdrant ES Gateway:

<video controls preload="metadata" poster="https://dstockton.github.io/qdrant-es-gateway/assets/qdrant-es-gateway-launch.jpg" width="960">
  <source src="https://dstockton.github.io/qdrant-es-gateway/assets/qdrant-es-gateway-launch.mp4" type="video/mp4">
  Your browser does not support embedded video. [Download the MP4](https://dstockton.github.io/qdrant-es-gateway/assets/qdrant-es-gateway-launch.mp4).
</video>

The story is simple: an existing application keeps its Elasticsearch-shaped client requests, while the gateway translates the supported application-search subset to the Qdrant engine. The video compares compatibility, retrieval, mixed workloads, memory, and the supported feature boundary.

See the [compatibility matrix](https://dstockton.github.io/qdrant-es-gateway/compatibility/) and [production notes](https://dstockton.github.io/qdrant-es-gateway/production/) for the detail behind the demo.


---

Source: https://dstockton.github.io/qdrant-es-gateway/markdown/2026-09-18-a-dress-a-tea-bag-and-a-null.md

# A dress, a tea bag, and a null

Real catalogues are less like neat databases and more like drawers everyone in the house has used.

The [H&M catalogue](https://huggingface.co/datasets/Qdrant/hm_ecommerce_products) starts politely enough: “Strap top”, black, white, off-white. Then the variants multiply, identical descriptions return in several colours, and one of them becomes “Strap top (1)”. An item is more than a sentence: it is a sentence, a colour, a department, a picture, and a suspiciously similar sibling.

[Open Food Facts](https://huggingface.co/datasets/openfoodfacts/product-database) is messier. It contains millions of products, dozens of languages, community edits, missing categories, unknown food names, incomplete ingredient percentages, and products whose environmental score is essentially shrug emoji-shaped. One tea bag arrives with `ciqual_food_name = unknown`; another has no nutrition data at all. The database puts the mess in a field and carries on.

That is why a search gateway needs real catalogues, not rows called `product-000001`. Search has to survive colour variants, multilingual labels, near-duplicate names, missing values, and the occasional tea bag with an identity crisis.

The useful benchmark question is therefore not “which engine sorted my toy data fastest?” It is: “can a customer still find the black top, the hazelnut spread, and the product whose category is null?”

That is where the engineering begins.