Blog
Short notes on compatibility, benchmarks, and the odd things that turn up in real search data.
Prefer plain text? Browse the Markdown index or complete text export.
Qdrant 1.20.0 through the gateway: almost identical read performance and an auth upgrade check
Qdrant released 1.20.0 on October 22. The proposed gateway stack update advances its development, example, test, and benchmark Qdrant images from 1.19.2 to 1.20.0. This article describes a candidate change under review; deployment stacks gain the new pin when the pull request is merged.
Script fields cannot be empty promises
The gateway accepted search bodies containing script_fields, did not execute the scripts, and returned ordinary hits without the requested fields object. That is a false success for clients that calculate display values or other per-hit data at search time.
Hit concurrency metadata cannot be invented
The gateway accepted version: true and seq_no_primary_term: true in search bodies but omitted the requested per-hit metadata. That is more serious than a cosmetic response difference: native clients can use sequence numbers and primary terms as optimistic-concurrency tokens, while the gateway’s approximate write fields do not carry Elasticsearch or OpenSearch’s shard history.
Requested doc values cannot silently disappear
The gateway accepted search bodies containing docvalue_fields but silently returned hits without the requested fields object. This can break clients that deliberately read keyword, numeric, or date values from doc values instead of _source; preserving _source does not satisfy that response contract.
Which dry plots need an irrigation inspection?
A market garden’s morning queue should include dry plots with recent sensor readings. An old low reading is a reason to check the sensor, not evidence to irrigate. These six synthetic snapshots deliberately include a threshold reading, stale data, a closed plot, and a missing reading age. The cutoffs are illustrative, not agronomic advice.
Query profiles cannot be empty
The gateway accepted profile: true in a search body but ignored it. That turns a diagnostic request into an ordinary search while still returning HTTP 200. It is especially misleading here because the gateway executes against Qdrant, not Lucene: a native-looking query and collector timing tree cannot be inferred from the returned hits.
Refreshing the production stack without opening old volumes
The development and benchmark stack now uses Rust 1.99.1, Qdrant 1.19.2, Valkey 9.1.2, HAProxy 3.4.7, Python 3.15.0, and Elasticsearch 9.5.5. The Rust client layer also moves to reqwest 0.13 and redis-rs 1.7. This is a maintenance update: the existing client and demo request shapes are unchanged.
Score explanations must explain the score
The gateway accepted explain: true in a search body but ignored it. That is not merely missing decoration: clients use each hit’s _explanation tree to debug ranking, and the gateway’s score comes from Qdrant rather than Lucene. Fabricating a Lucene-shaped explanation or silently omitting it would both misdescribe how the result was ranked.
Highlight snippets cannot vanish
The gateway accepted a search highlight object but ignored it. That changes the response contract: clients commonly request snippets to show where a result matched, and a successful response without those snippets can render an empty preview or force an unsafe fallback to the complete source text.
Which batteries belong on the dispatch shortlist?
An energy cooperative needs northern battery sites marked ready with at least 100 kWh available. Six synthetic snapshots include a site exactly at the threshold, one just below it, a southern site, a held site, and an offline site. Operator facets count sites, not stored energy.
AWS Retail Demo Store: compatible search requests, different trade-offs
The AWS Retail Demo Store is a useful reality check because its search service talks to OpenSearch through the ordinary opensearch-py API. Its product search uses dis_max, match_bool_prefix, category collapse with named inner_hits, and _source: false. Its similar-products route uses more_like_this.
Search controls cannot be decoration
The gateway accepted min_score, terminate_after, and per-search timeout fields but did not apply them. That is more dangerous than a clear unsupported-feature response: a caller could ask to remove low-scoring hits, stop collection early, or return within a deadline and instead receive an ordinary complete page.
An empty alias batch is not success
The aliases API is an administrative write boundary. The gateway previously treated {“actions”:[]} as a valid request: it opened and committed an empty catalogue transaction, then returned {“acknowledged”:true}. A client bug that accidentally dropped every action could therefore look like a completed routing change.
Testing a cache before building it
Would Valkey make this gateway fast enough to beat native search? Before adding a production cache, we tested a narrower question: is there enough latency headroom in an ideal warm hit to justify the work?
Find the next page across the whole catalogue
Sorting the first few candidates is not the same as finding the first few sorted results. Likewise, asking for a page after price 20 cannot work reliably if the gateway only fetched records priced below 20.
Two gateways need one view of their mappings
Adding a second gateway sounds straightforward: start another process and send it traffic. But if each process has its own SQLite file, an alias created through one process is invisible to the other. A restart-safe local database does not automatically become shared metadata.
Repeated searches get faster; fresh queries still cost more
The gateway now has a measured advantage for repeated, unchanged searches. With its optional shared cache enabled, warm search medians were 12–30% lower than the faster native engine across our eight search/concurrency comparisons. Point reads were close to parity, with only a 3–5% combined median advantage. That advantage was not consistent in every round: in the second enabled round, single-client point reads were about 2% slower than the faster native engine.
Rustls security fixes landed
The dependency update in PR #62 moved Rustls from 0.23.45 to 0.23.46. That patch is now on main. It strengthens the TLS layer used for HTTPS requests and encrypted shared-catalog connections, without changing the gateway’s search API.
Reusing catalog connections helps misses and writes, with a warm-read warning
The shared catalog was opening a new Valkey connection whenever a request loaded its metadata snapshot. We changed it to reuse a persistent multiplexed connection. A failed operation is still returned as a failure: reconnecting later must not silently replay a write whose outcome is uncertain.
Removing request-local SQLite reconstruction: modest gains, one worse tail
The shared-catalog read path used to rebuild an in-memory SQLite database for every uncached request. Document reads and searches now load the same validated snapshot into request-local maps. In our paired experiment, cold-query medians fell 4.5–5.5% with cache serving enabled and 4.3–7.6% with serving disabled. The enabled eight-client p95 worsened 10.5%. This is a modest reduction in request overhead, not an across-the-board latency win.
One Qdrant search thread improves tails, but hurts typical cold queries
Limiting Qdrant search pools to one thread produced a clear trade-off in our small CPU-budget experiment. With eight clients, gateway cold-query p95 fell from 45.610 to 11.088 ms, while p50 rose from 4.805 to 7.568 ms. That is a 75.7% reduction in the tail measurement and a 57.5% increase in typical latency. Throughput rose 20.1%. It is not an overall latency win or a reason to change the default everywhere.
Missing alias removals are not success
The gateway used to acknowledge every syntactically valid alias removal, even when the alias did not exist or belonged to another index. That made a stale deployment or rollover request look successful while leaving the catalogue unchanged. The risk is operational rather than cosmetic: a caller can continue believing that an old read or write route was detached.
Measure the request timeline before rewriting JSON handling
The gateway’s own diagnostic traces show that backend waiting occupies most of the measured cold-search dispatch interval. The median per-request fraction spent in the backend join was 79.4% with one client and 73.6% with eight. Cache-envelope JSON parsing took a single-client median of 0.017 ms, compared with 1.162 ms for the whole dispatch interval. These traces do not justify a JSON-handling rewrite as the next optimization.
Fewer requests did not make this query faster
Measurement note: these historical filter-aggregation trials recomputed the expected count in each client worker. The calculation was outside that request’s timer, but affected throughput and could delay concurrent requests. Later trials tagged precomputed-filter-count-v2 compute it once before timing. The figures here are preserved without adjustment; do not combine them with the corrected trials.
Can I borrow the large-print edition today?
A public-library reader wants a large-print book available at York today. A standard edition of the same title, an ebook, a copy at Leeds, and a fully borrowed title should stay out. These six synthetic holdings represent one title/format/branch combination per document; no borrower data is needed.
Inside a cold search: a backend diagnostic, not another engine benchmark
After reducing cache-fill overhead, we inspected three Qdrant operations used by a filtered, sorted search: exact counting, checking indexed-field coverage, and retrieving an ordered page. The diagnostic found that exact counting had the largest single-client server-reported time of those operations. It also showed a large gap between server-reported timings and parallel client wall time under concurrency. Neither observation identifies the entire gateway bottleneck.
Sorting twenty records should not fetch ten thousand
Measurement note: these historical filter-aggregation trials recomputed the expected count in each client worker. The calculation was outside that request’s timer, but affected throughput and could delay concurrent requests. Later trials tagged precomputed-filter-count-v2 compute it once before timing. The figures here are preserved without adjustment; do not combine them with the corrected trials.
Keep your sparse token IDs intact
A sparse embedding is a small dictionary: token IDs paired with weights. If the same token changes identity between indexing and searching, a successful HTTP response can hide the wrong answer.
Keep coordination durable, let response-cache fills be disposable
A response cache should be cheap to refill. Our cache entries shared Valkey’s append-only log with catalog metadata and writer coordination, so filling a cold response also added durable log traffic. We changed only response publication to skip incremental AOF propagation. Catalog updates, generations and writer tokens retain their required durability.
Asking for less should not make sorting slower
Measurement note: these historical filter-aggregation trials recomputed the expected count in each client worker. The calculation was outside that request’s timer, but affected throughput and could delay concurrent requests. Later trials tagged precomputed-filter-count-v2 compute it once before timing. The figures here are preserved without adjustment; do not combine them with the corrected trials.
Changing transport was not enough
Measurement note: these historical filter-aggregation trials recomputed the expected count in each client worker. The calculation was outside that request’s timer, but affected throughput and could delay concurrent requests. Later trials tagged precomputed-filter-count-v2 compute it once before timing. The figures here are preserved without adjustment; do not combine them with the corrected trials.
A reporting day is not always 24 hours
A sales report grouped by London day should move its boundaries when the clocks change. Adding 86,400 seconds to yesterday’s boundary gets that wrong twice a year.
One fewer catalog request, without a proven latency win
A quiet cache miss now gets its generation ticket and catalog snapshot in one Valkey operation. This removes a second metadata request before querying Qdrant. In our paired test, cold-query medians fell 3.0% with one client and 7.9% with eight clients. Native controls moved too, and warm reads were mixed. These measurements do not isolate a performance win from the change.
The cheapest item may be on the next page
A price filter should describe the matching catalogue, even when the shopper only asks for one product. The gateway previously calculated min, max, and filter counts from the hit candidates it had fetched. A small size could therefore produce a plausible but incomplete answer.
A shared catalog is not a snapshot
Moving metadata out of a gateway process makes replicas easier to run. It does not make every Elasticsearch consistency promise available.
Index names are a client contract
An invalid index name used to make it through the gateway’s metadata catalogue and into a normalized Qdrant collection name. That was more than cosmetic. An application could successfully provision state that the same Elasticsearch or OpenSearch client would expect a native cluster to reject, and distinct punctuation could then collide in Qdrant’s namespace.
Alias options must not disappear
The gateway deliberately implements unfiltered, unrouted aliases with one concrete target. Its alias action parser previously read only index and alias, but silently ignored every other metadata field. A request for a tenant filter, custom routing, or a write index could therefore return {“acknowledged”:true} while the stored alias exposed all documents and routed every operation normally. That is more dangerous than an unsupported-feature error because the response tells the caller that a data-access or write-routing boundary exists when it does not.
Who is still waiting for a first reply?
A support lead needs open billing tickets with no replies and no assigned agent. Seven synthetic tickets include assigned, answered, closed, and technical cases that must stay out of the queue. Missing and null assignees qualify; an empty string counts as present, so producers should normalize blank assignments to null.
Creating an index is not a mapping update
Provisioning code often treats “already exists” as a narrow, recoverable race. It can do that reliably only when the server identifies the condition correctly. The gateway already rejected a second PUT /products before contacting Qdrant, but it grouped the request with unsupported query features and returned qdrant_gateway_unsupported_query. That classification could make an Elasticsearch or OpenSearch client treat a normal create race as an incompatible request.
Alias add is not a retarget
The gateway stores one concrete index per alias. Its alias transaction used SQLite INSERT OR REPLACE for an add action, so adding an existing alias to a different index silently discarded the original target. That looked like a convenient retarget, but it changed the meaning of a standard Elasticsearch/OpenSearch request and could redirect application reads and writes without the caller expressing a swap.
HEAD must not hide storage failures
Document-existence checks have a deliberately small contract. The Elasticsearch 8 API reference and the OpenSearch Get Document API reference both describe HEAD /<index>/_doc/<id> as HTTP 200 when the document exists and HTTP 404 when it does not. A storage outage is neither result.
A missing point is not a failed upstream
The Elasticsearch and OpenSearch index APIs are upserts, but their response describes which side of the operation happened. Live probes against Elasticsearch 9.5.5 and OpenSearch 3.9.0 both returned HTTP 201 with result: “created” for the first PUT /<index>/_doc/<id>, then HTTP 200 with result: “updated” for the replacement.
Two passengers, one cancelled departure
A travel disruption desk needs departures from York with at least two seats, either direct to London or to Peterborough for an onward connection. Six synthetic services include a one-seat departure, a different origin, an unsuitable destination, and an already cancelled train. These are illustrative inventory snapshots, not live timetables.
Source filters should not leak sibling fields
Elasticsearch clients often use _source filtering to keep response payloads small or avoid returning fields a particular view does not need. The gateway accepted array-shaped top-level filters, but a string such as “profile.name” silently returned the complete source. Nested paths were compared only with top-level keys, so even some accepted filters returned the wrong shape. A caller asking for a profile name could therefore receive sibling contact fields or unrelated metadata.
Integer hit totals need the minus-one sentinel
OpenSearch retains a legacy response mode for clients that expect hits.total to be a number rather than the newer {value, relation} object. A live probe against OpenSearch 3.9.0 confirmed that rest_total_hits_as_int=true returns the exact integer for an ordinary search and the sentinel -1 when the request also sets track_total_hits: false. The same query parameter applies to every item in _msearch. The gateway previously discarded the query string, so these requests silently received the wrong response shape.
Disabled hit totals should skip the count
OpenSearch lets a search caller choose the cost and precision of hits.total. A live probe against OpenSearch 3.6.0 with 15 matching documents confirmed the three useful modes: the default returned 15/eq, track_total_hits: false omitted hits.total, a threshold of 5 returned 5/gte, and a threshold of 20 returned 15/eq. The gateway previously ignored all four request shapes and always paid for its exact Qdrant count on eligible filter-only searches.
Search-after needs more than one page
A deterministic catalogue replay against native Elasticsearch 9.5.5 exposed an empty second page from the gateway. Six products were sorted by ascending price, the first two sort values were returned as the cursor, and Elasticsearch returned the next two products. The gateway returned none.
OpenSearch wildcards keep the leading star
A catalogue replay against native OpenSearch 3.6.0 found a gateway-only empty result for {“wildcard”:{“title”:”keyboard”}}. The document was present and other filters worked. The defect was in the gateway’s wildcard-to-regular-expression translation: a leading * disappeared, turning a contains query into a starts-with query.
Is this coastal clip cleared for the UK edit?
A media producer needs coastal footage under 31 seconds with UK clearance. A French-only clip, a long coastal walk, and an uncleared aerial should stay out of the shortlist. These six synthetic archive records use topic and territory arrays; matching either array means matching any element.
Invalid pages should not query Qdrant
Pagination inputs sit on a resource boundary. Elasticsearch documents from as non-negative, size as non-negative, and search_after as an array of scalar sort values. It also limits the ordinary from plus size result window to 10,000 by default and directs deeper pagination to search_after.
Inner hits need the requested name
The AWS Retail Demo Store sends a normal Elasticsearch collapse request: collapse on category.keyword, name the inner result category_hits, request a bounded number of group members, and suppress _source. Its response code then reads item[“inner_hits”][“category_hits”] to distribute products across categories. The request is visible in the project’s search service, and the corresponding response access is a few lines below.
Unsupported post filters must fail closed
The gateway documents a deliberately small post_filter subset: term, terms, numeric range, exists, match-all, and bool combinations of those clauses. An unsupported clause previously fell through the hydrated-source predicate as false. The request then succeeded with no hits, making an unsupported query indistinguishable from a valid filter that matched nothing. A multi-field term clause had a related problem: only the first field was evaluated.
A post-filter is not an aggregation filter
Elasticsearch applies post_filter after aggregations. This lets a catalogue page show only one selected brand while its facet and metric aggregations still describe every document matched by the main query. The gateway applied the post-filter to its hydrated candidate hits before computing min, max, filter, and keyed-filter aggregations, so those aggregations described only the visible hits. Server-side terms facets did not have this bug, which made mixed aggregation responses especially misleading.
Packed, but where is the tracking number?
A dispatch coordinator in York needs packed parcels that still lack a tracking number before the carrier arrives. These five synthetic shipments include an omitted field, an explicit null, an already labelled parcel, another depot, and a dispatched parcel. Adding tracking must remove only the newly labelled parcel from the exception list.
Keyword multi-fields need one route
Elasticsearch clients and application code commonly address an exact-value view as brand.keyword. The gateway already treated that conventional suffix as the base brand field when it evaluated hydrated source, sorted results, collapsed groups, or requested a terms facet. Native Qdrant filters did not: a term, terms, range, or exists clause was sent upstream with the dotted key unchanged. With a base keyword payload, that request searched a field that was never written and could quietly return no hits.
A page is not a total
The gateway used to retrieve at most from + size filter matches and then report the length of that candidate window as an exact Elasticsearch hits.total. A request for zero hits could therefore say that zero documents matched even when the collection contained many matches. Pagination made the same error less obvious: a ten-hit page could claim the total was ten.
Sort syntax must not change the order
Elasticsearch’s sort reference permits a basic field sort as “price”, {“price”:”asc”}, or {“price”:{“order”:”desc”}}. The gateway previously understood only the direct-order object. It treated the standard options object as ascending regardless of its order, while a string field silently became _score descending. The same misread specification was then used to generate and consume search_after values.
An evening class with room for one more
An adult learner wants an evening Python class in York. A full course, a daytime class, or a place in Leeds is no help. These six synthetic course listings test that distinction, including the moment the last introductory place disappears.
Arrays are multi-valued fields, not scalar mismatches
Elasticsearch does not need a separate array field type: an ordinary field can carry several values, and a document matches when an applicable value satisfies the query. Qdrant payload filters use the same practical rule—an array filter succeeds when at least one value meets the condition (Qdrant payload documentation).
Optional should really is optional
Elasticsearch changes the default for minimum_should_match according to the other clauses in the same bool. A should-only query requires one match, but should becomes optional when must or filter is present. The Elasticsearch bool query reference documents the defaults as one and zero respectively.
Nested negation keeps its boundary
Elasticsearch bool clauses accept either one query object or an array. They also preserve the nested bool boundary: excluding a bool that requires both status = deleted and tenant = internal means NOT (status = deleted AND tenant = internal), not status != deleted AND tenant != internal.
Pattern counts must scan the source
The gateway evaluates wildcard, prefix, and regexp clauses against hydrated _source documents because Qdrant does not receive an equivalent native filter. Search already followed that rule, but _count did not: it removed the pattern while building the Qdrant filter and then called Qdrant’s count endpoint. A pattern-only count therefore returned the size of the whole collection.
The pump is fixed. Take it off the queue.
A maintenance planner at a Leeds factory needs open pump faults, not last week’s completed inspections or another plant’s backlog. These five synthetic work orders make that distinction testable. Closing a job should remove it from the queue while preserving its original notes.
Exists means not empty
The gateway translated an Elasticsearch exists query directly into Qdrant’s is_empty condition. That is the opposite predicate: documents with a missing, null, or empty-array value were selected, while documents with a real value were excluded.
Zero results still need a positive scroll
Elasticsearch clients commonly send “size”: 0 when they need aggregations but no document hits. The gateway passed that value through as Qdrant’s scroll limit, turning a valid search into an upstream error.
Mapping updates must not erase the map
Elasticsearch sends a mapping update as {“properties”: …}, while index creation nests the same object under mappings. The gateway reused the index-creation parser for both shapes. A normal PUT /products/_mapping therefore interpreted the update as an empty mapping and replaced the stored definition with {}, silently losing the fields used to build search vectors and filter payloads for later writes.
Invalid UTF-8 is not source data
The gateway used to convert every HTTP body with a lossy UTF-8 decoder before parsing JSON or bulk NDJSON. An invalid byte inside a quoted value therefore became the Unicode replacement character (U+FFFD). If the value was source data or a bulk _id, the request could remain syntactically valid and write data under a value the client never sent.
Alias batches are one routing change
Elasticsearch’s aliases API accepts a list of actions so clients can express a routing change as one request. The gateway previously applied each action to its SQLite catalogue while it was still parsing later actions. A valid add followed by a malformed remove could therefore return HTTP 400 after the new alias was already visible.
Bulk mutations need document IDs
Elasticsearch requires bulk update and delete actions to identify the document they mutate. The gateway previously generated a random ID whenever _id was absent. That was particularly misleading for deletes: a malformed request could report not_found for a generated ID even though the caller never selected a document.
Bulk action metadata is a write boundary
A bulk action line decides which operation, index, and document ID the gateway will mutate. Treating malformed metadata as if it were absent is therefore unsafe: a numeric _id, for example, previously fell through to generated-ID behavior for index and create actions. An action line containing more than one operation also silently selected whichever key the JSON object exposed first.
Unchanged updates should not write
Elasticsearch’s Update API detects no-op updates by default. The gateway already had to read the stored source to merge a partial doc, but it wrote the merged source back to Qdrant even when no value changed. Repeated idempotent updates therefore consumed write capacity and reported result: “updated” instead of result: “noop”.
Partial updates should not erase sibling fields
Elasticsearch describes an Update API doc as a partial document merged into the stored source. That merge is recursive for inner objects: changing details.warranty.years must preserve both details.manufacturer and details.warranty.region. Arrays and scalar values, by contrast, replace their previous values.
Bulk items should preserve the operation result
An Elasticsearch bulk response is more than a status code. Each successful item identifies the concrete index that handled the operation and describes whether the document was created, updated, deleted, or not found. That distinction matters when a request uses an alias: clients need the backing index in the response, not the alias they sent.
A partial document is not an upsert source
Elasticsearch’s Update API supports two source documents with different jobs. doc is merged into a document that already exists. A separate upsert object is inserted when the ID is absent. They need not contain the same fields:
A partial update must preserve the document
In the default embedded-source storage mode, a partial update to a non-text field took an optimized Qdrant payload path. That path wrote only the patch beneath _source. An update such as {“doc”:{“price”:125}} could therefore make the next document read return only price, dropping untouched fields such as title and brand. The mapped top-level price payload used by filters and sorts also remained stale.
Accurate bulk index results without per-item reads
Elasticsearch bulk index items distinguish inserts from replacements. A new ID reports HTTP 201 with result: “created”; an existing ID reports HTTP 200 with result: “updated”. The gateway previously returned 201 for every successful item, which made replacement-heavy imports look like all-new ingestion.
Standalone create now preserves existing documents
Elasticsearch’s PUT /<index>/_create/<id> and POST /<index>/_create/<id> endpoints are deliberately stricter than the ordinary index API: they create a missing document but return HTTP 409 rather than replace an existing ID. This matters for importers and job processors that use create-only writes as an idempotency guard.
Index results should describe the write
Elasticsearch’s index API is an upsert: the same request shape can either create a new document or replace an existing one. That distinction is observable. A create returns HTTP 201 with result: “created”; a replacement returns HTTP 200 with result: “updated”.
One index, one collection namespace
The gateway derives Qdrant collection names from Elasticsearch index names. Its original mapping kept ASCII letters, numbers, underscores, and hyphens, while replacing other characters with underscores. That makes catalog.v2 and catalog_v2 distinct Elasticsearch names but the same Qdrant collection name. A second edge case exists when document projection is enabled: catalog_documents uses the collection name reserved for catalog’s document store.
Bulk create must not become upsert
Elasticsearch gives index and create different meanings inside a bulk request. index may replace a document at the same ID, while create must fail with an item-level HTTP 409 conflict when that ID already exists.
An alias must route the request
The gateway has stored aliases durably from its first release, but persistence alone did not make them functional. Index lookup still derived the Qdrant collection name and deterministic point ID from the name in the request. A request to current-products/_doc/1 therefore looked in the alias namespace rather than the products namespace recorded as its target.
0.1.2: safer defaults for production traffic
Version 0.1.2 is a small operational release.
A missing update is not an upsert
Elasticsearch’s update API does not create a missing document unless the request explicitly asks for upsert behavior. The gateway previously lost that distinction: an update to an absent ID could return “result”: “updated”, even though Qdrant had no point to change. A text-field update could go further and create the document from an empty source.
A missing delete is not a successful delete
Deleting an absent document is intentionally idempotent, but Elasticsearch still tells the client that nothing was removed. The gateway previously returned HTTP 200 with “result”: “deleted” because Qdrant’s point-delete operation succeeds even when the point does not exist.
A timeout is a feature
What happens when Qdrant stops responding?
Oversized requests are not bad JSON
The gateway bounds request bodies with MAX_BODY_BYTES and gives bulk ingestion a separate MAX_BULK_BYTES ceiling. Those limits protect memory, but the rejection previously looked like a generic HTTP 400 compatibility error. A client could not reliably tell whether to fix malformed JSON or split a valid payload into smaller requests.
A missing document is a 404
Elasticsearch’s get-document API distinguishes a missing document at both layers of the response: the JSON body contains “found”: false, and the HTTP status is 404. The gateway previously returned the right body with HTTP 200.
One replica is the safe default
The Helm chart used to request two gateway replicas while also provisioning one ReadWriteOnce volume for the SQLite metadata database. That combination looked highly available but did not supply a highly available control plane.
No phantom indices after creation failures
Creating an index spans two stores: Qdrant owns the collections and payload indexes, while SQLite keeps the gateway’s mappings and aliases. That boundary needs a deliberate publication order.
A slash is part of the document ID
Elasticsearch clients put document IDs in URL path segments. An ID such as order/42 therefore reaches the server as order%2F42; spaces and non-ASCII text are encoded for the same reason. The gateway previously routed on the raw URI path, so it stored the encoded spelling as _id instead of the caller’s original value. A later request produced by a client from order/42 could still find that spelling, but responses exposed order%2F42, breaking identity round trips and the documented arbitrary-ID guarantee.
When bulk errors did not mean errors
Elasticsearch bulk responses have two levels: a top-level errors boolean and an items array whose action objects contain per-document results. The Bulk API contract defines errors as true when one or more operations did not complete successfully. Clients commonly inspect the boolean first and only walk the item details when it is true.
Three nodes, one disappears, and the search keeps moving
A search service is easy to benchmark when everyone is alive. The more useful question is what happens when one of them vanishes in the middle of lunch.
Making the bulk request limit real
The gateway has separate request-size controls for ordinary JSON requests and bulk NDJSON: MAX_BODY_BYTES defaults to 10 MiB, while MAX_BULK_BYTES defaults to 50 MiB. A useful bulk batch is often much larger than a search or single-document request.
Keep the Elasticsearch client, change the engine
Changing search engines can mean changing application code that already works. In a product catalogue, documentation site, jobs board, or support portal, the application may already know how to create an index, bulk documents, search with filters, and render _source through an Elasticsearch client.
Two collections, one stateless gateway
Reading a product’s current source and finding it through search are different operations. Qdrant ES Gateway can store the source separately from its searchable projection, allowing a write to wait for durable source storage while the search projection converges asynchronously.
Pattern queries without a client rewrite
Catalogue search often has two different jobs hiding behind one endpoint. Natural-language search wants token-aware retrieval. Operational search wants to find an SKU prefix, a serial number, or a wildcarded identifier.
Batching searches and paging through ordered results
An application may need several searches to populate a catalogue page, or successive pages from an ordered result set. Qdrant ES Gateway supports _msearch for batching searches and search_after for cursor pagination within its supported search API.
Compatibility includes the boring lifecycle calls
An application can have a perfectly ordinary search query and still fail a migration before it reaches that query. Production clients create and delete indexes, ask whether an index exists, refresh after an import, and sometimes close and reopen an index while applying settings.
Launch video: keep the Elasticsearch client, change the engine
Here is the short launch video for Qdrant ES Gateway:
A dress, a tea bag, and a null
Real catalogues are less like neat databases and more like drawers everyone in the house has used.