Anvi Search: Enterprise Commerce Search & Endeca Migration
Anvi Search is enterprise commerce search built as an Endeca migration instrument — delivering hybrid vector retrieval, a visual merchandiser workbench, and 100% in-VPC data sovereignty.
Engineered for multi-million SKU retailers and B2B distributors modernizing off legacy Oracle Endeca (MDEX) and Lucidworks Fusion (SolrCloud) without rewriting business merchandising rules, losing dynamic cartridges, or surrendering proprietary catalog data to multi-tenant cloud SaaS.
Open Engineering & Learning Laboratory: Anvi Search is an independent engineering research project built as a hobby, technical playground, and deep learning endeavor by a Principal Search Architect. It is not a commercial product for sale. Everything is engineered from first principles and shared openly to explore how modern hybrid retrieval, vector embeddings, and an explainable merchandiser workbench can solve enterprise search modernization.
Derived from the ancient Sanskrit root अन्वेषण (Anveshana) / अन्वी (Anvi) — meaning "To Seek, To Inquire, To Follow the Path of Knowledge, and To Explore Truth". In classical Indian thought, Anvi represents one who bridges and guides the way through deep, uncharted territory. Anvi Search was named to honor this pure spirit of discovery: engineering an open, transparent, and explainable search discovery engine from first principles.
The Enterprise Search Dilemma: Three Paths Forward
Thousands of enterprise retailers and B2B distributors run billions in commerce revenue through aging Oracle Endeca (MDEX) and Lucidworks Fusion (SolrCloud) clusters. Here is how staying, moving to SaaS, and Anvi Search compare:
Staying on Legacy Endeca or Lucidworks Fusion
Oracle MDEX is End-of-Life, and Lucidworks Fusion faces enterprise sunsetting and licensing uncertainty. Sustaining support is expensive, ZooKeeper quorum splits and JVM GC freezes cause outages, and finding engineers who know Forge or SolrCloud internals is nearly impossible.
- ❌ End-of-Life and platform sunsetting support risks
- ❌ Fragile ZooKeeper quorums, JVM memory bloat & 4-hour batch reindexing
- ❌ Lacks sub-12ms hybrid vector embeddings and native MCP agent protocols
Moving to Cloud SaaS (Coveo/Algolia)
Cloud search vendors promise turnkey AI, but require rewriting all your merchandising rules from scratch. Query logs, customer PII, and proprietary catalog margins leave your private network, and usage-based pricing scales to hundreds of thousands every year.
- ❌ $100,000 to $300,000+ recurring annual query fees
- ❌ Proprietary catalog and customer order data leaves your VPC
- ❌ Black-box ML models you cannot inspect or explain
Drop-In Parity for Endeca & Lucidworks + Modern Hybrid RAG
Built specifically as a dual migration instrument for Oracle Endeca (MDEX) and Lucidworks Fusion (SolrCloud). Keeps 100% of your business rules, dynamic cartridges, and query pipeline stages. Upgrades you to sub-12ms Hybrid Vector search, inside your own private VPC with zero recurring SaaS fees.
- ✓ 100% automated rule, cartridge & pipeline stage parity (0 retraining)
- ✓ 100% In-VPC data sovereignty (zero cloud query fees)
- ✓ Sub-12ms Hybrid BM25 + Vector RRF with explainable scoring
The 6 Core Superpowers of Anvi Search
Combining cutting-edge NLP, Machine Learning, and Contextual RAG with enterprise-grade Rule Management, Visual CMS Cartridges, and 100% On-Premise / In-VPC sovereignty.
1. Hybrid Vector RRF & Dynamic Swatches
Dense semantic embeddings meet lexical SKU precision in sub-12ms:
- Parent-Child SKU Collapse: Matches on child attributes (size, spec, color) while rolling up to 1 clean parent card to eliminate result clutter.
- Query-Aware Swatch Promotion: Searching "red running shoes" dynamically promotes the red variant image, rather than the default black thumbnail.
- Application-Layer RRF: Dual-branch BM25 + 384-dim dense vector fusion preserving sort stack determinism (Price, Margin) with neural MiniLM cross-encoder reranking.
2. Geo-Climate & Order History Personalization
Coveo-class relationship & location boosting with 100% In-VPC privacy:
- Geo-Climate Location Intent: Searching "jacket" in New York in November automatically boosts heavy down parkas (+300 pts), while in Dallas (72°F) it boosts lightweight fleece and windbreakers.
- First-Party Order History Affinity: Binds past brand loyalty (e.g. 3 Arc'teryx orders → +250 boost), saved size profiles (Size 10.5 → +150), and price tiers at query time.
- Installed-Base & Contract Graphs: Connects registered equipment (e.g. CAT 320D) to compatible parts (+500 pts) and B2B contract tiers with zero customer PII leakage to cloud SaaS.
3. Kafka Event Streaming (vs Push/Pull)
Replacing 4-hour batch ITL crawls and table locks with real-time CDC:
- Recommended Flagship Route: Enterprise ERP/PIM systems emit domain events to partitioned Kafka topics. Zero search cluster lock contention.
- Sub-20ms CDC Hot Path: Price and inventory changes stream continuously into in-memory document overlays.
- Pluggable Ingestion Adapters: Supports Debezium DB CDC (Oracle, Postgres, Mongo), REST Bulk Push, and S3/SFTP batch pull feeds.
4. Multi-Site & Multi-Locale Architecture
Hierarchical rule inheritance across brands, countries, and channels:
- Inheritance Rule Tree: Define global clearance suppression once at the Global Enterprise level; child country sites inherit automatically.
- Local Site Overrides: UK or German storefronts can override banners and campaigns without modifying parent rules (depth-before-breadth).
- Language-Family Cores: Dedicated tokenizers and embeddings for English, German, and Japanese, with site-filtered slices inside each core.
5. Store-Based BOPIS & Pricing Overlays
Omnichannel retail without the 700 million document explosion trap:
- In-Memory Store Bitsets: Supports 2,000 physical stores by layering store inventory over a single store-agnostic catalog core.
- Sub-Millisecond BOPIS Filtering: Shopper clicks "In Stock at Chicago Store #1042" → filter evaluates in 0.4ms with zero index bloat.
- Store-Specific Clearance Pricing: Overlays local store markdown prices over national online MSRP.
6. 100% On-Prem & In-VPC (Zero SaaS)
Complete data sovereignty, air-gapped resilience, and $0 query fees:
- Runs in Your Own Infrastructure: Deploy on AWS, Azure, GCP, or bare-metal Kubernetes. Customer queries and margins never leave your firewall.
- Zero SaaS Query Taxes: Flat, predictable infrastructure costs. High-traffic flash sales never generate surprise $200k SaaS bills.
- Air-Gapped & Transparent: Local CPU ONNX vector embeddings with full source code ownership and explainable scoring audits.
The Merchandiser Workbench: 8 Production Screens
Built in Django & Wagtail mounted directly inside the FastAPI engine (one process, one port, one unit). Merchandisers never touch terminal configs:
End-to-End Retrieval & Ingestion Pipelines
Inspect every microsecond of the 11-stage query execution lifecycle and the three multi-speed catalog ingestion lanes:
Executes dual retrieval concurrently: Branch A runs BM25 (Solr edismax) for exact SKU precision; Branch B runs dense vector k-NN (384-dim e5-small). Executes Solr parent-level field collapse: matches child SKU attributes (size, color, spec) while rolling up to 1 clean parent card.
- Rule AST & site_id filter
- Dense Query Vector
- Parent-Child SKU Grouping Spec
- Top-100 Lexical Candidates + Scores
- Top-100 Vector Candidates + Cosine
- Collapsed Parent Match Groups
Real-Time Kafka Hot-Path (CDC)
Sub-20ms Continuous Propagation
Async Embedding & Text Mutations
Continuous Micro-Batch (1-3s Freshness)
Shadow Core Baseline & Stream Replay
Stream Replay / Zero-Downtime Swap
Lane 1: Real-Time Kafka Hot-Path (CDC)
Topic: catalog.inventory-updates & catalog.price-changesCaptures instant row-level database mutations (stock drops, flash sale discounts, BOPIS store inventory) and streams them via Kafka into an in-memory document overlay with zero search table locks.
Oracle Endeca (MDEX 11.3)
Single-threaded query processing bound to proprietary MDEX binary. Vulnerable to memory fragmentation.
Brittle XML configuration files running multi-hour monolithic batch crawls.
Proprietary XML rule engine for boost/bury, slotting, and dimension precedence.
Proprietary query parameters embedded in ATG JSP / React storefront form handlers.
Anvi Search Platform
Concurrent multi-threaded C++ / Rust core executing dual BM25 + Vector HNSW retrieval.
Continuous event streaming via Kafka / Debezium with in-memory delta overlays.
1:1 drop-in XML import with real-time audit tracing and explainable rule collision visualizer.
Transparently accepts legacy Endeca query parameters and maps them to hybrid plans.
Dgraph Engine ➔ Stateless In-VPC Hybrid Core
Unpredictable latency spikes under heavy concurrent faceting.
Sub-10ms predictable p99 latency across millions of SKUs with horizontal auto-scaling.
Dynamic Tool Contract Discovery
Agent Tool Execution Call
Hallucination Shield & In-VPC Verification
Grounded Citation Payload Delivery
Dynamic Tool Contract Discovery
The AI Agent queries the MCP gateway to discover active catalog dimensions, available tools, and parameter schemas dynamically without hardcoded prompting.
{
"tools": [{
"name": "catalog_search",
"description": "Hybrid vector + BM25 catalog search with parametric facet filtering.",
"parameters": {
"query": { "type": "string" },
"filters": { "type": "object" }
}
}]
}Model Context Protocol (MCP) Server (GET /mcp/tools)
Standardized tool-calling contract allowing autonomous AI agents (Claude, OpenAI, Gemini) to query catalog facets, search products, and inspect inventory.
Agent Readiness & Grounding Shield (GET /readiness)
Guarantees zero AI hallucination by evaluating catalog attribute completeness and generating verified citation proofs for LLM outputs.
Graph-RAG Relational Knowledge Traversal
Traverses product compatibility matrices, variant trees, and bundle graphs to resolve relational questions.
Corrective RAG (CRAG) & Confidence Filtering
Evaluates retrieval confidence across hybrid branches; automatically filters out low-relevance items before feeding generation contexts.
In-Session Real-Time Intent Drift Tracking
Dynamically updates user intent vectors within the active shopping session based on click paths without requiring persistent PII.
Multimodal Vision-Language Search (SigLIP / CLIP)
Joint vision-language embedding space enabling visual similarity search combined with text attributes.
Query NLU & Token Normalization
Deep query normalization, Unicode case-folding, punctuation normalization, and token-level accent stripping.
Query-Side Synonyms Engine
Merchandiser-editable one-way and two-way equivalence synonym expansion running before rules evaluate.
NLP Intent & Entity Decomposition
Deconstructs compound queries into structured attribute filters (colors, sizes, brands, price boundaries).
Vocabulary-Aware Speller & Did-You-Mean
Spelling suggestion that validates against index vocabulary first, firing only when strict matches fail.
Lexical BM25 / Edismax Engine
High-throughput Lucene/Solr inverted index retrieval with configurable multi-match modes.
Dense Vector Semantic Search (kNN)
384-dimensional dense semantic embedding retrieval powered by quantized e5-small-v2 ONNX models.
Cross-Lingual Retrieval Engine
Retrieves English product catalog records from queries submitted in Spanish, French, or other languages.
Application-Layer RRF Hybrid Fusion
Reciprocal Rank Fusion in the application tier (Lexical 1.0 / Vector 0.4) preserving sort stacks.
Cross-Encoder Neural Reranker (ms-marco-MiniLM-L6-v2)
Deep sequence-pair cross-attention reranker providing +0.035 nDCG boost.
Learning to Rank (LTR / Click-Feedback)
Machine learning ranking (LambdaMART) trained on verified live click, add-to-cart, and purchase streams.
Landing Page & Trigger Resolution (Depth > Breadth)
Deterministic page and cartridge resolution walking the navigation state hierarchy.
Merchandising Rules Engine (Boost, Bury, Pin)
Layered business rules modifying product elevations, score multipliers, redirects, and banner slots.
The Resolution Inspector (Visual Debugger)
Interactive diagnostic workbench answering Why is this page showing? with complete candidate traces.
Guided Navigation & Dynamic Faceting
Multi-select facet counts, hierarchical dimension trees, and dynamic facet reordering.
Negotiable Retrieval on Zero Results
Analyzes binding constraints when queries return 0 results and suggests relaxed alternatives with live counts.
Sub-Millisecond Typeahead & Autocomplete
Ultra-fast prefix search honoring tenant, entitlements, and merchandiser exclusions.
Lane A: Batch Catalog Sync & Blue-Green Reindex
12-step full catalog reindex with validated configsets, monotonic guards, and node parity validation.
Lane B: Real-Time Price & Inventory Streaming (CDC)
Microsecond price, stock, and status updates via high-watermark PostgreSQL streaming.
Lane C: Dynamic Graph & Product Relations
Calculates and updates product bundles, cross-sells, and compatible accessories.
Query Health & Fallback Analytics
Monitors search fallback rates, zero-result frequency, and shallow rerank pool warnings.
Atomic Append-Only Audit Trail
Complete, immutable record of every merchandising change, rule edit, and API key action.
100% In-VPC Isolation & Data Protection
Runs entirely within your cloud VPC or bare-metal environment with zero external telemetry egress.
Measured on 4,998 Real Amazon Products (5,482 Human ESCI Judgements)
Search quality must be scientifically measured against real catalog data, not assumed. Here are the empirical measurements taken on running Anvi Search nodes:
| Retrieval Configuration | nDCG@10 Score | Latency (Laptop) | Production Status & Architectural Role |
|---|---|---|---|
| Lexical Alone (BM25 edismax) | 0.527 | 3.5 ms | Fast exact token, SKU, and keyword matching. |
| Vector Alone (e5-small k-NN) | 0.492 | 4.8 ms | Semantic recall; loses on exact SKUs when unassisted. |
| Anvi Hybrid Fusion (RRF) | 0.533 – 0.537 | 9.3 ms | Production Default: Zero-loss semantic + exact SKU recall. |
| Hybrid Fusion + Cross-Encoder Rerank | 0.569 (+0.035 nDCG) | 82.5 ms | Opt-In: Maximum precision reranking for top-25 candidates. |
Passionate About Search Architecture & Engineering?
Anvi Search is an open hobby and learning research endeavor. I love connecting with fellow software engineers, search architects, and technology enthusiasts to exchange ideas, discuss retrieval patterns, and share learnings.