Multilingual NLP in Coveo: Language Detection, Decompounding, Stemming, and Cross-Lingual Semantic Search

1. The Four Linguistic Challenges in Global Search

Building enterprise search for multilingual catalogs involves solving four major linguistic hurdles:

2. Ingestion-Time Language Architecture

When documents arrive via Push API, Stream API, or Crawlers, Coveo’s Language Processor executes three critical steps:

1. Automated Language Identification: Evaluates character n-gram frequencies across document title and body text. Assigns an explicit two-letter ISO language tag (e.g. @language == "de"). If a document contains multilingual translations, individual language variants are extracted into separate child records.
2. Language-Specific Field Indexing: For each detected language, text is indexed into dedicated linguistic fields (e.g. title_de, body_de). German text uses German decompounding rules; Japanese text uses Kuromoji dictionary segmentation.
3. Multilingual Vector Embedding: The document is passed to a multilingual transformer model (e.g. based on multilingual e5/BERT) generating a 384-dimensional or 768-dimensional dense vector representing the abstract concept regardless of source language.

3. Query-Time Multilingual Pipeline Execution

At query time, the incoming user request is processed through localized pipeline filters:

Step 1: Query Language Identification

Even if the user is on the English storefront, they may enter a query in French or Spanish. Coveo detects the language of the query in under 0.6 milliseconds using fast n-gram probability classifiers.

Step 2: Localized Thesaurus & Synonym Isolation

A critical architectural rule in multilingual search is Thesaurus Isolation:

• Synonyms must be scoped strictly to individual language locales (e.g., fr-FR vs. fr-CA).
• This prevents "false friend" semantic pollution. For example, the word "gift" in English means a present, while in German "Gift" means poison. Global, un-scoped synonym dictionaries cause catastrophic ranking errors.

Step 3: Cross-Lingual Semantic Retrieval

When a user searches in French ("remplacement filtre à huile"), Coveo executes a dual-branch retrieval strategy:

1. Lexical Branch: Searches the French index fields (@language == "fr") for exact localized matches.
2. Dense Vector Branch: Encodes the French query into the shared multilingual embedding vector. The vector engine searches across the entire knowledge base, retrieving English technical repair guides whose semantic vector matches with high cosine similarity.
3. Reciprocal Rank Fusion (RRF): Merges localized French results with cross-lingual English semantic hits, prioritizing native language matches when available while guaranteeing zero zero-result dead ends.

4. Localized Generative Answering & Smart Snippets

Coveo Relevance Generative Answering (CRGA) and Smart Snippets are fully multilingual:

• Native Extractive Answering: When an Italian user asks "Qual è la pressione massima per la pompa PX-3?", the neural reading model extracts the precise sentence from the Italian documentation card.
• Cross-Lingual Synthesis: If an answer only exists in English source documents, the generative answering engine synthesizes the final answer in Italian, citing the English source document for technical verification while shielding the shopper from language barriers.

5. Multilingual Query Pipeline Configuration Example

Below is an example of an enterprise multi-locale pipeline setup using Coveo Headless:

// services/coveoMultilingualSearch.ts
import axios from "axios";

interface MultilingualSearchProps {
  query: string;
  locale: string; // e.g. "de-DE", "ja-JP", "fr-CA"
  token: string;
}

export async function searchMultilingualCatalog({ query, locale, token }: MultilingualSearchProps) {
  const COVEO_ORG = process.env.COVEO_ORGANIZATION_ID;
  const isoLang = locale.split("-")[0]; // "de", "ja", "fr"

  const payload = {
    q: query,
    locale: locale,
    searchHub: `Storefront-${isoLang.toUpperCase()}`,
    // Prefer native language while keeping cross-lingual fallback active
    rankingFunctions: [
      {
        expression: `@language == "${isoLang}"`,
        normalizeWeight: true,
        modifier: 200
      }
    ],
    // Request localized Smart Snippet
    questionAnswering: {
      enableSmartSnippet: true,
      enableGenerativeAnswer: true
    },
    facets: [
      { facetId: `category_${isoLang}`, field: `category_${isoLang}`, numberOfValues: 8 },
      { facetId: "brand", field: "brand", numberOfValues: 6 }
    ]
  };

  const res = await axios.post(
    `https://${COVEO_ORG}.org.coveo.com/rest/search/v2`,
    payload,
    {
      headers: {
        "Authorization": `Bearer ${token}`,
        "Content-Type": "application/json"
      }
    }
  );

  return {
    results: res.data.results,
    snippet: res.data.questionAnswering?.smartSnippet,
    generativeAnswer: res.data.questionAnswering?.generativeAnswer
  };
}