Ajay Agrawal|Java · Kafka · Search Architect
TOPIC KNOWLEDGE HUB

Data Ingestion & Indexing Pipelines

Real-time CDC streaming, batch blue-green reindex swaps, schema mapping, monotonic guards, and node parity validation.

38 Technical Deep-DivesAll Legacy URLs Preserved (HTTP 200)Zero Data Loss
💡
Architectural Context: High-throughput catalog ingestion architectures maintaining sub-20ms price and stock freshness.

Technical Guides & Implementations (38)

Chronological engineering references
Indexing

Coveo Indexing Pipeline Architecture: Push API, Stream API, Document Extensions, and Security Identity Providers

Architecting scalable ingestion into Coveo using Push API v2, Stream API, custom Python document extension scripts, and early-binding Security Identity Provider…

Indexingfrom the archive

how to enable ancestor matches during record search for Hierarchical Dimension?

Search interface is generally used to do record search and searchable attributes needs to be added. In a ATG/Endeca integration, By Default product.category dim…

Indexingfrom the archive

Endeca 11.x : There was a problem with the zip file given

Behaviour :- Following Exception/Error comes during baseline or promoting content from Experience Manager. Error shows that " there was a problem with Zip File …

Indexingfrom the archive

Endeca 11.x : Property 'priority' must be a positive integer

Behaviour :- Following Exception/Error comes during Endeca experience manager content migration between environments or importing contents from zip file. SEVERE…

Indexingfrom the archive

ATG-Endeca Integration : How to Enable/Disable automatic indexing after BCC Post Deployment?

Endeca Indexing jobs getting triggered automatically just after BCC post deployment. IndexingDeploymentListners helps trigger indexing and part of /atg/epub/Dep…

Indexingfrom the archive

ATG Endeca Integration : Interview Questions and Answers- Part2

ATG Endeca Integrations - Interview Questions - Part-2 1. What are basic components used to Integrate ATG and Endeca? 2. What are Accessors and how it helps in …

Indexingfrom the archive

Can Endeca Partial Indexing be performed after inventory or direct SQL update?

Behaviour As per default behavior of ATG Endeca integration, Endeca Partial indexing does not update index with inventory update. ATG driven partial index looks…

Indexingfrom the archive

Oracle Endeca ITL and Expereince Manager High Availability Architecture

What are High Availability and Failover High availability as the ability for the system to continue functioning after the failure of one or more of the servers.…

Indexingfrom the archive

Endeca Basics : Interview Questions and Answers- Part1

Endeca Basics - Interview Questions - Part-1 1. What is Endeca and why do you use Endeca instead of other search products? 2. What are dimensions and properties…

Indexingfrom the archive

What are the different available deployment templates in Endeca?

Oracle Provides three different types of deployment template to create Endeca Applications. CAS-Forge based deployment template - This was being used until vers…

Indexingfrom the archive

Oracle Endeca 11.x : How to Migrate Endeca Application between environment?

As part of the development/debugging phase, Endeca Application needs to be migrated from QA to UAT or Production to Staging. Following items need to be migrated…

Indexingfrom the archive

Oracle Endeca 10.x : EndecaHow to prevent 404 error after baseline update due to workbench password change?

Behaviour :- Web application starts showing 404-page error after baseline update once admin password changed from change password window using Endeca workbench …

Indexingfrom the archive

How to Migrate Endeca Application between environment for forge-based Indexing?

As part of the development/debugging phase, Endeca Application needs to be migrated from QA to UAT or Production to Staging. Following items need to be migrated…

Indexingfrom the archive

Endeca 11.2 : Workbench login credentials are not found or found invalid credential type for key

Behaviour :- Following Exception/Error comes during Endeca Application initialize services or baseline/partial indexing process. [04.20.16 22:46:52] INFO: Provi…

Indexingfrom the archive

Endeca 11.x : How to Add manipulator using CAS?

Oracle Endeca 11.1 onwards forge less approach is used to index. Most of the data manipulation is done at the source level. Java Manipulators required if some d…

Indexingfrom the archive

Oracle Endeca 11.x - How to Add multiple record store using Endeca CAS ?

Oracle Endeca has introduced forge less baseline indexing approach to index the data to MDEX engine. In this process, CAS generates the MDEX compatible data. Su…

Indexingfrom the archive

Endeca 11.2 : Unauthorized (401): Unauthorized access to workbench

Behaviour :- Following Exception/Error comes during Endeca Application initialize services or baseline/partial indexing process. < [04.20.16 22:55:48] SEVERE…

Indexingfrom the archive

How to enable property for record filtering in Endeca - CAS based Indexing?

The Index Configuration modifies index configuration stored in the Endeca Configuration Repository for an application. index-config.json contains all dimension …

Indexingfrom the archive

CAS Based Indexing : What is Index-config.json

The Index Configuration modifies index configuration stored in the Endeca Configuration Repository for an application. This utility is typically used to modify …

Indexingfrom the archive

How to migrate CAS record store schema with data between environments?

As per ATG-Endeca Integration, All product catalog data gets stored in Endeca CAS record store during baseline process. Sometimes, these data needs to be migrat…

Indexingfrom the archive

Oracle Endeca 11.x : How Last Mile Crawl is used?

Endeca baseline update process invokes last mile crawl to create Dgidx-compatible data and passes it to Dgidx to generate binary files for MDEX engine. Find out…

Indexingfrom the archive

Endeca Basics : Partial update Process Limitation

A partial update modifies only those portions of the MDEX Engine index that have changed since the last baseline update. Partial updates can affect any number o…

Indexingfrom the archive

Endeca Basics : How to troubleshoot partial update is working as expected or not?

Endeca Partial indexing process uses only Forge process to generate XML file and Dgidx process gets skipped. The MDEX Engine processes partial updates concurren…

Indexingfrom the archive

Endeca Basics : How to Debug Endeca Pipeline Components transformation is working as expected?

A pipeline is composed of a collection of components. Each component performs a specific function during the transformation of source data into Endeca records. …

Indexingfrom the archive

Oracle Endeca Commerce 11.x : How to troubleshoot file based promote content in Distributed environment?

On an Oracle Endeca Commerce 11.1 and later version, promote_content script publishes content to a folder as zip file format and ATG store instance reads and ca…

Indexingfrom the archive

Forge Error : The requested file copy cannot be completed since it requires writing multiple files to the same location.

Behaviour :- Following Errors comes during baseline process. CRS.fetch_config_to_input_for_forge_Forge.log Mar 22, 2016 12:16:35 PM com.endeca.esf.delegate.task…

Indexingfrom the archive

SocketTimeoutException: Time out error during Read Records From CAS RecordStore

Behaviour :- Following errors comes during baseline update <SEVERE: Caught an exception while invoking method 'run' on object 'BaselineUpdate'. Releasing loc…

Indexingfrom the archive

Endeca Baseline Error : SEVERE: Unable to update Assembler

Behaviour:- Following Errors comes during baseline process SEVERE: Unable to update Assembler at URL: http://hostname:10280/dyn/admin/assemblerAdmin/admin org.a…

Indexingfrom the archive

Endeca CAS Last-Mile-Crawl : wrong value for property 'mergeAction' (UPDATE) when no imported entry found to merge with

Behaviour :- Following Exception comes while during baseline run Mar 11, 2015 8:20:25 PM com.endeca.soleng.eac.toolkit.Controller execute SEVERE: Caught an exce…

Indexingfrom the archive

Create new Crawler in Endeca 11.x application : Could not find the crawl Error

Behaviour :- Sometimes, "Could not find the crawl" or "crawler has been removed/corrupted" error comes during baseline run. Solution:- Run below command to see …

Indexingfrom the archive

ATG Endeca Integration : How to Troubleshoot Endeca Baseline update When Expected Data is Not Available in Endeca indices?

Business creates data and push into Endeca for Indexing. Some type of data that was added through ATG and Endeca baseline update ran successfully without any er…

Indexingfrom the archive

Endeca 11.x - EacCommunicationException Error After Running Initialize Services

The following error is being thrown for any new application trying to run initialize_services on: [01.22.16 16:33:09] INFO: Checking definition from AppConfig.x…

Indexingfrom the archive

Oracle Endeca 3.x : Live Site Outage after Experience Manager promote content

Following exception occurs during Experience Manger promote_content from authoring to Live User experience is impacted because the front-end application no long…

Indexingfrom the archive

Endeca Script : How To Run Dgidx from Command Prompt??

Dgidx is the indexing program that reads the tagged records that were prepared by Forge or CAS and creates the proprietary indexes for the MDEX Engine. Find out…

Indexingfrom the archive

Oracle Endeca 11.x : How to Setup Precedence Rules?

A Precedence Rule provides a way to hide the display of dimensions until someone click a dimension or a particular navigation state. Precedence rules are define…

Indexingfrom the archive

Oracle Endeca : How to display BCC ordering of products in category pages??

In ATG, BCC is used to merchandise product data and there are other multiple features as well. Business can create/edit product and assign a product to multiple…

Indexingfrom the archive

Oracle Endeca 11.1 : Migrate State file to Dimension Value ID Mapping

Migrate State file to Dimension Value ID Mapping From Forge-Based to CAS-Based Applications : - CAS based MDEX writable output concept is introduced from 11.1 o…

Indexingfrom the archive

Dynamic Ranking Not Applied in Guided Navigation in 11.1

Behavior - Selecting sorting option as frequency(Dynamic ranking) under refinement menu not sorting refinement as intended. Solutions :- There are two options a…