Oracle Endeca 11.x : How Last Mile Crawl is used?
Endeca baseline update process invokes last mile crawl to create Dgidx-compatible data and passes it to Dgidx to generate binary files for MDEX engine.
Find out following operations during last-mile-crawl
As per XML snapshot above, multiple record stores can be added for further processing. CAS Based indexing only support switch join between multiple Record store.
2. Add Manipulators
Java manipulators can be added into last-mile-crawl in case any data manipulation required e.g. remove comma and create multi-valued properties
3. Merges index-config
initialize_services.sh runs the following command to update the Endeca Configuration repository with the properties and dimensions mentioned in the ./index-config.json
"${WORKING_DIR}/index_config_cmd.sh" set-config -f "${WORKING_DIR}/../config/index_config/index-config.json" -o all
It's your Turn
Was this blog helpful for you? What do you think about this post? Any other topics that you want to cover in details.
Provide your valuable comments or response below.
Find out following operations during last-mile-crawl
- Merges index-config.json using system user and ATG user
- Merge Multiple record store defined in crawl configuration XML file.
- Processes product/article/store records. Data manipulation can be done if required using custom manipulators.
- Writes the schema and records in the MDEX-compatible format.
How Last Mile gets created
Endeca Application instance needs to create using deployment template. Following commands create last-mile-crawl
<<Endeca_App>>/control/initialize_services.sh
In turn the initialize_services.sh runs a following command
${CAS_ROOT}/bin/cas-cmd.sh createCrawls -h ${CAS_HOST} -p ${CAS_PORT} -f ${WORKING_DIR}/../config/cas/last-mile-crawl.xml
Features
1. Record Store Joins
Where the <<Endeca_App>>/config/cas/last-mile-crawl.xml sets up the CAS Crawl with the names of the CAS Recordstores,
Endeca Application instance needs to create using deployment template. Following commands create last-mile-crawl
<<Endeca_App>>/control/initialize_services.sh
In turn the initialize_services.sh runs a following command
${CAS_ROOT}/bin/cas-cmd.sh createCrawls -h ${CAS_HOST} -p ${CAS_PORT} -f ${WORKING_DIR}/../config/cas/last-mile-crawl.xml
Features
1. Record Store Joins
Where the <<Endeca_App>>/config/cas/last-mile-crawl.xml sets up the CAS Crawl with the names of the CAS Recordstores,
| <moduleProperties> <moduleProperty> <key>dataRecordStores</key> <value>CRS-data</value> <value>CRS-External-data</value> </moduleProperty> <moduleProperty> <key>dimensionValueRecordStores</key> <value>CRS-dimvals</value> </moduleProperty> </moduleProperties> |
As per XML snapshot above, multiple record stores can be added for further processing. CAS Based indexing only support switch join between multiple Record store.
2. Add Manipulators
Java manipulators can be added into last-mile-crawl in case any data manipulation required e.g. remove comma and create multi-valued properties
| <manipulatorConfig> <moduleId> <id>com.endeca.cas.extension.sample.manipulator. substring.SubstringManipulator</id> </moduleId> <moduleProperties> <moduleProperty> <key>sourceProperty</key> <value>Endeca.Document.Text</value> </moduleProperty> <moduleProperty> <key>targetProperty</key> <value>Short.Truncated.Text</value> </moduleProperty> <moduleProperty> <key>length</key> <value>20</value> </moduleProperty> </moduleProperties> <id>Create short truncated text property</id> </manipulatorConfig> |
3. Merges index-config
initialize_services.sh runs the following command to update the Endeca Configuration repository with the properties and dimensions mentioned in the ./index-config.json
"${WORKING_DIR}/index_config_cmd.sh" set-config -f "${WORKING_DIR}/../config/index_config/index-config.json" -o all
It's your Turn
Was this blog helpful for you? What do you think about this post? Any other topics that you want to cover in details.
Provide your valuable comments or response below.
Discussion (7)
Technical questions and architecture discussionsJoin the Discussion