Sunday, May 1, 2016

Oracle Endeca 11.x : How to Migrate Endeca Application between environment?

As part of the development/debugging phase, Endeca Application needs to be migrated from QA to UAT... thumbnail 1 summary

  • As part of the development/debugging phase, Endeca Application needs to be migrated from QA to UAT or Production to Staging.

    Following items need to be migrated between environments. Find out the description below for each item.

    • Record Store Record store contains all indexable records(Product catalog, content, record store etc.) and dimension information.
    • Workbench application Business creates content using Endeca Experience manager and gets stored in ECR(Endeca Configuration Repository). This application contains XM content, cartridges and other configuration like thesaurus etc.
    • Index-config - Index-config contains all properties, dimensions, precedence rules and dictionary related configuration in JSON format.
      • Dimension Value Id Manager - Dimension Value Id manager contains Endeca auto generated dimension values. Endeca N-value can be different if this does not copy to the environment.
      • Endeca Pipeline -  This Folder contains Endeca Application indexing configuration XML including properties, dimensions, search etc.
      • Editors Config This Folder contains configuration file to make a call to MDEX to show appropriate data in Experience manager cartridges.
      productdataservice.json Snapshot
      productdataservice.json

      As part of the migration process, the application needs to be exported first and the same set of files can be used to import into another environment.

      Export Endeca Application steps 

      1. Record Store - Click here to get more details. This Step can be optional if baseline triggers using /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin Component

      2. Dimension Value Id Manager - Navigate To CAS Bin folder and run following commands to export
      CAS_Installation_Path/CAS/11.1.0/bin->./cas-cmd.[bat|sh] exportDimensionValueIdMappings -m <<dimension-value-id-manager name>> -f /home/atg/dimvalid.csv

      3. Endeca Pipeline - Copy Endeca_App/config/mdex folder

      4. Editors Config - Copy Endeca_App/config/ifcr/configuration/tools/xmgr folder

      5. Workbench Application - Navigate To Endeca Application control folder and run following commands to export content
             <<Endeca_App>>/control->./runcommand.[bat|sh] IFCR exportApplication </path to generate file>
      This would generate the zip/exploded folder.


      6. Index-Config - Navigate To Endeca Application control folder and run following commands to export index-config.json
           <<Endeca_APP_PATH>>/control->./index_config_cmd.[bat|sh] get-config -o all-File_Path/index-config.json



      Import Endeca Application steps - Steps should be in the following order.

      1. Record Store Click here to get more details. This Step can be optional if baseline triggers using /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin Component.

      2. Dimension Value Id Manager - Run below command to import dimension values
      <CAS_Installation_Path>/CAS/11.1.0/bin->./cas-cmd.[bat|sh] importDimensionValueIdMappings -m <<dimension-value-id-manager name>> -f /home/atg/dimvalid.csv

      3. Endeca Pipeline - Use the backup of <<Endeca_App>>/config/mdex folder and move pipeline.

      4. Editors ConfigCopy editors_config from step 5 and change host and post information based on environment under each json file and run the following command
               <<Endeca_App>>/control -> set_editors_config.[bat|sh]

      5. Index-Config - Run below command to import index-config.json
      <<Endeca_APP_PATH>>/control->./index_config_cmd.[bat|sh] set-config -o all -f File_path/index-config.json


      6. Endeca Baseline Update-  Run below Command

            <<Endeca_App>>/control -> ./baseline_update.[bat|sh]

      7. Workbench Content - Run below command to import workbench content
              <<Endeca_APP_PATH>>/control->./runcommand.sh IFCR importApplication <<export_zip_file/folder_path>>

      8. Promote Content - Run the following command to promote content from Authoring to Live. Promote content can be run the Workbench as well.
               <<Endeca_App>>/control -> ./promote_content.[bat|sh]

        From Author
        Was this post resolve the solution you are looking for? are you looking for some other issues?Provide your comments

        Oracle Endeca 10.x : EndecaHow to prevent 404 error after baseline update due to workbench password change?

        Behaviour :- Web application starts showing 404-page error after baseline update once admin passw... thumbnail 1 summary
      • Behaviour :-

        Web application starts showing 404-page error after baseline update once admin password changed from change password window using Endeca workbench and its not updated WorkbenchConfig.xml.

        Workbench XML IFCR password change
        WorkbenchConfig.xml IFCR screenshot
        The following error can be seen in Endeca application logs

        INFO: [LiveMDEXHostA] Starting backup utility 'backup_log_dir_for_component_DgraphA1'.
        Apr 29, 2015 9:53:25 PM com.endeca.soleng.eac.toolkit.component.Component start
        INFO: [LiveMDEXHostA] Starting component 'DgraphA1'.
        Apr 29, 2015 9:53:33 PM sun.reflect.NativeMethodAccessorImpl invoke0
        INFO: Publishing Workbench 'live' configuration to MDEX 'DgraphA1'
        Apr 29, 2015 9:53:33 PM com.endeca.soleng.eac.toolkit.component.IFCRComponent pushLiveContentToDgraphById
        INFO: Pushing live content to dgraph: DgraphA1
        Apr 29, 2015 9:53:33 PM com.endeca.soleng.eac.toolkit.script.Script runBeanShellScript
        SEVERE: Unauthorized (401): Unauthorized access to workbench. Please check your credentials.
        Occurred while executing line 5 of valid BeanShell script: 
        [[

        2|       

        3|        compId = invokingObject.getElementId();
        4|        log.info("Publishing Workbench 'live' configuration to MDEX '" + compId + "'");
        5|        IFCR.pushLiveContentToDgraphById(compId);
        6|      
        7|    
        ]]
        Apr 29, 2015 9:53:33 PM com.endeca.soleng.eac.toolkit.component.ServerComponent runPostStartupScript
        WARNING: Caught an exception while executing post-startup script for server component 'DgraphA1'.
        com.endeca.soleng.eac.toolkit.exception.AppControlException: Error executing valid BeanShell script.
        at com.endeca.soleng.eac.toolkit.script.Script.runBeanShellScript(Script.java:179)
        at com.endeca.soleng.eac.toolkit.script.Script.run(Script.java:127)
        at com.endeca.soleng.eac.toolkit.component.ServerComponent.runPostStartupScript(ServerComponent.java:323)
        at com.endeca.soleng.eac.toolkit.component.ServerComponent.startInParallel(ServerComponent.java:132)
        at com.endeca.soleng.eac.toolkit.component.cluster.DgraphCluster.applyIndex(DgraphCluster.java:612)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.endeca.soleng.eac.toolkit.Controller.invokeRequestedMethod(Controller.java:917)
        at com.endeca.soleng.eac.toolkit.Controller.execute(Controller.java:255)
        at com.endeca.soleng.eac.toolkit.Controller.main(Controller.java:134)
        Caused by: com.endeca.soleng.eac.toolkit.utility.IFCRUtility$HttpStatusException: Unauthorized (401): Unauthorized access to workbench. Please check your credentials.
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.execute(IFCRUtility.java:785)
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.httpPostDoNotConsume(IFCRUtility.java:731)
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.httpPost(IFCRUtility.java:703)
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.httpPost(IFCRUtility.java:654)
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.mdexPublish(IFCRUtility.java:598)
        at com.endeca.soleng.eac.toolkit.component.IFCRComponent.pushLiveContentToDgraphById(IFCRComponent.java:442)
        at com.endeca.soleng.eac.toolkit.script.Script.runBeanShellScript(Script.java:165)
        ... 45 more
        Apr 29, 2015 9:53:33 PM com.endeca.soleng.eac.toolkit.utility.Utility start
        INFO: [LiveMDEXHostA] Starting shell utility 'rmdir_dgraph-input-old'.


        Solution:-

        This issue comes if Endeca workbench admin password is different than WorkbenchConfig.xml under Endeca_app/config/script file. As per logs, Index generated by Dgidx gets propagated to live Dgraph first than Endeca workbench contents gets published to live Dgraph. So if the password is different than publishing live content method gets failed and logs the error in the log but baseline process does not stop.

        The following code as per snapshot can be added to baseline process under Endeca_App/config/script/DataIngest.xml to prevent above issue even password is different.

        Endeca Baseline update process change to prevent 404 error



        Above code would first try to export site data and verify if workbench password and WorkbenchConfig.xml password both are same or not and export the XML.

        Note : Delete the generated XML after adding above code.


        From Author

        Was this post resolve the solution you are looking for? are you looking for some other issues?Provide your comments

        Wednesday, April 27, 2016

        How to Migrate Endeca Application between environment for forge-based Indexing?

        As part of the development/debugging phase, Endeca Application needs to be migrated from QA to UAT ... thumbnail 1 summary
      • As part of the development/debugging phase, Endeca Application needs to be migrated from QA to UAT or Production to Staging.

        Following items need to be migrated between environments. Find out the description below for each item.

        • Workbench content - Business creates content using Endeca Experience manager and gets stored in IFCR(InFront Configuration Repository)
        • State File - State file contains Endeca auto generated dimension values and FCM generated dimension value. Endeca N-value can be different if state files do not get copied to the environment.
        • Endeca Pipeline -  This Folder contains Endeca Application indexing configuration XML including properties, dimensions, search etc.
        • Cartridge Templates -  This folder contains all cartridge templates that is used to create content in Experience Manager.
        • Editors Config - This Folder contains configuration file to make a call to MDEX to show appropriate data in Experience manager cartridges.
        • Record Store Record store contains all indexable records(Product catalog, content, record store etc.) and dimension information.

        Endeca Dimension Cartridge Editors
        Cartridge Editors


        As part of the migration process, the application needs to be exported first and the same set of files can be used to import into another environment.

        Export Endeca Application steps 

        1. Record Store - Click here to get more details. This Step can be optional if baseline triggers using /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin Component

        2. State files - Copy <<Endeca_App>>/data/state folder.


        3. Endeca Pipeline - Copy <<Endeca_App>>/config/pipeline folder

        4. Cartridge Templates - Copy <<Endeca_App>>/config/cartridge_templates folder.

        5. Editors Config - Copy <<Endeca_App>>/config/editors_config folder

        6. Workbench ContentNavigate To Endeca Application control folder and run following commands to get export 
               <<Endeca_App>>/control->export_site.[bat|sh]
              This would generate the XML file and take the backup
        Endeca Export Site



        Import Endeca Application steps - Steps should be in the following order.

        1. Record Store Click here to get more details. This Step can be optional if baseline triggers using /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin Component.

        2. State files - Take the backup of <<Endeca_App>>/data/state folder and move state files copied from step-2  during export.

        3. Endeca Pipeline - Take the backup of <<Endeca_App>>/config/pipeline folder and move pipeline files copied from step-3  during export.

        4. Cartridge Templates - Copy cartridge_templates  from step 4 and run the following command to apply to Workbench. 
                <<Endeca_App>>/control -> set_templates.[bat|sh]
        Endeca Set Template command

        5. Editors Config - Copy editors_config from step 5 and change host and post information based on environment under each json file and run the following command
                 <<Endeca_App>>/control -> set_editors_config.[bat|sh]

        6. Endeca Baseline Update-  Run below Command

              <<Endeca_App>>/control -> ./baseline_update.[bat|sh]

        7. Workbench Content - Import generated XML from Step-6 (Import site) to the Workbench content u. Run the following command to  import XM content
                <<Endeca_App>>/control -> ./import_site.[bat|sh]<<File_Name>>

        Endeca Import Site command


        8. Promote Content - Run the following command to promote content from Authoring to Live. Promote content can be run the Workbench as well.
               <<Endeca_App>>/control -> ./promote_content.[bat|sh]

        Endeca Promote Content


        From Author

        Was this post resolve the solution you are looking for? are you looking for some other issues?Provide your comments

        Tuesday, April 26, 2016

        Endeca 11.2 : Workbench login credentials are not found or found invalid credential type for key

        Behaviour :- Following Exception/Error comes during Endeca Application initialize services or base... thumbnail 1 summary
      • Behaviour :-

        Following Exception/Error comes during Endeca Application initialize services or baseline/partial indexing process.


        [04.20.16 22:46:52] INFO: Provisioning site from prototype...
        [04.20.16 22:46:54] SEVERE: Workbench login credentials are not found or found invalid credential type for key <ifcr> and mapName <endecaToolsAndFrameworks>. Please update using <ENDECA_TOOLS_ROOT>/credential_store/bin/manage_credentials script with '--type password' as argument. Please refer to script usage.
        [04.20.16 22:46:54] SEVERE: Failed to initialize IFCRUtility.
        Occurred while executing line 3 of valid BeanShell script:
        [[
        1|
        2|    
        3|    IFCR.provisionSite();
        4|    CAS.importDimensionValueIdMappings("Discover-dimension-value-id-manager",
        5|         InitialSetup.getWorkingDir() + "/test_data/initial_dval_id_mappings.csv");
        6|    
        ]]

        [04.20.16 22:46:54] SEVERE: Caught an exception while invoking method 'run' on object 'InitialSetup'. Releasing locks.

        Caused by java.lang.reflect.InvocationTargetException
        sun.reflect.NativeMethodAccessorImpl invoke0 - null
        Caused by com.endeca.soleng.eac.toolkit.exception.AppControlException
        com.endeca.soleng.eac.toolkit.script.Script runBeanShellScript - Error executing valid BeanShell script.
        Caused by com.endeca.soleng.eac.toolkit.exception.AppConfigurationException
        com.endeca.soleng.eac.toolkit.component.IFCRComponent initIFCR - Failed to initialize IFCRUtility.


        Solution :-
        Above Exception comes as Endeca workbench password does not match with csfManager defined in WorkbenchConfig.xml. This happens if Endeca workbench password changed from change password screen and the same password not updated manually in CSF/OCS repository.


        Change Password using Endeca Workbench

        Find out command below to update Endeca CSF/OCS repository :-

        Run the command manage_credentials.bat add --user admin --key ifcr under <<ToolsAndFrameworks_PATH>>/<<version>>/credential_store.
         - Update the password with workbench password.
         - When it prompts to overwrite, type yes.


        From Author

        Was this post resolve the solution you are looking for? are you looking for some other issues?Provide your comments

        Monday, April 25, 2016

        Endeca 11.x : How to Add manipulator using CAS?

        Oracle Endeca 11.1 onwards forge less approach is used to index. Most of the data manipulation is do... thumbnail 1 summary
      • Oracle Endeca 11.1 onwards forge less approach is used to index. Most of the data manipulation is done at the source level. Java Manipulators required if some data manipulation required at CAS level.

        Find out the list of steps to create manipulator, deploy into CAS and use in the Endeca project.
        • Create a Manipulator similar to as per sample provided by Oracle endeca. Sample can be found under <installpath>\CAS\version\sample\cas-extensions\src\main\com\endeca\cas\extension\sample\manipulator\substring.
        • Create a jar file for classes created for the custom manipulator. Sample example gives ant script to create a jar file.
        • Stop CAS service. Create a new folder under “<Installpath>\CAS\version\lib\cas-server-plugins. Place newly created jar and dependent jar to support manipulators under the newly created directory.
        • Start CAS services. Newly created manipulator would be registered in CAS service. To verify, run below command to see if CAS has registered newly created manipulators or not :-
          • <Installpath>\CAS\11.1.0\bin>cas-cmd.sh listModules
        • Update “<Endeca App Path>\CRS\config\cas\last-mile-crawl.xml” file to inject newly created manipulator to manipulate the data. Sample file attached
        Manipulator Entry in Last-mile-crawl

        • Run the following command to update configuration in CAS 
          • <Installpath>\CAS\11.1.0\bin >./cas-cmd.sh updateCrawls -f <Endeca App Path>\CRS\config\cas\last-mile-crawl.xml
        • As per above example file, a “targetProperty” has created as “product.description123”. Mapping needs to be added for newly created property to Endeca indexing.
        • Add mapping in “<Endeca App path>\CRS\config\index_config\index-config.json”. find out the screenshot below
        New attribute added in index-config.json

        • Run the following command to update configuration in Endeca configuration repository(ECR).
          • <Endeca App Path>\CRS\control>./index_config_cmd.shset-config -f ..\config\index_config\index-config.json -o all
        • Run baseline indexing process using App/control/./baseline_update.sh. Changes should be visible in endeca_jspref after indexing is done. 
        Note :- Java Manipulators must be thread-safe. The CAS Server will invoke methods on this class from multiple-threads and potentially concurrently. 



        From Author

        Was this post resolve the solution you are looking for? are you looking for some other issues?Provide your comments

        Oracle Endeca 11.x - How to Add multiple record store using Endeca CAS ?

        Oracle Endeca has introduced forge less baseline indexing approach to index the data to MDEX engine... thumbnail 1 summary
      • Oracle Endeca has introduced forge less baseline indexing approach to index the data to MDEX engine. In this process, CAS generates the MDEX compatible data.

        Suppose there are requirements to add "content" from third-party and these data are different from products. In this case, another record store required for the new data source.

        Each Record under new record store should have "record.id" attribute. Endeca CAS uses record.id to make a switch join between multiple record store. The record would not be indexed if record.id is not available for any record.

        Find out steps below to add multiple Record store :-

        1. Create a new record store - Navigate to CAS/bin folder and run ./component-manager-cmd.sh create-component -t RecordStore -n CRS_content.

        2. Write data into record store using either CAS JAVA API or using XML file.

        3. Add Record store entry in last-mile-crawl.xml. This file can find out under <<Endeca App>>/config/cas folder.



        4. Run following command to update configuration in CAS 
         \CAS\11.1.0\bin >cas-cmd.bat updateCrawls -f <Endeca_App_Path>\config\cas\last-mile-crawl.xml

        5. Run baseline indexing process.


        From Author

        Was this post resolve the solution you are looking for? are you looking for some other issues?Provide your comments

        Sunday, April 24, 2016

        Endeca 11.2 : Unauthorized (401): Unauthorized access to workbench

        Behaviour :- Following Exception/Error comes during Endeca Application initialize services or baseline/partial indexing process. ... thumbnail 1 summary
      • Behaviour :-

        Following Exception/Error comes during Endeca Application initialize services or baseline/partial indexing process.


        <[04.20.16 22:55:48] SEVERE: Unauthorized (401): Unauthorized access to workbench. Please check your credentials in WorkbenchConfig.xml/OCS. If problem still persists, please contact your administrator.
        Occurred while executing line 3 of valid BeanShell script: 
        [[

        1|
        2|       
        3|    IFCR.provisionSite();
        4|    CAS.importDimensionValueIdMappings("Discover-dimension-value-id-manager", 
        5|         InitialSetup.getWorkingDir() + "/test_data/initial_dval_id_mappings.csv");
        6|      

        ]]

        [04.20.16 22:55:48] SEVERE: Caught an exception while invoking method 'run' on object 'InitialSetup'. Releasing locks.

        Caused by java.lang.reflect.InvocationTargetException
        sun.reflect.NativeMethodAccessorImpl invoke0 - null
        Caused by com.endeca.soleng.eac.toolkit.exception.AppControlException
        com.endeca.soleng.eac.toolkit.script.Script runBeanShellScript - Error executing valid BeanShell script.
        Caused by com.endeca.soleng.eac.toolkit.utility.IFCRUtility$HttpStatusException
        com.endeca.soleng.eac.toolkit.utility.IFCRUtility execute - Unauthorized (401): Unauthorized access to workbench. Please check your credentials in WorkbenchConfig.xml/OCS. If problem still persists, please contact your administrator.


        Solution :-
        Above Exception comes as Endeca workbench password does not match with csfManager defined in WorkbenchConfig.xml. This happens if Endeca ifcr password is different than workbench password.

        Find out command below to update Endeca CSF/OCS repository :-

        Run the command manage_credentials.bat add --user admin --key ifcr under <<ToolsAndFrameworks_PATH>>/<<version>>/credential_store.
         - Update the password with workbench password.
         - When it prompts to overwrite, type yes.


        From Author

        Was this post resolve the solution you are looking for? are you looking for some other issues?Provide your comments

        Text Widget