Thursday, February 23, 2017

Oracle Endeca ITL and Expereince Manager High Availability Architecture

What are High Availability and Failover   High availability  as the ability for the system to continue functioning after the failure of... thumbnail 1 summary
  • What are High Availability and Failover

    •  High availability as the ability for the system to continue functioning after the failure of one or more of the servers.
    • Failover as one possible implementation of high availability. Failover is defined as the ability for client connections to migrate from one server to another in the event of server failure so client applications can continue to operate.

    How to Setup ITL and XM Failover

    Two Information Transformation Layer (ITL) servers which host Tools and Framework required achieving high availability and another for Disaster recovery (DR) if required. 
    Only one ITL can be in active mode and remaining failover/backup should be passive mode . As per architecture, specific, in the diagram below, folders should be shared between primary and backup ITL servers using shared drive. DR generally hosted in the different data center so file system replication or other technique required for DR environment. The Shared drive can be setup during product and app installation.
    Following points needs to be taken care of during installation
    • Installation path should be same in all environments to avoid any conflicts and issues.
    • Endeca Application Initialize services script required to run from all the ITL servers for the first time.
    • Make sure to copy same Dgraph cluster XML with same naming convention to avoid duplicate Dgraph instance
    Find out block diagram below for HA understanding and setup

    What needs to be shared between active and passive ITL servers:-

    1.  Tools and Framework state folder
    2. Endeca application Data folder
    3. CAS workspace shared folder

    What happens if failover happens during mid-process 

    • Failover ITL would have to begin the process again if primary ITL Dgidx failed from Dgidx process.
    • Failover ITL would have to begin the entire process again if primary ITL forge failed.
    • Failover ITL would have to begin the entire process again if primary CAS indexing  failed.
    Note :  ATG properties need to be changed to redirect indexing to failover ITL in the event of primary ITL fault, but best practice would probably be to restart the indexing process again.
    For more Endeca related information visit:  http://www.ajayagrwal.com/

    Sunday, March 27, 2016

    Forge Error : The requested file copy cannot be completed since it requires writing multiple files to the same location.

    Behaviour :-  Following Errors comes during baseline process. CRS.fetch_config_to_input_for_forge_Forge.log Mar 22, 2016 12:16... thumbnail 1 summary
  • Behaviour :- 

    Following Errors comes during baseline process.
    CRS.fetch_config_to_input_for_forge_Forge.log
    Mar 22, 2016 12:16:35 PM com.endeca.esf.delegate.task.TaskLogger info
    INFO: [Central Server: 10.218.4.6] List files for path: /data/atg/endeca/apps/CRS/
    ./config/pipeline/\* recursively.
    Mar 22, 2016 12:16:35 PM com.endeca.esf.delegate.task.TaskLogger info
    INFO: [Central Server: 10.218.4.6] List files for path /data/atg/endeca/apps/CRS
    /./config/pipeline/\* recursive  returned 144 items.
    Mar 22, 2016 12:16:35 PM com.endeca.esf.delegate.task.TaskLogger severe
    SEVERE: [Central Server: 10.218.4.6] The copy failed:
    The requested file copy cannot be completed since it requires writing multiple files to the same location.
    The files /data/atg/endeca/apps/CRS/config/pipeline/CRS.languages.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.merch_rule_group_default.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.profiles.xml /data/atg/endeca/apps/CRS/config/
    pipeline/partial_pipeline.lyt /data/atg/endeca/apps/CRS/config/pipeline/CRS.search_chars.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.dimension_groups.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.thesaurus.xml /data/atg/endeca/
    apps/CRS/config/pipeline/CRS.record_filter.xml /data/atg/endeca/apps/CRS/config/pipeline/pipeline.lyt /data/atg/endeca/apps/CRS/config/pipeline/CRS.render_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.dimension_refs.xml /data/
    atg/endeca/apps/CRS/config/pipeline/configurationGenerator.epx /data/atg/endeca/apps/CRS/config/pipeline/crawl_profile_1_url_list.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.record_spec.xml /data/atg/endeca/apps/CRS/config/pipeline
    /CRS.dimsearch_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.key_props.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.crawl_profiles.xml /data/atg/endeca/apps/CRS/config/pipeline/partial_pipeline.epx /data/atg/endeca/apps/L
    LM/config/pipeline/CRS.dval_refs.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.rollups.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.merch_rules.xml /data/atg/endeca/apps/CRS/config/pipeline/crawl_profile_1.xml /data/atg/endeca/a
    pps/CRS/config/pipeline/CRS.dimsearch_index.xml /data/atg/endeca/apps/CRS/config
    /pipeline/CRS.prop_refs.xml /data/atg/endeca/apps/CRS/config/pipeline/crawl_profile_1_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.recsearch_inde
    xes.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.phrases.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.refinement_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.analytics_config.xml /data/atg/endeca
    /apps/CRS/config/pip
    eline/CRS.merch_rule_group_default_redirects.xml /data/atg/endeca/apps/CRS/config
    /pipeline/CRS.merchstyles.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.record_sort_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.crawler_glo
    bal_config.xml /data/atg/endeca/apps/CRS/config/pipeline/pipeline.epx /data/atg/endeca/apps/CRS/config/pipeline/CRS.stemming.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.relrank_strategies.xml /data/atg/endeca/apps
    /CRS/config/pipeli
    ne/CRS.merchzones.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.esp /data/atg/endeca/apps/CRS/config/pipeline/CRS.recsearch_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.derived_props.xml /data/atg/endeca/
    apps/CRS/config/p
    ipeline/CRS.crawler_defaults.properties /data/atg/endeca/apps/CRS/config/pipeline/
    CRS.precedence_rules.xml /data/atg/endeca/apps/CRS/config/pipeline/dimensions.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.stop_words.xml /data/atg/en
    deca/apps/CRS/config/pipeline/externaldimensions.xml /data/atg/endeca/apps/
    CRS/config/pipeline/CRS.dval_ranks.xml
    would all be copied to /data/atg/endeca/apps/CRS/data/processing
    Mar 22, 2016 12:16:35 PM com.endeca.esf.delegate.task.TaskLogger severe
    SEVERE: [Central Server: 10.218.4.6] com.endeca.esf.shared.EsfException:
    The requested file copy cannot be completed since it requires writing multiple files to the same location.
    The files /data/atg/endeca/apps/CRS/config/pipeline/CRS.languages.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.merch_rule_group_default.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.profiles.xml /data/atg/endeca/apps/CRS/config/
    pipeline/partial_pipeline.lyt /data/atg/endeca/apps/CRS/config/pipeline/CRS.search_chars.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.dimension_groups.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.thesaurus.xml /data/atg/endeca/
    apps/CRS/config/pipeline/CRS.record_filter.xml /data/atg/endeca/apps/CRS/config/pipeline/pipeline.lyt /data/atg/endeca/apps/CRS/config/pipeline/CRS.render_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.dimension_refs.xml /data/
    atg/endeca/apps/CRS/config/pipeline/configurationGenerator.epx /data/atg/endeca/apps/CRS/config/pipeline/crawl_profile_1_url_list.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.record_spec.xml /data/atg/endeca/apps/CRS/config/pipeline
    /CRS.dimsearch_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.key_props.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.crawl_profiles.xml /data/atg/endeca/apps/CRS/config/pipeline/partial_pipeline.epx /data/atg/endeca/apps/L
    LM/config/pipeline/CRS.dval_refs.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.rollups.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.merch_rules.xml /data/atg/endeca/apps/CRS/config/pipeline/crawl_profile_1.xml /data/atg/endeca/a
    pps/CRS/config/pipeline/CRS.dimsearch_index.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.prop_refs.xml /data/atg/endeca/apps/CRS/config/pipeline/crawl_profile_1_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.recsearch_inde
    xes.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.phrases.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.refinement_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.analytics_config.xml /data/atg/endeca/apps/CRS/config/pip
    eline/CRS.merch_rule_group_default_redirects.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.merchstyles.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.record_sort_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.crawler_glo
    bal_config.xml /data/atg/endeca/apps/CRS/config/pipeline/pipeline.epx /data/atg/endeca/apps/CRS/config/pipeline/CRS.stemming.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.relrank_strategies.xml /data/atg/endeca/apps/CRS/config/pipeli
    ne/CRS.merchzones.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.esp /data/atg/endeca/apps/CRS/config/pipeline/CRS.recsearch_config.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.derived_props.xml /data/atg/endeca/apps/CRS/config/p
    ipeline/CRS.crawler_defaults.properties /data/atg/endeca/apps/CRS/config/pipeline/CRS.precedence_rules.xml /data/atg/endeca/apps/CRS/config/pipeline/dimensions.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.stop_words.xml /data/atg/en
    deca/apps/CRS/config/pipeline/externaldimensions.xml /data/atg/endeca/apps/CRS/config/pipeline/CRS.dval_ranks.xml
    would all be copied to /data/atg/endeca/apps/CRS/data/processing
            at com.endeca.esf.shared.util.file.FileCopyDestinationResolver.calculateDestinations
    (FileCopyDestinationResolver.java:77)
            at com.endeca.esf.delegate.task.FileCopyTask.doTaskWork(FileCopyTask.java:70)
            at com.endeca.esf.delegate.task.AbstractTask$1.run(AbstractTask.java:90)



    Solution:-

    Above exception comes during Baseline->Forge Process. Sometimes <<Endeca_App>>/data/processing folder gets deleted or file name created as "processing" instead of the folder.

    Following steps would resolve forge exception:-

    1. Delete file "processing" under /data folder if any.
    2. Create "processing" folder /data folder.
    3. Run Baseline process once again.







    Saturday, March 26, 2016

    SocketTimeoutException: Time out error during Read Records From CAS RecordStore

    Behaviour :- Following errors comes during baseline update <SEVERE: Caught an exception while invoking method 'run' on object ... thumbnail 1 summary
  • Behaviour :-

    Following errors comes during baseline update

    <SEVERE: Caught an exception while invoking method 'run' on object 'BaselineUpdate'. Releasing locks.
    java.lang.reflect.InvocationTargetException
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at com.endeca.soleng.eac.toolkit.Controller.invokeRequestedMethod(Controller.java:931)
            at com.endeca.soleng.eac.toolkit.Controller.execute(Controller.java:269)
            at com.endeca.soleng.eac.toolkit.Controller.main(Controller.java:137)
    Caused by: com.endeca.soleng.eac.toolkit.exception.AppControlException: Error executing valid BeanShell script.
            at com.endeca.soleng.eac.toolkit.script.Script.runBeanShellScript(Script.java:179)
            at com.endeca.soleng.eac.toolkit.script.Script.run(Script.java:127)
            ... 7 more
    Caused by: com.endeca.soleng.eac.toolkit.exception.CasCommunicationException: Error starting baseline crawl 'stg1-last-mile-crawl'.
            at com.endeca.eac.toolkit.component.cas.ContentAcquisitionServerComponent.startBaselineCasCraw
    l(ContentAcquisitionServerComponent.java:400)
            at com.endeca.eac.toolkit.component.cas.ContentAcquisitionServerComponent.runBaselineCasCrawl
    (ContentAcquisitionServerComponent.java:308)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at bsh.Reflect.invokeMethod(Unknown Source)
            at bsh.BSHMethodInvocation.eval(Unknown Source)
            at bsh.BSHPrimaryExpression.eval(Unknown Source)
            at bsh.BSHPrimaryExpression.eval(Unknown Source)
            at bsh.BSHBlock.evalBlock(Unknown Source)
            at bsh.BSHBlock.eval(Unknown Source)
            at bsh.BSHBlock.eval(Unknown Source)
            at bsh.BSHIfStatement.eval(Unknown Source)
            at bsh.Interpreter.eval(Unknown Source)
            at bsh.Interpreter.eval(Unknown Source)
            at bsh.Interpreter.eval(Unknown Source)
            at com.endeca.soleng.eac.toolkit.script.Script.runBeanShellScript(Script.java:165)
            ... 8 more
    Caused by: java.net.SocketTimeoutException: Read timed out
            at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
            at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:154)
            at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
            at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
            at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
            at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
            at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
            at org.apache.axis.client.Call.invoke(Call.java:2767)
            at org.apache.axis.client.Call.invoke(Call.java:2443)
            at org.apache.axis.client.Call.invoke(Call.java:2366)
            at org.apache.axis.client.Call.invoke(Call.java:1812)
            at com.endeca.cas.wsdl.CasCrawlerServiceSoapBindingStub.startCrawl
    (CasCrawlerServiceSoapBindingStub.java:1202)




    Solution :- 

    By default the "httpSocketTimeout" is set to 60 secs. Increasing this resolves the read timeout error.

    Update the "httpSocketTimeout" value in DataIngest.xml in the deployed app. The value is in milliseconds and Restart CAS before running baseline.

    Saturday, March 19, 2016

    Endeca 11.x - EacCommunicationException Error After Running Initialize Services

    The following error is being thrown for any new application trying to run initialize_services on: [01.22.16 16:33:09] INFO: Checking definit... thumbnail 1 summary
  • The following error is being thrown for any new application trying to run initialize_services on:

    [01.22.16 16:33:09] INFO: Checking definition from AppConfig.xml against existing EAC provisioning.
    [01.22.16 16:33:10] INFO: Setting definition for application '<app_name>'.
    [01.22.16 16:33:10] SEVERE: Caught an exception while checking provisioning.

    Caused by com.endeca.soleng.eac.toolkit.exception.EacCommunicationException
    com.endeca.soleng.eac.toolkit.application.Application setDefinition - Caught exception while defining application '<app_name>'.
    Caused by com.endeca.eac.client.EACFault
    sun.reflect.NativeConstructorAccessorImpl newInstance0 - null

    SEVERE: Caught an exception while checking provisioning.
    com.endeca.soleng.eac.toolkit.exception.EacCommunicationException: Caught exception while defining application '<app_name>'.
           at com.endeca.soleng.eac.toolkit.application.Application.setDefinition(Application.java:141)
           at com.endeca.soleng.eac.toolkit.Controller.provision(Controller.java:667)
           at com.endeca.soleng.eac.toolkit.Controller.execute(Controller.java:235)
           at com.endeca.soleng.eac.toolkit.Controller.main(Controller.java:134)
    Caused by: An error occurred while trying to create the application: Error while running a hibernate transaction: could not insert: [com.endeca.esf.centralserver.service.ApplicationType]
           at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)


    Solution :-

    This issue can be caused by a db.lck file in the %ENDECA_CONF%\state\eacstore directory.

     Stop all services, delete db.lck file, restart the services and then run initialize_services again.

    Thursday, March 10, 2016

    Oracle Endeca 11.1 : Migrate State file to Dimension Value ID Mapping

    Migrate State file to Dimension Value ID Mapping From Forge-Based to CAS-Based Applications : - CAS based MDEX writable output concept i... thumbnail 1 summary

  • Migrate State file to Dimension Value ID Mapping From Forge-Based to CAS-Based Applications : -

    CAS based MDEX writable output concept is introduced from 11.1 onward. There are no way to migrate Endeca auto generated state file (Dimensions) and FCM(Forge Configuration Manager) file to dimension value id manager format in 11.1

    As of the release of Oracle Commerce 11.2, Forge-to-CAS Conversion Utility available and can be used to convert to dimension-value-id-manager CSV format..

    This utility transforms the current dval state information, data, dimensions and attribute configuration of a Forge pipeline-based application into a format that can be used by CAS.

    Further details and usage instructions can be found in below oracle documents.

    http://docs.oracle.com/cd/E67228_01/CAS.112/pdf/CASDevGuide.pdf#page=145


    It's your Turn
    Was this post help you to resolve the issue that you are looking for? Any other issues that you might have and need a solution?

    Click here for more ATG-Endeca Integration related issue.

    Provide your feedback in the comment section below.

    Wednesday, March 9, 2016

    Dynamic Ranking Not Applied in Guided Navigation in 11.1

    Behavior -   Selecting sorting option as frequency(Dynamic ranking) under refinement menu not sorting refinement as intended. Soluti... thumbnail 1 summary
  • Behavior -  

    Selecting sorting option as frequency(Dynamic ranking) under refinement menu not sorting refinement as intended.


    Solutions :-

    There are two options available as per my knowledge

    1. Add all dimensions in <<Endeca_App_Path>>/config/mdex/appname.refinement_config.xml and run base line. This would start showing the proper results in front-end.

    2, Use patch provided by Oracle Endeca to fix this.

    Patch 20961687

    Text Widget