Most Recent

Sunday, March 20, 2016

Endeca Basics : does Stemming change relevance ranking score?

Stemming feature allows expanding the search results from singular to plural and vice versa. Suppose, search term is "shoe", this... thumbnail 1 summary
    • Stemming feature allows expanding the search results from singular to plural and vice versa.

     Suppose, search term is "shoe", this gives the result for "shoe" and "shoes" both.  Find out example below:-

    Record -A 
    DGraph.WhyDidItMatch: product.keywords: shoe (Exact,Substring/Phrase)
    DGraph.WhyDidItMatch: product.longDescription: Perfect for a day at the beach or around town. Though simple in design, this is a great everyday shoe with cork heel for extra walking comfort during daily wear. (Substring/Phrase)

    Record- B
    DGraph.WhyDidItMatch: product.keywords: shoes (Stemming)
    DGraph.WhyDidItMatch: product.longDescription: With a suit or with jeans, these shoes will be a hit in either case. Made of supple leather and with a rubber sole. (Stemming)

     Both above records show, why did it match. Record-A got match based on exact match and Record-B based on stemming.

     Relevance ranking of Endeca provides one of the modules "Interp", this module ranks the exact/phrase matched product always on top than stemming matched.

    Click here to get more topics related to Endeca basic concepts.

    Saturday, March 19, 2016

    Endeca Basics : How to Add/Delete word from stemming dictionary??

    Oracle Endeca provides a dictionary for stemming for all languages. This dictionary is part of MDEX engine installation. Dictionary can be ... thumbnail 1 summary
    • Oracle Endeca provides a dictionary for stemming for all languages. This dictionary is part of MDEX engine installation. Dictionary can be found under \MDEX\<<vesion>>\conf\stemming directory.

    Why do we need to delete stemming dictionary? One of the use case below:-

    A user searching for search term "short" displays "shorts" results as well and vice versa.

    There are two ways to Add/Delete word from stemming dictionary:-

    1. Open specific language file  under \MDEX\<<vesion>>\conf\stemming folder and edit. This is not the recommended way to do it.

    2. Modify the default stemming dictionaries by running Dgidx with the --stemming-updates flag and specifying an XML file that contains the updates to the dictionary that you want to make. The update file can include both additions and deletions. Dgidx processes the file by adding and deleting entries in the static stemming dictionary file.

    Adding entries to a static stemming dictionary

    Each entry in a static stemming dictionary is defined by an <ADD_WORD_FORMS> element and its sub-element <WORD_FORMS_COLLECTION>. For example, the following entry adds apple and its plural form apples to the static stemming dictionary:
    <!DOCTYPE WORD_FORMS_COLLECTION_UPDATES SYSTEM "word_forms_collection_updates.dtd">
    <WORD_FORMS_COLLECTION_UPDATES>
    <ADD_WORD_FORMS>
    <WORD_FORMS_COLLECTION>
    <WORD_FORMS>
    <WORD_FORM>apple</WORD_FORM>
    <WORD_FORM>apples</WORD_FORM>
    </WORD_FORMS>
    </WORD_FORMS_COLLECTION>
    </ADD_WORD_FORMS>
    </WORD_FORMS_COLLECTION_UPDATES>


    Deleting entries from a static stemming dictionary

    <!DOCTYPE WORD_FORMS_COLLECTION_UPDATES SYSTEM "word_forms_collection_updates.dtd">
    <WORD_FORMS_COLLECTION_UPDATES>
    <REMOVE_WORD_FORMS_KEYS>
    <WORD_FORM>aalborg</WORD_FORM>
    </REMOVE_WORD_FORMS_KEYS>
    </WORD_FORMS_COLLECTION_UPDATES>


    Combining deletes and adds

    You can also specify a combination of deletes and adds. Deletes are processed before adds. For example,the following XML removes aachen and then adds it and several stemmed variants of it.
    <!DOCTYPE WORD_FORMS_COLLECTION_UPDATES SYSTEM "word_forms_collection_updates.dtd">
    <WORD_FORMS_COLLECTION_UPDATES>
    <REMOVE_WORD_FORMS_KEYS>
    <WORD_FORM>aachen</WORD_FORM>
    </REMOVE_WORD_FORMS_KEYS>
    <ADD_WORD_FORMS>
    <WORD_FORMS_COLLECTION>
    <WORD_FORMS>
    <WORD_FORM>aachen</WORD_FORM>
    <WORD_FORM>aachens</WORD_FORM>
    <WORD_FORM>aachenes</WORD_FORM>
    </WORD_FORMS>
    </WORD_FORMS_COLLECTION>
    </ADD_WORD_FORMS>
    </WORD_FORMS_COLLECTION_UPDATES>


    Processing the update file

    To process the stemming update file, run Dgidx with the --stemming-updates flag and specify the XML file that contains the stemming updates.
    For example:
    dgidx --stemming-updates myAppStemmingChanges.en.xml

    Endeca Basics : How to Disable Endeca Rule Manager trigger for Stemming?

    Stemming enables keyword search results to provide results for both singular and plural words. Stemming and spelling correction both trigg... thumbnail 1 summary
    • Stemming enables keyword search results to provide results for both singular and plural words.

    Stemming and spelling correction both triggers Endeca Rules in Experience manager.


     Stemming can not be disabled for Rule triggering. However, if stemming needs to be disabled for few search term, remove those words from Stemming dictionary.

     Stemming Dictionary can be found under \MDEX\<<version>>\conf\stemming

    Endeca Basics : What is Stemming?

    The Stemming feature expands search results to include root words and variants of root words.Stemming enables words with a common root (such... thumbnail 1 summary
    • The Stemming feature expands search results to include root words and variants of root words.Stemming enables words with a common root (such as the singular and plural forms of nouns) to be used interchangeably by searches. 

     Stemming enables the MDEX engine to consider alternate forms of individual words as equivalent for the purpose of search query matching.

     For example
    search results for the word "shirt" will include the plural form "shirts", while a search for shirts will include the singular form shirt.

     Enable Stemming

     Stemming can be enabled as follow :-
    1. Forge based system -  Open CRS.stemming.xml under <<Endeca_App>>/config/pipeline folder
    2. CAS based system - Open CRS.stemming.xml under <<Endeca_App>>/config/mdex folder

     add the following snippet to enable for English:

     <STEMMING>
      <STEM_EN_US ENABLE="TRUE" />

    </STEMMING>

     Note:-  Endeca OOTB provides the stemming dictionary for many languages.


    Text Widget