Endeca Basics : How to Add/Delete word from stemming dictionary??
Oracle Endeca provides a dictionary for stemming for all languages. This dictionary is part of MDEX engine installation. Dictionary can be found under \MDEX\<<vesion>>\conf\stemming directory.
Why do we need to delete stemming dictionary? One of the use case below:-
A user searching for search term "short" displays "shorts" results as well and vice versa.
There are two ways to Add/Delete word from stemming dictionary:-
1. Open specific language file under \MDEX\<<vesion>>\conf\stemming folder and edit. This is not the recommended way to do it.
2. Modify the default stemming dictionaries by running Dgidx with the --stemming-updates flag and specifying an XML file that contains the updates to the dictionary that you want to make. The update file can include both additions and deletions. Dgidx processes the file by adding and deleting entries in the static stemming dictionary file.
Adding entries to a static stemming dictionary
Each entry in a static stemming dictionary is defined by an <ADD_WORD_FORMS> element and its sub-element <WORD_FORMS_COLLECTION>. For example, the following entry adds apple and its plural form apples to the static stemming dictionary:
<!DOCTYPE WORD_FORMS_COLLECTION_UPDATES SYSTEM "word_forms_collection_updates.dtd">
<WORD_FORMS_COLLECTION_UPDATES>
<ADD_WORD_FORMS>
<WORD_FORMS_COLLECTION>
<WORD_FORMS>
<WORD_FORM>apple</WORD_FORM>
<WORD_FORM>apples</WORD_FORM>
</WORD_FORMS>
</WORD_FORMS_COLLECTION>
</ADD_WORD_FORMS>
</WORD_FORMS_COLLECTION_UPDATES>
Deleting entries from a static stemming dictionary
<!DOCTYPE WORD_FORMS_COLLECTION_UPDATES SYSTEM "word_forms_collection_updates.dtd">
<WORD_FORMS_COLLECTION_UPDATES>
<REMOVE_WORD_FORMS_KEYS>
<WORD_FORM>aalborg</WORD_FORM>
</REMOVE_WORD_FORMS_KEYS>
</WORD_FORMS_COLLECTION_UPDATES>
Combining deletes and adds
You can also specify a combination of deletes and adds. Deletes are processed before adds. For example,the following XML removes aachen and then adds it and several stemmed variants of it.
<!DOCTYPE WORD_FORMS_COLLECTION_UPDATES SYSTEM "word_forms_collection_updates.dtd">
<WORD_FORMS_COLLECTION_UPDATES>
<REMOVE_WORD_FORMS_KEYS>
<WORD_FORM>aachen</WORD_FORM>
</REMOVE_WORD_FORMS_KEYS>
<ADD_WORD_FORMS>
<WORD_FORMS_COLLECTION>
<WORD_FORMS>
<WORD_FORM>aachen</WORD_FORM>
<WORD_FORM>aachens</WORD_FORM>
<WORD_FORM>aachenes</WORD_FORM>
</WORD_FORMS>
</WORD_FORMS_COLLECTION>
</ADD_WORD_FORMS>
</WORD_FORMS_COLLECTION_UPDATES>
Processing the update file
To process the stemming update file, run Dgidx with the --stemming-updates flag and specify the XML file that contains the stemming updates.
For example:
dgidx --stemming-updates myAppStemmingChanges.en.xml
Discussion (8)
Technical questions and architecture discussionsJoin the Discussion