Sitecore Creating custom search index

Do you have an issue in your live system with indexing whereby a re-index of data can take long hours and the index frequently becomes corrupted forcing more regular re-indexes, a timely and intensive process?

Some items are used in a very distinct manner on the site to show logged in users the items that relate specifically to their account.  There are no filters for this information so the indexed data need only be relatively simple.

• Extract items into distinct indexes

As a precursor to this work, it will be necessary to implement the audit recommendation to review and restructure Sitecore content and media library to reduce the number of folders at the root level and placing the directories at a high level and distinct from the rest of the media library and content so that they can be indexed independently:

Ex:
If we have the following structure in the Sitecore tree and Custom Item A include up to 100000 images and  Custom Item B has 1000000 images then Sitecore rebuild indexes will take hours to complete indexing.

• Sitecore
• Content
• Home
• Item1
• Media Library
• Custom Item A
• 2015
• 2016
• 2017
• 12
• 11
• 10
• 9
• image1
• image 2
• image3
• image4
• image5
• image6
• image 100000
• Custom Item B
• image1
• imag2
• image3
• image 4
• image5
• image6
• 
  
• image100000

• /sitecore/media library/Documents/Custom Item A 
• /sitecore/media library/Documents/Custom Item B

The first step will then be to extract the custom items into distinct indexes.  This will allow them to be managed independently of the main website data (news, events, etc.) reducing the overhead on the servers and consequently the time required to re-index this data when needed.  As there will be less overall data held in each of these indexes it should, in turn, to reduce the number of corruptions that occur.

The new indexes should be configured with as few of the computed fields if needed.

This will require changes to the media library content structure in Sitecore, the index configurations and the code that accesses the index in various touch points across the site code to utilise the new indexes.

Steps to create a custom index configuration file:

• Create custom config file ex: Stecore.ContentSearch.Lucene.CstomIndexConfiguration.config
• Replace default index configuration tag <DefaultIndexConfiguration> with new custom name under <index configurations>

<CustomIndexConfiguration type="Sitecore.ContentSearch.LuceneProvider. LuceneIndexConfiguration,Sitecore.ContentSearch.LuceneProvider">

• Add included templates to "Include template" tag

ex:

 <include hint="list:IncludeTemplate">

            <CustomTemplateId>{0603F166-35B8-469F-8123-E8D87BEDC171}</CustomTemplateId>

          </include>

• Add computed fields used in the search

Steps to create a custom index file:

• Create new index file  ex: Sitecore.ContentSearch.Lucene.Index.Custom.config
• In this file add custom index name to 
•   <index id="[index name]" type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex, Sitecore.ContentSearch.LuceneProvider">
• Navigate to Configuration tab to update the path to use new configuration tag to use custom index tag added to index configuration file which was CustomIndexConfiguration in our example 
• Add root ID for the items included in this index by add parent ID <Root> tag 

At the end the file will look like the follwoing

• Exclude Custom  indexed items from the main index

To take the advantage of distinct indexing, we will need to make changes to the existing index configuration by excluding the template used for sub items if they have their separate templates or exclude these items by ID from the default index and also that requires the implementation of the audit review of the media library structure to make this feasible.

From Default Lucene configuration file Exclude templates which have custom index configuration

• Sitecore Search Indexing for custom indexed items

• After creating configuration files a new index will be added to indexing manager which can be used to index all items included in that index
• From the code, it's all about getting a new index to search items in the custom index.
var index = ContentSearchManager.GetIndex("custom_index_id") As index ID is the one added to    <index id="custom-news_index" type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex, Sitecore.ContentSearch.LuceneProvider">