Home > PowerShell, SharePoint 2010, SharePoint 2010 Maintenance > Manage Deletion of Index Items in SharePoint 2010

Manage Deletion of Index Items in SharePoint 2010

In the environment I work SharePoint Search is considered a number #1 service for our customers, so refining the Search Service with quick returns, and limited errors is one of my top priority.

As many might already know SharePoint 2010 Enterprise Search can sometimes become a “pain” in configuring to meet high demands, you sometimes need a full time job with Search just to get it environmentally accepted.

I think i’ve had to completely blow away search 5+ times already in my production environment due to all types of reasons,  and then having to recreate and configure.

With SharePoint 2010 Enterprise Search you might encounter Crawls (Incremental and Full) that return all kinds of errors.  What i’ve noticed is when there is tons of errors in a Search the crawls tends to complete slower, and usually take more time to complete.

My job was to get my Incremental crawls to return results in 15 minutes or less, every 15 mins.  I was able to accomlish this.

One of the major issues that was slowing my crawl rate down was the amount of errors that were returning during each crawl (Incremental and Full).    My farm has over 800,000+ searchable items, which is not a whole lot compared to larger organizations, and was returning 2000+ errors, which is not bad at all.

However having those 2000+ errors was spiking my incremental crawls above the 15 minute return window.

Looking at the Crawl Error Log I noticed that majority if not all the errors I was receiving was due to SharePoint 2010 Crawl not cleaning up deleting items in the Index.

Errors such as (Item not found or Access Denied) were showing up.  By default SharePoint 2010 Enterprise Search cleans up (deletes items) from the Index only  if that item is returned in 30 separate crawls (Incremental and Full) every 30 days.  So even if that item no longer exists SharePoint will still count it towards the crawl results as an error, which in return slows the crawl return rate down.

I wanted to remove these (Item not found or Access Denied) errors but did not want to have to click through every single error in my crawl log and select ‘Remove the item from Index’.  That would take forever.

Luckily in SharePoint 2010 they have now have allowed us to Manage the deletion of index items in search with the use of PowerShell.

http://technet.microsoft.com/en-us/library/hh127009(v=office.14).aspx

Below is how I utilized the PowerShell cmdlt to redefine Enterprise Search deletion policy to fit in my environment.

=========================================================================================================

Deletion policy name String Default value
Delete policy for access denied or file not found ErrorDeleteCountAllowed

ErrorDeleteIntervalAllowed

30

720 Hours (30 days)

Delete policy for all other errors ErrorDeleteAllowed

ErrorIntervalAllowed

100

1440 Hours (60 days)

Delete unvisited policy DeleteUnvisitedMethod 1
Re-crawl policy for SharePoint content RecrawlErrorCount

RecrawlErrorInterval

10

360 Hours (15 days)

Deletion policy name Description
Delete policy for access denied or file not found When the crawler encounters an access denied or a file not found error, the item is deleted from the index if the error was encountered in more than ErrorDeleteCountAllowed consecutive crawls AND the duration since the first error is greater than ErrorDeleteIntervalAllowed hours. If both conditions are not met, the item is retried.
Delete policy for all other errors When the crawler encounters errors of types other than access denied or file not found, the item is deleted from the index if the error was encountered in more than ErrorDeleteAllowed consecutive crawls AND the duration since the first error is greater than ErrorIntervalAllowed hours.
Delete unvisited policy During a full crawl, the crawler executes a delete unvisited operation in which it deletes items that are in the crawl history that are not found in the current full crawl. You can use the DeleteUnvisitedMethod property to specify what items get deleted. You can specify the following three values:

  • 0, all unvisited items are deleted.
  • 1 (default), unvisited items that have the same host as the start address specified in the content source are retained, and unvisited items that were discovered by following links to other hosts are deleted.
  • 2, none of the unvisited items are deleted.
Re-crawl policy for SharePoint content This policy applies only to SharePoint content. If the crawler encounters errors when fetching changes from the SharePoint content database for RecrawlErrorCount consecutive crawls AND the duration since first error is RecrawlErrorInterval hours, the system re-crawls that content database.

$SearchApplication = Get-SPEnterpriseSearchServiceApplication -Identity “<SearchServiceApplicationName>”
$SearchApplication.SetProperty(“ErrorDeleteCountAllowed“, 1) #Set the DeleteCountAllow policy for 1 crawl
$SearchApplication.SetProperty(“ErrorDeleteIntervalAllowed”, 1) #Set the DeleteIntervalAllowed policy to 1 hr.
SearchApplication.SetProperty(“ErrorDeleteAllowed“, 1) #Set the DeleteAllowed policy for 1 crawl
$SearchApplication.SetProperty(“ErrorIntervalAllowed“, 1) #Set the IntervalAllowed policy to 1 hr.

After setting the deletion policy I fired off a new Incremental crawl.  During this crawl the items marked for deletion in my Index were deleted.  After the incremental crawl completed I fired off one more Incremental crawl, and this time my Incremental crawl completely within a few minutes (3 mins) with only 2 errors.

============================================================================================================

 

 

Advertisements
  1. Jens
    May 15, 2014 at 1:26 pm

    Hi James,
    I was trying to set all the parameters you (and Technet) discuss. All is fine except the parameter “ErrorDeleteAllowed”. Both properties (Get and Set) fail:

    + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : DotNetMethodException

    Do you have an idea what’s wrong with that property?
    Do I have a chance to list all properties of my Search Service Application?

    Thanks in advance, Jens

    • May 16, 2014 at 8:31 am

      Hi Jens,

      Do you mind posting the PS script you are trying to execute? Ideally the “ErrorDeleteAllowed”?

      Thanks

      v/r
      JShidell

      • Oldenburg, Jens
        May 16, 2014 at 9:39 am

        Hi JShidell,

        sure, please find my scripts attached.

        Thanks for your Support, Jens

        ________________________________________

      • May 16, 2014 at 11:44 am

        Hi Jens,

        I’m not seeing any attached scripts. Am I missing something?

        Thanks

        v/r
        JShidell

      • Jens
        May 22, 2014 at 12:06 pm

        Hi there, how can I send or upload an attachement? Or let me do it here:

        1. the script:
        if ( (Get-PSSnapin -Name ‘Microsoft.SharePoint.PowerShell’ -ErrorAction SilentlyContinue) -eq $null)
        {Add-PSSnapin -Name ‘Microsoft.SharePoint.PowerShell’}

        $essa = “TUBASearchServiceApplication”
        $SearchApplication = Get-SPEnterpriseSearchServiceApplication -Identity $essa
        write-host “get Parameters of SearchServiceApplication “”$essa”””

        write-host ” ”
        write-host “Delete policy for access denied or file not found”
        write-host “default: ErrorDeleteCountAllowed 30”
        write-host “set here: ” $SearchApplication.GetProperty(“ErrorDeleteCountAllowed”)
        write-host “default: ErrorDeleteIntervalAllowed 720 Hours (30 days)”
        write-host “set here: ” $SearchApplication.GetProperty(“ErrorDeleteIntervalAllowed”)

        write-host ” ”
        write-host “Delete policy for all other errors”
        write-host “default: ErrorDeleteAllowed=100”
        write-host “set here: ” $SearchApplication.GetProperty(“ErrorDeleteAllowed”)
        write-host “default: ErrorIntervalAllowed 1440 Hours (60 days)”
        write-host “set here: ” $SearchApplication.GetProperty(“ErrorIntervalAllowed”)

        write-host ” ”
        write-host “Delete unvisited policy”
        write-host “default: DeleteUnvisitedMethod 1”
        write-host “set here: ” $SearchApplication.GetProperty(“DeleteUnvisitedMethod”)

        write-host ” ”
        write-host “Re-crawl policy for SharePoint content”
        write-host “default: RecrawlErrorCount 10”
        write-host “set here: ” $SearchApplication.GetProperty(“RecrawlErrorCount”)
        write-host “default: RecrawlErrorInterval 360 Hours (15 days)”
        write-host “set here: ” $SearchApplication.GetProperty(“RecrawlErrorInterval”)

        2. the result:
        get Parameters of SearchServiceApplication “TUBASearchServiceApplication”

        Delete policy for access denied or file not found
        default: ErrorDeleteCountAllowed 30
        set here: 1
        default: ErrorDeleteIntervalAllowed 720 Hours (30 days)
        set here: 1

        Delete policy for all other errors
        default: ErrorDeleteAllowed=100
        Ausnahme beim Aufrufen von “GetProperty” mit 1 Argument(en): “Das System kann die angegebene Datei nicht finden. ”
        Bei C:\Scripts\Admin\SharePoint\SearchTuning\GetSearchParameters.ps1:18 Zeichen:55
        + write-host “set here: ” $SearchApplication.GetProperty <<<< ("ErrorDeleteAllowed")
        + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
        + FullyQualifiedErrorId : DotNetMethodException

        default: ErrorIntervalAllowed 1440 Hours (60 days)
        set here: 1

        Delete unvisited policy
        default: DeleteUnvisitedMethod 1
        set here: 0

        Re-crawl policy for SharePoint content
        default: RecrawlErrorCount 10
        set here: 1
        default: RecrawlErrorInterval 360 Hours (15 days)
        set here: 1

        should I translate the message?

        Kind regards, Jens

      • Jens
        June 5, 2014 at 2:01 pm

        Hi there, we have updated our farm with CU 04/2014. Now we have BuildVersion 14.0.7015.1000. But we’re still unable to Get or Set the property “ErrorDeleteAllowed”.
        Now we have more than 21,000 item marked for deletion. Does anyone have an idea?

        Thanks in advance, Jens

      • Jens
        June 5, 2014 at 2:18 pm

        Hi there, found the error.
        The name of that parameter by MS is “ErrorDeleteAllowed” (see http://technet.microsoft.com/en-us/library/hh127009%28v=office.14%29.aspx). But the correct property is “ErrorCountAllowed” (thanks Hal at http://blogs.technet.com/b/tothesharepoint/archive/2011/01/20/managing-crawl-deletion-policies-for-sharepoint-server-2010.aspx and Joseph at http://sprider.org/2014/03/05/manage-sharepoint-incremental-crawl-index-deletion/).
        Now I hope to reduce the count of items marked for deletion.

        Cheers, Jens

  2. Dan
    May 15, 2015 at 5:58 am

    Hey mate… this was really helpful.

    But just for someone else benefit these changes will “force” a full crawl (even if its called an incremental).

    So keep this in mind everyone! 🙂

    A friend just found out the hard way.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: