Farm backup fails on search - Object Crawl-1 (D: on SERVER) failed in event OnBackupComplete

I finally resolved an issue I've had with two environments.
When doing a farm backup, the backup kept failing on the Search Service Application.

[18.07.2013 12:13:21] FatalError: Object Search Service Application failed in event OnBackup. For more information, see the spbackup.log or sprestore.log file located in the backup directory.
    FaultException: Management called failed with System.InvalidOperationException: 'Job failed: Have tried to perform backup/restore operation twice on all in-sync members in cluster SPe760d89b26bc.0, but none succeeded. Last failure message: Microsoft.Ceres.SearchCore.Seeding.SnapshotTransferException: Could not send chunk ms\%default\gen.0000000000000016.state: Localpath: [0-983> to target BackupDirectoryTarget[directory=\\server\share\spbr0004\I.0.0,validateTransfers=False]
   at Microsoft.Ceres.SearchCore.Seeding.SnapshotSender.SendChunks(ISnapshot snapshot, ISeedSource source, ISeedTarget target, SeedStatus status, Func`1 checkAborted, Int32 targetFragIndex)
   at Microsoft.Ceres.SearchCore.Seeding.SnapshotSender.FirstPhaseTransfer(ISeedSource source, ISeedTarget target, Action`1 updateProgress, Func`1 shouldAbort)
   at Microsoft.Ceres.SearchCore.Seeding.BackupWorker.BackupWork.DoFirstPhaseWork()' at    at Microsoft.Ceres.SearchCore.IndexController.BackupService.ThrowOnFailure(JobStatus status)
   at Microsoft.Ceres.SearchCore.IndexController.BackupService.ProgressFirstPhase(String handle)
   at Microsoft.Ceres.SearchCore.IndexController.IndexControllerManagementAgent.WrapCall[T](Func`2 original)
[18.07.2013 12:13:21] Debug:
Server stack trace:
   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

My topology

2 app servers in one vlan
Crawler, Admin,  Analytics, Content

2 web servers in one vlan:
Query and Index

Sql server in one vlan.
Hosting backupshare

The Resolution:

After opening the SMB port 445 from all servers hosting search comonents to the backupshare, the backup completed successfully.

I had only opened that port on app1 where the backup was running from. Apparently, the backup send files directly from server where index is stored (web1 and 2) to the backupshare, instead of sending via the server executing the backup (app1).


Why isn't this described clearly in the documentation?? Two months and a Microsoft case later I resolved it myself...
GRRR!

Failed to run flow Microsoft.CustomDictionaryDeployment

I've been stuck on a petty search-related issue in SharePoint 2013 for a while now, and FINALLY solved the shit!

You see the following failed timer jobs in Central Admin:

 

Eventlog is spammed every 10 minutes:


The Execute method of job definition Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition (ID 5a7f11db-d042-4051-b52c-dd28b4bb5dca) threw an exception. More information is included below.

Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: 521b299c-baf4-d067-82a7-d6b3f1c19aef.

ULS logs shows the following error:

Information      Custom dictionary deployment timer job started. SearchComponent: OWSTIMER.EXE, SearchServiceApplication: {f3303f37-5871-470c-8dd1-6a22d31998f0}.    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Medium      Found settings: Dictionaries last checked for changes 01/01/1754 00:00:00    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Medium      WcfSendRequest: RemoteAddress: 'http://server:32843/2522386c09e545599461de90989c02b9/MetadataWebService.svc' Channel: 'Microsoft.SharePoint.Taxonomy.IMetadataWebServiceApplication' Action: 'http://schemas.microsoft.com/sharepoint/taxonomy/soap/IDataAccessReadOnly/GetChangedTermSets' MessageId: 'urn:uuid:b1c14f44-c081-4a11-89f1-ed9fcb379b11'    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Information    Dictionary deployment started. SearchComponent: OWSTIMER.EXE, CorrelationId: e77d30dd-a7cd-4708-8743-81011e7cf57b, SearchServiceApplication: {f3303f37-5871-470c-8dd1-6a22d31998f0}, DictionaryName: Microsoft.UserDictionaries.EntityExtraction.Companies.Inclusions, TermSetId: c866ca65-f095-4a16-9249-028d500f7703, FlowId: {34e1f2d1-b957-4a7b-9ba9-e80cc73c926e}, FlowName: Microsoft.CustomDictionaryDeployment    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Warning     Dictionary deployment failed. SearchComponent: OWSTIMER.EXE, CorrelationId: e77d30dd-a7cd-4708-8743-81011e7cf57b, SearchServiceApplication: {f3303f37-5871-470c-8dd1-6a22d31998f0}, DictionaryName: Microsoft.UserDictionaries.EntityExtraction.Companies.Inclusions, Component:FlowExecution, FailureMessage: Deployment (flow:Microsoft.CustomDictionaryDeployment) for custom dictionary c866ca65-f095-4a16-9249-028d500f7703 did not complete successfully. Flow execution state was: Failed. Failure: Evaluation failure. An evaluation thread threw an exception. (Exception type: Microsoft.Ceres.Evaluation.DataModel.EvaluationException. Exception message: Evaluation failed in operator RepoWriter of type DictionaryRepositoryWriter), Cause: Microsoft.Ceres.ContentEngine.Services.ContentIntegrationEngine.FailC...    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Warning     ...auseException: Evaluation failed in operator RepoWriter of type DictionaryRepositoryWriter.    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Warning     Custom dictionary deployment timer job failed. SearchComponent: OWSTIMER.EXE, SearchServiceApplication: {f3303f37-5871-470c-8dd1-6a22d31998f0}, FlowName: Microsoft.CustomDictionaryDeployment, CorrelationId: {6a1c299c-9afa-d067-82a7-d7a23f3bb19d}, Failure: System.InvalidOperationException: Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: e77d30dd-a7cd-4708-8743-81011e7cf57b.     at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.DoExecute(CtsConnection connection)     at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.ExecuteTimerJob().    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Critical    The Execute method of job definition Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition (ID 5a7f11db-d042-4051-b52c-dd28b4bb5dca) threw an exception. More information is included below.  Failed to run flow Microsoft.CustomDictionaryDeployment. Correlation Id: 6a1c299c-9afa-d067-82a7-d7a23f3bb19d.    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Unexpected    Exception stack trace:    at Microsoft.Office.Server.Search.Administration.CustomDictionaryDeploymentJobDefinition.ExecuteTimerJob()     at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Medium      Leaving Monitored Scope (Timer Job Search Custom Dictionaries Update for Search Application f3303f37-5871-470c-8dd1-6a22d31998f0.). Execution Time=2674.81076505534    6a1c299c-9afa-d067-82a7-d7a23f3bb19d

Solution:
I started the SharePoint Search Host Controller Service on every server in the farm, and suddenly the timer job finished successfully.

My topology:
4 app servers
3 web servers


The SSHC service was not started on web1 and web2, however there are no other search components on those servers... I guess there must be some old info in the configuration db which is looking for info on web1 and 2...
The SSHC service should only be enabled on servers with search components activated as far as I know, so this is really not a good solution.


Also: Make sure you have given your search service account read permission to the Managed Metadata Service Application. Or else you could get errors aswell. See following post:
http://get-sharepoint.com/tag/error/

My Site creation stuck at “We’re almost ready!” - SP2013

After changing the MySite host to a dedicated web app, the user's MySite did no longer get provisioned after visiting the users page.

It turns out that the managed path "personal" was not created automatically, as it is in SP2010.
Here is the solution:

• Create a new web app
• Create a site collection at the root using the My Site Host template
• Assuming you want users’ MySites to be created at http://mySiteHost/personal/[John_Smith], create a Managed Path for the web app. The path should be personal, and the type should be Wildcard Inclusion.
• In User Profile Service Application, set My Site Host location to http://mySiteHost, and Personal Site Location to personal.
• Enable Self-Service Site Creation for the web app

Application pools recycle when memory limits are exceeded - SP2010

Each time I configure a new SharePoint farm, I have to wrestle with the Health Analyzer rules.
A good one is:
SharePoint 2010 – Application pools recycle when memory limits are exceeded.

If you check the provided technet article in the rule, you'll get the following (quite misleading) solution.

1. In the Internet Information Services management console, in the Connections pane, expand the tree view, and then click Application Pools.
   
2. In the Application Pools list, right-click the application pool on which you want to disable the memory limits, and then click Recycling.
   
3. In the Edit Application Pool Recycling Settings dialog box, in the Memory Based Maximums section, clear the Virtual memory usage (in KB) and Private memory usage (in KB) check boxes, and then click Next.
    

I discovered this rule on a quite fresh install, hence no changes done to Virtual memory usage in IIS.
Turns out that the reason for this issue was that I had stopped the service "SharePoint Foundation Web Application" on my app servers.
This lead to the rule not finding all application pools as it was expecting to find.

Since you can't set rules for specific servers (not even in sp2013), the easiset solution is to disable the rule.



Next simplest solution is to add the missing application pool to the servers.
The following powershell script looks for missing application pools:

$contentService = [Microsoft.SharePoint.Administration.SPWebService]::ContentService
$contentService.ApplicationPools | fl Name