When doing a farm backup, the backup kept failing on the Search Service Application.
[18.07.2013 12:13:21] FatalError: Object Search Service Application failed in event OnBackup. For more information, see the spbackup.log or sprestore.log file located in the backup directory.
FaultException: Management called failed with System.InvalidOperationException: 'Job failed: Have tried to perform backup/restore operation twice on all in-sync members in cluster SPe760d89b26bc.0, but none succeeded. Last failure message: Microsoft.Ceres.SearchCore.Seeding.SnapshotTransferException: Could not send chunk ms\%default\gen.0000000000000016.state: Localpath: [0-983> to target BackupDirectoryTarget[directory=\\server\share\spbr0004\I.0.0,validateTransfers=False]
at Microsoft.Ceres.SearchCore.Seeding.SnapshotSender.SendChunks(ISnapshot snapshot, ISeedSource source, ISeedTarget target, SeedStatus status, Func`1 checkAborted, Int32 targetFragIndex)
at Microsoft.Ceres.SearchCore.Seeding.SnapshotSender.FirstPhaseTransfer(ISeedSource source, ISeedTarget target, Action`1 updateProgress, Func`1 shouldAbort)
at Microsoft.Ceres.SearchCore.Seeding.BackupWorker.BackupWork.DoFirstPhaseWork()' at at Microsoft.Ceres.SearchCore.IndexController.BackupService.ThrowOnFailure(JobStatus status)
at Microsoft.Ceres.SearchCore.IndexController.BackupService.ProgressFirstPhase(String handle)
at Microsoft.Ceres.SearchCore.IndexController.IndexControllerManagementAgent.WrapCall[T](Func`2 original)
[18.07.2013 12:13:21] Debug:
Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
My topology
2 app servers in one vlan
Crawler, Admin, Analytics, Content
2 web servers in one vlan:
Query and Index
Sql server in one vlan.
Hosting backupshare
The Resolution:
After opening the SMB port 445 from all servers hosting search comonents to the backupshare, the backup completed successfully.
I had only opened that port on app1 where the backup was running from. Apparently, the backup send files directly from server where index is stored (web1 and 2) to the backupshare, instead of sending via the server executing the backup (app1).
Why isn't this described clearly in the documentation?? Two months and a Microsoft case later I resolved it myself...
GRRR!
