This article provides instruction for troubleshooting virtual machine backup failures in Acronis Backup Advanced virtualization solutions.
Follow these steps to troubleshoot virtual machine backup failures:
- Get familiar with technology of virtual machine backup in the About part.
- Check for known issues in the Known issues part.
- Follow to Troubleshooting part to troubleshoot and resolve the issue.
Major steps of the virtual machine backup in Acronis Backup Advanced (Acronis Backup & Recovery) are the following.
- Backup virtual machines at hypervisor level to off-load the guest OS.
- Quiescing and VSS should be working properly for each VM with transactional applications like databases and email servers.
- Do not overload the backup agent. See the system requirements for 5-10 simultaneous VM backup with Agent for VMware vSphere ESX(i) (Virtual Appliance). Use Agent for ESX(i) (Windows) to off-load the ESX(i) host. In Acronis Backup 12.5, you can also limit the total number of simultaneously backed-up virtual machines.
- Plan backups carefully. Distribute the backups over time, agents, hosts, Storage Nodes so that none of the resources is overused and neither the backup servers nor the infrastructure degrade in performance.
See more about virtual machine backup in following articles:
- Acronis Backup and Acronis Backup Advanced for VMware: Backup Algorithm on ESX Environment
- Acronis Backup Advanced: Agent-based and Agentless Backup of Virtual Machines
In order for the agentless virtual machine backup to work correctly, following conditionds should be fulfilled. Complete all steps before proceeding to error troubleshooting step.
1.1 VM snapshot should work correctly
Acronis software uses snapshots executed by the hypervisor while taking the virtual machine backup. It is necessary that these snapshots work correctly. You can check the consistency of the snapshot in the following way.
Create a virtual machine snapshot
- Open the vSphere connection to the ESX host.
- Right click on the machine > take snapshot > check the "Quiesce guest file system"
- Uncheck the "Snapshot the virtual machine's memory" and click OK.
If you backup a virtual machine with transactional applications installed (SQL, Exchange evt.), then:
- VMware Tools should be installed inside that guest OS.
- VSS service should not be disabled inside the guest OS.
In case the guest OS event logs contain errors or warnings for VSS operations during creation of the snapshot, troubleshoot the VSS issues inside the guest OS.
If a virtual machine snapshot fails due to incorrect guest file system quiescing (snapshot without option Quiesce quest file system is successful), for virtual machines with no transcational applications installed you can workaround the issue by uninstalling the VMware Tools for the guest OS or (Acronis Backup Cloud or Acronis Backup 12 and above) by disabling Volume Shadow Copy (VSS) for Virtual Machines backup option.
Keep in mind the VMware ESX(i) virtualization platform limitations: VMware vSphere does not support snapshots of fault tolerant virtual machines, independent disks, and Raw Device Mapping (RDM) disks in physical compatibility mode. To overcome these limitations install an agent into the guest OS.
Create a VSS snapshot of the Hyper-V server and check that the Hyper-V VSS writer shows no errors. Follow instructions in Using DiskShadow to Determine Issues with VSS to create a VSS snapshot with Microsoft Hyper-V VSS writer verification.
If you backup a virtual machine with transactional applications installed (SQL, Exchange etc.), then:
- Hyper-V Integration Services should be installed inside that guest OS.
- VSS service should not be disabled inside the guest OS.
Keep in mind the Hyper-V virtualization platform limitations: Microsoft Hyper-V does not provide control over pass-through disks to a host system. As a result, the Microsoft Software Shadow Copy provider cannot provide Agent for Hyper-V with snapshots of pass-through disks. To vercome these limitations install an agent into the guest OS.
1.2 Appropriate license should be used
To perform an agentless backup of virtual machines one of the following agents should be installed:
- Agent for ESX(i) (Virtual Appliance)/Agent for VMware (Virtual Appliance)
- Agent for ESX(i) (Windows)/Agent for VMware (Windows)
- Agent for Hyper-V
These agents require a license for Acronis Backup (Advanced) for Virtual Host to work. One license is valid for one physical host.
1.3 Grant sufficient permissions
Provide the agent user and backup user with sufficient privileges to backup the virtual machines, if the user has been changed after installation.
To backup the VMware ESX(i) virtual machines, the user you specify in the agent for the vCenter/ESX(i) Host connection must have privileges listed in Privileges for VM backup and recovery help topic.
Also make sure that the TCP ports are open on the ESX(i) hosts for all operations. See this Prerequisites help topic.
To backup clustered Hyper-V machines the user must be a domain user and have administrative privileges on all cluster nodes. If the user has been changed after installation, then it should be specified explicitly in every backup and recovery task. See Backing up clustered Hyper-V machines help topic for more information.
If all steps above have been executed and issue still persists, go to the error troubleshooting step.
2. Error troubleshooting
Find and analyse the error messages to resolve the issue. Complete all steps before proceeding to collect information step.
2.1 Isolate the VM and the error message
Find the error message corresponding to the virtual machine backup failure.
Beware that the error message 'TOL cumulative completion result.' that appears in the backup plan error details is a generic error. You have to find an exact error for the failed task.
Look for the error event in the log or open the failed activity and locate the failed task and it's log.
Based on the information taken from the error message, define the virtual machine that could not be backed up. If the error message does not provide you with sufficient information on how to resolve the issue, check that this virtual machine fulfills the agentless backup criteria in prerequisites step and proceed to step 2.2.
2.2 Isolate and troubleshoot the VM backup stage
According to the error message in log, define at which backup stage the failure happens.
2.2.1 Snapshot creation
At this stage the agent send a request to the host to create the snapshot and the host executes the snapshot creation.
Applicable to Acronis Backup 11.5/11.7 and lower only: create a new backup plan and go to the backup source. Select the VM and click on List Disks for this VM. This will force the VM snapshot in the same way as it is created during the backup.
Applicable to Acronis Backup Cloud and Acronis Backup 12 and above: take manual quiesced snapshot of the VM via vSphere client.
If the message contains connection errors, double-check that the necessary permissions were granted to the user and the necessary port are open on the host. Go back to step 1.3.
Analyze the error and go to step 2.3 to check for known issues.
If no error appears while creating the snapshot and disks are listed correctly, then the failure happens at the other backup stage. Proceed to the next step and continue troubleshooting.
2.2.2 Reconfiguration of the VM (applicable only to ESX(i) VM backup)
At this stage the virtual disks or their links are attached to the agent. In case of LAN-free backup with Virtual Appliance (or Agent for VMware (Windows) installed inside a VM), virtual disks are attached and appear in the datastore folder where the Virtual Appliance resides. Possible reasons for the failure are the following:
- The blocksize of the datastore of the Virtual Appliance does not support the file size of the virtual disk being attached. See Acronis Backup & Recovery and Acronis vmProtect: Virtual machine backup fails with "Awaiting task 'CreateSnapshot' has failed. Reason: File is larger than the maximum size supported by datastore".
- The Virtual Appliance folder contains the virtual disk files with the same name, that were not deleted after the last backup. See troubleshooting described in https://kb.acronis.com/content/32611
Additional VM configuration files are also downloaded from the datastore at this stage. If backup fails with an error similar to "Failed to download additional file", this would mean, that the agent failed to access additional VM configuration files (.vmx, .nvram). Copy the path specified in the error message (http://VC-hostname/FolderName/FileName) to the internet browser and make sure you can download the files mentioned in the error message. If the download fails, refer to this VMware KB article.
If the virtual disk can be copied to the datastore successfully and the failure cause is unknown, proceed to the next step and continue troubleshooting.
2.2.3 VM data backup
At this stage data is read from the virtual disks.
Reading data from the virtual disks may give errors if the vdisk filesystem contains errors. Troubleshoot such errors accordingly, investigating the consistency of the filesystem of the guest OS. Refer to this article in such case: Operation with Acronis Product Fails with "Run list corrupted" or "MFT bitmap corrupted"
If backup is performed via LAN and network errors or file read errors appear, troubleshoot the connection between the agent and the ESXi host controlling the datastore where the virtual disks reside.
If the issue persists, proceed to the next step.
2.2.4 Links/disks are deleted from the agent (applicable only to ESX(i) VM backup)
At this stage the Virtual Appliance removes the backed up virtual disks from appliance VM configuration. Make sure nothing locks the disks in the Appliance's folder on the datastore and the Appliance can access and delete them.
2.2.5 Snapshot consolidation (applicable only to ESX(i) VM backup)
At the last stage of backup, ESX(i) Agent sends a command to the ESX(i) host to delete and consolidate the snapshots and delete the vmdk-delta files. If backup fails at this stage, you will find non-consolidated snapshots left after backup failure or delta-files in the datastore folder of the virtual machine that is being backed up.
Make sure that the consolidation works correctly in vSphere console for the snapshots of this virtual machine.
If the error message does not provide you with sufficient information on how to resolve the issue, proceed to step 2.3.
2.3 Check for known issues
Isolate the error message in the log. Go to known issues part and check for known solutions.
If a known solution has been applied, but the issue still persists, go to prerequisites step to troubleshoot the new issue.
If all steps above have been executed and issue still persists, go to collect information step.
3. Collect information
Collect following information and contact Acronis Customer Central.
3.1 Version of Hypervisor.
3.2 System information (Acronis system report)
Hyper-V: Collect System Report from both the Hyper-V host and from the guest OS of the faulty virtual machine.
ESX(i): Collect System Report from the Agent and from the guest OS of the faulty virtual machine.
- How to collect system information in Acronis Backup 12.5 and Acronis Backup 12
- How to collect system information in Acronis Backup 11.7 and 11.5
- How to collect system information in Acronis Backup Cloud
- How to collect system information from the VM guest OS
3.3 Additional information
You might be asked to collect additional information for virtual machine backup troubleshooting.
- Acronis Backup & Recovery 11.7/11.5: Collecting Cluster Logs
- Acronis Backup: Collecting SnapAPI Logs
- VSS Requestor Logs
- Collecting diagnostic information for VMware vCenter Server (VMware KB).
- Virtual machine log
- where ESXI logs are stored https://kb.vmware.com/s/article/1021806?r=2&Quarterback.validateRoute=1&...
If agentless virtual machine backup fails, you can install an agent inside the quest OS and backup the system or files as a workaround.