This article provides instruction for troubleshooting virtual machine backup failures in Acronis Cyber Backup virtualization solutions.
Solution
Follow these steps to troubleshoot virtual machine backup failures:
- Get familiar with technology of virtual machine backup in the About part.
- Check for known issues in the Known issues part.
- Follow to Troubleshooting part to troubleshoot and resolve the issue.
About
Major steps of the virtual machine backup in Acronis Cyber Backup (Acronis Cyber Backup & Recovery) are the following.
Best practices:
- Backup virtual machines at hypervisor level to off-load the guest OS.
- Quiescing and VSS should be working properly for each VM with transactional applications like databases and email servers.
- Do not overload the backup agent. See the system requirements for 5-10 simultaneous VM backup with Agent for VMware vSphere ESX(i) (Virtual Appliance). Use Agent for ESX(i) (Windows) to off-load the ESX(i) host. In Acronis Cyber Backup 12.5, you can also limit the total number of simultaneously backed-up virtual machines.
- Plan backups carefully. Distribute the backups over time, agents, hosts, Storage Nodes so that none of the resources is overused and neither the backup servers nor the infrastructure degrade in performance.
See more about virtual machine backup in the following articles:
- Acronis Cyber Backup: Backup Algorithm on ESX Environment
- Acronis Cyber Backup 12.5: agent-based and agentless backup of virtual machines
Known Issues
Troubleshooting
Follow step-by-step instructions below to troubleshoot the issue with VM backup failure. Complete prerequisites step before proceeding to the error troubleshooting step.
1. Prerequisites
In order for the agentless virtual machine backup to work correctly, the following conditions should be fulfilled. Complete all steps before proceeding to the error troubleshooting step.
1.1 VM snapshot should work correctly
Acronis software uses snapshots executed by the hypervisor while taking the virtual machine backup. It is necessary that these snapshots work correctly. You can check the consistency of the snapshot in the following way.
ESX(i)
Create a virtual machine snapshot
- Open the vSphere connection to the ESX host.
- Right-click on the machine > take snapshot > check the "Quiesce guest file system"
- Uncheck the "Snapshot the virtual machine's memory" and click OK.
If you backup a virtual machine with transactional applications installed (SQL, Exchange etc.), then:
- VMware Tools should be installed inside that guest OS.
- VSS service should not be disabled inside the guest OS.
In case the guest OS event logs contain errors or warnings for VSS operations during creation of the snapshot, troubleshoot the VSS issues inside the guest OS.
If a virtual machine snapshot fails due to incorrect guest file system quiescing (snapshot without option Quiesce quest file system is successful), for virtual machines with no transactional applications installed you can workaround the issue by uninstalling the VMware Tools for the guest OS or (Acronis Cyber Backup Cloud or Acronis Cyber Backup 12 and above) by disabling Volume Shadow Copy (VSS) for Virtual Machines backup option.
Keep in mind the VMware ESX(i) virtualization platform limitations: Agent for VMware backs up a fault tolerant machine only if fault tolerance was enabled in VMware vSphere 6.0 and later. If you upgraded from an earlier vSphere version, it is enough to disable and enable fault tolerance for each machine. If you are using an earlier vSphere version, install an agent in the guest operating system.
Hyper-V
Create a VSS snapshot of the Hyper-V server and check that the Hyper-V VSS writer shows no errors. Follow instructions in Using DiskShadow to Determine Issues with VSS to create a VSS snapshot with Microsoft Hyper-V VSS writer verification.
If you backup a virtual machine with transactional applications installed (SQL, Exchange etc.), then:
- Hyper-V Integration Services should be installed inside that guest OS.
- VSS service should not be disabled inside the guest OS.
Keep in mind the Hyper-V virtualization platform limitations: Microsoft Hyper-V does not provide control over pass-through disks to a host system. As a result, the Microsoft Software Shadow Copy provider cannot provide Agent for Hyper-V with snapshots of pass-through disks. To overcome these limitations install an agent into the guest OS.
1.2 Appropriate license should be used
To perform an agentless backup of virtual machines one of the following agents should be installed:
- Agent for ESX(i) (Virtual Appliance)/Agent for VMware (Virtual Appliance)
- Agent for ESX(i) (Windows)/Agent for VMware (Windows)
- Agent for Hyper-V
These agents require a license for Acronis Cyber Backup (Advanced) for Virtual Host to work. One license is valid for one physical host.
1.3 Grant sufficient permissions
Provide the agent user and backup user with sufficient privileges to back up the virtual machines, if the user has been changed after installation.
ESX(i):
To backup the VMware ESX(i) virtual machines, the user you specify in the agent for the vCenter/ESX(i) Host connection must have privileges listed in Privileges for VM backup and recovery help topic.
Also make sure that the TCP ports are open on the ESX(i) hosts for all operations. See this Prerequisites help topic.
Hyper-V:
To backup clustered Hyper-V machines the user must be a domain user and have administrative privileges on all cluster nodes. If the user has been changed after installation, then it should be specified explicitly in every backup and recovery task. See Backing up clustered Hyper-V machines help topic for more information.
If all steps above have been executed and the issue still persists, go to the error troubleshooting step.
2. Error troubleshooting
Find and analyze the error messages to resolve the issue. Complete all steps before proceeding to collect information step.
2.1 Isolate the VM and the error message
Find the error message corresponding to the virtual machine backup failure.
Beware that the error message 'TOL cumulative completion result.' that appears in the backup plan error details is a generic error. You have to find an exact error for the failed task.
Look for the error event in the log or open the failed activity and locate the failed task and its log.
Based on the information taken from the error message, define the virtual machine that could not be backed up. If the error message does not provide you with sufficient information on how to resolve the issue, check that this virtual machine fulfills the agentless backup criteria in prerequisites step and proceed to step 2.2.
2.2 Isolate and troubleshoot the VM backup stage
According to the error message in log, define at which backup stage the failure happens.
2.2.1 Snapshot creation
At this stage the agent sends a request to the host to create the snapshot and the host executes the snapshot creation.
Applicable to Acronis Cyber Backup 11.5/11.7 and lower only: create a new backup plan and go to the backup source. Select the VM and click on List Disks for this VM. This will force the VM snapshot in the same way as it is created during the backup.
Applicable to Acronis Cyber Backup Cloud and Acronis Cyber Backup 12 and above: take a manual quiesced snapshot of the VM via vSphere client.
If the message contains connection errors, double-check that the necessary permissions were granted to the user and the necessary port are open on the host. Go back to step 1.3.
Analyze the error and go to step 2.3 to check for known issues.
If no error appears while creating the snapshot and disks are listed correctly, then the failure happens at the other backup stage. Proceed to the next step and continue troubleshooting.
2.2.2 Reconfiguration of the VM (applicable only to ESX(i) VM backup)
At this stage the virtual disks or their links are attached to the agent. In case of LAN-free backup with Virtual Appliance (or Agent for VMware (Windows) installed inside a VM), virtual disks are attached and appear in the datastore folder where the Virtual Appliance resides. Possible reasons for the failure are the following:
- The blocksize of the datastore of the Virtual Appliance does not support the file size of the virtual disk being attached. See Acronis Products: Virtual machine backup fails with "Awaiting task 'CreateSnapshot' has failed. Reason: File is larger than the maximum size supported by datastore".
- The Virtual Appliance folder contains the virtual disk files with the same name, that were not deleted after the last backup. See troubleshooting described in https://kb.acronis.com/content/32611
Additional VM configuration files are also downloaded from the datastore at this stage. If backup fails with an error similar to "Failed to download additional file", this would mean, that the agent failed to access additional VM configuration files (.vmx, .nvram). Copy the path specified in the error message (http://VC-hostname/FolderName/FileName) to the internet browser and make sure you can download the files mentioned in the error message. If the download fails, refer to this VMware KB article.
If the virtual disk can be copied to the datastore successfully and the failure cause is unknown, proceed to the next step and continue troubleshooting.
2.2.3 VM data backup
At this stage data is read from the virtual disks.
Reading data from the virtual disks may give errors if the vdisk filesystem contains errors. Troubleshoot such errors accordingly, investigating the consistency of the filesystem of the guest OS. Refer to this article in such case: Operation with Acronis Product Fails with "Run list corrupted" or "MFT bitmap corrupted"
If backup is performed via LAN and network errors or file read errors appear, troubleshoot the connection between the agent and the ESXi host controlling the datastore where the virtual disks reside.
If the issue persists, proceed to the next step.
2.2.4 Links/disks are deleted from the agent (applicable only to ESX(i) VM backup)
At this stage the Virtual Appliance removes the backed up virtual disks from appliance VM configuration. Make sure nothing locks the disks in the Appliance's folder on the datastore and the Appliance can access and delete them.
2.2.5 Snapshot consolidation (applicable only to ESX(i) VM backup)
At the last stage of backup, ESX(i) Agent sends a command to the ESX(i) host to delete and consolidate the snapshots and delete the vmdk-delta files. If backup fails at this stage, you will find non-consolidated snapshots left after backup failure or delta-files in the datastore folder of the virtual machine that is being backed up.
Make sure that the consolidation works correctly in vSphere console for the snapshots of this virtual machine.
If the error message does not provide you with sufficient information on how to resolve the issue, proceed to step 2.3.
2.3 Check for known issues
Isolate the error message in the log. Go to known issues part and check for known solutions.
If a known solution has been applied, but the issue still persists, go to prerequisites step to troubleshoot the new issue.
If all steps above have been executed and the issue still persists, go to collect information step.
3. Collect information
Collect the following information and contact Acronis Support.
3.1 Version of Hypervisor.
3.2 System information (Acronis system report)
Hyper-V: Collect System Report from both the Hyper-V host and from the guest OS of the faulty virtual machine.
ESX(i): Collect System Report from the Agent and from the guest OS of the faulty virtual machine.
- How to collect system information in Acronis Cyber Backup 12.5 and Acronis Cyber Backup 12
- How to collect system information in Acronis Cyber Backup 11.7 and 11.5
- How to collect system information in Acronis Cyber Backup Cloud
- How to collect system information from the VM guest OS
3.3 Additional information
You might be asked to collect additional information for virtual machine backup troubleshooting.
Hyper-V:
ESX(i):
- Collecting diagnostic information for VMware vCenter Server (VMware KB).
- Virtual machine log
- where ESXI logs are stored https://kb.vmware.com/s/article/1021806?r=2&Quarterback.validateRoute=1&...
More information
If agentless virtual machine backup fails, you can install an agent inside the guest OS and backup the system or files as a workaround.