This article describes what to do if after a backup failure the backed up virtual disks are left attached to Acronis Virtual Appliance
(!) This article applies to cases when the backup is performed using the Hot-add method, i.e. attaching disks to Acronis Virtual Appliance for LAN-free backup. See also Acronis vmProtect: Supported Environments.
(!) Most of the steps outlined in the article require a vSphere client connected to vCenter/ESX(i) host where Acronis Virtual Appliance is deployed.
- You notice that Acronis Virtual Appliance (VA) contains additional virtual disks in configuration even if there is no backup active at this time. See the below example screenshot (the backed up VM is w2k_Lenok):
(!) It is normal to see this picture while the backup is running. The disks are normally deleted from Acronis Virtual Appliance (VA) configuration after the backup finishes.
- After the backup there is a snapshot left in the VM snapshot manager:
- Removing this snapshot will not delete the disks deltas from the datastore (thus causing impact on VM performance):
- The future backups of this VMs start to fail with error similar to below one (note the bold line). This happens as a result of a previous failure since while the disks are already attached to the VA, the new backup attempt fails to attach them twice:
12 Information 6/8/12 6:20:41 AM Adding virtual hard disk '[datastore] SERVER/SERVER.vmdk' to '0:1'.
13 Error 6/8/12 6:20:48 AM Failed to open raw hard disk data file '[datastore] SERVER/SERVER.vmdk'.
Error code: 94
Message: Failed to open raw hard disk data file '[datastore] SERVER/SERVER.vmdk'.
Error code: 64
Message: VMware error: 'An internal error.'.
Error code: 52
Message: Awaiting task 'AttachScsiHardDisk' has failed. Reason: fault.FileAlreadyExists.summary.
14 Information 6/8/12 6:20:49 AM Remove snapshot (snapshot-3927).
15 Warning 6/8/12 6:20:49 AM The operation 'Backing up' has failed.
Error code: 26
Message: The operation 'Backing up' has failed.
Error code: 1080
Message: Failed to execute the operation.
The main cause of this behavior is that Acronis Virtual Appliance backup engine has crashed or the Virtual Appliance had been hardly reset (also could be caused by crash). The solution section below describes the troubleshooting steps as well as clean-up steps required to revert from this situation back to normal workflow.
This issue has been fixed in Acronis Backup for VMware 9 (Acronis vmProtect 9).
To solve the issue in an older version without updating, clean-up the Virtual Appliance:
- Remove the unnecessary disks (with underscores in their names) which were unexpectedly left attached from the VA. You should choose the Delete from disk option which will remove the links from the VA folder:
This step will unlock the snapshot delta files which are left in the backed up VM folder on the datastore so now we can get rid of these deltas.
(!) If the virtual disk path is not in Acronis Virtual Appliance folder then you should _NOT_ delete it from disk - choose the 1st option ("Remove from virtual machine") instead.
- Reboot (or reset) Acronis Virtual Appliance to reset the disk attach traces inside it.
- Create a temporary simple snapshot (without quiescing or memory) of the affected source VM:
- Go to Snapshot Manager and click "Delete All" button:
- Since the snapshots were unlocked before, the delta files will be merged (consolidated) into the base disk:
(!) The snapshot files may be locked by ddb.deletable flag in rare cases. To get rid of this flag and allow the snapshots (deltas) to be deleted please follow the instructions from https://kb.vmware.com/s/article/1012383
After cleaning up the traces we should investigate the reasons for the crash by collecting the crash dump from the VA:
- Install the WinSCP utility which you can get from http://winscp.net/eng/download.php. WinSCP (Windows Secure CoPy) is a free and open source SFTP, SCP, and FTP client for Microsoft Windows. Its main function is to secure file transfer between a local and a remote computer. Beyond this, WinSCP offers basic file manager and file synchronization functionality.
- Log in to vSphere Client and open the console of Acronis vmProtect Virtual Appliance. Write down or remember the IP address of it.
- Press CTRL-SHIFT-ALT-SPACE-F1 and then ALT-F2 (this will bring you to the command line interface);
- In the console please type this command: /bin/sshd
- Launch the WinSCP utility on the machine where you installed it;
- In the host machine field type your Acronis vmProtect Virtual Appliance IP (recorded on Step 2), the same as in the vSphere console. The default login:password is root:root (the password may differ if you changed it via web console at Configure -> Agent Password).
- The protocol should be set to SCP and then click on log in.
- You will get a window with all the directories of the Appliance (Windows Commander-like interface), we are interested in the /tmp/core file and /var/lib/Acronis/vmProtect/ folder. Please copy both file and the folder, pack them into .zip and contact Acronis Supportwith the collected information and a reference to this article.
The symptoms described in this article can be emulated by resetting Acronis Virtual Appliance while the backup task is running (typically the reason for such symptoms is backup daemon crash, or Virtual Appliance crash).
Acronis Virtual Appliance has crash detection mechanism which puts a warning into the log file when the crash occurs. However, the backup daemon crash cannot be detected this way and in this case the backup log will be truncated (incomplete).