This Article applies to:

• Acronis Software

Description

This Article describes how to collect dumps of hanging processes on a Virtual Appliance or a linux-based Bootable media. You will need a machine to remotely connect to the bootable media. If you don´t have such machine, see instructions at the bottom.

1. Determine which version of gdb you need:
   •  if you are on Acronis Bootable Media, then you need gdb32 for 32 bit and gdb64 for 64 bit media. To determine which media you are on, proceed with the below steps and once you are in shell, execute command cat /proc/version. If the output contains "x86_64", you need gdb64, otherwise gdb32.
   • If you are on Virtual Appliance in Acronis Cyber Backup 11.5/11.7 then you need gdb32
   • If you are on Virtual Appliance in Acronis Cyber Backup 11.7 and use custom agent from KB59602 you need gdb64
   • If you are on Virtual Appliance in Acronis Cyber Protect 15, Acronis Cyber Backup 12 or 12.5 then you need gdb64
2. Download gdb32.zip or gdb64.zip attached to this article depending on step 1.
3. Extract it.
4. Download WinSCP. Download Putty.
5. Get into shell (Shell is basically command prompt in Linux OS):
   1. On Acronis Cyber Backup and Recovery bootable media or Acronis Cyber Backup and Recovery Virtual Appliance or Acronis Snap Deploy bootable agent press Ctrl+Shift+F2
   2. On vmProtect Virtual Appliance Press  Ctrl+Shift+Alt+Space+F1, then Alt+F2
6. Start ssh daemon by issuing the command /bin/sshd (If you see any error just click "Enter") 
   
7. Check the IP of the bootable media. If there is no IP - your NIC might not be detected. Try a newer version of the bootable media, for more information see Acronis Bootable Media: Troubleshooting NIC Detection
8. Use WinSCP to connect to the IP that you've checked above. Login is "root". Password for everything except vmProtect Virtual Appliance is "root". For vmProtect Virtual Appliance the password is the same that you use to login into vmProtect web console. File protocol: "SCP". Click "yes" when it asks you about certificate.
   
9. Transfer files using WinSCP: file "gdb"to the root folder (The folder name is "/"), lib* files to the /lib/ folder. If transfer fails with "not enough space left on device" then 
   1. Go back to the shell and execute command df -h. If "Available" on /dev/root is 0 or close to it, the reason of process hanging is most likely lack of space.
      
      2. Please contact Acronis Support. You can shut down the appliance and save it to OVF Template. (File -> Export to OVF in your vSphere console) and provide it to support with reference to this article.
10. Use Putty to connect to the above IP using the same username and password. Putty is optional in this case. It just makes all your work in Shell more convenient especially if you are connected using several remote sessions. Use the same IP, credentials as in step 7. Click "Yes" when asked about certificate.
    
11. Make gdb executable and accessible issuing command in Shell chmod 0777 /gdb 
12. Go into root folder issuing command cd /
13. Check the PID (Process ID) of the hanging process: 
    • Bootable media: usually the process in question is "/bin/product". The command to check would be ps | grep /bin/product
    • Virtual Appliance: if the GUI and agent itself hangs the process is "/bin/product". However if the backup/replication/cleanup/recovery hangs in Acronis Cyber Backup & Recovery Virtual Appliance, the process name is "/bin/service_process". For vmProtect processes see Acronis vmProtect 9: Processes and Services. The command would be "ps | grep service_process". If you see more that 1 service process running you will have to create a dump of each.
      
       Note that on this picture there is only one "/bin/service_process" that you need to dump and the PID of it is 1018. Grep /bin/service_process is basically the process generated by your command. /bin/sh -c /bin/service_process is a shell instance that initiates service process. There is no need to dump it.
    • All-in-One appliance: use commands ps aux | grep service_process and ps aux | grep mms to determine the ID of the corresponding process.
14. Switch to GDB shell with the following command:
    ./gdb -pid <process_id>
    where <process_id> is the identificator of the hanging process. It will start loading symbols and will eventually show you (gdb).
    
15. To write a core file, issue:
    gcore <FILE_NAME_OF_THE_DUMP>
16. Once the core file is written, issue:
    detach
17. Repeat steps 14-16 several times to get more dumps. Make sure you specify different  FILE_NAME_OF_THE_DUMP to prevent overwriting previous dumps.
    
18. Quit GDB by issuing quit
    
19. Dumps will be generated in the root folder "/". They will have the names that you gave them in step 14.
    
20. Use WinSCP to download them to your machine.
21. Zip the dumps and contact Acronis Support with reference to this article.

(!) If network does not work or if you don´t have second machine (for example, the problem is reproduced on Acronis Bootable Media on your home PC\the only PC available), mount USB stick to save the coredump:

Attach USB flash drive before booting into Acronis Bootable Media.
Mount locally, for example for USB flash drive:

fdisk -l

you will see an output similar to this:

Device Boot Start End Blocks Id System
/dev/sdb1 1 64 480000000 83 Linux

use the commands to mount USB stick:

mkdir /tmp/dongle

mount -t auto /dev/sdb1 /tmp/dongle

Now you can repeat the steps above and then copy the files to the mountpoint we created. Also remember that the FAT32 file size limit is 4 GB. If it is possible, you can format your USB drive in ext4 FS for example.

To copy the files after coredumps have been created:

cp /core* /tmp/dongle

Do not remove the USB drive until you shut down or reboot machine or until you unmount drive with the following command:

umount /tmp/dongle