This article describes the items to check and basic troubleshooting of Backup Storage service (ABGW) in Acronis Cyber Infrastructure (ACI).
Check the status of the nodes and service in WebCP > Infrastructure > Nodes > All nodes and WebCP > Storage services > Backup Storage. It is possible to review the list of nodes included into Backup Storage service, the status of the services on nodes and its performance statistic. Healthy state indicates that ABGW service is up and running:
When service on node is unhealthy, node will be marked as 'Failed' with 'Red' status of the Backup storage service on the corresponding UI screens:
In that case further check of service via CLI will be required.
For more details on monitoring refer to the following documentation:
Check WebCP > Monitoring > Alerts for alerts related to Backup Storage, see the full list of alerts in the documentation: Administrator guide - Backup storage alerts.
Backup Storage certificates status
Check the validity of the Backup Storage certificate:
- if ACI version is below 5.1 certificate file should be checked manually on any ACI node in cluster via CLI:
# openssl x509 -noout -text -in /mnt/vstorage/vols/acronis-backup/certs/reg.crt | grep -i valid -A 2
- if ACI version is 5.1 and higher, certificate expiration date may be checked in WebCP > Backup Storage > Registrations, e.g.:
Or via CLI:
# vstorage-abgw-register list | grep expiration_date
In case if certificates expiration date is close or already reached, the corresponding alert is expected to be observed in WebCP > Monitoring > Alerts. Expired certificates should be updated manually, see KB for reference:
Backup Storage Throttling status
Check Append throttle chart in WebCP > Storage Services > Backup Storage > Overview.
In certain conditions ABGW may start throttle writing operations which is observed with decreased speed of backups. More details on Backup Storage throttling mechanisms may be found in the following KB:
Starting since ACI 5.1 version, the corresponding alert is expected to be observed in WebCP > Monitoring > Alerts in case if throttling is active.
Service status in CLI
- The main service responsible for backup storage requests processing is called vstorage-abgw.service - it is enabled, configured and running on all nodes included into ABGW. Service listen port 44445 by default in order to accept connections from backup agent and another ABGW nodes in cluster (expected to be externaly available). Additionally service listen on port 44444 reporting it's metrics to Prometheus monitoring service in ACI, not exposed externally.
- Starting since ACI 4.7.1 version additional service abgw-setting.service was introduced, it is required for delivering Immutability settings from Acronis Cyber Cloud (ACC) platform to ABGW. Service listen on port 40440 by default and running on all nodes included into ABGW.
- The ports required for ABGW are being opened on ACI node interfaces by the means of automatically configured iptables rules. Rules are set per network interface in accordance with assigned Infrastructure network having Backup (ABGW) public traffic type. More details on traffic types management may be found in the documentation - Administrator Guide: Managing regular traffic types.
How to check if service is running
To check the status of the service:
# systemctl status vstorage-abgw.service
Example output of active services:
To check the journal logs of the service activities and events:
# journalctl -u vstorage-abgw.service | tail
How to check if service is accepting network connections
To check if the service listen ABGW port (default is 44445) and if there are any external connections established:
# netstat -plant | grep 44445
Or to check for the process ID of vstorage-abgw service and review all connections:
# netstat -plant | grep $(pidof vstorage-abgw)
telnet utility may be used locally issued from the ACI node to check if service is responding and accepting network connections:
# telnet localhost 44445
Example output showing that local connection works:
In case of issues with service (service in failed status, not responding, auto-restarting in the loop) proceed with service logs investigation.
Service log information
- investigate the log for the timeframe of service failure in accordance with systemctl and journalctl messages;
- in case if service is constantly auto-restarting there would be recurring error messages in ABGW logs on each start attempt;
- if the issue with service leading to it's crash check if there are any reports generated for ABGW processes by ABRT tool using abrt-cli ls command, for example:
[root@abgw-01 ~]# abrt-cli ls
reason: vstorage-abgw killed by SIGABRT
time: Wed 23 Nov 2022 05:15:19 PM EET
cmdline: /usr/bin/vstorage-abgw -c /etc/vstorage/abgw.config
uid: 978 (abgw)
In case if such report is generated - mention it on support ticket submission.
Contact Acronis Support in case if technical assistance with investigation of the service issues is required.