68726: Acronis Cyber Infrastructure: Backup Storage service status check

use Google Translate

    Last update: 24-11-2022

    Introduction

    This article describes the items to check and basic troubleshooting of Backup Storage service (ABGW) in Acronis Cyber Infrastructure (ACI).

    Solution

    Cluster status

    Check the status of the nodes and service in WebCP > Infrastructure > Nodes > All nodes and WebCP > Storage services > Backup Storage. It is possible to review the list of nodes included into Backup Storage service, the status of the services on nodes and its performance statistic. Healthy state indicates that ABGW service is up and running:

    When service on node is unhealthy, node will be marked as 'Failed' with 'Red' status of the Backup storage service on the corresponding UI screens:

    In that case further check of service via CLI will be required.

    For more details on monitoring refer to the following documentation:

    Administrator Guide - Monitoring backup storage

    Administrator Guide - Monitoring the storage cluster

    Cluster Alerts

    Check WebCP > Monitoring > Alerts for alerts related to Backup Storage, see the full list of alerts in the documentation: Administrator guide - Backup storage alerts.

    Backup Storage certificates status

    Check the validity of the Backup Storage certificate:

    • if ACI version is below 5.1 certificate file should be checked manually on any ACI node in cluster via CLI:
       

      # openssl x509 -noout -text -in /mnt/vstorage/vols/acronis-backup/certs/reg.crt | grep -i valid -A 2

    • if ACI version is 5.1 and higher, certificate expiration date may be checked in WebCP > Backup Storage > Registrations, e.g.:

      Or via CLI:

      # vstorage-abgw-register list | grep expiration_date
              "certs_expiration_date": "2025-02-13T09:33:31Z",
              "certs_expiration_date_ts": 1739439211,

    In case if certificates expiration date is close or already reached, the corresponding alert is expected to be observed in WebCP > Monitoring > Alerts. Expired certificates should be updated manually, see KB for reference:

    59997: Acronis Cyber Infrastructure: how to update backup storage certificate

    Backup Storage Throttling status

    Check Append throttle chart in WebCP > Storage Services > Backup Storage > Overview.

    In certain conditions ABGW may start throttle writing operations which is observed with decreased speed of backups. More details on Backup Storage throttling mechanisms may be found in the following KB:

    62823: Acronis Cyber Infrastructure: Throttling on backup storage

    Starting since ACI 5.1 version, the corresponding alert is expected to be observed in WebCP > Monitoring > Alerts in case if throttling is active.

    Service status in CLI

    Common information

    • The main service responsible for backup storage requests processing is called vstorage-abgw.service - it is enabled, configured and running on all nodes included into ABGW. Service listen port 44445 by default in order to accept connections from backup agent and another ABGW nodes in cluster (expected to be externaly available). Additionally service listen on port 44444 reporting it's metrics to Prometheus monitoring service in ACI, not exposed externally.
    • Starting since ACI 4.7.1 version additional service abgw-setting.service was introduced, it is required for delivering Immutability settings from Acronis Cyber Cloud (ACC) platform to ABGW. Service listen on port 40440 by default and running on all nodes included into ABGW.
    • The ports required for ABGW are being opened on ACI node interfaces by the means of automatically configured iptables rules. Rules are set per network interface in accordance with assigned Infrastructure network having Backup (ABGW) public traffic type. More details on traffic types management may be found in the documentation - Administrator Guide: Managing regular traffic types.

    How to check if service is running

    To check the status of the service:

    # systemctl status vstorage-abgw.service

    Example output of active services:

    To check the journal logs of the service activities and events:

    # journalctl -u vstorage-abgw.service | tail

    How to check if service is accepting network connections

    To check if the service listen ABGW port (default is 44445) and if there are any external connections established:

    # netstat -plant | grep 44445

    Or to check for the process ID of vstorage-abgw service and review all connections:

    # netstat -plant | grep $(pidof vstorage-abgw)

    Example outputs:

    telnet utility may be used locally issued from the ACI node to check if service is responding and accepting network connections:

    # telnet localhost 44445

    Example output showing that local connection works:

    In case of issues with service (service in failed status, not responding, auto-restarting in the loop) proceed with service logs investigation.

    Service log information

    Tips:

    • investigate the log for the timeframe of service failure in accordance with systemctl and journalctl messages;
    • in case if service is constantly auto-restarting there would be recurring error messages in ABGW logs on each start attempt;
    • if the issue with service leading to it's crash check if there are any reports generated for ABGW processes by ABRT tool using abrt-cli ls command, for example:

      [root@abgw-01 ~]# abrt-cli ls

      id c824b453b2a8eab2003b90a440c98cde54dc03f5
      reason:         vstorage-abgw killed by SIGABRT
      time:           Wed 23 Nov 2022 05:15:19 PM EET
      cmdline:        /usr/bin/vstorage-abgw -c /etc/vstorage/abgw.config
      package:        vstorage-abgw-2.15.39-2.as7
      uid:            978 (abgw)
      count:          1
      Directory:      /var/spool/abrt/ccpp-2022-11-23-17:15:19-979448

    In case if such report is generated - mention it on support ticket submission.

    See also: 61905: Acronis Cyber Infrastructure: how to open Acronis Backup Gateway log

    More information

    Contact Acronis Support in case if technical assistance with investigation of the service issues is required.

    Tags: