62823: Acronis Cyber Infrastructure: Throttling on backup storage

use Google Translate

    Last update: 23-02-2023


    As available physical space on underlying storage reaches a certain soft threshold, Acronis Backup Gateway (ABGW) starts to throttle writing operations. Throttling intensity depends on consumed space and increases until the hard threshold is reached. When the used space passes the hard threshold, throttling works with maximum intensity in order to delay overflow of the storage till more storage space is added. Once the storage space is sufficient (less than soft threshold), ABGW is expected to automatically disable the throttling shortly.

    Starting since 5.1.1-68 ACI version alert Backup storage throttling is activated was added to notify about ongoing throttling in WebCP > Monitoring > Alerts.

    More details on thresholds and throttling behavior applicable for different Acronis Cyber Infrastructure (ACI) versions are listed below.

    Throttling in ACI version 5.1 and higher

    Default soft and hard thresholds in versions 5.1 and higher are universal for any type of storage backend and are set on 85% and 93% of used storage space accordingly.

    Tuning of throttling threshold

    Default thresholds are suitable for the most of the Backup Storages, so manual adjustment of thresholds is not required unless carefully planned.

    ABGW parameter storage.space_low_watermark may be used in order to tune the throttling thresholds.

    In case of custom value for the parameter thresholds are calculated as the following:

    • soft threshold = ( 100 - storage.space_low_watermark ) %
    • hard threshold =  ( 100 - storage.space_low_watermark/2 ) %

    E.g.: if storage.space_low_watermark = 10 then soft threshold is 90% and hard threshold is 95%.

    Since the storage size and backup load for different Backup Storages differs, the following should be taken into account when selecting custom value of the threshold:

    • in accordance with daily backup usage increase, there should be enough time for adding more storage space between reaching soft and hard thresholds;
    • the minimal recommended value is storage.space_low_watermark = 5, setting smaller value may impact storage performance. 
    • full disabling of throttling is not possible by design.

      Manually the value can be specified in the following way:

      [root@abgw01 ~]# echo "storage.space_low_watermark = 10" > /mnt/vstorage/vols/acronis-backup/conf.d/throttling_params

      After specifying the custom value it will be required to manually restart vstorage-abgw.service on each node included into Backup Gateway to apply the parameter:

      [root@abgw01 ~]# systemctl restart vstorage-abgw

      Throttling in ACI versions from 4.6 to 5.0

      Thresholds in versions 4.6 till 5.0 depend on the storage backend type and amount of the nodes in cluster, full table may be found in the corresponding documentation: Acronis Cyber Infrastructure Administrator Guide - Monitoring backup storage

      Throttling in ACI versions 4.5 and below

      Thresholds in versions 4.5 and below depend on the type of storage backend and throttling suspends writing operations completely when hard threshold is reached:

      • For Acronis Backup Gateway with local Acronis Cyber Infrastructure cluster and NFS backend soft and hard thresholds are 80% and 95% accordingly.
      • For Acronis Backup Gateway with Object Storage backend (S3, Azure, Swift, etc.) soft and hard thresholds 75% and 87% accordingly.

      Throttling with Public Cloud backend

      In case if ABGW is configured with Public Cloud (Object Storage) backend, apart from common throttling based on storage cluster capacity additional throttling mechanism based on performance of object storage backlog processing is implemented, see more details in the following KB article:

      68284: Acronis Cyber Infrastructure: Append throttling on ABGW installations with public cloud backend

      Throttling with NFS backend

      When ABGW is configured with NFS backend, throttling is based on available space of NFS share which is being automatically mounted on path /mnt/vstorage/vols/acronis-backup/storage, e.g.:

      [root@abgw-node ~]# df -h /mnt/vstorage/vols/acronis-backup/storage
      Filesystem                 Size  Used Avail Use% Mounted on   100T   90T  10T  90% /mnt/vstorage/vols/acronis-backup/storage

      Note, that if more space was added on NFS server for the specified share, restart of vstorage-abgw.service may be required to re-mount NFS share properly to reflect a new NFS share size.

      Throttling monitoring

      Starting since ACI 4.6 throttling may be monitored in WebCP > Storage services > Backup storage > Overview > Append throttling chart.

      Additionally, since ACI version 3.0 and higher it is also possible to check if the throttling is enabled fetching the following ABGW metric:

      # curl -s | grep "abgw_append_throttle_delay_ms" 

      The command should be executed on any node included into Backup Gateway, if output is different from 0 - throttling is enabled, e.g.:

      [root@abgw01 ]# curl -s | grep "abgw_append_throttle_delay_ms" 
      # HELP abgw_append_throttle_delay_ms_total sum of delays injected into appends because of low storage space 
      # TYPE abgw_append_throttle_delay_ms_total counter 
      abgw_append_throttle_delay_ms_total{} 156477800 

      More information

      Contact Acronis support in case of the issues regarding the topic or if some assistance will be required.