Description
As available physical space on underlying storage reaches a certain soft threshold, Acronis Backup Gateway (ABGW) starts to throttle writing operations. Throttling intensity depends on consumed space and increases until the hard threshold is reached. When the used space passes the hard threshold, throttling works with maximum intensity in order to delay overflow of the storage till more storage space is added. Once the storage space is sufficient (less than soft threshold), ABGW is expected to automatically disable the throttling shortly.
Starting since 5.1.1-68 ACI version alert Backup storage throttling is activated was added to notify about ongoing throttling in WebCP > Monitoring > Alerts.
More details on thresholds and throttling behavior applicable for different Acronis Cyber Infrastructure (ACI) versions are listed below.
Throttling in ACI version 5.1 and higher
Default soft and hard thresholds in versions 5.1 and higher are universal for any type of storage backend and are set on 85% and 93% of used storage space accordingly.
Tuning of throttling threshold
ABGW parameter storage.space_low_watermark may be used in order to tune the throttling thresholds.
In case of custom value for the parameter thresholds are calculated as the following:
- soft threshold = ( 100 - storage.space_low_watermark ) %
- hard threshold = ( 100 - storage.space_low_watermark/2 ) %
E.g.: if storage.space_low_watermark = 10 then soft threshold is 90% and hard threshold is 95%.
- in accordance with daily backup usage increase, there should be enough time for adding more storage space between reaching soft and hard thresholds;
- the minimal recommended value is storage.space_low_watermark = 5, setting smaller value may impact storage performance.
- full disabling of throttling is not possible by design.
Manually the value can be specified in the following way:
[root@abgw01 ~]# echo "storage.space_low_watermark = 10" > /mnt/vstorage/vols/acronis-backup/conf.d/throttling_params
After specifying the custom value it will be required to manually restart vstorage-abgw.service on each node included into Backup Gateway to apply the parameter:
[root@abgw01 ~]# systemctl restart vstorage-abgw
Throttling in ACI versions from 4.6 to 5.0
Thresholds in versions 4.6 till 5.0 depend on the storage backend type and amount of the nodes in cluster, full table may be found in the corresponding documentation: Acronis Cyber Infrastructure Administrator Guide - Monitoring backup storage
Throttling in ACI versions 4.5 and below
Thresholds in versions 4.5 and below depend on the type of storage backend and throttling suspends writing operations completely when hard threshold is reached:
- For Acronis Backup Gateway with local Acronis Cyber Infrastructure cluster and NFS backend soft and hard thresholds are 80% and 95% accordingly.
- For Acronis Backup Gateway with Object Storage backend (S3, Azure, Swift, etc.) soft and hard thresholds 75% and 87% accordingly.
Throttling with Public Cloud backend
In case if ABGW is configured with Public Cloud (Object Storage) backend, apart from common throttling based on storage cluster capacity additional throttling mechanism based on performance of object storage backlog processing is implemented, see more details in the following KB article:
Throttling with NFS backend
When ABGW is configured with NFS backend, throttling is based on available space of NFS share which is being automatically mounted on path /mnt/vstorage/vols/acronis-backup/storage, e.g.:
[root@abgw-node ~]# df -h /mnt/vstorage/vols/acronis-backup/storage
Filesystem Size Used Avail Use% Mounted on
192.168.1.100:/nfs_share 100T 90T 10T 90% /mnt/vstorage/vols/acronis-backup/storage
Note, that if more space was added on NFS server for the specified share, restart of vstorage-abgw.service may be required to re-mount NFS share properly to reflect a new NFS share size.
Throttling monitoring
Starting since ACI 4.6 throttling may be monitored in WebCP > Storage services > Backup storage > Overview > Append throttling chart.
Additionally, since ACI version 3.0 and higher it is also possible to check if the throttling is enabled fetching the following ABGW metric:
# curl -s http://127.0.0.1:44444/metrics | grep "abgw_append_throttle_delay_ms"
The command should be executed on any node included into Backup Gateway, if output is different from 0 - throttling is enabled, e.g.:
[root@abgw01 ]# curl -s http://127.0.0.1:44444/metrics | grep "abgw_append_throttle_delay_ms"
# HELP abgw_append_throttle_delay_ms_total sum of delays injected into appends because of low storage space
# TYPE abgw_append_throttle_delay_ms_total counter
abgw_append_throttle_delay_ms_total{} 156477800
More information
Contact Acronis support in case of the issues regarding the topic or if some assistance will be required.