Effective validation and repair of TIBX/Archive3/Format12 backup archives using CLI tool archive_ctl

Scenario

You have a backup archive (TIBX) and operations with it (either continuing to back up to it, or restoring/extracting data from it) fail with errors that the archive is corrupted.

Goal: You want to run diagnosis (validate) the archive to see what is damaged in it and to what extent, and, if possible, you want to repair the archive (at least partially).

Note: The archive_ctl CLI tool, as well as some other CLI tools which are very useful in many situations can be downloaded from here or here: astor-1.14.zip

When it comes to validation, exploration, and repair attempts of TIBX archives, archive_ctl is often superior to other approaches such as:

- acrocmd

- validate + auto-repair archive on Windows by setting environment variable:

Press Win+R, type sysdm.cpl into the input field and press Ok. System properties window opens. Go to the Advanced tab and click on Environment variables
In System variables section, click New
Create a new variable with the following parameters:
Variable name: ARCHIVE3_AUTO_REPAIR
Variable value: 1
Save changes and close System properties window
In Protection Console, go to Backup Storage tab, select the archive that causes the error, and click Show backups. While accessing the backup, Agent will attempt to find the last good state and fix the archive.

... because using the archive_ctl tool provides much more direct info (outputs) about what is wrong with the archive and how wrong it is.

Dealing with corrupted archives using archive_ctl can be done in several stages.

Please note that the archive validations/inspection/repair attempts over an archive require reading/scanning the archive several times end-to-end, therefore - if possible - when the archive is stored on a network share (e.g. SMB share), it is recommended to run the following operations from a location / machine on the LAN where connectivity to the share is high-speed, or, if the share is on a Windows box and not on some NAS appliance, to run the validation locally there to skip having to go over SMB protocol/network.

If running from the machine where the agent is, or from any other machine on the LAN (i.e. NOT locally on the machine where the SMB share is), you'll need to specify the values for these parameters:

Network login options:
--network-login network login name
--network-password network password

...or, alternatively, the network share can be mapped as a network drive like e.g. Z: and that path could be used.

If the archive is on a cloud storage location (no matter partner's or in an Acronis DC), you'll need to specify the values for these parameters:

--astor <host:port> connect to an astorage gw at host:port
--cert <fname> use a client certificate in file fname

The certificate is the cloud access certificate described in https://kb.acronis.com/content/60082

Step 1A: Get basic details about the archive:

archive_ctl --info --password <your_pass_for_the_archive> -f <full_absolute_path_to_archive>/<archive_name>.tibx > archive_info_log.txt

Step 1B: Get some details about the archive:

Relevant options/flags:

-s, --slice-list list slices in archive
--show-deleted show deleted slices
--show-complete-only show only complete slices
--show-hidden show hidden differential slices
--reversed-order list slices from latest to oldest

Command:
archive_ctl --slice-list --show-deleted --show-hidden --reversed-order --password <your_pass_for_the_archive> -f <full_absolute_path_to_archive>/<archive_name>.tibx > slice_list_all_log.txt

Step 2: Run FULL (metadata + user data + sizes/refs/linkage of segments/objects and pointers):

--validate-all operation will download (stream) all the archive data while validating; take this into consideration when validating a cloud archive

Local/network storage:

archive_ctl --validate-all --password <your_pass_for_the_archive> -f <full_absolute_path_to_archive>/<archive_name>.tibx > validate_all_log.txt

Cloud storage:

archive_ctl --validate-all --astor <host:port> --cert <certificate.crt> --password <pass_for_the_archive> -f /1/<archive_name>.tibx > validate_all_log.txt

Alternatively you may pipe all archive_ctl output to log file using combination of -d and --log switches (where -d stands for log level; 7 is max available log level). Having log file is useful as all output will be written to log that helps in troubleshooting

archive_ctl --validate-all --astor <host:port> --cert <certificate.crt> --password <pass_for_the_archive> -f /1/<archive_name>.tibx -d 7 --log validate.log

Step 3: Attempting to repair/fix (what can be fixed of) the archive:

Options to attempt fix:

--last-good find last good state in corrupted archive
--last-good=fix if last good state was found perform metadata validation and fix archive
--validate-all in addition to meta validate all data at last good archive state

Recommended:
Try fix with:
--last-good=fix --validate-all

Command:
archive_ctl --last-good --last-good=fix --validate-all --password <your_pass_for_the_archive> -f <full_absolute_path_to_archive>/CAV-APSERVER4.caverton.local-2286BA7F-3F51-436E-B35F-53653383AC60-8D76B665-0887-496B-B0D6-B04899B6821BA.tibx > try_fix_all_log.txt

NOTE: The syntax for the "last-good" family of commands is a bit special. You MUST start with --last-good as the 1st-level/leftmost switch (it tells the tool to search-for-consistent-points mode), and then with the 2nd-level --last-good=fix switch you tell the tool to go into rewrite mode. If you run "archive_ctl --last-good ..." or "archive_ctl --last-good --validate-all ..." this will perform a dry-run/read-only search of the archive for valid points (valid slices).

Step 4: run fsck / chkdsk to try to repair the underlying filesystem on which the network share resides

preferably with full-check/surface-check and free-space checks (as much as the tool allows)

NOTE: I did not recommend this as the first step, as this may further break or wipe the archive file - depending on many factors (what the underlying OS/FS are, what the fsck/chkdsk tool version does and how it behaves, what options are used for it...)

Step 5: REFRESH

After all repairs, REFRESH the storage location and REFRESH the list of slices (recovery points) in the archive that are shown in the Backup Console > Backup Storage web GUI.

Then try using the archive/restore points.

Further notes:

The archive_ctl CLI tool, as well as some other CLI tools which are very useful in many situations can be downloaded from: https://dl.acronis.com/u/kb/622.zip

NOTE: You may need to install / copy to the tool's unzipped directory the MS VisualC/C++ Runtime Library (MSVCRT120.dll).

MSVCR120.dll Missing - Microsoft Community
https://answers.microsoft.com/en-us/windows/forum/windows_10-windows_ins...

You can get the list of all options and flags which archive_ctl has by running: archive_ctl --help .

The options / arguments / flags, as of 2021-01-25, are:

Click to expand

C:\Users\victor\Downloads\archive_tool - Copy>archive_ctl --help
archive_ctl tool
Common options:
-f, --file <archive> path to archive file
-m, --mode=rewrite|append|readonly archive open mode
--page-size <bytes> set page size for newly created archives
--page-cache <bytes> set page cache size
--read-rate <bytes per second> read rate limit
--write-rate <bytes per second> write rate limit
--compression none|low|normal|high set default archive compression level
--autocommit[=0|1] disable/enable autocommit (default=on)
--autocommit-size <bytes> sets size of autocommit
--autocommit-delay <ms> sets size of delay of autocommit
--commit-fault-inj <num> sets number of failed commit
--punch-min <bytes> minimal size of hole to punch in bytes
--punch-ratio <0-999> how many space in 1/10 percent of allocated size must be unused to trigger punch hole procedure
--punch-threshold <bytes> how many space in bytes must be unused to trigger punch hole procedure
--split <bytes> set archive split size
--vstorage enable quirks for Acronis Storage
--disable-multivolumes don't split archive on multiple files
--tape-cache open tape cache from local directory. Archive path (-f) should point to directory with slices, file name is ignored.
Example: archive_ctl -f "./TapeLocation/5B5F42A3-9B77-447A-AE22-E8628E063262/foo.tibx" --tape-cache -i
--encr-alg aes-{128|192|256}-{gcm|cbc} set encryption algorithm
-P, --password [<username>:]<password> set encryption password
--keyfile <fname> file with public or private keys
--keyfile-ext <fname> file with public keys for extended security model
--io-latency <min>[-<max>] emulate io latency min-max milliseconds (default max=min)
--cache-level[=1|2|3] create <archive>.pcache file containing only pages with level >= cache-level (default=3)
--clone <path> clone all read pages into files in directory at path
--memory-limit <bytes> set total item cache limit
--network-timeouts <msec> set timeouts for network operations
--infinite-timeouts set infinite timeouts for I/O operations
--ignore-crc ignore page crc mismatch errors
--allow-upgrade allow archive version upgrade on write
--allow-write-mode-change allow mixing append and rewrite mode when writing archive
--no-direct don't use direct I/O with archive files
--sequential <path> perform read operations sequentially using map file in directory at path
--reuse-delay <seconds> set archive space reuse or deallocation delay
-q, --quiet do not show progress info (useful for scripts)
-h, --help print this help message
-v, --version print archive3 library version

Acronis cloud storage:
--astor <host:port> connect to an astorage gw at host:port
--cert <fname> use a client certificate in file fname
--jwt <jwt> use a json web token
--srv-cert-digest <digest> expected SHA-256 digest of server certificate
in format 09[:]af[:]AF...
--http-proxy <host>:<port> connect to a HTTP proxy at host:port
--http-proxy-user <username> proxy user name
--http-proxy-pass <password> proxy password
--astor-use-sni enable SNI when connecting to a gw
--astor-protos 1,2 list of allowed protocols
--astor-fault-inj=random_seed enable fault injection in libastor_client (=random_seed is optional)
--astor-latency-inj enable latency injection in libastor_client

Object storages:
--azure <option>=<value> connect to Azure object storage
--azure help show Azure options list
--google <option>=<value> connect to Google Cloud Storage
--google help show Google Cloud Storage options list
--s3 <option>=<value> connect to S3 object storage
--s3 help show S3 options list
--swift <option>=<value> connect to Swift object storage
--swift help show Swift options list
--onedrive <option>=<value> connect to OneDrive object storage
--onedrive help show OneDrive options list
--ostor-emulate emulate object storage
--cold-storage <min>:<max> set cold storage params: minimum and maximum day before today
--time-shift <msec> add given value to current time to simulate events in past/future

Deduplication:
--dedup enable data deduplication
--hash-width <num> width of the hash window used for hash-based chunking
in range [1; 4096], default 4096
--hash-step <num> step of window used for hash-based chunking, data
segments are aligned on step, default 4096
--hash-bits <num> number of bits of the hash used as a border condition
in chunking algorithms, the more the value the more the
expected segment size, must be in range [1; 16], default 4

Options to open archive with non-last CI:
Archive is opened scanning backward starting with offset set by options.
WARNING! Archive opened with the option in write mode is truncated to the size known in used header.
--open-at-volume <val> volume with target CI (optional if the archive has single volume)
--open-at-offset <val> offset after target CI end inside the volume (optional)

Archive statistics:
--show-io-stat show archive io statistics on exit
--show-lsm-stat show lsm performance counters on exit
--show-commit-stat show commit performance counters on exit
--show-write-stat show write performance counters on exit
--show-bottleneck show current performance stats and bottleneck
--show-alloc-stat show space allocation stats
--show-seg-stat show segments stats
--show-metrics show Prometheus metrics

Logging options:
-d, --debug-level <num> log level
-L, --log <file> log file
-R, --log-rotate 10x100K log file rotate parameters

Network login options:
--network-login network login name
--network-password network password

Slice operation commands:
-C, --slice-create=full|inc create new full or incremental slice

-F, --slice-finish finish current archive slice

-N, --new-chain=full|diff create new full of differential slice chain

--slice-cleanup cleanup invisible extents in slice

-D, --slice-rm <num> delete slice <num> from archive

--replicate-from <archive> replicate slices from another archive
-S, --slice-id <num> slice id to replicate (by default, all slices)
--dst-copy-encr-setup copy encryption setup from src archive
Common options (network and encryption settings) after --replicate-from
would be applied only to source archive.

Item operation commands:
-p, --put <name> put item <name> in archive
--src-file <fname> source file with data to write (by default, item name is used as source file name)
--offset <num> start position in source file (put cmd option)
--size <num> num of bytes to put (put cmd option)
--punch-hole punch a hole in range specified via --offset and --size (sparse)
--stream <num> stream id inside the item (0 by default)
--stream-size <num> stream size in bytes (by default, equal to source file size)
--not-complete don't complete item after data is written
--user-val set LSM user value of item only
--hash <hex> notarize item

-g, --get <fname> extract file <fname> from archive (use -o to specify destination)
-o, --output <file> specify destination file or folder for extracted archive files (- for stdout)
--sha1 calculate and print sha1 hash
--offset <num> start position in source stream (get cmd option)
--size <num> num of bytes to get (get cmd option)
-S, --slice-id <num> slice id to get specific file version
--stream <num> stream id inside the item (0 by default)
--user-val get LSM user value of item only

-r, --rm <fname> delete file <fname> from archive
--recursive delete item and it's children recursively
--stream <num> delete only stream id inside the item
--complete if stream id is specified, mark item as complete after stream delete

--undo <fname> undo all changes to item in current slice

-l, --list list items in archive
--start-name <name> starting name for items listing
-S, --slice-id <num> slice id to list items
--show-changes show changes in selected slice
--show-deleted show deleted items
--show-complete-only show only complete items
--parent-name <name> parent name for show children/direct children options
--show-children show only child names for the given starting name
--show-direct-children show only direct child names for the given starting name
--show-last-complete show only last complete items

--list-versions <name> list versions of given item name
--slice-id slice id from which to start listing
--show-deleted show deleted items
--show-complete-only show only complete items

--list-links list all names of item
--item-id <num> item id to list names
-S, --slice-id <num> slice id to list item names

--find-name <name> find last slice contaning item with given name
--min-slice-id <num> minimal slice id (optional)
--max-slice-id <num> maximal slice id (optional)

--show-stream-changes <name> list item stream changes between specified and last slices
-S, --slice-id <num> slice id to be compared with last slice
(if absent, show allocated regions in last slice)
--stream <num> stream id inside the item (0 by default)

--rename renames item <src-name> to <dst-name>
--src-name old item name (should belong to already finished slice)
--dst-name new item name in current slice

--link creates hardlink with name <dst-name> to item <dst-name>
--src-name source item name
--dst-name new item hardlink name

--commit commit archive changes

Notary operation commands

--notary-start start new notary tree
--notary-complete complete current notary tree
--notary-root get current notary tree root (must be completed)
--notarize <name> notarize item in the current notary tree
--hash <hex> hash of the item
--notary-proof <name> get notary proof of the item, tree where this item was notarized, must be completed

Archive operation commands
--list-archives <directory> list all archives in a given directory
--archive-rm <archive> delete archive
-i, --info print general archive information
--set <key>=<value> set archive meta (printed by --info)
--compact tries to move content and truncate file size
--resparse force punch holes on deallocated archive extents

Archive validation and fixup commands:
--validate-pages validate CRC of archive pages
--validate-trees validate CRC of tree pages
--validate-segments validate segments are readable, valid and do not overlap each other
-S, --slice-id <num> slice id to validate data (optional)
--validate-objects validate that all objects in object storage archive are present
--validate-hash validate segment hashes
--validate-refs validate that reference count of segments in data map matches segment map
--validate-orphan-exts validate if archive has orphan extents not refered to by any item name
--validate-user-size validate user size of all slices
--validate-notary[=name] validate notarization hash matches item(s) data
-S, --slice-id <num> slice id to validate data (optional)
--validate-all perform all validations
--clean-orphan clean orphan segments
--fix-user-size re-calculate and fix user size of all slices
--fix-segment-refs fix segment refs to match data map
--fix-space-usage fix unused and deallocated space counters
--last-good find last good state in corrupted archive
--last-good=fix if last good state was found perform metadata validation and fix archive
--validate-all in addition to meta validate all data at last good archive state
--no-validate skip validation
-c, --count <num> number of last states to validate (default=1)

Low-level debugging commands:
--dump-pages[=options] dump archive pages
--volume [start][:<end>] volume range. All volumes are used by default.
If end volume is not specified, only start volume will be used
and two additional options are available:
--offset <val> page offset (optional)
--length <val> range length (optional)
Dump pages options (you can specify multiple options separated by comma):
page dump page headers (default)
ci show commit info
fixed show fixed header part
hdr show full archive header info
lsm show lsm headers
segment show segment headers
golomb show golomb headers
all show all page types info

--dump-headers dump archive headers

--dump-unused dump unused extents
--show-deleted show deleted (re-used) extents
--offset <val> range offset (optional)
--length <val> range length (optional)

--dump-data-map dump data extents
-S, --slice-id <num> slice id to dump extents (optional)
--item-id <num> item id to dump extents (optional)
--show-changes show changes in selected slice
--show-deleted show deleted items
--show-complete-only show only complete items

--dump-notary dump notary map

--dump-segments dump data segments
--get-segment get segment payload
--offset <val> segment offset
--length <val> segment length
-o, --output <file> specify destination file for segment data (- for stdout)
--raw don't decompress segment data (optional)
--query-segment <hash> query segment hash in dedup map

--dump-ctrees[=name] dump number of extents and size of lsm ctree(s)
--verbose dump all ctree extents (optional)

--dump-sequential[=name] dump sequential map of item(s)
-S, --slice-id <num> slice id to list items
--stream <num> max stream id to include into sequential map (default=all)

Other options:
--script <fname> file with list of archive commands (one command per line)
--add-password [<username>:]<password> add encryption password
--sleep <seconds> wait specified number of seconds

==============

Example read-only/dry-run:

Click to expand

D:\archive_tool - Copy>archive_ctl --last-good --validate-all -f D:\rec-deb-lvm01-03F771E2-E485-42EB-B576-F61D7FFC79CF-52517B7E-CFB5-2C04-6144-279225C334C0A.tibx
2021-02-18 11:02:02.384 using default page cache size 134217728
2021-02-18 11:02:02.386 Searching last 1 good archive state(s)...
ar#1: ci: commit_seq=3 open-at-offs=793985024 open-at-vol=0
2021-02-18 11:02:02.429 ar#1: invalid header at 793976832 in vol 0 (buffer 4186112/4194304)
2021-02-18 11:02:02.430 ar#1: archive has 1 volume, chain starts at 0
ar#1: hdr found: commit_seq=3 open-at-vol=0 open-at-offs=793985024 hdr=793972736
ar#1: ci: commit_seq=2 open-at-offs=793972736 open-at-vol=0
2021-02-18 11:02:02.432 ar#1: invalid header at 793964544 in vol 0 (buffer 4173824/4194304)
ar#1: hdr ignored: commit_seq=2 open-at-vol=0 open-at-offs=793972736 hdr=793960448
ar#1: ci: commit_seq=1 open-at-offs=793350144 open-at-vol=0
2021-02-18 11:02:02.433 archive_ctl: read: 4194304, written: 0, 0.5%
2021-02-18 11:02:03.442 archive_ctl: read: 150994944, written: 0, 19.0%
2021-02-18 11:02:04.455 archive_ctl: read: 301989888, written: 0, 38.0%
2021-02-18 11:02:05.458 archive_ctl: read: 390070272, written: 0, 49.1%
2021-02-18 11:02:06.469 archive_ctl: read: 536870912, written: 0, 67.6%
2021-02-18 11:02:07.492 archive_ctl: read: 687865856, written: 0, 86.6%
2021-02-18 11:02:08.218 ar#1: invalid header at 16384 in vol 0 (buffer 16384/1261568)
ar#1: ci: commit_seq=0 open-at-offs=12288 open-at-vol=0
2021-02-18 11:02:08.222 ar#1: invalid header at 4096 in vol 0 (buffer 4096/1261568)
2021-02-18 11:02:08.226 archive_ctl: read: 793985024, written: 0, 100.0%
2021-02-18 11:02:08.229 ar#1: found 4 hdrs, 0 have no ci, 2 unused ci
2021-02-18 11:02:08.231 Found 1 archive state(s)...

Checking ver 3...
2021-02-18 11:02:08.240 ar#0: enable autocommit
2021-02-18 11:02:08.243 ar#1: opening archive path="\\?\D:\/rec-deb-lvm01-03F771E2-E485-42EB-B576-F61D7FFC79CF-52517B7E-CFB5-2C04-6144-279225C334C0A.tibx" in readonly mode
2021-02-18 11:02:08.254 ar#1: ci page found at 0x2f533000
2021-02-18 11:02:08.257 ar#1: opened archive path="\\?\D:\/rec-deb-lvm01-03F771E2-E485-42EB-B576-F61D7FFC79CF-52517B7E-CFB5-2C04-6144-279225C334C0A.tibx" mode=readonly reopen=0:0 (commit_seq=3, reuse_seq=3, file_size=793985024, uuid=9a604e7e030a1a35efa6729591f4aa1c)
2021-02-18 11:02:08.260 Validating trees...
2021-02-18 11:02:08.263 archive_ctl: read: 0, written: 0, 0.7%
2021-02-18 11:02:08.285 archive_ctl: read: 0, written: 0, 100.0%
2021-02-18 11:02:08.288 ar#1: Successfully validated 3 ctrees, 528KB total
2021-02-18 11:02:08.289 Validating segments...
2021-02-18 11:02:08.395 archive_ctl: read: 798720, written: 0, 0.1%
2021-02-18 11:02:09.403 archive_ctl: read: 146812928, written: 0, 18.5%
2021-02-18 11:02:10.408 archive_ctl: read: 231518208, written: 0, 29.1%
2021-02-18 11:02:11.413 archive_ctl: read: 377307136, written: 0, 47.5%
2021-02-18 11:02:12.420 archive_ctl: read: 456081408, written: 0, 57.5%
2021-02-18 11:02:13.440 archive_ctl: read: 557875200, written: 0, 70.3%
2021-02-18 11:02:14.449 archive_ctl: read: 622780416, written: 0, 78.5%
2021-02-18 11:02:15.459 archive_ctl: read: 687869952, written: 0, 86.7%
2021-02-18 11:02:16.431 archive_ctl: read: 793182208, written: 0, 100.0%
2021-02-18 11:02:16.432 ar#1: Successfully validated 8577 data segments
2021-02-18 11:02:16.433 ar#1: seg=793.18 MB avg=90.31 KB user=2088.24 MB ratio=36.98% pad=19064355 (2.40%)

Successfully found last good state in archive
2021-02-18 11:02:16.434 ar#1: archive closing
2021-02-18 11:02:16.435 ar#1: archive close (commit_seq=3, reuse_seq=3, file_size=793985024, uuid=9a604e7e030a1a35efa6729591f4aa1c)
2021-02-18 11:02:16.436 archive cache: total (7111/26MB) == active (186/0MB) + inactive (6925/27MB) + decommit (0/0MB) + locked (0/0MB) + priv (101MB) <= limit (128MB)
2021-02-18 11:02:16.437 archive cache: locked (0/0MB) == user (0/0MB) + rd (0/0MB) + wr (0/0MB), error (0/0MB)
2021-02-18 11:02:16.438 archive cache: pool free (3379/13MB) decommitted (9434/36MB)
2021-02-18 11:02:16.439 archive cache: recommitted (0/0MB) lost (0/0MB) limit 12224MB

Knowledge Base

67951: Effective validation and repair of TIBX/Archive3/Format12 backup archives using CLI tool archive_ctl

use Google Translate

Applies to: