71091: Acronis Cyber Protect: Backup to QNAP NAS may fail at the very start of the backup

use Google Translate

    Last update: 09-11-2022

    Symptoms

    Backups to QNAP NASes may fail at the very start of the backup with high-level user-visible errors like "error 0x29b1399: Network operation failed" (at the bottom of the detailed error stack, when viewed in the Acronis Backup Console UI).

    It is possible that some backups to the same NAS work fine, while only a few or even one backup/backup plan fails.

    Most often creating a new backup plan and using a new TIBX does NOT help.

    Details

    On the Agent, you may see the error:

    | error 0x1350016: TOL: Failed to execute the command. Backing up
    | line: 0x8d165e86fb8195bd
    | file: d:\366\enterprise\common\tol\command\command.cpp:495
    | function: Tol::`anonymous-namespace'::MakeFailResult
    | CommandID: 8F01AC13-F59E-4851-9204-DE1FD77E36B4
    | $module: gtob_backup_command_addon_vsa64_30025
    |
    | error 0x1490003: Backup has failed.
    | line: 0x1cd98aae889424f9
    | file: d:\366\enterprise\managers\gtob\util\impl\convert_batch_result.cpp:58
    | function: Gtob::Backup::ConvertBatchResult
    | $module: disk_bundle_vsa64_30025
    |
    | error 0x10424: Failed to prepare an operation.
    | line: 0x145a191bfe9050c1
    | file: d:\366\processor\batch\batch.cpp:108
    | function: Processor::BatchOperation::Execute
    | $module: disk_bundle_vsa64_30025
    |
    | error 0x40015: Network disconnected.
    | line: 0x8af64b2c0920f802
    | file: d:\366\core\resizer\archive3\archive3_error.cpp:196
    | function: `anonymous-namespace'::ConvertArchive3Error
    | $module: disk_bundle_vsa64_30025
    |
    | error 0x29b1399: Network operation failed
    | line: 0xc3bc4c1eb2299bf9
    | file: d:\366\enterprise\applications\archiving\storages\archive3\src\archive3.cpp:591
    | function: `anonymous-namespace'::CoOpenArchive
    | function: archive_open
    | path: agentmachine.company.local-33D9766C-29E9-4FD0-B45F-138C466AEC91-F49A5FB2-F0DD-4217-9519-90ECFC97F2F3A.tibx
    | $module: disk_bundle_vsa64_30025
     

    In the PCS log of the agent (C:\ProgramData\Acronis\BackupAndRecovery\MMS\pcs.0.log.gz), there are entries like this:

    2022-07-18T13:27:58:450+03:00 24920 I00000000: service_process(20212): ar#1: opening archive path="\\?\UNC\the_QNAP_NAS\some_backup_name.tibx" in rewrite mode (create)
    2022-07-18T13:27:58:546+03:00 24920 I00000000: service_process(20212): pcs_co_file_getfsize("\\?\UNC\the_QNAP_NAS\some_backup_name.tibx") failed: 59 (pcs_err=4)
    2022-07-18T13:27:58:546+03:00 24920 E00000000: service_process(20212): ar#1: failed to open archive path="\\?\UNC\the_QNAP_NAS\some_backup_name.tibx" mode=rewrite uuid=00000000000000000000000000000000, err=-5017 (Network operation failed)

    More detailed symptoms, when viewing a packet trace in Wireshark, show that the NAS side resets the TCP connection on SMB port 445 immediately after the backup agent requests the NAS to enable parseness on the TIBX file.

    Setting/requesting sparse mode on a file is a standard part of the SMB protocol, therefore no correct implementation of it should crash - the SMB server can return an error code if it does not support sparse files for some reason (i.e. if the underlying filesystem the share/NAS uses doesn't support sparse files, which would be extremely rare, as in 2022 practically all filesystems used on NASes and on any fileserver computers do support sparse files).

    Inspection of some of the logs of the QNAP NAS itself show crashes of the samba daemon, which is the the server process providing SMB protocol support.

    ===============================================================
    [2022/07/25 09:38:29.150627,  0] ../../lib/util/fault.c:163(smb_panic_log)
      INTERNAL ERROR: Signal 11: Segmentation fault in pid 17745 (4.13.17)
    [2022/07/25 09:38:29.150640,  0] ../../lib/util/fault.c:168(smb_panic_log)
      If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
    [2022/07/25 09:38:29.150665,  0] ../../lib/util/fault.c:169(smb_panic_log)
      ===============================================================
    [2022/07/25 09:38:29.150676,  0] ../../lib/util/fault.c:171(smb_panic_log)
      PANIC (pid 17745): Signal 11: Segmentation fault in 4.13.17
    [2022/07/25 09:38:29.150945,  0] ../../lib/util/fault.c:275(log_stack_trace)
      BACKTRACE: 4 stack frames:
       #0 /usr/local/samba/lib/libsamba-util.so.0(log_stack_trace+0x2b) [0x7fc137e212fb]
       #1 /usr/local/samba/lib/libsamba-util.so.0(smb_panic+0x24) [0x7fc137e21574]
       #2 /usr/local/samba/lib/libsamba-util.so.0(+0x18774) [0x7fc137e21774]
       #3 /lib/libpthread.so.0(+0x106f0) [0x7fc1344916f0]
    [2022/07/25 09:38:29.151034,  0] ../../source3/lib/dumpcore.c:315(dump_core)
      dumping core in /var/log/cores/smbd

    Note: There can be slight variations across different NAS models, different versions of the firmware, different versions of the samba daemon and its libraries, etc.

    Cause

    Issue in the implementation and version of the samba daemon which the QNAP NAS firmware ships.

    Solution

    A firmware update was published by QNAP around mid-August 2022, which resolves the samba crashes, thus solves the problem.

    Applicability to other NAS (non-QNAP)

    The issue MAY be applicable to other, non-QNAP NASes. Similar samba implementations and versions are used on e.g. Synology units, and on those of other NAS manufactures. Usually, the samba software component on NASes is not the most up-to-date (compared to the "upstream" versions which the SAMBA project publishes, and compared to what most current Linux distributions (Debian, Ubuntu, RedHat-derived ones ...) ship with in the package repositories - these usually track the "upstream" faster and closer than what NAS manufacturers use). This means that sometimes it takes a fairly long time for bugfixes and stability improvements to propagate and to be available in a NAS firmware update for your particular NAS  brand and model. (There's also the complication that not all NAS models receive firmware updates at the same time even for the same manufacturer, and that not all manufacturers include fixes at the same pace, plus different manufacturers usually publish firmware updates at different frequencies.)

    Tags: