Symptoms

1. Attempt to update a cluster from 3.0.5-69 to 3.5.x version failed with 'Internal error' in UI and shortly after that cluster nodes started to be displayed as 'Offline' and cluster state is 'Unavailable' on main WebCP dashboard.

2. The following errors can be seen in logs on the Management Node revealing issues with ui-backend certificate:

/var/log/vstorage-ui-backend/software-updates.log:

ERROR 2020-04-22 08:52:16,104 r-53d0ef6b8f2fca95 backend/business/models/software_updates/managers.py:56:PrepareUpdateManager.prepare Failed to prepare to update: Traceback (most recent call last):
...
backend.business.models.certificates.exceptions.CertificateException: Cannot generate certificate: touch: cannot touch ‘/usr/libexec/vstorage-ui-backend/.rnd’: Permission denied
chown: changing ownership of ‘/usr/libexec/vstorage-ui-backend/.rnd’: Operation not permitted

/var/log/vstorage-ui-backend/celery_periodic.log:

[2020-05-20 21:30:02,366: ERROR/MainProcess] Task backend.tasks.stat.collect[3a065557-6e08-4594-9651-042297f881cf] raised unexpected: Error([('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'PEM lib')],)

/var/log/vstorage-ui-backend/gen_certificate.log:

writing new private key to '/usr/libexec/vstorage-ui-backend/ca/client.key'
-----
Using configuration from /usr/libexec/vstorage-ui-backend/ca/conf/ca.conf
/usr/libexec/vstorage-ui-backend/ca/conf/serial: No such file or directory
error while loading serial number
140122305857424:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/usr/libexec/vstorage-ui-backend/ca/conf/serial','r')
140122305857424:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to write 'random state'
Cannot create x509 certificate
[ERROR][30320] 2020-04-22 08:52:47,122 Cannot create x509 certificate

3. File /usr/libexec/vstorage-ui-backend/.rnd can be owned by root user:

[root@node01 ~]# ll /usr/libexec/vstorage-ui-backend/.rnd
-rw------- 1 root root 1024 May 21 00:47 /usr/libexec/vstorage-ui-backend/.rnd

4. File /usr/libexec/vstorage-ui-backend/ca/conf/serial is absent on node:

[root@node01 ~]# ll /usr/libexec/vstorage-ui-backend/ca/conf/serial
ls: cannot access /usr/libexec/vstorage-ui-backend/ca/conf/serial: No such file or directory

Cause

Issue in the previous versions of the product. Incorrect permissions of the file /usr/libexec/vstorage-ui-backend/.rnd and/or absent file /usr/libexec/vstorage-ui-backend/ca/conf/serial causing issue with UI backend certificate generation upon start of cluster update to 3.5.x version.

Solution

The issue has been fixed in 3.0 Update 5.1.

To fix the issue for the older versions follow the steps below to re-generate certificate manually; they should be executed on the node currently having Management Node role:

[root@node01 ~]# chown vstoradmin:vstoradmin /usr/libexec/vstorage-ui-backend/.rnd
[root@node01 ~]# openssl rand -hex 8 > "/usr/libexec/vstorage-ui-backend/ca/conf/serial"
[root@node01 ~]# sh /usr/libexec/vstorage-ui-backend/libexec/gen-certificate.sh -m
[root@node01 ~]# systemctl restart vstorage-ui-backend
[root@node01 ~]# systemctl restart nginx

After that it should be possible to manage cluster in UI and re-attempt upgrade.