When "Guest OS is not supported" error appears while creating an application-aware backup task, make sure that the .vmx of the backed up VM contains proper guestOS value which corresponds the real guest OS installed.