The Rebex error message indicates that the key algorithm string, received in the encoded SSH public key, is incorrect. For ed25519, it should be "ssh-ed25519", but that was apparently not the case.
A log from Rebex Ssh object at LogLevel.Verbose would contain the raw SSH public key and make it possible to tell whether this is actually the case. However, we have no idea how to create such logs with RoyalTS.
Update: According to the verbose log, this is indeed caused by a server-side bug.
The log shows that when "rsa-sha2-512" host key algorithm has been negotiated, the server responds with a wrongly-encoded SSH public key that identifies the key format as "rsa-sha2-512" instead of "ssh-rsa". That violates the specification in RFC 8332 which says:
Since RSA keys are not dependent on the choice of hash function, the
new public key algorithms reuse the "ssh-rsa" public key format as
defined in [RFC4253]:
string "ssh-rsa"
mpint e
mpint n
And then it repeats that requirement once more:
If one of the two host key algorithms is
negotiated, the server sends an "ssh-rsa" public key as part of the
negotiated key exchange method (e.g., in SSH_MSG_KEXDH_REPLY) and
encodes a signature with the appropriate signature algorithm name --
either "rsa-sha2-256" or "rsa-sha2-512"But the server you are trying to connect to uses "rsa-sha2-512" for both the public key and the signature algorithm name.
Therefore, this is clearly a server-side issue. If the current version of the server is still affected, it should be reported to the server vendor. As a temporary workaround, disable "rsa-sha2-256" and "rsa-sha2-512" host key algorithms with affected servers.