The stack trace in your log indicates that the error did actually occur in a Rebex library. The error was raised by Cryptware NCryptoki library's RSA provider, which you plugged into Rebex SFTP to perform RSA operations.
at Cryptware.NCryptoki.NCryptokiRSAProvider.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
at Rebex.Security.Cryptography.CryptoHelper.dhnn(RSA bkb, Byte[] bkc, String bkd)
at Rebex.Security.Cryptography.CryptoHelper.dhnm(RSA bjy, Byte[] bjz, String bka)
at pnxh.ycpm(Byte[] gwy, pnur gwz)
at Rebex.Security.Cryptography.AsymmetricKeyAlgorithm.SignMessage(Byte[] message, SignatureParameters parameters)
at Rebex.Net.SshPrivateKey.iudu(Byte[] adr, String ads, Boolean adt)
at tern..ctor(String st, String su, Byte[] sv, SshPrivateKey sw, String sx, Boolean sy, Boolean sz)
at Rebex.Net.SshSession.ayoe(String ahb, String ahc, SshPrivateKey ahd, SshGssApiCredentials ahe, Boolean ahf)
at Rebex.Net.SshSession.Authenticate(String userName, String password, SshPrivateKey privateKey)
This means that "System.NotSupportedException: Algorithm 2.16.840.1.101.3.4.2.3 is not supported" error comes from NCryptoki's NCryptokiRSAProvider.SignHash method.
The OID of 2.16.840.1.101.3.4.2.3 represents the SHA-512 hash algorithm. Along with the rest of the log, this indicates that Rebex SFTP attempted to create an RSA/SHA-512 signature in order to authenticate using a key, but NCryptoki probably does not support this algorithm.
To resolve this, either resolve this NCryptoki issue (uprade their library or contact Cryptware support), or instruct Rebex SFTP to use another signature algorithm that is supported by your version of NCryptoki library.
However, it looks like you are using an older version of Rebex SFTP library, which does not make this directly configurable. Instead, try disabling "rsa-sha2-512" host key algorithm, which will disable key authentication using that algorithm as well:
var sftp = new Sftp();
...
string[] algs = SshParameters.GetSupportedHostKeyAlgorithms().Where(alg => alg != "rsa-sha2-512").ToArray();
sftp.Settings.SshParameters.SetHostKeyAlgorithms(algs);
...
sftp.Connect(host, portNumber);