This looks like something very strange is going on. In the SFTP protocol (and therefore Rebex File Server), files are transfered in multiple chunks of several tens of kilobytes in length each, which means that memory usage should be virtually the same when transferring a 1 MB or 20 GB file, and it should stay pretty much the same for the duration of the transfer.
To make sure this is indeed the case with Rebex File Server, I launched our GUI File Server - FileServerWinForm sample and observed the memory usage using Windows Task Manager. As expected, the memory usage stayed almost constant the whole time, with the working set staying around 34 MB and private working set staying around 10 MB.
Could you please give this sample app a try and let us know whether your results differ in any way? Or supply an application that reproduces the issue? Thanks!