Improve performance of GetFiles with a wildcard on a directory with a large number of files

I am noticing that GetList is somewhat slow on directories that contain a large amount of files. I am currently getting a list of how many matches there are in the remote directory before i decide weather to do multiple file transfer or a single file transfer inside of a for loop.

Is there a way to combine the GetList (with a wildcard) with a Linq expression containing a sort and limit on how many items to return?

Is there any other way to improve performance?

Here is the code

//Check if remote directory exists and has at least one file
fileCount = client.GetList(remoteSearch).Count;
if (client.DirectoryExists(remoteFolder) && fileCount > 0)
{
     //Check if there are more than 20 files to download
     if (fileCount > 20)
     {
          //download oldest 20 files in folder that match criteria                                                                                                
          var sftpItems = client.GetList(remoteSearch).Cast<SftpItem>()
               .Where(w => w.IsFile)
               .OrderBy(o => o.LastWriteTime)
               .Take(20);

          foreach (SftpItem item in sftpItems)
          {
               try
               {
                    client.Download(item.Path, tempDirectory, TraversalMode.MatchFilesShallow,                                TransferMethod.Move, ActionOnExistingFiles.OverwriteAll);
               }
               catch (Exception ex)
               {
                    throw ex;
               }                            
          }
     }
     else
     {
          //download all files in folder that match criteria                        
          client.Download(remoteSearch, tempDirectory, TraversalMode.MatchFilesShallow,
                        TransferMethod.Move, ActionOnExistingFiles.OverwriteAll);                        
     }                      
}