This project is read-only.

SFtp Streaming

Mar 6, 2011 at 2:03 AM

To take advantage of reading large amounts of data from another server, it would be helpful if user-code could pull the data and process it.

For it to do this, it would be beneficial to "Open" a remote file, thus returning a Stream object, much like the File.Open methods, or generally all file processing in .NET.

With this user-code can easily stream information directly between the application and the server without any push-pull middle buffer or local file on disk.

The best way I can see to get similar results is to pass a MemoryStream object to the Sftp.Get methods, and hope that I can process the memory stream faster than SSH.NET can read it in from the server, or maybe writing my own DownloadFileCommand.

Coordinator
Mar 6, 2011 at 2:12 AM

Hi,

 

Thanks you for suggestion and I thought about doing that but the way protocol work its not exactly possible as it comes encrypted.

So I always need to get a block, decrypt it, then, in case of SFTP extract SFTP message from SSH message and only then I push it into the stream, which is provided by the client.

I think I dont have too many SFTP examples but this is how I was testing it:

            sftp.Connect();

            var mem = File.OpenRead("C:\\Test\\TestFile0.bin");
            var mem1 = File.OpenRead("C:\\Test\\TestFile1.bin");
            var mem2 = File.OpenRead("C:\\Test\\TestFile2.bin");
            var mem3 = File.OpenRead("C:\\Test\\TestFile3.bin");
            var mem4 = File.OpenRead("C:\\Test\\TestFile4.bin");
            var asynch = sftp.BeginUploadFile(mem, "/home/oleg/test1.txt", null, null);
            var asynch1 = sftp.BeginUploadFile(mem1, "/home/oleg/test2.txt", null, null);
            var asynch2 = sftp.BeginUploadFile(mem2, "/home/oleg/test3.txt", null, null);
            var asynch3 = sftp.BeginUploadFile(mem3, "/home/oleg/test4.txt", null, null);
            var asynch4 = sftp.BeginUploadFile(mem4, "/home/oleg/test5.txt", null, null);

            var sftpASynch = asynch as SftpAsyncResult;
            var sftpASynch1 = asynch1 as SftpAsyncResult;
            var sftpASynch2 = asynch2 as SftpAsyncResult;
            var sftpASynch3 = asynch3 as SftpAsyncResult;
            var sftpASynch4 = asynch4 as SftpAsyncResult;
            while (!sftpASynch.IsCompleted && !sftpASynch1.IsCompleted && !sftpASynch2.IsCompleted && !sftpASynch3.IsCompleted && !sftpASynch4.IsCompleted)
            {
                Console.Write(string.Format("\rUploaded {0:#########} KB Uploaded {1:#########} KB Uploaded {2:#########} KB Uploaded {3:#########} KB Uploaded {4:#########} KB", (sftpASynch.UploadedBytes / 1024), (sftpASynch1.UploadedBytes / 1024), (sftpASynch2.UploadedBytes / 1024), (sftpASynch3.UploadedBytes / 1024), (sftpASynch4.UploadedBytes / 1024)));
                Thread.Sleep(100);
            }

            sftp.EndUploadFile(asynch);
            sftp.EndUploadFile(asynch1);
            sftp.EndUploadFile(asynch2);
            sftp.EndUploadFile(asynch3);
            sftp.EndUploadFile(asynch4);

            mem = File.Create("C:\\Test\\TestFile0_out.bin");
            mem1 = File.Create("C:\\Test\\TestFile1_out.bin");
            mem2 = File.Create("C:\\Test\\TestFile2_out.bin");
            mem3 = File.Create("C:\\Test\\TestFile3_out.bin");
            mem4 = File.Create("C:\\Test\\TestFile4_out.bin");

            asynch = sftp.BeginDownloadFile("/home/oleg/test1.txt", mem, null, null);
            asynch1 = sftp.BeginDownloadFile("/home/oleg/test2.txt", mem1, null, null);
            asynch2 = sftp.BeginDownloadFile("/home/oleg/test3.txt", mem2, null, null);
            asynch3 = sftp.BeginDownloadFile("/home/oleg/test4.txt", mem3, null, null);
            asynch4 = sftp.BeginDownloadFile("/home/oleg/test5.txt", mem4, null, null);

            sftpASynch = asynch as SftpAsyncResult;
            sftpASynch1 = asynch1 as SftpAsyncResult;
            sftpASynch2 = asynch2 as SftpAsyncResult;
            sftpASynch3 = asynch3 as SftpAsyncResult;
            sftpASynch4 = asynch4 as SftpAsyncResult;
            Console.WriteLine("Downloading");
            while (!sftpASynch.IsCompleted && !sftpASynch1.IsCompleted && !sftpASynch2.IsCompleted && !sftpASynch3.IsCompleted && !sftpASynch4.IsCompleted)
            {
                Console.Write(string.Format("\rDownloaded {0:#########} MB Downloaded {1:#########} MB Downloaded {2:#########} MB Downloaded {3:#########} MB Downloaded {4:#########} MB", (sftpASynch.DownloadedBytes / 1024), (sftpASynch1.DownloadedBytes / 1024), (sftpASynch2.DownloadedBytes / 1024), (sftpASynch3.DownloadedBytes / 1024), (sftpASynch4.DownloadedBytes / 1024)));
                Thread.Sleep(100);
            }

            sftp.EndDownloadFile(asynch);
            sftp.EndDownloadFile(asynch1);
            sftp.EndDownloadFile(asynch2);
            sftp.EndDownloadFile(asynch3);
            sftp.EndDownloadFile(asynch4);

            mem.Close();
            mem1.Close();
            mem2.Close();
            mem3.Close();
            mem4.Close();

            Console.WriteLine("Checksums");
            for (int i = 0; i < 5; i++)
            {
                var name = string.Format("C:\\Test\\TestFile{0}_out.bin", i);
                var md5 = GetMD5HashFromFile(name);

                Console.WriteLine("name: {0}, checksum: {1}", name, md5);
            }
            
            sftp.Disconnect();

Hope it helps, also, please download latest changeset, I didnt publish it yet but I improved performance there for upload and download.

 

Thanks,

Oleg