This project is read-only.

Heisenbug when uploading over sftp

Jul 23, 2012 at 10:30 PM
Edited Jul 24, 2012 at 5:14 PM

I am using the latest version (17229, Jun 18) and Visual Studio 2010 Ultimate, targeting .net 4.

When I run my application with the debugger attached everything works as expected and my file is uploaded successfully.

When I run my application without the debugger (hitting ctrl-f5) it will connect and begin the upload, but it will stop after ~1MB. The stopping point is consistent to the byte and varies based on the buffer size.

I get this exception: "Exception: A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied "

relevant code:

 

        public void UploadRecordFile(string sftp_host, string sftp_user, string sftp_pass, int sftp_port)
        {
            Console.WriteLine("Attempting to upload file. Any errors will result in a run-time exception and the file may or may not be uploaded.");
            Renci.SshNet.SftpClient sftpclient = new Renci.SshNet.SftpClient(sftp_host, sftp_port, sftp_user, sftp_pass);
            sftpclient.Connect();                        
            System.IO.FileStream stmStreamToImage = System.IO.File.OpenRead(this.SaveFile);
            sftpclient.UploadFile(stmStreamToImage, "/TARGET");            
            sftpclient.Disconnect(); 
        }
It may be worth noting that it works just fine on my localhost sftp server, but it fails on the slower remote server.
Jul 25, 2012 at 8:26 PM
Edited Jul 25, 2012 at 8:41 PM

Running my program with the catch removed and no debugger attached, I got the following data when it crashed:

 

System.Net.Sockets.SocketException was unhandled
  Message=A request to send or receive data was disallowed because the socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied
  Source=System
  ErrorCode=10057
  NativeErrorCode=10057
  StackTrace:
       at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
       at Renci.SshNet.Session.SocketWrite(Byte[] data) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\Session.NET.cs:line 134
       at Renci.SshNet.Session.SendMessage(Message message) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\Session.cs:line 732
       at Renci.SshNet.Channels.Channel.SendMessage(ChannelDataMessage message) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\Channels\Channel.cs:line 455
       at Renci.SshNet.Channels.Channel.SendData(Byte[] buffer) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\Channels\Channel.cs:line 212
       at Renci.SshNet.Sftp.SubsystemSession.SendData(Byte[] data) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\SubsystemSession.cs:line 105
       at Renci.SshNet.Sftp.SftpSession.SendMessage(SftpMessage sftpMessage) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\Sftp\SftpSession.cs:line 80
       at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\Sftp\SftpSession.cs:line 255
       at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 offset, Byte[] data, EventWaitHandle wait) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\Sftp\SftpSession.cs:line 406
       at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, SftpUploadAsyncResult asynchResult, Flags flags) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\SftpClient.cs:line 1293
       at Renci.SshNet.SftpClient.UploadFile(Stream input, String path, Boolean canOverride) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\SftpClient.cs:line 525
       at Renci.SshNet.SftpClient.UploadFile(Stream input, String path) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\Renci.SshNet\SftpClient.cs:line 499
       at MyProejct.Worker.UploadRecordFile(String sftp_host, String sftp_user, String sftp_pass, Int32 sftp_port) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\MyProject\Worker.cs:line 156
       at MyProject.Program.Main(String[] args) in C:\Users\username\documents\visual studio 2010\Projects\MyProject\MyProject-console\Program.cs:line 156
  InnerException: 

 

Program.cs:156 calls the method in my first post

Worker.cs:156 is the sftpclient.UploadFile(stmStreamToImage, "/TARGET") call

Coordinator
Aug 10, 2012 at 2:56 PM

Hmm,

 

It sounds that server somehow drops connection, but it should be caught somewhere else gracefully, so I assume there is another problem that I probably didnt handle.

 

Does this problem exists if you connect to different server/ server versions or it consistent regardless of the server you connecting to?

 

Sorry for late response, simply busy with other project and dont have much time for this one lately.

 

Thanks,

Oleg

Aug 10, 2012 at 3:30 PM

The problem only exists on this one server, when running without the debugger.

Clients tested: Only my dev machine

Servers tested: Coreftp on my dev machine, remote production server

The production server is controlled by an outside company and I have no information on it. I can't even SSH in to poke around("Server refused to allocate pty"). SFTP works with filezilla and another sftp .net library.

Coordinator
Aug 10, 2012 at 7:26 PM

One guess, while it works in debugger and now when you run it is because a synch issue, so this server probably sends messages too fast and doesn't have time to process them in real time.

Another idea is that internal buffer that used to hold incoming data before is filled in faster then it can be read from.

So one thing to try, in Session.cs file, where it sends messages to the server, try to put it to sleep for like 200 or 500 ms and see if it works at a run time. This is to test that theory.

 

What happens during SFTP, it can send multiple messages without receiving confirmation that previous messages was received, this is to improve transfer speed and eliminate need to wait for package confirmation. So now when I think about it, it also could be that when you upload file it issues multiple SFTP WRITE requests, so may be what happens, your server cannot handle more then 1MB of outstanding WRITE request and they need to be queued.

One way to do it is to try modify RequestWrite method in SftpSession.cs file like this:

Instead of:

        internal void RequestWrite(byte[] handle, UInt64 offset, byte[] data, EventWaitHandle wait)
        {
            var maximumDataSize = 1024 * 32 - 38;

            if (data.Length < maximumDataSize + 1)
            {
                var request = new SftpWriteRequest(this.NextRequestId, handle, offset, data,
                    (response) =>
                    {
                        if (response.StatusCode == StatusCodes.Ok)
                        {
                            if (wait != null)
                                wait.Set();
                        }
                        else
                        {
                            this.ThrowSftpException(response);
                        }
                    });

                this.SendRequest(request);

                if (wait != null)
                    this.WaitHandle(wait, this._operationTimeout);
            }
            else
            {
                var block = data.Length / maximumDataSize + 1;

                for (int i = 0; i < block; i++)
                {
                    var blockBufferSize = Math.Min(data.Length - maximumDataSize * i, maximumDataSize);
                    var blockBuffer = new byte[blockBufferSize];

                    Buffer.BlockCopy(data, i * maximumDataSize, blockBuffer, 0, blockBufferSize);

                    var request = new SftpWriteRequest(this.NextRequestId, handle, offset + (ulong)(i * maximumDataSize), blockBuffer,
                        (response) =>
                        {
                            if (response.StatusCode == StatusCodes.Ok)
                            {
                                if (wait != null)
                                    wait.Set();
                            }
                            else
                            {
                                this.ThrowSftpException(response);
                            }
                        });

                    this.SendRequest(request);

                    if (wait != null)
                        this.WaitHandle(wait, this._operationTimeout);
                }
            }
        }

Make it :

        internal void RequestWrite(byte[] handle, UInt64 offset, byte[] data, EventWaitHandle wait)
        {
            var maximumDataSize = 1024 * 32 - 38;

            if (data.Length < maximumDataSize + 1)
            {
                var request = new SftpWriteRequest(this.NextRequestId, handle, offset, data,
                    (response) =>
                    {
                        if (response.StatusCode == StatusCodes.Ok)
                        {
                            if (wait != null)
                                wait.Set();
                        }
                        else
                        {
                            this.ThrowSftpException(response);
                        }
                    });

                this.SendRequest(request);

                if (wait != null)
                    this.WaitHandle(wait, this._operationTimeout);
            }
            else
            {
                var block = data.Length / maximumDataSize + 1;

                for (int i = 0; i < block; i++)
                {
                    var blockBufferSize = Math.Min(data.Length - maximumDataSize * i, maximumDataSize);
                    var blockBuffer = new byte[blockBufferSize];

                    Buffer.BlockCopy(data, i * maximumDataSize, blockBuffer, 0, blockBufferSize);

                    EventWaitHandle test = new ManualResetEvent(false);

                    var request = new SftpWriteRequest(this.NextRequestId, handle, offset + (ulong)(i * maximumDataSize), blockBuffer,
                        (response) =>
                        {
                            test.Set();
                            if (response.StatusCode == StatusCodes.Ok)
                            {
                                if (wait != null)
                                    wait.Set();
                            }
                            else
                            {
                                this.ThrowSftpException(response);
                            }
                        });

                    this.SendRequest(request);
                    test.WaitOne();

                    if (wait != null)
                        this.WaitHandle(wait, this._operationTimeout);
                }
            }
        }

 

Please not text event wait handle.

 

Please let me know if it works, so I could make this change as an optional for future use.

 

Thanks,

Oleg

Aug 27, 2012 at 6:13 PM

Sorry for the delayed response.

After implementing your suggestion, I still receive the same exception.

It did substantially increase the uploaded filesize to around 600kb-3MB instead of around 98kb. It is no longer consistently failing after the same amount of data.