This project is read-only.

Payload cannot be more then 32768 bytes

Apr 20, 2011 at 8:48 PM

I tried uploading some larger files to see what would happen and I got this error: Payload cannot be more then 32768 bytes.  Each file saved to the ftp server with a size of 0 bytes.  Is there a file size limit or am I doing something wrong?

     Private Sub Upload(ByVal sftp As Renci.SshClient.SftpClient)
        ' make a list of remote files on ftp server
        Dim ftpFilesList As New List(Of String)
        For Each file In sftp.ListDirectory(_FtpDir)
            If file.IsRegularFile Then
                ftpFilesList.Add(file.Name.ToUpper)
            End If
        Next

        For Each lclFile In Directory.GetFiles(_LocalDir)
            lclFile = Path.GetFileName(lclFile)

            If ftpFilesList.Contains(lclFile.ToUpper) Then
                ' file exists, log issue
                TraceFunc.WriteLine(lclFile + " exists on server")
            Else
                ' Upload the file
                Dim fs As New IO.FileStream(_LocalDir + lclFile, FileMode.Open, FileAccess.Read)
                TraceFunc.WriteLine("Uploading to " + _FtpDir + "/" + lclFile)
                sftp.UploadFile(fs, _FtpDir + "/" + lclFile)
                TraceFunc.WriteLine("Upload complete")
                fs.Close()
                ' Archive the file
                ArchiveUploaded(lclFile)
            End If
        Next
    End Sub

Coordinator
Apr 20, 2011 at 8:57 PM

No,

It there is no limit file upload as I was doing tests with 100GB files and it worked correctly.

Are you changin SftpClient.BufferSize property by any chance to something bigger then 1024 * 32 - 38 ?

Apr 20, 2011 at 10:00 PM

No, I didn't touch the buffer size, should I be?  I just tried a 255 kb file and got the same error.

Coordinator
Apr 21, 2011 at 2:16 PM

Hmm, then it strange behavior then :(

Can you set BufferSize to 1024 * 16 and see if it works?

Thanks.

Oleg

Jul 26, 2011 at 3:35 AM

Hi, i found the same problem.

 

I've tried to change the buffer size, but now it gives me "Connection Was Lost" exception.
And then "Session Operation Has Timed Out"
This is my code :
Dim

 

 

 

sftp As Renci.SshNet.

SftpClient

 

Try

sftp =

New Renci.SshNet.SftpClient

(hosttectia, 22, usernametectia, passwordtectia)

sftp.Connect()

sftp.ChangeDirectory(

"IN"

)

sftp.BufferSize = 1024 * 16

 

Dim file As IO.FileStream = New IO.FileStream(filename, IO.FileMode

.Open)

sftp.UploadFile(file, fileshort)

MsgBox(

"Sending " & filename & " to " & hosttectia & "\" & fileshort & " succesfully transferred via FTP", MsgBoxStyle.Information, "Succesful"

)

 

Catch ex As

Exception

MsgBox(

"Sending " & filename & " to " & hosttectia & "\" & fileshort & " failed." & vbCrLf & "Detail : "

& ex.Message)

 

Finally

sftp.Disconnect()

sftp.Dispose()

 

End

Try

 

Coordinator
Jul 26, 2011 at 2:58 PM

How big is your file?

Is it binary or ASCII?

Does it happen only with one specific file or with any file ?

Does it happens with only specific SFTP server if so what version?

 

Thanks,

Oleg

Jul 27, 2011 at 4:18 AM

My file is a zip file,
The filesize is 1,5Mb.
It's always fail when i try to send any file larger than 32Kb.
I have succesfully sent 5Kb file
but I failed sending 416Kb, 624Kb, and 1,5 Mb.
I don't have any other SFTP server, I can only try it in my Tectia server.

Coordinator
Jul 27, 2011 at 5:31 PM

Sorry, but I have follow up questions.

Can you SFTP using putty to this server?

If so can you save the log file and send it to me or may be its possible to get temporary access to that server so I could debug the problem, if yes, you can send me that info privately.

Also, can you try to connect to different SSH server just for test purposes and see if it works for you on different server?

I am just trying to figure out where is the problem, cause in the past I had also problems with different SSH server implementation so I am trying to isolate the problem.

 

Thanks,

Oleg

Jul 28, 2011 at 3:27 AM

Hi Oleg,

I myself don't have access to that server,
I always ask someone to test to send files.
I can't give you temporary access to that server,
and I can't try to send to different SFTP server here.

But, they can send any files using their Tectia SSH Client.
The problem is, they want to me to make some application that automatically send those files.

Sorry I can't be much help for you to isolate the problem.

Thanks,

Bonar

Coordinator
Jul 29, 2011 at 3:36 PM

Unfortunately for me to truck it down I need some way to recreate problem here, that's why I wanted to try to connect to your server.

Because, it looks like, it doesn't happen to other people, who use different server, I would suspect SSH server implementation however it doesn't mean I cannot fix something in my library to make it work with this server too.

 

Thanks,

Oleg

Aug 8, 2011 at 12:57 PM

Hi

I'm getting the same error when I upload files. I can upload the same file to the server using an sftp client program. Unfortunately this is not my server so I cannot give you access to it.

I read through the code and it seems the problem comes from the Session.SendMessage method. The file length is tested against the constant Session.MAXIMUM_PAYLOAD_SIZE. If the file is larger that that in bytes then the InvalidOperationException is thrown with the message 'Payload cannot be more then {0} bytes' where {0} is Session.MAXIMUM_PAYLOAD_SIZE. I changed the Session.MAXIMUM_PAYLOAD_SIZE from 1024*32 to 1024*1024 and recompiled the source code. After that my 450k test file uploaded with no errors.

So it seems like this problem has nothing to do with the server at all. It's just a check that is made before anything is transfered at all. Should not be hard to fix. I really wonder how did you manage to upload the 100GB test file.

Coordinator
Aug 8, 2011 at 3:39 PM

I just did a test with 10MB file and it worked just fine, well on servers that I am testing it against.

The reason I have this restriction is because based on the SSH 2.0 protocol, maximum SSH packet size is 32K.

So in theory, if your server implementation supports and uses packets bigger then 32K then it will work fine, if you take restrication away, but on others it can fail.

I actually intresting to know how and why I am trying to send packets which I bigger then 32K, as it should be prevented by the code somewhere but apparently I am missing something somewhere.

If you can think of the way I could recreate this problem here I would be glad to fix it.

Also, as a way to check something, can you add this line to show message length in SendMessage method:

            var messageData = message.GetBytes();
 
            System.Diagnostics.Debug.WriteLine(messageData.Length);

And post the output

Thanks,

Oleg

Aug 9, 2011 at 10:07 AM

Ok. I did as you suggested and added the WriteLine. True message length being sent appears to be 32,772 bytes while the 32*1024 makes 32,768. So each and every time (except when there is no more data to send) 32,772>32,768 holds and the exception is thrown just before the very first upload message is being sent and so nothing gets done.

Now that I know this much I may as well add some speculation. Can this extra payload be caused by some extra stuff that is agreed with the server? Your server does not need it and so you can't reproduce the problem? I tried to debug this a bit and it appears that in SshData.cs in the Method WriteBinaryString there is 8 bytes of data in the buffer before the 32,763 of payload is added which makes together 32,771 which already is enough to cause the exception later.  The one extra byte must be some sort of end marker.

I'm not sure if this helps at all but I hope so. The files get uploaded after I changed the MAXIMUM_PAYLOAD_SIZE but if the standard does not allow messages larger than the original maximum this does not sound like a proper way to solve the problem.

Aug 9, 2011 at 10:55 AM

After a little bit more debugging I found out that ChannelDataMessage.SaveData calls base.SaveData which means ChannelMessage.SaveData and that throws this.LocalChannelNumber (4 bytes) into the buffer. Then ChannelDataMessage.SaveDate calls this.WriteBinaryString that actually is the protected method SshData.WriteBinaryString. That throws data.Length (4 bytes) into the buffer. Those are the 8 bytes that are in the buffer before the real payload is added.

The buffer size that is read from the file is 32*1024-38=32,730 bytes. The constructor of the class SftpDataMessage adds 33 bytes to the beginning of the buffer and makes the total length of the message 32,763 bytes and then there is that another extra 8 bytes I mentioned earlier. The SftpDataMessage constructor calls SftpMessage.GetBytes that calls (not directly though but via SshData.GetBytes) SftpWriteRequest.SaveData and that adds SftpMessageType (1 byte). SftpRequest.SaveData then adds this.RequestId (4 bytes). Then SftpWriteRequest.SaveData adds this.Handle to the buffer via WriteBinaryString (length and Handle make together 4+8=12 bytes). Then SftpRequest.SaveData adds this.Offset to the buffer using this.Write (length and Offset make together 8 bytes). Then the 32,730 bytes of data is added via WriteBinaryString (length and data make 4+32,730=32,734 bytes). At this point the buffer has 32,759 bytes. Then the SftpDataMessage constructor makes a buffer of 32,759+4 bytes and adds the data length and the data to it. And there we have a buffer of 32,763 bytes.

I'm not sure which of these parts depend of the actual server being used. At this point the way I see it the original buffer size that is used to read the data from the original local file should actually be 32*1024-46 bytes so that the message size would stay within the bounds.

Coordinator
Aug 11, 2011 at 12:29 PM

Thanks for this research.

It looks like I need to decrease size of SFTP package by a little more.

I will try to fix it as soon as I can then. Want to dig in other code sources to see how they solve this issue if I can.

 

Thanks,

Oleg

Aug 12, 2011 at 4:09 PM

Hey guys, I am having the same issue and also was able to increase the Session.MAXIMUM_PAYLOAD_SIZE and was able to get it to work as a short term.

Thanks

Kevin

Coordinator
Aug 17, 2011 at 4:08 PM

Hey everybody,

 

So I want to resolve this issue somehow and I am thinking of several options and wanted to hear a feedback as far as which way to go.

So after diggin into putty source code, they do it very simple, they create 4K buffer and then read data from the file and write this buffer to the server, so since its always 4K they never have a problem.

I dont want to lock down the buffer to be 4K only since I think it could be beneficial for huge file transfer so I would like to keep this option open to be able to specify buffer size.

So so far I have to solutions:

1. Is to throw an exception when buffer would be bigger then 32*1024-38=32,730 ( or in fact I will allow maximum of 32*1024 and will do 38 bytes deduction internally if needed).

2. Is to implement additional logic that will handle write operation for buffers that more then 32730. The problem with that I will have to break data that needed to be transferred into smaller chunks of data that will fit into the packet but then it could be unefficient since last packet could be very small but still require to be transmitted. And of course this scenario could be avoided by specifying smaller buffer in the first place.

 

Please let me know what you think.

 

Also,

Even so I managed recreate the problem, I would like to know if anybody changed BufferSize property for SftpClient?

Since this is the only case where I can think you could get this error. (internally I already setting buffer to be 32730

this.BufferSize = 1024 * 32 - 38;

Thanks,

Oleg

Aug 23, 2011 at 1:53 PM
Edited Aug 23, 2011 at 1:55 PM

Hi Oleg,

 

I am trying to upload file using SFTP. file size is Maximum 1 - 10 MB.

i  could not upload the file.

if file size more the KB then loop  is coming into else

 

  if (bytesRead < this.BufferSize)
                {
                    var data = new byte[bytesRead];
                    Array.Copy(buffer, data, bytesRead);
                    this._sftpSession.RequestWrite(handle, offset, data);
                    uploadCompleted = true;
                }
                else
                {
                    this._sftpSession.RequestWrite(handle, offset, buffer);
                }

But keep on waiting the response. finally i have wait till 1 and 1/2 hours also.. there is no reponse from server.

i think no data is writing in to the server. if i tried 12 and 25 KB files it is successfully writing the files.

Where i need to change the code to upload  1 mb file. to get response from server.

i tried changes for

 

protected const int MAXIMUM_PACKET_SIZE = 150000;

 

 

 

 but not able to success. :(

 

 Could you please let me know what are the changes required to upload 1- 10 mb files.

 

 Thanks

Rambhopal reddy E

 

Coordinator
Aug 23, 2011 at 3:06 PM

Can you please download latest source code?

 

Recently I made some changes to address this issue.

 

Also, if it doesn't work or you want to use existing version, you can decrease SFTP buffer by setting BufferSize to something less then 32K.

For example, putty SFTP utility uses 4K buffer.

 

Hope it helps,

Thanls,

Oleg

Nov 24, 2011 at 11:47 PM
Edited Nov 25, 2011 at 12:28 AM

edit. mistaken post..

Aug 28, 2012 at 4:10 PM

I ran into this same problem with the 3.5 version to get around it  I did this

 

First: ( notice the smaller buffer size of 16k on the stream reader at the end )

sr = New StreamReader(<file name>, System.Text.Encoding.ASCII, True, (1024 * 16)) 
 

Second: (set the BufferSize on the SFTPClient object to the same as the StreamReader)

sftp.BufferSize = (1024 * 16)

By setting the smaller 16k buffer I didn't hit the 32k ceiling 

Nov 20, 2012 at 6:16 AM

I'm getting the same error too.

However, the error was solved when specifying buffer size.

Thanks guys.

Dec 10, 2012 at 10:40 AM

Hello, I've experinced the same problem today with following code:

using (var ms = new MemoryStream(file))
{
   sftp.UploadFile(ms, name);
}
where file is array o bytes byte[]

 

 

Coordinator
Dec 20, 2012 at 1:36 PM

Hey guys,

 

I changed default buffer size to 16K now since it seems to be an issue for many so hopefully it will not happen anymore.

 

Thanks,

Oleg