2
Vote

SCP : Missing files when using DirectoryUpload

description

Hi,I have the following environment:- Win7 / 64- .NET 3.5- MPC-Target :OpenSSH_6.0p1, OpenSSL 1.0.0j 10 May 2012- SSH.NET commit 19813I want to upload a directory structure (recursive, depth = 2) from the PC to a MPC basedLinux remote system. The total amount of data is about 8MB with 23 files from55 bytes to 4MB.Calling mScp.Upload(new DirectoryInfo(localDir), remoteDir) does not result anyerror and the Uploading event shoes uploading all files. But after Upload() returnedThere are a random number of missing files on the target (between 1 and 5).It tried the same with a PC based Linux as target what resulted in the same behaviour(OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011). It seems theproblem is caused anywhere within SSH.NET.Does anyone have a solution?Extension:I tried to workaround by replacing Upload(new DirectoryInfo(...)) by several singlecalls of Upload(new FileInfo(...)). This worked but is too slow for my application.

file attachments

comments

olegkap wrote Dec 21, 2012 at 12:06 AM

Can you post a code that you using to do it?
I don't think I added support to upload directory using Upload function.

Thanks,
Oleg

TheCloudlessSky wrote Feb 26, 2013 at 4:16 PM

@Oleg,
  1. This issue is still not fixed (even in the latest source). The issue is not the same as being able to upload a rooted directory. I have the exact same issue as the OP: I'm uploading a directory that has multiple files/nested directories and files randomly do not get uploaded:

    using (var scp = new ScpClient("host", "user", "password"))
    {
    scp.Connect();
    
    scp.Uploading += (s, e) =>
    {
        Console.WriteLine("UPLOAD -> " + e.Filename + ", Size=" + e.Size + ", Uploaded=" + e.Uploaded);
    };
    
    var directory = new DirectoryInfo(@"c:\toupload");
    
    scp.Upload(directory, "xyz");
    
    scp.Disconnect();
    }
This prints the upload message for every file - but doesn't actually upload all of the files!

TheCloudlessSky wrote Feb 26, 2013 at 4:17 PM

Sorry for the messed up formatting...

Schmid wrote Feb 27, 2013 at 5:53 AM

Hi Oleg,

I think you really intended to support uploading / downloading a directory:
ScpClient.Upload(DirectoryInfo directoryInfo, string filePath);
ScpClient.Download(string filePath, DirectoryInfo directoryInfo);

Please, please fix the problem.

Thanks,
Ulrich

TheCloudlessSky wrote Mar 6, 2013 at 12:32 PM

Oleg,

Any chance of getting this fixed? This has completely blocked us from using your library and we've had to resolve to using the old Tamir.SharpSSH library that is terribly slow!

olegkap wrote Mar 6, 2013 at 1:12 PM

Hey guys,
I briefly looked at it but under one scenario that I checked it worked so I think what happens here is that I need to think about all possible scenarios and address them appropriately.

I guess I might look into Tamir library for solution example but unfortunately I don't have much free time right now.

I will defiantly will try to fix it as soon as possible.

Thanks,
Oleg

TheCloudlessSky wrote Mar 6, 2013 at 1:56 PM

Oleg,

I think having multiple nested directories/files might be the cause of this. I really do appreciate a fix for this when you have the time. Cheers!

Schmid wrote Mar 7, 2013 at 9:54 AM

Oleg,

I also still have to live with the very slow workaround by transfering file by file.
Please spend some more time for finding and fixing the bug.

Thanks
Ulrich

olegkap wrote Mar 7, 2013 at 3:18 PM

Hi,

Can you please check 23466 commit and see if it fixed that?
I made some changes to upload logic and I hope it should resolved issue of missing files too, since I cannot duplicate this issue locally.

Thanks,
Oleg

TheCloudlessSky wrote Mar 7, 2013 at 5:42 PM

Hi Oleg,

That commit did not fix the missing file issue :(. I don't know a lot about the SSH/SCP protocols, but I'm going to spend some time adding some debug statements to see what's up.

TheCloudlessSky wrote Mar 7, 2013 at 6:19 PM

Also - I would be glad to privately send you the files we're uploading that seem to "cause issues".

olegkap wrote Mar 7, 2013 at 7:13 PM

Yea, sure, feel free to send me your files so I could try it here as well.

Also, I would suggest to put debug meesages after lines where it sends D0755, C0644 and E commands.
This could give a better picture of what files or directories it tried to copy, then if you can, attach this output with your directory tree structure so I could try to analyze it.

Thanks,
OIeg

Schmid wrote Jun 4, 2013 at 4:57 AM

Hi Oleg,

you set the status to fixed and closed.
Which revision number fixed the bug?

Thanks
Ulrich

Schmid wrote Jun 6, 2013 at 7:40 AM

Hi Oleg,

I updated to revision 25006.
This did not fix the problem. Thus for me the bug is not fixed, yet.

Ulrich.

drieseng wrote Mar 1 at 7:01 PM

Reopened since issue is not fixed for Ulrich.

Schmid wrote Mar 12 at 6:24 AM

Hi drieseng,

I'm pleased to see that one is going to analyze the bug again.
Please let me know if you need any further information or support.

I still need a solution

Thanks
Ulrich

drieseng wrote Mar 14 at 6:11 AM

Ulrich,

Please provide a simple repro (code only), and any files necessary to reproduce this issue.

Thanks!
Gert

Schmid wrote Mar 24 at 9:07 AM

Gert,

Please apologize the delay. I did not have the time for providing a demo the last days.
The demo creates a temporary file structure an cyclically uploads it to the remote device.
I'm referencing Renci.SshNet.NET35.dll and running the release build.

I'm using revision 25006 in which I already integrated a patch from a SSH.NET discussion for
using absolute path strings (see attached file).

Thank you for reopening the issue.

Ulrich

using System;
using System.Text;
using Renci.SshNet;
using System.IO;


namespace ConsoleApplication1
{
  class Program
  {
    const string address = "169.254.177.104";
    const string user = "root";
    const string pass = "password";

    const string mRemoteDir = "/tmp/test/";


    static SshClient mSsh;
    static ScpClient mScp;

    //************************************************************************
    static int Execute(string command)
    {
      return Execute(command, false);
    }

    //************************************************************************
    static int Execute(string command, bool silent)
    {
      if (!silent)
      {
        Console.WriteLine(string.Format("SSH.Execute : {0}", command));
      }

      SshCommand cmd = mSsh.RunCommand(command);
      if (!silent)
      {
        Console.WriteLine(String.Format("ExitCode: {0}", cmd.ExitStatus));
        if (!String.IsNullOrEmpty(cmd.Result))
        {
          Console.WriteLine(string.Format("Result:\n{0}", cmd.Result));
        }
        if (!String.IsNullOrEmpty(cmd.Error))
        {
          Console.WriteLine(string.Format("Error:\n{0}", cmd.Error));
        }
      }
      return cmd.ExitStatus;
    }

    //************************************************************************
    static void Main(string[] args)
    {
      var localDir = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());
      try
      {
        Directory.CreateDirectory(localDir);
        CreateTestData(localDir);

        using (mSsh = new SshClient(address, user, pass))
        {
          mSsh.Connect();

          using (mScp = new ScpClient(address, user, pass))
          {
            mScp.Connect();
            mScp.ErrorOccurred += new EventHandler<Renci.SshNet.Common.ExceptionEventArgs>(mScp_ErrorOccurred);
            mScp.Uploading += new EventHandler<Renci.SshNet.Common.ScpUploadEventArgs>(mScp_Uploading);

            for (int i = 0; i < 10; ++i)
            {
              UploadDir(localDir, mRemoteDir);
            }
          }
        }
      }
      catch (System.Exception ex)
      {
        Console.WriteLine(String.Format("Exception:\n{0}", ex));
      }
      finally
      {
        if (Directory.Exists(localDir))
        {
          Directory.Delete(localDir);
        }
      }

      Console.WriteLine("Done");
      Console.ReadLine();
    }


    //*************************************************************************
    static void mScp_ErrorOccurred(object sender, Renci.SshNet.Common.ExceptionEventArgs e)
    {
      Console.WriteLine(String.Format("Error: {0}", e));
    }


    //*************************************************************************
    static void mScp_Uploading(object sender, Renci.SshNet.Common.ScpUploadEventArgs e)
    {
      //Console.WriteLine(String.Format("Uploading {0} ({1}/{2})"
      //                                , e.Filename
      //                                , e.Uploaded
      //                                , e.Size));
    }

    //*************************************************************************
    private static void UploadDir(string localDir, string remoteDir)
    {
      string remoteDirPath = remoteDir.TrimEnd('/');
      Execute(String.Format("rm -rf {0}", remoteDirPath), true);

      Console.WriteLine("Upload to " + remoteDir);
      DateTime startTime = DateTime.Now;
      mScp.Upload(new DirectoryInfo(localDir)
                 , remoteDirPath);

      Console.WriteLine("{0}", DateTime.Now - startTime);

      Console.ForegroundColor = ConsoleColor.Red;
      Console.WriteLine(String.Format("Valid : {0}"
                                     , ValidateUpload(localDir, remoteDirPath)));
      Console.ResetColor();
    }

    //*************************************************************************
    private static void CreateTestData(string localDir)
    {
      CreateTestFiles(localDir);
      for (int subDirIndex = 0; subDirIndex < 1; ++subDirIndex)
      {
        var subDir = Path.Combine(localDir, subDirIndex.ToString());
        Directory.CreateDirectory(subDir);
        CreateTestFiles(subDir);
      }
    }

    //*************************************************************************
    private static void CreateTestFiles(string dir)
    {
      var fileContent = new StringBuilder("HelloWorld\n");

      for (int fileIndex = 1; fileIndex < 20; ++fileIndex)
      {
        var fileName = String.Format("{0}.txt", fileIndex);
        var filePath = Path.Combine(dir, fileName);
        fileContent.Append(fileContent.ToString());
        File.WriteAllText(filePath, fileContent.ToString());
      }
    }

    //*************************************************************************
    private static bool DoRecursive( string localDir
                                    , string remoteDir
                                    , Func<string, string, bool> directoryDelegate
                                    , Func<string, string, bool> fileDelegate)
    {
      bool result = true;

      result &= directoryDelegate(localDir, remoteDir);

      foreach (string localFile in Directory.GetFiles(localDir))
      {
        string remoteFile = String.Format("{0}/{1}"
                                         , remoteDir
                                         , Path.GetFileName(localFile));
        result &= fileDelegate(localFile, remoteFile);
      }

      foreach (string localSubDir in Directory.GetDirectories(localDir))
      {
        string remoteSubDir = String.Format("{0}/{1}"
                                           , remoteDir
                                           , Path.GetFileName(localSubDir));
        result &= DoRecursive( localSubDir
                             , remoteSubDir
                             , directoryDelegate
                             , fileDelegate);
      }

      return result;
    }


    //*************************************************************************
    private static bool ValidateUpload(string localDir, string remoteDir)
    {
      return DoRecursive(localDir, remoteDir
                        , (locDir, rmtDir) =>
                          {
                            bool validDir= true;
                            if (0 != Execute(String.Format("[ -d {0} ]", rmtDir), true))
                            {
                              Console.WriteLine(String.Format("Missing remote directory {0}"
                                                             , rmtDir));
                              validDir = false;
                            }
                            return validDir;
                          }
                        , (locFile, rmtFile) =>
                          {
                            bool validFile = true;
                            if (0 != Execute(String.Format("[ -f {0} ]", rmtFile), true))
                            {
                              Console.WriteLine(String.Format("Missing remote file {0}"
                                                             , rmtFile));
                              validFile = false;
                            }
                            return validFile;
                          }
                        );
    }
  }
}
The output is completely curious:
Upload to /tmp/test/
00:00:09.0710000
Missing remote file /tmp/test/0/8.txt
Missing remote file /tmp/test/0/9.txt
Valid : False
Upload to /tmp/test/
00:00:09.0390000
Missing remote file /tmp/test/0/9.txt
Valid : False
Upload to /tmp/test/
00:00:09.0390000
Valid : True
Upload to /tmp/test/
00:00:09.0460000
Missing remote file /tmp/test/0/8.txt
Missing remote file /tmp/test/0/9.txt
Valid : False

drieseng wrote Mar 26 at 7:09 PM

Ulrich,

I have not yet been able to reproduce the problem.
I'll perform more tests tomorrow.

Cheers,
Gert

Schmid wrote Today at 10:36 AM

Gert,

it seams the behaviour is not reproducable without our embedded device.
We would like to send you such one. Would you please send me your
contact?

Thanks
Ulrich