[SOLVED] Retrieving a LOT of output from a command...

Jun 16, 2014 at 9:52 PM
Edited Jun 16, 2014 at 10:55 PM
Experts,

This is an interesting one for me. Previously, we saw this discussed in this post, and I will try that as well.

For now, I can say that with the below SS instantiation, I am maxing out at (wildly approximate, based on line length) 50,000 bytes/characters:

Me.ss = Me.Client.CreateShellStream("dumb", 80, 24, 800, 600, 102400)

Each line is ~50characters, and one output I read has over 50,000 such 50 character lines.

I will try what uugan posted as an update now:
 var cmd = client.CreateCommand("show log ");   
            var asynch = cmd.BeginExecute();

            var reader = new StreamReader(cmd.OutputStream);

            while (!asynch.IsCompleted)
            {
                var result = reader.ReadToEnd();
                if (string.IsNullOrEmpty(result))
                    continue;
                Console.Write(result);
            }
            cmd.EndExecute(asynch);
...but I'd also appreciate any insights from those doing this as well, and if you do find a solution, then what DO you set your SS to (102400 in my current setting above)?
Jun 16, 2014 at 10:55 PM
Edited Jun 16, 2014 at 10:55 PM
Well, I think I officially failed to get uuleg's example working:

"Failed to open a channel after 10 attempts."

Using:
Public Function SendGetLots(Cmd As String) As String
    Dim result As New StringBuilder
    '
    Try
        '
        Dim sshCmd As SshCommand = Client.CreateCommand(Cmd)
        Dim asynch = sshCmd.BeginExecute()    <=== this line (165) fails 

        Dim reader = New StreamReader(sshCmd.OutputStream)

        Do While Not asynch.IsCompleted
            ' Dim result = reader.ReadToEnd()
            result.AppendLine(reader.ReadToEnd())
            If String.IsNullOrEmpty(result.ToString) Then
                Continue Do
            End If
            Console.Write(result)
        Loop
        sshCmd.EndExecute(asynch)
    Catch ex As Exception
        log.Fatal("SendGetLots(" & Cmd & ") exception caught message: " & vbCrLf & ex.Message())
        log.Fatal("SendGetLots(" & Cmd & ") exception caught toString: " & vbCrLf & ex.ToString())
        log.Fatal("SendGetLots(" & Cmd & ") exception caught stackTrace: " & vbCrLf & ex.StackTrace())
    End Try
    Return result.ToString
End Function
yields:

[2014-06-16 17:46:26,330] SendCommand(sh run object-group id AE) exception caught message:
Failed to open a channel after 10 attempts., Last Milliseconds: 1.0001 [FATAL] Machine: DOOZY User: Pat Process: 64 CallingMethod: SendGetLots LineNumber: 165
[2014-06-16 17:46:27,883] SendCommand(sh run object-group id AE) exception caught toString:
Renci.SshNet.Common.SshException: Failed to open a channel after 10 attempts.
at Renci.SshNet.Channels.ChannelSession.Open()
at Renci.SshNet.SshCommand.BeginExecute(AsyncCallback callback, Object state)
at Renci.SshNet.SshCommand.BeginExecute()
at SendGetLots(String Cmd) in C:...\SecureComms.vb:line 165, Last Milliseconds: 1 [FATAL] Machine: DOOZY User: Pat Process: 64 CallingMethod: SecureComms.SendGetLots LineNumber: 165
[2014-06-16 17:46:29,460] SendCommand(sh run object-group id AE) exception caught stackTrace:
at Renci.SshNet.Channels.ChannelSession.Open()
at Renci.SshNet.SshCommand.BeginExecute(AsyncCallback callback, Object state)
at Renci.SshNet.SshCommand.BeginExecute()
at SecureComms.SendGetLots(String Cmd) in C:...\SecureComms.vb:line 165, Last Milliseconds: 1 [FATAL] Machine: DOOZY User: Pat Process: 64 CallingMethod: SecureComms.SendGetLots LineNumber: 165

I really have no experience with async calls with Renci, so any hints/pointers appreciated. I think maybe there was some other code he used that wasn't included in the snippet that wold have made it work! :)

TIA!

pat
:)
Jun 18, 2014 at 2:36 AM
Experts,

OK... I'd like to ask you for feedback/opinions on the following, seeking a better way to do this. You know the requirement: Capture a TON of output from a command...

The standard Execute (synchronous) truncated output at the buffer limit of the shell. In this self-contained example, there is no ShellStream():

[Top level Sub()]
Sub TopBatch()
    Using client As SshClient = New SshClient("IPOrHostname", "userName", "userPassword")
        Try
            client.Connect()
            If client.IsConnected Then
                Dim result1 As String = __DoCommand__(client, "enable" & vbLf & "enablePass" & vbLf & "term pager 0" & vbLf & "sh run" & vbLf)
                Console.WriteLine("-1-" & vbCrLf & result1 & vbCrLf & "-1-")
                Console.WriteLine(vbCrLf & "result 1 is " & result1.Length & " characters long..")
            End If
            client.Disconnect()
        Catch ex As Exception
            Console.WriteLine("EXCEPTION: " & ex.ToString)
        End Try
    End Using
End Sub
That sub calls this routine... At the time of this test, I didn't know if I'd be calling the following once or multiple times... Come to find out that with this approach, you need to stack the commands like I did above:
Dim result1 As String = DoCommand(client, "enable" & vbLf & "enablePass" & vbLf & "term pager 0" & vbLf & "sh run" & vbLf)
This is the DoCommand function:
Function DoCommand(ByRef client As SshClient, TheCommand As String) As String
    Dim tsAsyncTimeout As TimeSpan = TimeSpan.FromSeconds(8)
    Dim swAsyncTimeout As New Stopwatch
    Dim sbResult As New StringBuilder
    Try
        Dim cmd As SshCommand = client.CreateCommand(TheCommand)
        Dim asynch As System.IAsyncResult = cmd.BeginExecute()
        Using reader As StreamReader = New StreamReader(cmd.OutputStream)
            Dim res As String = ""
            swAsyncTimeout.Start()
            Do While Not asynch.IsCompleted
                res = reader.ReadToEnd()
                If String.IsNullOrEmpty(res) Then
                    If swAsyncTimeout.Elapsed.TotalSeconds >= tsAsyncTimeout.TotalSeconds Then
                        Console.WriteLine(String.Format("Cancelling async execution.. idle for {0} seconds [{1}]...", swAsyncTimeout.Elapsed.TotalSeconds.ToString("0.00"), tsAsyncTimeout.ToString))
                        cmd.CancelAsync() ' turns asynch.IsCompleted to True; same as doing:
                        ' Exit Do
                    End If
                Continue Do
                Else
                    sbResult.Append(res)
                    swAsyncTimeout.Restart() ' every time we get something, we re-start the timer...
                End If
            Loop
        End Using
        ' cmd.EndExecute(asynch)
    Catch ex As Exception
        Console.WriteLine("EXCEPTION: " & ex.ToString)
    End Try
    Return sbResult.ToString
End Function
I could have passed the timeout to the Function, but I'm lazy & it's just a test... The line where you define the timeout is here:
Dim tsAsyncTimeout As TimeSpan = TimeSpan.FromSeconds(8)
Since I couldn't get the async loop to terminate properly (asynch.IsCompleted was never true (unless I manually set it with cmd.CancelAsync(), so after all text returned, the loop was just run even though it clearly was done with the command. Probably not a bug, just my ignorance.

So I used:
If String.IsNullOrEmpty(res) Then
...to tell me that output has stopped/paused. If this was not the case, I would ReStart() the stopwatch (timer), so any time the command returned even one line, I would reset the timeout stopwatch (swAsyncTimeout)... With a lot of output, it would be silly to try to estimate how long it would take to download it all and then try to create a timeout based on the largest file. I think this approach is a little more robust, and easily adjustable in code at runtime. I'm interested to see how you guys are handling timeouts, though, as I'm not relying on anything internal.

I tried to use cmd.Timeout() but it was ignored in the loop of the example in the help chm. Also, I found the ExtendedOutputStream() hung on ReadToEnd(), where using OutputStream() got what it could & continued (which is desired).

OK. So on to the output. The program ran, getting the output from the router for about 1 minute, then spit it out to console (I wanted the test/proof to be easy for you to run, so I didn't use any fancy loggers):

In the code, you can see that the last line of output asks the program how many characters were returned:
[...]
result 1 is 2887590 characters long..
End...
Before this exercise, I was left with just the first 1/2 of whatever the buffer in my ShellStream was set to, the rest was tossed/truncated. So, I'd say this is a pretty darned good proof! I could show you my 'before' code, but you wouldn't be impressed.

I will tell you this, though. This has caused me to completely re-work my programs using SSH. Unless I'm wrong, here is the difference:
  • If you use ShellStream(), and define a terminal, you can interact with the shell, command by command, and then shut the shell down when done. This is the way I'm doing things today.
  • If you do it the way earlier in this post (via IAsyncResult()), though, like this:
Dim cmd As SshCommand = client.CreateCommand(TheCommand)
Dim asynch As System.IAsyncResult = cmd.BeginExecute()
Using reader As StreamReader = New StreamReader(cmd.OutputStream)
    [...]
...you have to line up all your commands, and fire them all off for processing.

Which one to use?

As usual, "It Depends", is what I found out so far.
  • If you want to enter commands based on the output of the previous command, and they are short, brief back-n-forths, then ShellStream() is the way to go.
  • If, on the other hand, you need to do everything in one session (lifetime: Client.Connect() to Client.Disconnect()), and you expect a BLAST of output, then you can use the method shown here, as it works (for me).
-> All I want to ask the community for are ways to make this better, or your examples (C# is fine, I use both) of how you are doing it.

Sound fair? :)

Thanks!

pat
:)

P.S. Thanks to the authors & developers for keeping the community happy, but more importantly engaged... :)
Marked as answer by BogusException on 6/18/2014 at 4:30 PM
Jul 12, 2014 at 7:09 PM
I have figured out recently on the Cisco switches on the network I work on the "term pager 0" doesn't work.
Dim result1 As String = DoCommand(client, "enable" & vbLf & "enablePass" & vbLf & "term pager 0" & vbLf & "sh run" & vbLf)
Instead I had to use "terminal length 0" to get the same result.
Dim result1 As String = DoCommand(client, "enable" & vbLf & "enablePass" & vbLf & "term len 0" & vbLf & "sh run" & vbLf)
I hope this helps someone out. Your example is a great one for learning and has helped me get started working with this library. Tip my hat off to you!
Jul 16, 2014 at 8:23 PM
@chipgraphics,

What a nice note!

Thanks!

It was a stupid change for cisco, and makes no sense whatsoever... it's the same IOS, for cryin' out loud...