Channel sessions not cleaning up (Avoiding Memory Leaks)

May 28, 2013 at 6:50 PM
Edited May 29, 2013 at 1:56 AM
I have an application which requires very long term execution. It creates a series of SshCommand objects (Each with a specific command) then executes those (using command.execute()) once every few seconds for multiple weeks. The string results are used to populate various parsers and GUI elements.

I have not been able to isolate the reasons, but what I see are millions of left over ChannelSession objects resulting in a system crash (Out of memory).

I suspect I simply am not using the library correctly since there's no mention of this in recent discussions. Is there a good reference example of how to use the library to perform long term polling (Via SshCommand)?

Generally, my technique is:
  • Create a ConnectionInfo object conn
  • Create a SshClient(conn) object client
  • Call client.Connect()
  • Add SshCommand (s) to client using client.CreateCommand()
  • Periodically execute the SshCommands using command.Execute()
  • When the program is done, call client.Disconnect() and client.Dispose()
In WinDbg.exe, I see persistent ChannelSession objects which should have been garbage collected. Included with these are many events such as AutoReset and ManualReset events.

If my technique is correct, why is ChannelSession not being garbage collected after each Execute()? A bit of debugging makes me think that SshCommand is a bit mixed up on re-executions. A re-execution creates a new channel without cleaning up the old channel, even if it has closed.

Attempting to manually dispose() the SshCommand doesn't appear to affect its persistence either (If the session isn't also disposed).

Thanks,

Robert
May 29, 2013 at 4:27 PM
Edited May 29, 2013 at 7:45 PM
The code in SshCommand 2013.4.7 (EndExecute):
        public string EndExecute(IAsyncResult asyncResult)
        {
            if (this._asyncResult == asyncResult && this._asyncResult != null)
            {
                lock (this._endExecuteLock)
                {
                    if (this._asyncResult != null)
                    {
                        //  Make sure that operation completed if not wait for it to finish
                        this.WaitHandle(this._asyncResult.AsyncWaitHandle);

                        if (this._channel.IsOpen)
                        {
                            this._channel.SendEof();

                            this._channel.Close();
                        }

                        this._channel = null;
                        this._asyncResult = null;

                        return this.Result;
                    }
                }
            }

            throw new ArgumentException("Either the IAsyncResult object did not come from the corresponding async method on this type, or EndExecute was called multiple times with the same IAsyncResult.");
        }
If _channel is set to null here, it appears _channel will not be properly disposed (Because SshCommand's Dispose() checks for _channel being null).

In the 2013.01.27 release, this type of memory leak did not occur. I suspect it has something to do with the change from AutoResetEvent to ManualResetEvent in channel.cs and session.cs (That's the only major difference I can see).
May 30, 2013 at 5:45 PM
Hours of experimentation in attempting to get SshCommand and/or ChannelSessions to properly dispose have led me nowhere. I obviously don't understand the full extent of the SshNet design (How session relates to SshCommand relates to Channel relates to ChannelSession).

At this time, unless there's some movement on suggestions, the only way I've found to actually get the leaks under control is to completely disconnect the SshClient and dispose of it. Then to re-create it and re-connect/authenticate to the server. Given the millions of commands I need to send during my long-term testing, this is not feasible for every command since OpenSsh will think it's a DDoS and throttle. Re-connecting every minute or so (With hundreds of SshCommand.Execute() calls in between) seems to keep the memory leak under some level of control at least. This is not an optimal solution of course, but it works.
Aug 21, 2013 at 1:17 PM
Exactly my issue, I don't use millions of commands...but when I get to 10 channels...timeout....I'm happy to see another kindred spirit, I have come to the same conclusion (disconnect/reconnect sshClient). but my problem...I have to use an RSA key :-( I am working on corporate policy to get an alternate login situation.....anyway.

If you don't mind, I would appreciate seeing code on how you Re-Connect every minute or so. If I can solve my RSA key issue, i'll still need a way to clear/dispose the channels that stay open. It sounds like your method is good. Thank you!
Aug 22, 2013 at 8:12 AM
XDotNet wrote:
Exactly my issue, I don't use millions of commands...but when I get to 10 channels...timeout....I'm happy to see another kindred spirit, I have come to the same conclusion (disconnect/reconnect sshClient). but my problem...I have to use an RSA key :-( I am working on corporate policy to get an alternate login situation.....anyway.

If you don't mind, I would appreciate seeing code on how you Re-Connect every minute or so. If I can solve my RSA key issue, i'll still need a way to clear/dispose the channels that stay open. It sounds like your method is good. Thank you!
This is not an issue with this lib. 10 channels is the default maximum of the ssh-daemon.
See MaxSessions 10 in your sshd-config
Aug 22, 2013 at 6:34 PM
I have a test application which makes a single long-lasting connection to a servers (Well, a single connection to each server). This connection is 'maintained' via a function that is called in-between sets of command updates (Refreshes). Very single threaded approach...
    private bool MaintainConnection()
    { 
        bool connectionEstablished = false;
        bool refreshConnection = false;

        DateTime connectionRefreshTime = DateTime.Now.Add(new TimeSpan(0,0,-CONNECTIONREFRESHDELAY));
        DateTime reconnectTime = DateTime.Now.Add(new TimeSpan(0,0,-RECONNECTDELAY));

        //Disconnect for refresh
        if (lastConnectAttempt < connectionRefreshTime)
        {
            if(this.client != null)
            {
                this.client.DisconnectSSH();  //Clean out connection
                this.client = null;
                refreshConnection = true;
            }
        }

        //Connection or reconnection attempt
        if (this.client == null)
        {
            if (lastConnectAttempt < reconnectTime)
                Connect();
            else
                return false;  //Bail out until it's time to reconnect
        }

        //Check the new connection
        if(this.client != null && this.client.IsConnected())
        {
            //Do something that proves the connection works like sending data...
            connectionEstablished = true;
        }

        //Check if we just lost the connection
        if (this.isConnected && !connectionEstablished)
        {
            //Do something about it
        }

        //If the conneciton is dead, dispose of it
        if(!connectionEstablished)
        {
            if(this.client != null)
            {   
                this.client.DisconnectSSH();
                this.client = null;
            }
        }

        return connectionEstablished;
    }
    #endregion Private methods
}
I have a wrapper for my client with a few helper functions to assist in using SshNet:
    public virtual bool IsConnected()
    {
        return (this.client != null) && (this.client.IsConnected);
    }

    public bool ConnectSSH(bool pingCheck = false)
    {
        if (valid)
        {
            if (pingCheck)
            {
                PingReply reply = pinger.Send(hostname, 1000);
                if (reply.Status != IPStatus.Success)
                {
                    log.ERROR(
                        "Failed to connect SSH session, can not reach " +
                        this.hostname + "(Error = " +
                        reply.Status.ToString() + " )");

                    return false;
                }
            }

            client = new SshClient(conn);

            try
            {
                client.Connect();
                return client.IsConnected;
            }
            catch (SshAuthenticationException e)
            {
                        log.ERROR(e.Message + 
                            " Check if the correct agent key is loaded for " + 
                            this.username + "@" + this.hostname);
            }
            catch (System.Net.Sockets.SocketException e)
            {
                    log.ERROR(e.Message);
            }
            catch (Exception e)
            {
                    log.ERROR(e.Message);
            }
        }
        return false;
    }

    public void DisconnectSSH()
    {
        try
        {
            if (this.client != null)
            {
                this.client.Disconnect();
                this.client.Dispose();
                this.client = null;
            }
        }
        catch
        {
            //Ignore any errors
            log.ERROR("SSH Client session did not disconnect cleanly.");
        }
    }

What you describe though doesn't sound like the problem I am having (Was having until I coded around it). I'm only creating a single channelsession for each server, but those were not garbage collected unless the entire client was disposed of.
Sep 4, 2013 at 10:10 PM
Edited Sep 5, 2013 at 8:24 PM
See issues 1756 through 1761.

[Proposed Fix] Timeout waiting for KeyExchange with multi-threaded connections
[Proposed Fix] Chasing a serious Event Handle leak,... Part I
[Proposed Fix] Chasing a serious Event Handle leak,... Part II
[Proposed Fix] Chasing a serious Event Handle leak,... Part III
[Proposed Fix] Chasing a serious Event Handle leak,... Part IV

Herein are proposed fixes for a set of issues ranging from hangs, timeouts, failed connections, resource leaks and dispose issues. Taken in order it should help explain the causes and the remedies. Until the changes are adopted, you will have to make them in your own copy of the source. Let me know if any further explianation is needed.