2

Closed

Orphaned threads when port forwarding

description

I'm not sure if this is the same issue as here.

In the situation where a port forward has been established and actively used, therefore a listener thread has been created, if the network is disconnected (by pulling out the cable) and reconnected sometime later then the internal thread will remain there even after everything else has been disposed properly. This causes problems, especially when trying to exit the application since it will hang.

With a fair amount of Console.WriteLineing it was established that the internal listener thread, consisting mostly of the Bind function in ChannelDirectTcpip, eventually exits its main loop correctly but then stalls at this point forever (well at least until a kill -9 is issued):
System.Threading.WaitHandle.WaitAny(new WaitHandle[] { this._channelEof });
I do not know much about the inner workings of SSH.NET and therefore do not know the best solution to the problem, but a workaround that seems to do the job for us is to simply allow that wait to timeout:
System.Threading.WaitHandle.WaitAny(new WaitHandle[] { this._channelEof }, 5000);
After that, the inner thread exits properly and no more hangs on exit.
Closed Apr 6 at 2:56 PM by drieseng

comments

olegkap wrote Mar 7, 2013 at 8:03 PM

Hi,

I am trying to look into this problem now and I think I found couple issues there.
The only thing is that ti use 5000 timeout in this case will mean that all data, during port forwarding will have to occur under 5 sec, which is not always the case so I need to something differently here.
One thing in mind is to handle connection drop gracefully which I will try to resolve as soon as possible.

Thanks,
Oleg

olegkap wrote Mar 7, 2013 at 8:37 PM

Hi,

Can you try 23481 commit?

I identified two areas actually which can cause for thread to hang in case of errors so I think I handle it now.

Please let me know if it fixed your problem or you still having issues.

Thanks,
Oleg

teadriven wrote Mar 7, 2013 at 11:27 PM

Hi,

I wasn't suggesting the timeout as a solution, it was simply a hack to get it working for us in the meantime since our application should only be run on a local network for the now and therefore 5 seconds is reasonable.

I pulled down the source for 23481 and copied a build of the library over to my test bed (a server and a client VM, both linux, with simulated network cable disconnections to test the ability to maintain a tunnel).

Unfortunately there is still a runaway thread that causes the processor usage to ramp up when exiting the application.

Another similar effect is that the IsConnected property blocks for a very long time if the network cable is disconnected (it does not honour the ConnectionInfo.Timeout value). To help with that situation we have to launch a Task to check the connection and attempt a reconnect which means I can then Wait on the Task with a sensible timeout (yet another hack I'm afraid).

Regards,
Dan

olegkap wrote Mar 8, 2013 at 1:05 AM

Hi,

Ok, yea, no problem, just wanted to mentioned about this 5000 hack and make sure you aware of side effect.

As far as locking in IsConnected, there is no locking there with exception of this._socket.Poll method call which could potentially wait for a while.

I guess another idea, can you try this commit: 23468?
This is one of my ideas to fix this problem, this way I dont test for is connected but have a special variable to test for disposing situation.

Since it seems you can easily reproduce this error, can you try this commit and let me know if it works?
If it does, I guess I will make this as a solution.

Thanks,
Oleg

teadriven wrote Mar 9, 2013 at 1:31 AM

Oleg,

I'm afraid the 23468 build does not appear to deal with physical disconnects any differently. The IsConnected property still blocks and the tunnels still leave behind threads.

Regards,
Dan

olegkap wrote Mar 9, 2013 at 2:05 PM

Hmm,
Well, 23468 doesnt handle disconnect differently, it only insures that IsConnected is not called during disposing.
So may be a problem somewhere else :(:(.

Is it possible for you to create a test case that I can run locally to recreate the problem, or may be if possible to connect to test server that you have where I could recreate this problem. You can send me this info privately if such scenario is possible,

I guess one more idea, to prevent isConnected from blocking is to replace this:
                isConnected = (!this._isDisconnecting && this._socket != null && this._socket.Connected && this._isAuthenticated && this._messageListenerCompleted != null)
                    && this._socket.Poll(-1, SelectMode.SelectWrite);
with this:
                isConnected = (!this._isDisconnecting && this._socket != null && this._socket.Connected && this._isAuthenticated && this._messageListenerCompleted != null);
And see if it still hangs on IsConnected


Thanks,
Oleg