This project is read-only.

Odd reoccurring exception

Jul 15, 2011 at 7:32 PM
Edited Jul 15, 2011 at 7:33 PM

Hey all,

Every once in a while Ill get an exception on line 421 of Session.cs:

this._socket = new Socket(ep.AddressFamily, SocketType.Stream, ProtocolType.Tcp);

It doesnt happen all the time, even between runs with no changes made anywhere...  The exception is below.

Thanks, this one is confusing...

System.AccessViolationException was unhandled
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=System
  StackTrace:
       at System.Net.UnsafeNclNativeMethods.OSSOCK.WSASocket(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType, IntPtr protocolInfo, UInt32 group, SocketConstructorFlags flags)
       at System.Net.SafeCloseSocket.InnerSafeCloseSocket.CreateWSASocket(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType)
       at System.Net.Sockets.Socket..ctor(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType)
       at Renci.SshNet.Session.Connect() in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\Renci.SshNet\Renci.SshNet\Session.cs:line 421
       at Renci.SshNet.BaseClient.Connect() in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\Renci.SshNet\Renci.SshNet\BaseClient.cs:line 103
       at WorkMan_Release_Monitor.UnixConnection.SendUnixCommand(PasswordConnectionInfo connectionInfo, String serverIP, String command) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\UnixConnection.cs:line 66
       at WorkMan_Release_Monitor.UnixConnection..ctor(UnixServer server, Boolean shell, String command) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\UnixConnection.cs:line 37
       at WorkMan_Release_Monitor.ModuleStatus.CheckPrimaryPair(WorkManModule module, UnixServer server1, UnixServer server2) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\ModuleStatus.cs:line 178
       at WorkMan_Release_Monitor.ModuleStatus.CheckModuleStatus(WorkManModule module) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\ModuleStatus.cs:line 94
       at WorkMan_Release_Monitor.ModuleStatus.WorkerThread_DoWork(Object sender, DoWorkEventArgs e) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\ModuleStatus.cs:line 66
       at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
       at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
       at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
       at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.DoAsyncCall()
       at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.ThreadPoolCallBack(Object o)
       at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
  InnerException: 
 

Jul 17, 2011 at 9:40 AM
Edited Jul 17, 2011 at 9:42 AM

Hi, I suspect that this might be the result of your system configuration. I will outline some steps to try get the full picture.

Try to log each time this exception occurs, and see if there is a pattern. You can simply use notepad and press F5 every time it happens, or check the Edit menu, it should be the last in the list.

  • Are you using some kind of network limiter software?
  • Can you check in msinfo32.exe ->Components -> Network -> WinSock what is listed there
  • Are you getting this in the debugger?
  • Are you running your application outside the debugger?
  • Is it a debug build? (So you can attach debugger to it)
  • Can you get a list of loaded modules by the application? (In VS, Debug -> Windows -> Modules)
  • If I remember correctly, I believe VS 2010 ships with Parallel Stack, can you take a screenshot of that, it should be under Debug -> Windows -> Parallel Stack, if memory serves me. This is to see if there might be any threading issues.
  • Under what privileges are you running the application? (Normal user or as an Administrator)
  • What .NET Framework versions do you have? Check %windir%\Microsoft.NET\Framework\
  • Is your Windows up-to-date?
  • Do you have a firewall, antivirus or similar that provides network protection of sorts?
  • Check the event log for network related entries. (Simply run eventvwr.msc -> Windows Logs)

 

Jul 18, 2011 at 10:07 PM
kenneth_aa wrote:

Quite the list, Ill do my best to anwser

  • Are you using some kind of network limiter software?
    • Not sure, wouldn't surprise me though
  • Can you check in msinfo32.exe ->Components -> Network -> WinSock what is listed there
    • winsock.dll, version 3.10.0.103 and wsock32.dll, version 6.1.7600.16385
  • Are you getting this in the debugger?
    • Yes
  • Are you running your application outside the debugger?
    • Yes, no exception is thown, but that connection fails so the update it was supposed to do doesn't happen
  • Is it a debug build? (So you can attach debugger to it)
    • VS2010 C# Express, so its always in debug mode haha
  • Can you get a list of loaded modules by the application? (In VS, Debug -> Windows -> Modules)
    • VS2010 C# Express doesn't have that option
  • If I remember correctly, I believe VS 2010 ships with Parallel Stack, can you take a screenshot of that, it should be under Debug -> Windows -> Parallel Stack, if memory serves me. This is to see if there might be any threading issues.
    • VS2010 C# Express doesn't ahve that option
  • Under what privileges are you running the application? (Normal user or as an Administrator)
    • Normal (80% sure), not sure how they set up my account at work
  • What .NET Framework versions do you have? Check %windir%\Microsoft.NET\Framework\
    • Latest, 4.0
  • Is your Windows up-to-date?
    • Yep
  • Do you have a firewall, antivirus or similar that provides network protection of sorts?
    • Yes, but I dont think this could be it as it only seems to fail when Im sending out multiple connects rapidly.
  • Check the event log for network related entries. (Simply run eventvwr.msc -> Windows Logs)
    • Ill get to this one later

 

 

Jul 18, 2011 at 10:11 PM

New one

System.AccessViolationException was unhandled
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=System
  StackTrace:
       at System.Net.UnsafeNclNativeMethods.OSSOCK.WSASocket(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType, IntPtr protocolInfo, UInt32 group, SocketConstructorFlags flags)
       at System.Net.SafeCloseSocket.InnerSafeCloseSocket.CreateWSASocket(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType)
       at System.Net.Sockets.Socket..ctor(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType)
       at Renci.SshNet.Session.Connect() in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\Renci.SshNet\Renci.SshNet\Session.cs:line 421
       at Renci.SshNet.BaseClient.Connect() in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\Renci.SshNet\Renci.SshNet\BaseClient.cs:line 103
       at WorkMan_Release_Monitor.TailBackgroundWorker.TailBackgroundWorker_DoWork(Object sender, DoWorkEventArgs e) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\Threading.cs:line 71
       at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
       at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
       at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
       at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.DoAsyncCall()
       at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.ThreadPoolCallBack(Object o)
       at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
  InnerException:

Jul 19, 2011 at 6:56 AM
Edited Jul 19, 2011 at 6:58 AM

Hi

It could be that you are hitting the limited ten half-open connection in tcpip.sys connection limit, read more http://social.technet.microsoft.com/Forums/en/itprovistanetworking/thread/9c268fd2-2566-4c65-ba57-fc339709f4db

This limit applies for all Windows versions after Windows XP SP2. 

Hi,
I would like to share the following information with you.

The limited TCP connections helps to limit the speed at which malicious programs, such as viruses and worms, spread to uninfected computers. Malicious programs often attempt to reach uninfected computers by opening simultaneous connections to random IP addresses. Most of these random addresses result in a failed connection, so a burst of such activity on a computer is a signal that it may have been infected by a malicious program.

There is no workaround from Microsoft to override this limitation. It is hard coded into TCPIP.SYS. There are third party workarounds, but these are not support by Microsoft and these workarounds may break if a Windows Vista update or Service Pack replaces the TCP/IP drivers.

Thank you for your understanding.

 

You can check which modules are loaded by using process monitor from sysinternals, this also might give you stack information. http://technet.microsoft.com/en-us/sysinternals/bb896645

To check if you are running as an admin, I think visual studio adds [Administrator] in the window title.

How many outgoing connections are you trying to make? Does it only happen when you are connecting to several hosts simultaniously?

What Firewall vendor are you using?

Jul 20, 2011 at 3:33 PM
Edited Jul 20, 2011 at 3:54 PM

So now this error is crashing my program even while handling exceptions...

As far as connections.  I have 6 items that each run on 4 different servers.  I must get a value from a file on each server for each item, for a total of 24 connections.  The way I do it is by launching a thread for each of the 6 items.  Each item then checks the file it needs on the 4 servers (4 connections per thread).  However its a super fast check and I kinda assumed that they would finish and close before any issues happened.

If you need more detail let me know

EDIT: Also, I just checked the System Event Log, which, according to our IT fella logs any instances of throttling TCP connections based on the change since XP SP2 you alluded to and there are no TCP errors or warnings, which brings me back to thinking it might be a bug in the library...

Jul 20, 2011 at 9:15 PM
Edited Jul 20, 2011 at 9:18 PM

Another common one Im getting:

This is driving me bonkers... The worst part is that sometimes itll work fine and sometimes itll throw the above and below exceptions and sometimes it just straight crashes out and dies no exceptions... I cant tell ifs its my code, the library or some unholy combo of both

System.AccessViolationException was unhandled
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=System
  StackTrace:
       at System.Net.UnsafeNclNativeMethods.OSSOCK.setsockopt(SafeCloseSocket socketHandle, SocketOptionLevel optionLevel, SocketOptionName optionName, Int32& optionValue, Int32 optionLength)
       at System.Net.Sockets.Socket.SetSocketOption(SocketOptionLevel optionLevel, SocketOptionName optionName, Int32 optionValue, Boolean silent)
       at System.Net.Sockets.Socket.SetSocketOption(SocketOptionLevel optionLevel, SocketOptionName optionName, Int32 optionValue)
       at System.Net.Sockets.Socket.SetSocketOption(SocketOptionLevel optionLevel, SocketOptionName optionName, Boolean optionValue)
       at Renci.SshNet.Session.Connect() in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\Renci.SshNet\Renci.SshNet\Session.cs:line 424
       at Renci.SshNet.BaseClient.Connect() in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\Renci.SshNet\Renci.SshNet\BaseClient.cs:line 103
       at WorkMan_Release_Monitor.UnixConnection.SendUnixCommand(PasswordConnectionInfo connectionInfo, String serverIP, String command) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\UnixConnection.cs:line 87
       at WorkMan_Release_Monitor.UnixConnection..ctor(UnixServer server, Boolean shell, String command) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\UnixConnection.cs:line 54
       at WorkMan_Release_Monitor.ModuleStatus.CheckPrimaryPair(UnixServer server1, UnixServer server2) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\ModuleStatus.cs:line 104
       at WorkMan_Release_Monitor.ModuleStatus.CheckModuleStatus() in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\ModuleStatus.cs:line 48
       at WorkMan_Release_Monitor.Utilities.UpdateModule_DoWork(Object module) in C:\Users\lhersman\Documents\Visual Studio 2010\Projects\WorkMan Release Monitor [DEV]\WorkMan Release Monitor\Utilities.cs:line 305
       at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
  InnerException:

Jul 20, 2011 at 9:16 PM
Edited Jul 20, 2011 at 9:46 PM

New interestingness: So the main UI thread is what runs the loop that spawns the 6 threads that each do thier 4 server connections.... Now, when I did:

foreach (ModuleStatus module in moduleList)
{
    ThreadPool.QueueUserWorkItem(new WaitCallback(UpdateModule_DoWork), module);
    Thread.Sleep(100);
}

It seems to work every time (with the trade off of the UI thread being hung for .6 seconds)... So to fix that I tried putting the the sleep in the first like of the UpdateModule_DoWork function:

private static void UpdateModule_DoWork(object sender, System.ComponentModel.DoWorkEventArgs e)
{
    Thread.Sleep(100);
    ((ModuleStatus)e.Argument).CheckModuleStatus();
}

...and predicitably it either produces those about exceptions or crashes, but then I realized that this makes sense because the 6 threads are still starting at the same time (just .1 seconds later than with no sleeps anywhere).  This leads me to belive that it is either the framework not being able to handle that many connections or is something windows based and not my code.

Jul 28, 2011 at 3:43 PM

Any thoughts on this one... The fact that I have to timeout between thread starts kinda confirms its probably a library issue...

Coordinator
Jul 29, 2011 at 3:29 PM

Here my thought.

Because the error that you get is "AccessViolationException" and in System.Net.UnsafeNclNativeMethods ... makes me to think that something happen on lower level since I do not use any unsafe method in the library.

Just a guess, may be there is something going on with the UI thread.

The only thing I guess you should be aware in this case is that I limit number of simultaneous connection attempts to 10, I believe. The reason is that most SSH server detect such a behaviour and simply reject any connection, or do something about it, to avoid so many connections at the same time. I guess as a defense from cyber attack.

 

Try to add some kind of synchronization if it make sense.

 

Oleg

May 21, 2013 at 10:37 AM
Apologies for digging up an old thread, but I'm getting a similar issue.. but only when running on 64-bit (havent seen it on 32-bit OS so far). It's 100% reproducible on 64-bit, so hopefully we can track it down:

Application: NodeService.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
Stack:
at Renci.SshNet.ShellStream.Channel_DataReceived(System.Object, Renci.SshNet.Common.ChannelDataEventArgs)
at Renci.SshNet.Channels.Channel.OnData(Byte[])
at Renci.SshNet.Channels.Channel.OnChannelData(System.Object, Renci.SshNet.MessageEventArgs`1<Renci.SshNet.Messages.Connection.ChannelDataMessage>)
at Renci.SshNet.Session.OnChannelDataReceived(Renci.SshNet.Messages.Connection.ChannelDataMessage)
at Renci.SshNet.Session.HandleMessage(Renci.SshNet.Messages.Connection.ChannelDataMessage)
at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid2[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Runtime.CompilerServices.CallSite, System.__Canon, System.__Canon)
at Renci.SshNet.Session.HandleMessageCore(Renci.SshNet.Messages.Message)
at Renci.SshNet.Session.MessageListener()
at Renci.SshNet.Session.<Connect>b__4()
at Renci.SshNet.Session+<>c__DisplayClass3d.<ExecuteThread>b__3c(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

Machine was running Windows 7 64-bit, with .Net 4.5.

I'll try installing VS on the machine and see if I can get any insight.

thx
Steve