Saber Karmous created TINKERPOP-2090:
----------------------------------------
Summary: After running backend for a day or so
System.IO.IOException keep throwing
Key: TINKERPOP-2090
URL: https://issues.apache.org/jira/browse/TINKERPOP-2090
Project: TinkerPop
Issue Type: Bug
Components: dotnet
Affects Versions: 3.4.0
Environment: .NET Core 2.1.5
Microsoft Azure
Reporter: Saber Karmous
.NET Core 2.1.5
Gremlin.NET 3.4.0-rc2
We're using the latest RC of the Gremlin client. And we have a gremlin client
that's being injected as a singleton through out IoC container. After running
the backend for a day or two it keeps throwing System.IO.IOExceptions. If we
restart the application it works again.
We use Polly for out retry strategy, and retrying for 9 times. But it keeps
failing.
I added the stack trace below. Reproducing is a bit of a pain in the behind,
you have to wait for a day or two for the exception to occur.
{noformat}
*no* System.IO.IOException:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Net.Security.SslStreamInternal+<<WriteSingleChunk>g__CompleteAsync|36_1>d`1.MoveNext
(System.Net.Security, Version=4.1.1.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Net.Security.SslStreamInternal+<<WriteAsyncInternal>g__ExitWriteAsync|35_0>d`1.MoveNext
(System.Net.Security, Version=4.1.1.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.WebSocketConnection+<SendMessageAsync>d__7.MoveNext
(Gremlin.Net, Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.Connection+<SendAsync>d__13.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.Connection+<SubmitAsync>d__8`1.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.ProxyConnection+<SubmitAsync>d__3`1.MoveNext
(Gremlin.Net, Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.GremlinClient+<SubmitAsync>d__6`1.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.GremlinClientExtensions+<SubmitAsync>d__4`1.MoveNext
(Gremlin.Net, Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
boskalis.world.data.repository.Services.AzureCosmosDBGremlinProvider+<>c__DisplayClass9_0`1+<<Query>b__5>d.MoveNext
(boskalis.world.data.logic, Version=1.0.0.0, Culture=neutral,
PublicKeyToken=nullboskalis.world.data.logic, Version=1.0.0.0, Culture=neutral,
PublicKeyToken=null:
D:\a\1\s\src\boskalis.world.data.logic\Services\AzureCosmosDBGremlinProvider.csboskalis.world.data.logic,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 222)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
Polly.Policy+<>c__DisplayClass181_0`1+<<ExecuteAsyncInternal>b__0>d.MoveNext
(Polly, Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
Polly.RetrySyntaxAsync+<>c__DisplayClass25_1+<<WaitAndRetryAsync>b__1>d.MoveNext
(Polly, Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Polly.Retry.RetryEngine+<ImplementationAsync>d__1`1.MoveNext (Polly,
Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
Inner exception System.Net.Sockets.SocketException handled at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
{noformat}
I'm going to implement a workaround by recycling the client every couple of
hours.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
