[
https://issues.apache.org/jira/browse/TINKERPOP-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682928#comment-16682928
]
Florian Hockmann commented on TINKERPOP-2090:
---------------------------------------------
{quote}It is returned to the pool through Dispose() method which is called even
in the case of exception.
{quote}
You're right of course. Don't know why I didn't think of that.
{quote}Were you actually sending query through the network? Or perhaps
keep-alive was handled differently compared to Cosmos DB. Not sure about this,
I suggested this method because it worked for me.
{quote}
I used an application with Gremlin.Net on my machine and Gremlin Server inside
a Docker container / inside a VM. I tried both approaches by removing either
the network connection from the VM or detaching the Docker container from its
network, but both only led to temporary exceptions and everything worked again
after 1 or 2 requests after the network was back.
{quote}Stopping server most likely gracefully closes connections.
{quote}
I mainly wanted to simulate a situation with that were the whole machine goes
down like it was described in the response Greg received from Microsoft and I
assume that they shut down machines gracefully. However, I just tried it again
by killing the process without a graceful shutdown (with {{kill -9 [pid]}}) and
I got the same result.
It would be good to get more input from the Cosmos DB team here as the issue
seems to be specific to that database. Just to be clear: I don't want to
exclude the possibility here that we might need to improve the way Gremlin.Net
handles connection failures, but it's hard to find the cause of the problem
until we can reproduce it, ideally with just Gremlin Server.
I'll reach out to the Cosmos DB team to ask them for their take on this issue.
> After running backend for a day or so System.IO.IOException keep throwing
> -------------------------------------------------------------------------
>
> Key: TINKERPOP-2090
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2090
> Project: TinkerPop
> Issue Type: Bug
> Components: dotnet
> Affects Versions: 3.4.0
> Environment: .NET Core 2.1.5
> Microsoft Azure
> Reporter: Saber Karmous
> Priority: Critical
>
> .NET Core 2.1.5
> Gremlin.NET 3.4.0-rc2
> We're using the latest RC of the Gremlin client. And we have a gremlin client
> that's being injected as a singleton through out IoC container. After running
> the backend for a day or two it keeps throwing System.IO.IOExceptions. If we
> restart the application it works again.
> We use Polly for out retry strategy, and retrying for 9 times. But it keeps
> failing.
> I added the stack trace below. Reproducing is a bit of a pain in the behind,
> you have to wait for a day or two for the exception to occur.
> {noformat}
> *no* System.IO.IOException:
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Net.Security.SslStreamInternal+<<WriteSingleChunk>g__CompleteAsync|36_1>d`1.MoveNext
> (System.Net.Security, Version=4.1.1.0, Culture=neutral,
> PublicKeyToken=b03f5f7f11d50a3a)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Net.Security.SslStreamInternal+<<WriteAsyncInternal>g__ExitWriteAsync|35_0>d`1.MoveNext
> (System.Net.Security, Version=4.1.1.0, Culture=neutral,
> PublicKeyToken=b03f5f7f11d50a3a)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.WebSocketConnection+<SendMessageAsync>d__7.MoveNext
> (Gremlin.Net, Version=3.4.0.0, Culture=neutral,
> PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.Connection+<SendAsync>d__13.MoveNext (Gremlin.Net,
> Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.Connection+<SubmitAsync>d__8`1.MoveNext (Gremlin.Net,
> Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.ProxyConnection+<SubmitAsync>d__3`1.MoveNext
> (Gremlin.Net, Version=3.4.0.0, Culture=neutral,
> PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.GremlinClient+<SubmitAsync>d__6`1.MoveNext
> (Gremlin.Net, Version=3.4.0.0, Culture=neutral,
> PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.GremlinClientExtensions+<SubmitAsync>d__4`1.MoveNext
> (Gremlin.Net, Version=3.4.0.0, Culture=neutral,
> PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> boskalis.world.data.repository.Services.AzureCosmosDBGremlinProvider+<>c__DisplayClass9_0`1+<<Query>b__5>d.MoveNext
> (boskalis.world.data.logic, Version=1.0.0.0, Culture=neutral,
> PublicKeyToken=nullboskalis.world.data.logic, Version=1.0.0.0,
> Culture=neutral, PublicKeyToken=null:
> D:\a\1\s\src\boskalis.world.data.logic\Services\AzureCosmosDBGremlinProvider.csboskalis.world.data.logic,
> Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 222)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> Polly.Policy+<>c__DisplayClass181_0`1+<<ExecuteAsyncInternal>b__0>d.MoveNext
> (Polly, Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> Polly.RetrySyntaxAsync+<>c__DisplayClass25_1+<<WaitAndRetryAsync>b__1>d.MoveNext
> (Polly, Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Polly.Retry.RetryEngine+<ImplementationAsync>d__1`1.MoveNext (Polly,
> Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
> Inner exception System.Net.Sockets.SocketException handled at
> System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
> {noformat}
> I'm going to implement a workaround by recycling the client every couple of
> hours.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
