[
https://issues.apache.org/jira/browse/TINKERPOP-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681613#comment-16681613
]
Florian Hockmann commented on TINKERPOP-2090:
---------------------------------------------
[~Papapep], thanks for reaching out to Microsoft to get some input from their
side on this.
{quote}What is generally happening is that a request is made to cosmosDb and
1-10 minutes later this error is returned
{quote}
If you're getting this error in less than 10 minutes, then how could it be
explained by the possibilities Microsoft mentioned? I guess they don't perform
updates that often that require restarts of nodes or introduce new firewall
rules which you violate.
{quote}Can the client be enhanced to check if a connection is active prior to
using it? Or maybe close connections on websocket errors?
{quote}
Gremlin.Net already does this. After using a connection, it only returns it
into the pool [if it's still
open|https://github.com/apache/tinkerpop/blob/master/gremlin-dotnet/src/Gremlin.Net/Driver/ConnectionPool.cs#L79].
It also only returns a connection to be used for a request [if the connection
wasn't
closed|https://github.com/apache/tinkerpop/blob/master/gremlin-dotnet/src/Gremlin.Net/Driver/ConnectionPool.cs#L64]
while sitting idle in the pool.
To me it sounds more like we need to get to the reason why the connections are
closed by Cosmos DB before we can think about how to handle the situation in
Gremlin.Net.
Can someone who uses Cosmos DB try out whether the same problem also occurs
with other GLVs? (Ideally with Gremlin-Java as that's probably still the most
advanced driver.)
> After running backend for a day or so System.IO.IOException keep throwing
> -------------------------------------------------------------------------
>
> Key: TINKERPOP-2090
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2090
> Project: TinkerPop
> Issue Type: Bug
> Components: dotnet
> Affects Versions: 3.4.0
> Environment: .NET Core 2.1.5
> Microsoft Azure
> Reporter: Saber Karmous
> Priority: Critical
>
> .NET Core 2.1.5
> Gremlin.NET 3.4.0-rc2
> We're using the latest RC of the Gremlin client. And we have a gremlin client
> that's being injected as a singleton through out IoC container. After running
> the backend for a day or two it keeps throwing System.IO.IOExceptions. If we
> restart the application it works again.
> We use Polly for out retry strategy, and retrying for 9 times. But it keeps
> failing.
> I added the stack trace below. Reproducing is a bit of a pain in the behind,
> you have to wait for a day or two for the exception to occur.
> {noformat}
> *no* System.IO.IOException:
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Net.Security.SslStreamInternal+<<WriteSingleChunk>g__CompleteAsync|36_1>d`1.MoveNext
> (System.Net.Security, Version=4.1.1.0, Culture=neutral,
> PublicKeyToken=b03f5f7f11d50a3a)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Net.Security.SslStreamInternal+<<WriteAsyncInternal>g__ExitWriteAsync|35_0>d`1.MoveNext
> (System.Net.Security, Version=4.1.1.0, Culture=neutral,
> PublicKeyToken=b03f5f7f11d50a3a)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.WebSocketConnection+<SendMessageAsync>d__7.MoveNext
> (Gremlin.Net, Version=3.4.0.0, Culture=neutral,
> PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.Connection+<SendAsync>d__13.MoveNext (Gremlin.Net,
> Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.Connection+<SubmitAsync>d__8`1.MoveNext (Gremlin.Net,
> Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.ProxyConnection+<SubmitAsync>d__3`1.MoveNext
> (Gremlin.Net, Version=3.4.0.0, Culture=neutral,
> PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.GremlinClient+<SubmitAsync>d__6`1.MoveNext
> (Gremlin.Net, Version=3.4.0.0, Culture=neutral,
> PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Gremlin.Net.Driver.GremlinClientExtensions+<SubmitAsync>d__4`1.MoveNext
> (Gremlin.Net, Version=3.4.0.0, Culture=neutral,
> PublicKeyToken=d2035e9aa387a711)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> boskalis.world.data.repository.Services.AzureCosmosDBGremlinProvider+<>c__DisplayClass9_0`1+<<Query>b__5>d.MoveNext
> (boskalis.world.data.logic, Version=1.0.0.0, Culture=neutral,
> PublicKeyToken=nullboskalis.world.data.logic, Version=1.0.0.0,
> Culture=neutral, PublicKeyToken=null:
> D:\a\1\s\src\boskalis.world.data.logic\Services\AzureCosmosDBGremlinProvider.csboskalis.world.data.logic,
> Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 222)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> Polly.Policy+<>c__DisplayClass181_0`1+<<ExecuteAsyncInternal>b__0>d.MoveNext
> (Polly, Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at
> Polly.RetrySyntaxAsync+<>c__DisplayClass25_1+<<WaitAndRetryAsync>b__1>d.MoveNext
> (Polly, Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
> (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
> PublicKeyToken=7cec85d7bea7798e)
> at Polly.Retry.RetryEngine+<ImplementationAsync>d__1`1.MoveNext (Polly,
> Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
> Inner exception System.Net.Sockets.SocketException handled at
> System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
> {noformat}
> I'm going to implement a workaround by recycling the client every couple of
> hours.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
