Sami created TINKERPOP-2043:
-------------------------------
Summary: CloseAndRemoveAllConnections bubbles up
System.Net.Http.WinHttpException and doesn't recover
Key: TINKERPOP-2043
URL: https://issues.apache.org/jira/browse/TINKERPOP-2043
Project: TinkerPop
Issue Type: Bug
Components: dotnet
Affects Versions: 3.4.0
Environment: Gremlin.Net 3.4.0-rc1
Microsoft.NetCore.App v2.0
Reporter: Sami
We have a .Net Core web api with an Azure Cosmos db backend.
Our data access layer calls
_gremlinConnection.Client.SubmitAsync<dynamic>(query) using a singleton gremlin
connection. This has worked well except when the API is called with many
parallel web api requests. The endpoints that we provide make around 5 gremlin
calls (some to read, some to write cosmos db edges).
During performance testing we noticed System.Net.Http.WinHttpException
Looking into the issue, we found that NrConnections was at 96 and the stack
trace showed that Gremlin.Net called
ConnectionPool.CloseAndRemoveAllConnections.
We would expect that Gremlin.Net would recover after a call to
CloseAndRemoveAllConnections but instead it bubbled up the
System.Net.Http.WinHttpException and subsequent calls to our API failed with
the same System.Net.Http.WinHttpException.
Would you agree that Gremlin.Net's ConnectionPool.CloseAndRemoveAllConnections
should recover on its own and +complete+ its Client.SubmitAsync call or do you
expect the consumer of Gremlin.Net to provide some work-around? If so, how can
the pool be re-initiated or reset from outside of the Gremlin.Net library?
Here is the stack trace. Please note that we are ready for a release and this
is holding us back. By the way, we do use Gremlin.Net 3.4 RC1 but 3.3.3 also
proved to have the same issue.
{code:java}
// code placeholder
ProjectX.Exceptions.PersistenceFailedException:
at ProjectX.Repository.Cosmos.UnitOfWork+<DeletePersonAsync>d__32.MoveNext
(<...removed...>)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at ProjectX.ModelAPI.Controllers.PersonsController+<RemovePerson>d__7.MoveNext
(<...removed...>)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeActionMethodAsync>d__12.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeNextActionFilterAsync>d__10.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60)
at
Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker+<InvokeInnerFilterAsync>d__14.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeNextExceptionFilterAsync>d__23.MoveNext
(Microsoft.AspNetCore.Mvc.Core, Version=2.0.2.0, Culture=neutral,
PublicKeyToken=adb9793829ddae60)
Inner exception System.Net.Http.WinHttpException handled at
ProjectX.Repository.Cosmos.UnitOfWork+<DeletePersonAsync>d__32.MoveNext:
at System.Net.WebSockets.WinHttpWebSocket.InternalCloseAsync
(System.Net.WebSockets.Client, Version=4.1.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a)
at System.Net.WebSockets.WinHttpWebSocket+<CloseAsync>d__30.MoveNext
(System.Net.WebSockets.Client, Version=4.1.0.0, Culture=neutral,
PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.WebSocketConnection+<CloseAsync>d__4.MoveNext
(Gremlin.Net, Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.Connection+<CloseAsync>d__10.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.ConnectionPool+<TeardownAsync>d__16.MoveNext
(Gremlin.Net, Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Process.Utils.WaitUnwrap (Gremlin.Net, Version=3.4.0.0,
Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at Gremlin.Net.Driver.ConnectionPool.CloseAndRemoveAllConnections (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at Gremlin.Net.Driver.ConnectionPool.AddConnectionIfOpen (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at Gremlin.Net.Driver.ProxyConnection.Dispose (Gremlin.Net, Version=3.4.0.0,
Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at Gremlin.Net.Driver.GremlinClient+<SubmitAsync>d__6`1.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at Gremlin.Net.Driver.GremlinClientExtensions+<SubmitAsync>d__4`1.MoveNext
(Gremlin.Net, Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at ProjectX.Repository.Cosmos.Writer+<ExecuteGremlinCommand>d__7.MoveNext
(<...removed...>)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=7cec85d7bea7798e)
at ProjectX.Repository.Cosmos.UnitOfWork+<DeletePersonAsync>d__32.MoveNext
(<...removed...>)
{code}
Thank you so much for looking into this.
We're hoping for a reasonably quick fix or good work-around strategy.
Sami Abbushi
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
