Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-04-19 Thread Stephen Mallette
I just thought I'd close this discussion up a bit as the PR is now merged. I think it's worth noting that as long as the TinkerPop Transaction API does not have a universally accepted method for passing around transaction state among threads we will always have a situation where we need to keep a s

Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-04-05 Thread Stephen Mallette
I've opened a PR for the UnifiedChannelizer: https://github.com/apache/tinkerpop/pull/1414 Over the weekend I passed 10s of millions of messages through the UnifiedChannelizer and the server stayed quite stable throughout it all. With all the tests passing nicely now in travis, docker and locally

Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-04-02 Thread Stephen Mallette
I've solved the Travis issue. Some tests simply required more threads in gremlinPool than the default which is set to the available cores. On Travis you probably get a machine with 2 cores available and some tests required 3 or more to keep that many sessions open at once. Silly problem though the

Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-04-01 Thread Stephen Mallette
I've been fussing with Travis for a while now haven't had much luck figuring out what is wrong. Decided to start profiling the server to see if that provided any hints (and it was something I intended to do in any case). Two issues came up: 1. shouldEventuallySucceedOnSameServerWithDefault() had s

Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-03-29 Thread Stephen Mallette
I can't seem to get Travis to pass the build for this branch with the UnifiedChannelizer enabled. https://travis-ci.com/github/apache/tinkerpop/builds/221427334 It just hangs without much information. I can't seem to get it to behave otherwise and it doesn't fail for me locally (of course). If an

Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-03-25 Thread Stephen Mallette
It took me a long time to get to the final steps of this change. Uncovered some unexpected bugs in the WsAndHttpChannelizer and that was gnarly to sort out. Also found some problems with authorization through the SaslAndHttpBasicAuthenticationHandler. I can't say I really like how these combined ch

Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-03-23 Thread Stephen Mallette
More reasonable progress with the UnifiedChannelizer. The short version is that I have a pretty major quantity of the server integration tests passing under this new Channelizer. The ones that are failing are typically ones that assert some specific log messages or exception message which may not n

Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-03-22 Thread Stephen Mallette
I've started to play with some ideas for pushing all requests scripts/bytecode sessions/sessionless down one unified pipeline with one shared thread pool. The most interesting thing I've seen so far is how much the code could potentially simplify under this approach. The GremlinExecutor was the wro

Re: [DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-03-18 Thread Divij Vaidya
> > I suppose the > question is how would we ensure that each request for a session ends up > being executed on the same thread from the previous request if jobs are > randomly submitted to a worker pool? I haven't thought through the details, but on top of my head, we would have to maintain some

[DISCUSS] Converging the worker pool was: [DISCUSS] Remote Transactions

2021-03-18 Thread Stephen Mallette
I started a fresh thread for this topic Divij brought up, with more context: > In a scenario where we have both > session-less and sesion-ful requests being made to the server, the resource > allocation will not be fair and may lead to starvation for some requests. > This problem exists even today