[
https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5802:
--------------------------------------
Description:
During an UPSERT SELECT query or client-side DELETE, we clone the existing
Phoenix connection and call
[mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
which eventually calls either
[upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
or
[deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
When getting iterators, if we encounter any exception, then on calling
[close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
we attempt to cancel any queued work, while [accumulating already-started task
futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
Later we attempt to block on getting the result (iterator) of already-started
tasks and [close each
iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
Here however, if any of the upsert-select or client-side delete tasks had
thrown an exception _E_, we wouldn't be able to close the underlying iterator
(which in-turn would've [closed the cloned
connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
This manifests as [these
logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
{code:java}
"Failed to execute task during cancel", <exception _E_ stack trace>
{code}
This is leading to a connection leak. Because we use CQSI to create the Phoenix
Connection here, it also gets accounted for during open Phoenix connection
throttling, thus leaving the client application with less connections that can
be opened.
was:
During an UPSERT SELECT query or client-side DELETE, we clone the existing
Phoenix connection and call
[mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
which eventually calls either
[upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
or
[deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
When getting iterators, if we encounter any exception, then on calling
[close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
we attempt to cancel any queued work, while [accumulating already-started task
futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
Later we attempt to block on getting the result (iterator) of already-started
tasks and [close each
iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
Here however, if any of the upsert-select or client-side delete tasks had
thrown an exception _E_, we wouldn't be able to close the underlying iterator
(which in-turn [closes the cloned
connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
This manifests as [these
logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
{code:java}
"Failed to execute task during cancel", <exception _E_ stack trace>
{code}
This is leading to a connection leak. Because we use CQSI to create the Phoenix
Connection here, it also gets accounted for during open Phoenix connection
throttling, thus leaving the client application with less connections that can
be opened.
> Connection leaks in UPSERT SELECT/DELETE paths due to
> MutatingParallelIteratorFactory iterator not being closed
> ---------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-5802
> URL: https://issues.apache.org/jira/browse/PHOENIX-5802
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.0.0, 4.15.0
> Reporter: Chinmay Kulkarni
> Assignee: Chinmay Kulkarni
> Priority: Major
> Fix For: 5.1.0, 4.16.0
>
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing
> Phoenix connection and call
> [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
> which eventually calls either
> [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
> or
> [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
> When getting iterators, if we encounter any exception, then on calling
> [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
> we attempt to cancel any queued work, while [accumulating already-started
> task
> futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
>
> Later we attempt to block on getting the result (iterator) of
> already-started tasks and [close each
> iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
> Here however, if any of the upsert-select or client-side delete tasks had
> thrown an exception _E_, we wouldn't be able to close the underlying iterator
> (which in-turn would've [closed the cloned
> connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
> This manifests as [these
> logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
> {code:java}
> "Failed to execute task during cancel", <exception _E_ stack trace>
> {code}
> This is leading to a connection leak. Because we use CQSI to create the
> Phoenix Connection here, it also gets accounted for during open Phoenix
> connection throttling, thus leaving the client application with less
> connections that can be opened.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
