[jira] [Assigned] (PHOENIX-5803) Add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802
[ https://issues.apache.org/jira/browse/PHOENIX-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni reassigned PHOENIX-5803: - Assignee: Chinmay Kulkarni > Add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802 > - > > Key: PHOENIX-5803 > URL: https://issues.apache.org/jira/browse/PHOENIX-5803 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Major > Labels: phoenix-hardening > Fix For: 5.1.0, 4.16.0 > > > WhereConstantParser should be in the util package rather than coprocessor. > We should also refactor, remove anonymous classes, etc. in > BaseResultIterators, MutatingResultIteratorFactory, UpsertCompiler, etc. > Also need to add unit tests for all these classes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5806) Add a framework to assert relevant metric values between runs of each test
[ https://issues.apache.org/jira/browse/PHOENIX-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5806: -- Description: We should ensure that certain metrics are the expected value in between each test run. For example, each test should ensure resources (such as connections/statements) are closed after their use and can assert that the total Phoenix connection count is zero. This Jira broadly aims at 2 things: # Modifying existing tests so that we do not leak resources in the test itself # Adding a common framework/interface, etc. which all "relevant" tests can inherit behavior to assert specific values for metrics and resource usage like query failure counts, open phoenix connection counts, etc. The idea is to have something similar as is aimed in [PHOENIX-5296|https://issues.apache.org/jira/browse/PHOENIX-5296] The idea is to use this Jira to brainstorm various ideas. This will also help uncover resource leaks/incorrect metric updates throughout the code base. was: We should ensure that certain metrics are the expected value in between each test run. For example, each test should ensure resources (such as connections/statements) are closed after their use and can assert that the total Phoenix connection count is zero. This Jira broadly aims at 2 things: # Modifying existing tests so that we do not leak resources in the test itself # Adding a common framework/interface, etc. which all "relevant" tests can inherit behavior from to assert specific values for metrics and resource usage like query failure counts, open phoenix connection counts, etc. The idea is to have something similar as is aimed in [PHOENIX-5296|https://issues.apache.org/jira/browse/PHOENIX-5296] The idea is to use this Jira to brainstorm various ideas. This will also help uncover resource leaks/incorrect metric updates throughout the code base. > Add a framework to assert relevant metric values between runs of each test > -- > > Key: PHOENIX-5806 > URL: https://issues.apache.org/jira/browse/PHOENIX-5806 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: phoenix-hardening > > We should ensure that certain metrics are the expected value in between each > test run. For example, each test should ensure resources (such as > connections/statements) are closed after their use and can assert that the > total Phoenix connection count is zero. > This Jira broadly aims at 2 things: > # Modifying existing tests so that we do not leak resources in the test > itself > # Adding a common framework/interface, etc. which all "relevant" tests can > inherit behavior to assert specific values for metrics and resource usage > like query failure counts, open phoenix connection counts, etc. The idea is > to have something similar as is aimed in > [PHOENIX-5296|https://issues.apache.org/jira/browse/PHOENIX-5296] > The idea is to use this Jira to brainstorm various ideas. This will also help > uncover resource leaks/incorrect metric updates throughout the code base. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5806) Add a framework to assert relevant metric values between runs of each test
[ https://issues.apache.org/jira/browse/PHOENIX-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5806: -- Description: We should ensure that certain metrics are the expected value in between each test run. For example, each test should ensure resources (such as connections/statements) are closed after their use and can assert that the total Phoenix connection count is zero. This Jira broadly aims at 2 things: # Modifying existing tests so that we do not leak resources in the test itself # Adding a common framework/interface, etc. which all "relevant" tests can inherit behavior to assert specific values for metrics and resource usage like query failure counts, open phoenix connection counts, etc. We can implement something similar as is aimed in [PHOENIX-5296|https://issues.apache.org/jira/browse/PHOENIX-5296] The idea is to use this Jira to brainstorm various ideas. This will also help uncover resource leaks/incorrect metric updates throughout the code base. was: We should ensure that certain metrics are the expected value in between each test run. For example, each test should ensure resources (such as connections/statements) are closed after their use and can assert that the total Phoenix connection count is zero. This Jira broadly aims at 2 things: # Modifying existing tests so that we do not leak resources in the test itself # Adding a common framework/interface, etc. which all "relevant" tests can inherit behavior to assert specific values for metrics and resource usage like query failure counts, open phoenix connection counts, etc. The idea is to have something similar as is aimed in [PHOENIX-5296|https://issues.apache.org/jira/browse/PHOENIX-5296] The idea is to use this Jira to brainstorm various ideas. This will also help uncover resource leaks/incorrect metric updates throughout the code base. > Add a framework to assert relevant metric values between runs of each test > -- > > Key: PHOENIX-5806 > URL: https://issues.apache.org/jira/browse/PHOENIX-5806 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: phoenix-hardening > > We should ensure that certain metrics are the expected value in between each > test run. For example, each test should ensure resources (such as > connections/statements) are closed after their use and can assert that the > total Phoenix connection count is zero. > This Jira broadly aims at 2 things: > # Modifying existing tests so that we do not leak resources in the test > itself > # Adding a common framework/interface, etc. which all "relevant" tests can > inherit behavior to assert specific values for metrics and resource usage > like query failure counts, open phoenix connection counts, etc. We can > implement something similar as is aimed in > [PHOENIX-5296|https://issues.apache.org/jira/browse/PHOENIX-5296] > The idea is to use this Jira to brainstorm various ideas. This will also help > uncover resource leaks/incorrect metric updates throughout the code base. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5806) Add a framework to assert relevant metric values between runs of each test
[ https://issues.apache.org/jira/browse/PHOENIX-5806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5806: -- Labels: phoenix-hardening (was: ) > Add a framework to assert relevant metric values between runs of each test > -- > > Key: PHOENIX-5806 > URL: https://issues.apache.org/jira/browse/PHOENIX-5806 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: phoenix-hardening > > We should ensure that certain metrics are the expected value in between each > test run. For example, each test should ensure resources (such as > connections/statements) are closed after their use and can assert that the > total Phoenix connection count is zero. > This Jira broadly aims at 2 things: > # Modifying existing tests so that we do not leak resources in the test > itself > # Adding a common framework/interface, etc. which all "relevant" tests can > inherit behavior from to assert specific values for metrics and resource > usage like query failure counts, open phoenix connection counts, etc. The > idea is to have something similar as is aimed in > [PHOENIX-5296|https://issues.apache.org/jira/browse/PHOENIX-5296] > The idea is to use this Jira to brainstorm various ideas. This will also help > uncover resource leaks/incorrect metric updates throughout the code base. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5806) Add a framework to assert relevant metric values between runs of each test
Chinmay Kulkarni created PHOENIX-5806: - Summary: Add a framework to assert relevant metric values between runs of each test Key: PHOENIX-5806 URL: https://issues.apache.org/jira/browse/PHOENIX-5806 Project: Phoenix Issue Type: Improvement Affects Versions: 4.15.0, 5.0.0 Reporter: Chinmay Kulkarni We should ensure that certain metrics are the expected value in between each test run. For example, each test should ensure resources (such as connections/statements) are closed after their use and can assert that the total Phoenix connection count is zero. This Jira broadly aims at 2 things: # Modifying existing tests so that we do not leak resources in the test itself # Adding a common framework/interface, etc. which all "relevant" tests can inherit behavior from to assert specific values for metrics and resource usage like query failure counts, open phoenix connection counts, etc. The idea is to have something similar as is aimed in [PHOENIX-5296|https://issues.apache.org/jira/browse/PHOENIX-5296] The idea is to use this Jira to brainstorm various ideas. This will also help uncover resource leaks/incorrect metric updates throughout the code base. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
[
https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5802:
--
Labels: phoenix-hardening (was: )
> Connection leaks in UPSERT SELECT/DELETE paths due to
> MutatingParallelIteratorFactory iterator not being closed
> ---
>
> Key: PHOENIX-5802
> URL: https://issues.apache.org/jira/browse/PHOENIX-5802
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: Chinmay Kulkarni
>Assignee: Chinmay Kulkarni
>Priority: Major
> Labels: phoenix-hardening
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5802-4.x-v1.patch, PHOENIX-5802-4.x-v2.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing
> Phoenix connection and call
> [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
> which eventually calls either
> [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
> or
> [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
> When getting iterators, if we encounter any exception, then on calling
> [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
> we attempt to cancel any queued work, while [accumulating already-started
> task
> futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
>
> Later we attempt to block on getting the result (iterator) of
> already-started tasks and [close each
> iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
> Here however, if any of the upsert-select or client-side delete tasks had
> thrown an exception _E_, we wouldn't be able to close the underlying iterator
> (which in-turn would've [closed the cloned
> connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
> This manifests as [these
> logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
> {code:java}
> "Failed to execute task during cancel",
> {code}
> This is leading to a connection leak. Because we use CQSI to create the
> Phoenix Connection here, it also gets accounted for during open Phoenix
> connection throttling, thus leaving the client application with less
> connections that can be opened.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PHOENIX-5803) Add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802
[ https://issues.apache.org/jira/browse/PHOENIX-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5803: -- Summary: Add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802 (was: Refactor and add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802) > Add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802 > - > > Key: PHOENIX-5803 > URL: https://issues.apache.org/jira/browse/PHOENIX-5803 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: phoenix-hardening > Fix For: 5.1.0, 4.16.0 > > > WhereConstantParser should be in the util package rather than coprocessor. > We should also refactor, remove anonymous classes, etc. in > BaseResultIterators, MutatingResultIteratorFactory, UpsertCompiler, etc. > Also need to add unit tests for all these classes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5805) Verify GlobalClientMetric values and add more testing around various metrics
[ https://issues.apache.org/jira/browse/PHOENIX-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5805: -- Labels: phoenix-hardening (was: ) > Verify GlobalClientMetric values and add more testing around various metrics > > > Key: PHOENIX-5805 > URL: https://issues.apache.org/jira/browse/PHOENIX-5805 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Blocker > Labels: phoenix-hardening > Fix For: 5.1.0, 4.16.0 > > > We have seen situations where Phoenix GlobalClientMetrics show up with zero > values (for ex, 'tq') and no tests catch this. This Jira is to track efforts > to verify metric values are being updated correctly and add more rigorous > testing around the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5805) Verify GlobalClientMetric values and add more testing around various metrics
[ https://issues.apache.org/jira/browse/PHOENIX-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5805: -- Description: We have seen situations where Phoenix GlobalClientMetrics show up with zero values (for ex, 'tq') and no tests catch this. This Jira is to track efforts to verify metric values are being updated correctly and add more rigorous testing around the same. (was: We have seen situations where Phoenix GlobalClientMetrics show up with zero values and no tests catch this. This Jira is to track efforts to verify metric values are being updated correctly and add more rigorous testing around the same. ) > Verify GlobalClientMetric values and add more testing around various metrics > > > Key: PHOENIX-5805 > URL: https://issues.apache.org/jira/browse/PHOENIX-5805 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Blocker > Fix For: 5.1.0, 4.16.0 > > > We have seen situations where Phoenix GlobalClientMetrics show up with zero > values (for ex, 'tq') and no tests catch this. This Jira is to track efforts > to verify metric values are being updated correctly and add more rigorous > testing around the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5805) Verify GlobalClientMetric values and add more testing around various metrics
Chinmay Kulkarni created PHOENIX-5805: - Summary: Verify GlobalClientMetric values and add more testing around various metrics Key: PHOENIX-5805 URL: https://issues.apache.org/jira/browse/PHOENIX-5805 Project: Phoenix Issue Type: Improvement Affects Versions: 4.15.0, 5.0.0 Reporter: Chinmay Kulkarni Fix For: 5.1.0, 4.16.0 We have seen situations where Phoenix GlobalClientMetrics show up with zero values and no tests catch this. This Jira is to track efforts to verify metric values are being updated correctly and add more rigorous testing around the same. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5748) Simplify index update generation code for consistent global indexes
[ https://issues.apache.org/jira/browse/PHOENIX-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kadir OZDEMIR updated PHOENIX-5748: --- Attachment: PHOENIX-5748.master.001.patch > Simplify index update generation code for consistent global indexes > > > Key: PHOENIX-5748 > URL: https://issues.apache.org/jira/browse/PHOENIX-5748 > Project: Phoenix > Issue Type: Improvement >Reporter: Kadir OZDEMIR >Assignee: Kadir OZDEMIR >Priority: Major > Fix For: 5.1.1, 4.14.4, 4.16.0 > > Attachments: PHOENIX-5748-4.x-HBase-1.5.001.patch, > PHOENIX-5748-4.x.001.patch, PHOENIX-5748.master.001.patch > > > The implementation of the new global index design by PHOENIX-5156 essentially > introduced two coprocessors, IndexRegionObserver and GlobalIndexChecker. > IndexRegionObserver is the counterpart of the existing Indexer coprocessor > that the previous global indexing feature uses. It implements the indexing > write path. GlobalIndexChecker implements the read verification and read > repair that happens on the read path. One of the main objectives of the > design behind new global indexing was to leverage as much existing indexing > code as possible. This objective has been achieved greatly as the entire > index table update generation code implemented by various classes (including > PhoenixIndexBuilder, CachedLocalTable, NonTxIndexBuilder, IndexUpdateManager, > LocalTableState, ScannerBuilder, IndexMemStore and PhoenixIndexCodec) is > leveraged as it is mainly. This objective has served us well to deliver the > new indexing feature quickly. The leveraged code is very complex, over > engineered, and inefficient, and is not bug free. It is very hard to > maintain. It is time to replace the complex set of classes with something > drastically simpler and more efficient for the new design. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
[
https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5802:
--
Attachment: PHOENIX-5802-4.x-v2.patch
> Connection leaks in UPSERT SELECT/DELETE paths due to
> MutatingParallelIteratorFactory iterator not being closed
> ---
>
> Key: PHOENIX-5802
> URL: https://issues.apache.org/jira/browse/PHOENIX-5802
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: Chinmay Kulkarni
>Assignee: Chinmay Kulkarni
>Priority: Major
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5802-4.x-v1.patch, PHOENIX-5802-4.x-v2.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing
> Phoenix connection and call
> [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
> which eventually calls either
> [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
> or
> [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
> When getting iterators, if we encounter any exception, then on calling
> [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
> we attempt to cancel any queued work, while [accumulating already-started
> task
> futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
>
> Later we attempt to block on getting the result (iterator) of
> already-started tasks and [close each
> iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
> Here however, if any of the upsert-select or client-side delete tasks had
> thrown an exception _E_, we wouldn't be able to close the underlying iterator
> (which in-turn would've [closed the cloned
> connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
> This manifests as [these
> logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
> {code:java}
> "Failed to execute task during cancel",
> {code}
> This is leading to a connection leak. Because we use CQSI to create the
> Phoenix Connection here, it also gets accounted for during open Phoenix
> connection throttling, thus leaving the client application with less
> connections that can be opened.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PHOENIX-5803) Refactor and add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802
[ https://issues.apache.org/jira/browse/PHOENIX-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5803: -- Labels: phoenix-hardening (was: newbie) > Refactor and add unit testing for classes changed in PHOENIX-5801 and > PHOENIX-5802 > -- > > Key: PHOENIX-5803 > URL: https://issues.apache.org/jira/browse/PHOENIX-5803 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: phoenix-hardening > Fix For: 5.1.0, 4.16.0 > > > WhereConstantParser should be in the util package rather than coprocessor. > We should also refactor, remove anonymous classes, etc. in > BaseResultIterators, MutatingResultIteratorFactory, UpsertCompiler, etc. > Also need to add unit tests for all these classes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5803) Refactor and add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802
[ https://issues.apache.org/jira/browse/PHOENIX-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5803: -- Description: WhereConstantParser should be in the util package rather than coprocessor. We should also refactor, remove anonymous classes, etc. in BaseResultIterators, MutatingResultIteratorFactory, UpsertCompiler, etc. Also need to add unit tests for all these classes. was:WhereConstantParser should be in the util package rather than coprocessor. Also need to add unit tests for this class. > Refactor and add unit testing for classes changed in PHOENIX-5801 and > PHOENIX-5802 > -- > > Key: PHOENIX-5803 > URL: https://issues.apache.org/jira/browse/PHOENIX-5803 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: newbie > Fix For: 5.1.0, 4.16.0 > > > WhereConstantParser should be in the util package rather than coprocessor. > We should also refactor, remove anonymous classes, etc. in > BaseResultIterators, MutatingResultIteratorFactory, UpsertCompiler, etc. > Also need to add unit tests for all these classes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5803) Refactor and add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802
[ https://issues.apache.org/jira/browse/PHOENIX-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5803: -- Summary: Refactor and add unit testing for classes changed in PHOENIX-5801 and PHOENIX-5802 (was: Refactor and add unit testing for PHOENIX-5801 and PHOENIX-5802) > Refactor and add unit testing for classes changed in PHOENIX-5801 and > PHOENIX-5802 > -- > > Key: PHOENIX-5803 > URL: https://issues.apache.org/jira/browse/PHOENIX-5803 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: newbie > Fix For: 5.1.0, 4.16.0 > > > WhereConstantParser should be in the util package rather than coprocessor. > Also need to add unit tests for this class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5803) Refactor and add unit testing for PHOENIX-5801 and PHOENIX-5802
[ https://issues.apache.org/jira/browse/PHOENIX-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5803: -- Summary: Refactor and add unit testing for PHOENIX-5801 and PHOENIX-5802 (was: Refactor and add testing to WhereConstantParser) > Refactor and add unit testing for PHOENIX-5801 and PHOENIX-5802 > --- > > Key: PHOENIX-5803 > URL: https://issues.apache.org/jira/browse/PHOENIX-5803 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: newbie > Fix For: 5.1.0, 4.16.0 > > > WhereConstantParser should be in the util package rather than coprocessor. > Also need to add unit tests for this class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5804) Categorize the index verification results based on empty column value
Swaroopa Kadam created PHOENIX-5804: --- Summary: Categorize the index verification results based on empty column value Key: PHOENIX-5804 URL: https://issues.apache.org/jira/browse/PHOENIX-5804 Project: Phoenix Issue Type: Improvement Reporter: Swaroopa Kadam Assignee: Swaroopa Kadam In case of the initial rebuild, empty column value will be x and for other unverified rows it will be x02 so we want to make sure that the appropriate counts are reflected when inline verification is run for an index with BEFORE/AFTER/ONLY/BOTH options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5803) Refactor and add testing to WhereConstantParser
[ https://issues.apache.org/jira/browse/PHOENIX-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5803: -- Labels: newbie (was: ) > Refactor and add testing to WhereConstantParser > --- > > Key: PHOENIX-5803 > URL: https://issues.apache.org/jira/browse/PHOENIX-5803 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: newbie > Fix For: 5.1.0, 4.16.0 > > > WhereConstantParser should be in the util package rather than coprocessor. > Also need to add unit tests for this class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5803) Refactor and add testing to WhereConstantParser
[ https://issues.apache.org/jira/browse/PHOENIX-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5803: -- Description: WhereConstantParser should be in the util package rather than coprocessor. Also need to add unit tests for this class. > Refactor and add testing to WhereConstantParser > --- > > Key: PHOENIX-5803 > URL: https://issues.apache.org/jira/browse/PHOENIX-5803 > Project: Phoenix > Issue Type: Improvement >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Priority: Major > Fix For: 5.1.0, 4.16.0 > > > WhereConstantParser should be in the util package rather than coprocessor. > Also need to add unit tests for this class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5803) Refactor and add testing to WhereConstantParser
Chinmay Kulkarni created PHOENIX-5803: - Summary: Refactor and add testing to WhereConstantParser Key: PHOENIX-5803 URL: https://issues.apache.org/jira/browse/PHOENIX-5803 Project: Phoenix Issue Type: Improvement Affects Versions: 4.15.0, 5.0.0 Reporter: Chinmay Kulkarni Fix For: 5.1.0, 4.16.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5748) Simplify index update generation code for consistent global indexes
[ https://issues.apache.org/jira/browse/PHOENIX-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kadir OZDEMIR updated PHOENIX-5748: --- Attachment: PHOENIX-5748-4.x.001.patch > Simplify index update generation code for consistent global indexes > > > Key: PHOENIX-5748 > URL: https://issues.apache.org/jira/browse/PHOENIX-5748 > Project: Phoenix > Issue Type: Improvement >Reporter: Kadir OZDEMIR >Assignee: Kadir OZDEMIR >Priority: Major > Fix For: 5.1.1, 4.14.4, 4.16.0 > > Attachments: PHOENIX-5748-4.x-HBase-1.5.001.patch, > PHOENIX-5748-4.x.001.patch > > > The implementation of the new global index design by PHOENIX-5156 essentially > introduced two coprocessors, IndexRegionObserver and GlobalIndexChecker. > IndexRegionObserver is the counterpart of the existing Indexer coprocessor > that the previous global indexing feature uses. It implements the indexing > write path. GlobalIndexChecker implements the read verification and read > repair that happens on the read path. One of the main objectives of the > design behind new global indexing was to leverage as much existing indexing > code as possible. This objective has been achieved greatly as the entire > index table update generation code implemented by various classes (including > PhoenixIndexBuilder, CachedLocalTable, NonTxIndexBuilder, IndexUpdateManager, > LocalTableState, ScannerBuilder, IndexMemStore and PhoenixIndexCodec) is > leveraged as it is mainly. This objective has served us well to deliver the > new indexing feature quickly. The leveraged code is very complex, over > engineered, and inefficient, and is not bug free. It is very hard to > maintain. It is time to replace the complex set of classes with something > drastically simpler and more efficient for the new design. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5791) Eliminate false invalid row detection due to concurrent updates
[ https://issues.apache.org/jira/browse/PHOENIX-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kadir OZDEMIR updated PHOENIX-5791: --- Attachment: PHOENIX-5791.4.x-HBase-1.5.002.patch > Eliminate false invalid row detection due to concurrent updates > > > Key: PHOENIX-5791 > URL: https://issues.apache.org/jira/browse/PHOENIX-5791 > Project: Phoenix > Issue Type: Sub-task >Reporter: Kadir OZDEMIR >Assignee: Kadir OZDEMIR >Priority: Major > Attachments: PHOENIX-5791.4.x-HBase-1.5.001.patch, > PHOENIX-5791.4.x-HBase-1.5.002.patch > > Time Spent: 5h 40m > Remaining Estimate: 0h > > IndexTool verification generates an expected list of index mutations from the > data table rows and uses this list to check if index table rows are > consistent with the data table. To do that it follows the following steps: > # The data table rows are scanned with a raw scan. This raw scan is > configured to read all versions of rows. > # For each scanned row, the cells that are scanned are grouped into two > sets: put and delete. The put set is the set of put cells and the delete set > is the set of delete cells. > # The put and delete sets for a given row are further grouped based on their > timestamps into put and delete mutations such that all the cells in a > mutation have the timestamp. > # The put and delete mutations are then sorted within a single list. > Mutations in this list are sorted in ascending order of their timestamp. > The above process assumes that for each data table update, the index table > will be updated with the correct index row key. However, this assumption does > not hold in the presence of concurrent updates. > From the consistent indexing design (PHOENIX-5156) perspective, two or more > pending updates from different batches on the same data row are concurrent if > and only if for all of these updates the data table row state is read from > HBase under the row lock and for none of them the row lock has been acquired > the second time for updating the data table. In other words, all of them are > in the first update phase concurrently. For concurrent updates, the first two > update phases are done but the last update phase is skipped. This means the > data table row will be updated by these updates but the corresponding index > table rows will be left with the unverified status. Then, the read repair > process will repair these unverified index rows during scans. > Since expected index mutations are derived from the data table row after > these concurrent mutations are applied, the expected list would not match > with the actual list of index mutations. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5801) Connection leak when creating a view with a where condition
[ https://issues.apache.org/jira/browse/PHOENIX-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-5801: -- Attachment: PHOENIX-5801-4.x-v2.patch > Connection leak when creating a view with a where condition > --- > > Key: PHOENIX-5801 > URL: https://issues.apache.org/jira/browse/PHOENIX-5801 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5801-4.x-v1.patch, PHOENIX-5801-4.x-v2.patch > > Time Spent: 40m > Remaining Estimate: 0h > > When you create a view with a where condition, we open a connection-less > Phoenix connection, which is never closed. This is a connection leak and also > unnecessarily contributes to the GLOBAL_OPEN_PHOENIX_CONNECTIONS metric > (which is incremented on opening any PhoenixConnection). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5734) IndexScrutinyTool should not report rows beyond maxLookBack age
[ https://issues.apache.org/jira/browse/PHOENIX-5734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Geoffrey Jacoby updated PHOENIX-5734: - Attachment: PHOENIX-5734-4.x-addendum.patch > IndexScrutinyTool should not report rows beyond maxLookBack age > --- > > Key: PHOENIX-5734 > URL: https://issues.apache.org/jira/browse/PHOENIX-5734 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 4.16.0 > > Attachments: PHOENIX-5734-4.x-addendum.patch, PHOENIX-5734-4.x.patch, > PHOENIX-5734-4.x.v2.patch, PHOENIX-5734.4.x-HBase-1.3.v1.patch > > Time Spent: 6h > Remaining Estimate: 0h > > Index Scrutiny tool should not report row mismatch if the row gets rewritten > during the run and the last version around is beyond max look back age which > will then get removed by compaction. > > Or add another type of counters to separate invalid rows into > BEYOND_MAX_LOOKBACK or similar -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (PHOENIX-5775) Make PreCommit build run all Phoenix tests
[ https://issues.apache.org/jira/browse/PHOENIX-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni resolved PHOENIX-5775. --- Resolution: Not A Problem > Make PreCommit build run all Phoenix tests > -- > > Key: PHOENIX-5775 > URL: https://issues.apache.org/jira/browse/PHOENIX-5775 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0, 5.1.0 >Reporter: Chinmay Kulkarni >Priority: Major > Labels: phoenix-hardening > > Currently, it looks like the Hadoop QA PreCommit build only runs tests in > phoenix-core. This skips running tests in phoenix-pherf and other modules. We > should ideally run all tests in the project. > For ex: > https://builds.apache.org/job/PreCommit-PHOENIX-Build/3546//testReport/ for > PHOENIX-5607 master branch, > shows that the build only ran phoenix-core tests, however phoenix-pherf tests > like PherfTest.java and ResourceTest.java fail and this was not captured. > These tests may have been failing for a long time (not necessarily related to > 5607 changes). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
[
https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5802:
--
Attachment: PHOENIX-5802-4.x-v1.patch
> Connection leaks in UPSERT SELECT/DELETE paths due to
> MutatingParallelIteratorFactory iterator not being closed
> ---
>
> Key: PHOENIX-5802
> URL: https://issues.apache.org/jira/browse/PHOENIX-5802
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: Chinmay Kulkarni
>Assignee: Chinmay Kulkarni
>Priority: Major
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-5802-4.x-v1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing
> Phoenix connection and call
> [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
> which eventually calls either
> [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
> or
> [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
> When getting iterators, if we encounter any exception, then on calling
> [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
> we attempt to cancel any queued work, while [accumulating already-started
> task
> futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
>
> Later we attempt to block on getting the result (iterator) of
> already-started tasks and [close each
> iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
> Here however, if any of the upsert-select or client-side delete tasks had
> thrown an exception _E_, we wouldn't be able to close the underlying iterator
> (which in-turn would've [closed the cloned
> connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
> This manifests as [these
> logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
> {code:java}
> "Failed to execute task during cancel",
> {code}
> This is leading to a connection leak. Because we use CQSI to create the
> Phoenix Connection here, it also gets accounted for during open Phoenix
> connection throttling, thus leaving the client application with less
> connections that can be opened.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
[
https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5802:
--
Description:
During an UPSERT SELECT query or client-side DELETE, we clone the existing
Phoenix connection and call
[mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
which eventually calls either
[upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
or
[deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
When getting iterators, if we encounter any exception, then on calling
[close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
we attempt to cancel any queued work, while [accumulating already-started task
futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
Later we attempt to block on getting the result (iterator) of already-started
tasks and [close each
iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
Here however, if any of the upsert-select or client-side delete tasks had
thrown an exception _E_, we wouldn't be able to close the underlying iterator
(which in-turn would've [closed the cloned
connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
This manifests as [these
logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
{code:java}
"Failed to execute task during cancel",
{code}
This is leading to a connection leak. Because we use CQSI to create the Phoenix
Connection here, it also gets accounted for during open Phoenix connection
throttling, thus leaving the client application with less connections that can
be opened.
was:
During an UPSERT SELECT query or client-side DELETE, we clone the existing
Phoenix connection and call
[mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
which eventually calls either
[upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
or
[deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
When getting iterators, if we encounter any exception, then on calling
[close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
we attempt to cancel any queued work, while [accumulating already-started task
futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
Later we attempt to block on getting the result (iterator) of already-started
tasks and [close each
iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
Here however, if any of the upsert-select or client-side delete tasks had
thrown an exception _E_, we wouldn't be able to close the underlying iterator
(which in-turn [closes the cloned
connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
This manifests as [these
logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
{code:java}
"Failed to execute task during cancel",
{code}
This is leading to a connection leak. Because we use CQSI to create the Phoenix
Connection here, it also gets accounted for during open Phoenix connection
throttling, thus leaving the client application with less connections that can
be opened.
> Connection leaks in UPSERT SELECT/DELETE
[jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
[
https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5802:
--
Description:
During an UPSERT SELECT query or client-side DELETE, we clone the existing
Phoenix connection and call
[mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
which eventually calls either
[upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
or
[deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
When getting iterators, if we encounter any exception, then on calling
[close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
we attempt to cancel any queued work, while [accumulating already-started task
futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
Later we attempt to block on getting the result (iterator) of already-started
tasks and [close each
iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
Here however, if any of the upsert-select or client-side delete tasks had
thrown an exception _E_, we wouldn't be able to close the underlying iterator
(which in-turn would've [closed the cloned
connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
This manifests as [these
logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
{code:java}
"Failed to execute task during cancel",
{code}
This is leading to a connection leak. Because we use CQSI to create the Phoenix
Connection here, it also gets accounted for during open Phoenix connection
throttling, thus leaving the client application with less connections that can
be opened.
was:
During an UPSERT SELECT query or client-side DELETE, we clone the existing
Phoenix connection and call
[mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
which eventually calls either
[upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
or
[deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
When getting iterators, if we encounter any exception, then on calling
[close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
we attempt to cancel any queued work, while [accumulating already-started task
futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
Later we attempt to block on getting the result (iterator) of already-started
tasks and [close each
iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
Here however, if any of the upsert-select or client-side delete tasks had
thrown an exception _E_, we wouldn't be able to close the underlying iterator
(which in-turn would've [closed the cloned
connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
This manifests as [these
logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
{code:java}
"Failed to execute task during cancel",
{code}
This is leading to a connection leak. Because we use CQSI to create the Phoenix
Connection here, it also gets accounted for during open Phoenix connection
throttling, thus leaving the client application with less connections that can
be opened.
> Connection leaks in UPSERT SELECT/DELETE
[jira] [Updated] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
[
https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinmay Kulkarni updated PHOENIX-5802:
--
Description:
During an UPSERT SELECT query or client-side DELETE, we clone the existing
Phoenix connection and call
[mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
which eventually calls either
[upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
or
[deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
When getting iterators, if we encounter any exception, then on calling
[close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
we attempt to cancel any queued work, while [accumulating already-started task
futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
Later we attempt to block on getting the result (iterator) of already-started
tasks and [close each
iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
Here however, if any of the upsert-select or client-side delete tasks had
thrown an exception _E_, we wouldn't be able to close the underlying iterator
(which in-turn [closes the cloned
connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]).
This manifests as [these
logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]:
{code:java}
"Failed to execute task during cancel",
{code}
This is leading to a connection leak. Because we use CQSI to create the Phoenix
Connection here, it also gets accounted for during open Phoenix connection
throttling, thus leaving the client application with less connections that can
be opened.
> Connection leaks in UPSERT SELECT/DELETE paths due to
> MutatingParallelIteratorFactory iterator not being closed
> ---
>
> Key: PHOENIX-5802
> URL: https://issues.apache.org/jira/browse/PHOENIX-5802
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 5.0.0, 4.15.0
>Reporter: Chinmay Kulkarni
>Assignee: Chinmay Kulkarni
>Priority: Major
> Fix For: 5.1.0, 4.16.0
>
>
> During an UPSERT SELECT query or client-side DELETE, we clone the existing
> Phoenix connection and call
> [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59]
> which eventually calls either
> [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178]
> or
> [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126].
>
> When getting iterators, if we encounter any exception, then on calling
> [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377],
> we attempt to cancel any queued work, while [accumulating already-started
> task
> futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453].
>
> Later we attempt to block on getting the result (iterator) of already-started
> tasks and [close each
> iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465].
>
> Here however, if any of the upsert-select or client-side delete tasks had
> thrown an exception _E_, we wouldn't be able to close the underlying iterator
> (which in-turn [closes the cloned
>
[jira] [Created] (PHOENIX-5802) Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed
Chinmay Kulkarni created PHOENIX-5802: - Summary: Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed Key: PHOENIX-5802 URL: https://issues.apache.org/jira/browse/PHOENIX-5802 Project: Phoenix Issue Type: Bug Affects Versions: 4.15.0, 5.0.0 Reporter: Chinmay Kulkarni Assignee: Chinmay Kulkarni Fix For: 5.1.0, 4.16.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
