[ https://issues.apache.org/jira/browse/PHOENIX-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chinmay Kulkarni updated PHOENIX-5802: -------------------------------------- Labels: phoenix-hardening (was: ) > Connection leaks in UPSERT SELECT/DELETE paths due to > MutatingParallelIteratorFactory iterator not being closed > --------------------------------------------------------------------------------------------------------------- > > Key: PHOENIX-5802 > URL: https://issues.apache.org/jira/browse/PHOENIX-5802 > Project: Phoenix > Issue Type: Bug > Affects Versions: 5.0.0, 4.15.0 > Reporter: Chinmay Kulkarni > Assignee: Chinmay Kulkarni > Priority: Major > Labels: phoenix-hardening > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5802-4.x-v1.patch, PHOENIX-5802-4.x-v2.patch > > Time Spent: 3h > Remaining Estimate: 0h > > During an UPSERT SELECT query or client-side DELETE, we clone the existing > Phoenix connection and call > [mutate|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L59] > which eventually calls either > [upsertSelect|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/UpsertCompiler.java#L178] > or > [deleteRows|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/DeleteCompiler.java#L126]. > When getting iterators, if we encounter any exception, then on calling > [close|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1377], > we attempt to cancel any queued work, while [accumulating already-started > task > futures|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1453]. > > Later we attempt to block on getting the result (iterator) of > already-started tasks and [close each > iterator|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1464-L1465]. > Here however, if any of the upsert-select or client-side delete tasks had > thrown an exception _E_, we wouldn't be able to close the underlying iterator > (which in-turn would'veĀ [closed the cloned > connection|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/compile/MutatingParallelIteratorFactory.java#L101]). > This manifests as [these > logs|https://github.com/apache/phoenix/blob/73a86be9b588353457cd2c9de41239211330e7a7/phoenix-core/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java#L1470]: > {code:java} > "Failed to execute task during cancel", <exception E stack trace> > {code} > This is leading to a connection leak. Because we use CQSI to create the > Phoenix Connection here, it also gets accounted for during open Phoenix > connection throttling, thus leaving the client application with less > connections that can be opened. -- This message was sent by Atlassian Jira (v8.3.4#803005)