[ https://issues.apache.org/jira/browse/IMPALA-10414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522605#comment-17522605 ]
ASF subversion and git services commented on IMPALA-10414: ---------------------------------------------------------- Commit dfc2f175bdd10106e6984fc150ede9891a03eeb0 in impala's branch refs/heads/master from xqhe [ https://gitbox.apache.org/repos/asf?p=impala.git;h=dfc2f175b ] IMPALA-10414: fix memory leak when canceling the retried query The query retry launches in a separate thread. This thread may not finishes when deleting the query from the given QueryDriverMap if the query retry was failed launched. In this case, the resources for the query retry thread will not release. So the reference count of QueryDriver (via the shared_ptr) will not go to 0 and it will not be destroyed. We need wait until the query retry thread execution has completed when deleting the query from the given QueryDiverMap. Testing: Modify the test_query_retries.py to verify memory leak by checking the debug web UI of memz. Change-Id: If804ca65da1794c819a6b2e6567ea7651ab5112f Reviewed-on: http://gerrit.cloudera.org:8080/17735 Reviewed-by: Wenzhe Zhou <wz...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-by: Quanlong Huang <huangquanl...@gmail.com> > Retrying failed query may cause memory leak > ------------------------------------------- > > Key: IMPALA-10414 > URL: https://issues.apache.org/jira/browse/IMPALA-10414 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 4.0.0 > Reporter: Xianqing He > Assignee: Xianqing He > Priority: Minor > Fix For: Impala 4.1.0 > > > When cancel the retrying failed query, the qrery may not close and cause > memory leak. > {code:java} > Process: Limit=7.23 GB Total=137.45 MB Peak=254.58 MB > Buffer Pool: Free Buffers: Total=0 > Buffer Pool: Clean Pages: Total=0 > Buffer Pool: Unused Reservation: Total=0 > Control Service Queue: Limit=74.07 MB Total=0 Peak=52.05 KB > Data Stream Service Queue: Limit=370.35 MB Total=0 Peak=69.51 KB > Data Stream Manager Early RPCs: Total=0 Peak=0 > TCMalloc Overhead: Total=31.12 MB > RequestPool=default-pool: Total=0 Peak=110.10 MB > Query(bf406dca85fc951d:7cd2b15d00000000): Total=0 Peak=0 > Query(c146e822b2b670ad:fe8fc42e00000000): Reservation=0 > ReservationLimit=5.79 GB OtherMemory=0 Total=0 Peak=110.10 MB > Fragment c146e822b2b670ad:fe8fc42e00000006: Reservation=0 OtherMemory=0 > Total=0 Peak=3.21 MB > HDFS_SCAN_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=3.05 > MB > KrpcDataStreamSender (dst_id=9): Total=0 Peak=148.44 KB > Fragment c146e822b2b670ad:fe8fc42e00000003: Reservation=0 OtherMemory=0 > Total=0 Peak=3.21 MB > HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=3.05 > MB > KrpcDataStreamSender (dst_id=7): Total=0 Peak=148.44 KB > CodeGen: Total=0 Peak=239.50 KB > Fragment c146e822b2b670ad:fe8fc42e00000001: Reservation=0 OtherMemory=0 > Total=0 Peak=88.13 MB > HDFS_SCAN_NODE (id=0): Reservation=0 OtherMemory=0 Total=0 Peak=87.97 > MB > KrpcDataStreamSender (dst_id=6): Total=0 Peak=128.81 KB > CodeGen: Total=0 Peak=239.50 KB > CodeGen: Total=0 Peak=239.50 KB > Fragment c146e822b2b670ad:fe8fc42e00000004: Reservation=0 OtherMemory=0 > Total=0 Peak=14.72 MB > HASH_JOIN_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=3.05 > MB > Hash Join Builder (join_node_id=3): Total=0 Peak=21.12 KB > EXCHANGE_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=11.61 > MB > KrpcDeferredRpcs: Total=0 Peak=35.93 KB > EXCHANGE_NODE (id=7): Reservation=0 OtherMemory=0 Total=0 Peak=112.00 > KB > KrpcDeferredRpcs: Total=0 Peak=0 > KrpcDataStreamSender (dst_id=8): Total=0 Peak=133.47 KB > Fragment c146e822b2b670ad:fe8fc42e00000000: Reservation=0 OtherMemory=0 > Total=0 Peak=56.00 KB > AGGREGATION_NODE (id=11): Total=0 Peak=16.00 KB > NonGroupingAggregator 0: Total=0 Peak=8.00 KB > EXCHANGE_NODE (id=10): Reservation=0 OtherMemory=0 Total=0 Peak=32.00 > KB > KrpcDeferredRpcs: Total=0 Peak=0 > PLAN_ROOT_SINK: Total=0 Peak=0 > CodeGen: Total=0 Peak=6.50 KB > CodeGen: Total=0 Peak=1.54 MB > Fragment c146e822b2b670ad:fe8fc42e00000007: Reservation=0 OtherMemory=0 > Total=0 Peak=7.33 MB > AGGREGATION_NODE (id=5): Total=0 Peak=32.00 KB > NonGroupingAggregator 0: Total=0 Peak=8.00 KB > HASH_JOIN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=3.06 > MB > Hash Join Builder (join_node_id=4): Total=0 Peak=21.12 KB > EXCHANGE_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=4.14 MB > KrpcDeferredRpcs: Total=0 Peak=0 > EXCHANGE_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=112.00 > KB > KrpcDeferredRpcs: Total=0 Peak=0 > KrpcDataStreamSender (dst_id=10): Total=0 Peak=16.00 KB > CodeGen: Total=0 Peak=1.43 MB > Untracked Memory: Total=106.33 MB > {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org