[jira] [Commented] (IMPALA-7727) compute stats child query errors no longer propagate to parent query
[ https://issues.apache.org/jira/browse/IMPALA-7727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655993#comment-16655993 ] Michael Brown commented on IMPALA-7727: --- We've been hit by bugs like this before, like IMPALA-4981. Suggestion is an end-to-end test that does the following: 1. Run compute stats child queries directly (as real queries) with a low memory limit and make sure the error string has a reasonable match. 2. Run compute stats with the same low memory limit and assert the same error message is propagated. > compute stats child query errors no longer propagate to parent query > > > Key: IMPALA-7727 > URL: https://issues.apache.org/jira/browse/IMPALA-7727 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Michael Brown >Assignee: bharath v >Priority: Blocker > Labels: regression, stress > > [~bharathv] since you have been dealing with stats, please take a look. > Otherwise feel free to reassign. This bug prevents the stress test from > running with compute stats statements. It triggers in non-stressful > conditions, too. > {noformat} > $ impala-shell.sh -d tpch_parquet > [localhost:21000] tpch_parquet> set mem_limit=24m; > MEM_LIMIT set to 24m > [localhost:21000] tpch_parquet> compute stats customer; > Query: compute stats customer > WARNINGS: Cancelled > [localhost:21000] tpch_parquet> > {noformat} > The problem is that the child query didn't have enough memory to run, but > this error didn't propagate up. > {noformat} > Query (id=384d37fb2826a962:f4b10357): > DEBUG MODE WARNING: Query profile created while running a DEBUG build of > Impala. Use RELEASE builds to measure query performance. > Summary: > Session ID: d343e1026d497bb0:7e87b342c73c108d > Session Type: BEESWAX > Start Time: 2018-10-18 15:16:34.036363000 > End Time: 2018-10-18 15:16:34.177711000 > Query Type: QUERY > Query State: EXCEPTION > Query Status: Rejected query from pool default-pool: minimum memory > reservation is greater than memory available to the query for buffer > reservations. Memory reservation needed given the current plan: 128.00 KB. > Adjust either the mem_limit or the pool config (max-query-mem-limit, > min-query-mem-limit) for the query to allow the query memory limit to be at > least 32.12 MB. Note that changing the mem_limit may also change the plan. > See the query profile for more information about the per-node memory > requirements. > Impala Version: impalad version 3.1.0-SNAPSHOT DEBUG (build > 9f5c5e6df03824cba292fe5a619153462c11669c) > User: mikeb > Connected User: mikeb > Delegated User: > Network Address: :::127.0.0.1:46458 > Default Db: tpch_parquet > Sql Statement: SELECT COUNT(*) FROM customer > Coordinator: mikeb-ub162:22000 > Query Options (set by configuration): MEM_LIMIT=25165824,MT_DOP=4 > Query Options (set by configuration and planner): > MEM_LIMIT=25165824,NUM_SCANNER_THREADS=1,MT_DOP=4 > Plan: > > Max Per-Host Resource Reservation: Memory=512.00KB Threads=5 > Per-Host Resource Estimates: Memory=146MB > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B > thread-reservation=1 > PLAN-ROOT SINK > | mem-estimate=0B mem-reservation=0B thread-reservation=0 > | > 03:AGGREGATE [FINALIZE] > | output: count:merge(*) > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB > thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 03(GETNEXT), 01(OPEN) > | > 02:EXCHANGE [UNPARTITIONED] > | mem-estimate=0B mem-reservation=0B thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 01(GETNEXT) > | > F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=4 > Per-Host Resources: mem-estimate=136.00MB mem-reservation=512.00KB > thread-reservation=4 > 01:AGGREGATE > | output: sum_init_zero(tpch_parquet.customer.parquet-stats: num_rows) > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB > thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 01(GETNEXT), 00(OPEN) > | > 00:SCAN HDFS [tpch_parquet.customer, RANDOM] >partitions=1/1 files=1 size=12.34MB >stored statistics: > table: rows=15 size=12.34MB > columns: all >extrapolated-rows=disabled max-scan-range-rows=15 >mem-estimate=24.00MB mem-reservation=128.00KB thread-reservation=0 >tuple-ids=0 row-size=8B cardinality=15 >in pipelines: 00(GETNEXT) > > Estimated Per-Host Mem: 153092096 > Per Host Min Memory Reservation: mikeb-ub162:22000(0) > mikeb-ub162:22001(128.00 KB) > Re
[jira] [Commented] (IMPALA-7727) compute stats child query errors no longer propagate to parent query
[ https://issues.apache.org/jira/browse/IMPALA-7727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656134#comment-16656134 ] bharath v commented on IMPALA-7727: --- Thanks [~mikesbrown]. Will look into this. > compute stats child query errors no longer propagate to parent query > > > Key: IMPALA-7727 > URL: https://issues.apache.org/jira/browse/IMPALA-7727 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Michael Brown >Assignee: bharath v >Priority: Blocker > Labels: regression, stress > > [~bharathv] since you have been dealing with stats, please take a look. > Otherwise feel free to reassign. This bug prevents the stress test from > running with compute stats statements. It triggers in non-stressful > conditions, too. > {noformat} > $ impala-shell.sh -d tpch_parquet > [localhost:21000] tpch_parquet> set mem_limit=24m; > MEM_LIMIT set to 24m > [localhost:21000] tpch_parquet> compute stats customer; > Query: compute stats customer > WARNINGS: Cancelled > [localhost:21000] tpch_parquet> > {noformat} > The problem is that the child query didn't have enough memory to run, but > this error didn't propagate up. > {noformat} > Query (id=384d37fb2826a962:f4b10357): > DEBUG MODE WARNING: Query profile created while running a DEBUG build of > Impala. Use RELEASE builds to measure query performance. > Summary: > Session ID: d343e1026d497bb0:7e87b342c73c108d > Session Type: BEESWAX > Start Time: 2018-10-18 15:16:34.036363000 > End Time: 2018-10-18 15:16:34.177711000 > Query Type: QUERY > Query State: EXCEPTION > Query Status: Rejected query from pool default-pool: minimum memory > reservation is greater than memory available to the query for buffer > reservations. Memory reservation needed given the current plan: 128.00 KB. > Adjust either the mem_limit or the pool config (max-query-mem-limit, > min-query-mem-limit) for the query to allow the query memory limit to be at > least 32.12 MB. Note that changing the mem_limit may also change the plan. > See the query profile for more information about the per-node memory > requirements. > Impala Version: impalad version 3.1.0-SNAPSHOT DEBUG (build > 9f5c5e6df03824cba292fe5a619153462c11669c) > User: mikeb > Connected User: mikeb > Delegated User: > Network Address: :::127.0.0.1:46458 > Default Db: tpch_parquet > Sql Statement: SELECT COUNT(*) FROM customer > Coordinator: mikeb-ub162:22000 > Query Options (set by configuration): MEM_LIMIT=25165824,MT_DOP=4 > Query Options (set by configuration and planner): > MEM_LIMIT=25165824,NUM_SCANNER_THREADS=1,MT_DOP=4 > Plan: > > Max Per-Host Resource Reservation: Memory=512.00KB Threads=5 > Per-Host Resource Estimates: Memory=146MB > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B > thread-reservation=1 > PLAN-ROOT SINK > | mem-estimate=0B mem-reservation=0B thread-reservation=0 > | > 03:AGGREGATE [FINALIZE] > | output: count:merge(*) > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB > thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 03(GETNEXT), 01(OPEN) > | > 02:EXCHANGE [UNPARTITIONED] > | mem-estimate=0B mem-reservation=0B thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 01(GETNEXT) > | > F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=4 > Per-Host Resources: mem-estimate=136.00MB mem-reservation=512.00KB > thread-reservation=4 > 01:AGGREGATE > | output: sum_init_zero(tpch_parquet.customer.parquet-stats: num_rows) > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB > thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 01(GETNEXT), 00(OPEN) > | > 00:SCAN HDFS [tpch_parquet.customer, RANDOM] >partitions=1/1 files=1 size=12.34MB >stored statistics: > table: rows=15 size=12.34MB > columns: all >extrapolated-rows=disabled max-scan-range-rows=15 >mem-estimate=24.00MB mem-reservation=128.00KB thread-reservation=0 >tuple-ids=0 row-size=8B cardinality=15 >in pipelines: 00(GETNEXT) > > Estimated Per-Host Mem: 153092096 > Per Host Min Memory Reservation: mikeb-ub162:22000(0) > mikeb-ub162:22001(128.00 KB) > Request Pool: default-pool > Admission result: Rejected > Query Compilation: 126.903ms >- Metadata of all 1 tables cached: 5.484ms (5.484ms) >- Analysis finished: 16.104ms (10.619ms) >- Value transfer graph computed: 32.646ms (16.542ms) >- Single node plan created: 61.289ms (28.642ms) >
[jira] [Commented] (IMPALA-7727) compute stats child query errors no longer propagate to parent query
[ https://issues.apache.org/jira/browse/IMPALA-7727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657183#comment-16657183 ] Michael Brown commented on IMPALA-7727: --- I played around with this a little bit more and the *Query Status* is what isn't propagating in 3.1 vs. 2.12. I'll attach profiles to this Jira. > compute stats child query errors no longer propagate to parent query > > > Key: IMPALA-7727 > URL: https://issues.apache.org/jira/browse/IMPALA-7727 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.1.0 >Reporter: Michael Brown >Assignee: bharath v >Priority: Blocker > Labels: regression, stress > > [~bharathv] since you have been dealing with stats, please take a look. > Otherwise feel free to reassign. This bug prevents the stress test from > running with compute stats statements. It triggers in non-stressful > conditions, too. > {noformat} > $ impala-shell.sh -d tpch_parquet > [localhost:21000] tpch_parquet> set mem_limit=24m; > MEM_LIMIT set to 24m > [localhost:21000] tpch_parquet> compute stats customer; > Query: compute stats customer > WARNINGS: Cancelled > [localhost:21000] tpch_parquet> > {noformat} > The problem is that the child query didn't have enough memory to run, but > this error didn't propagate up. > {noformat} > Query (id=384d37fb2826a962:f4b10357): > DEBUG MODE WARNING: Query profile created while running a DEBUG build of > Impala. Use RELEASE builds to measure query performance. > Summary: > Session ID: d343e1026d497bb0:7e87b342c73c108d > Session Type: BEESWAX > Start Time: 2018-10-18 15:16:34.036363000 > End Time: 2018-10-18 15:16:34.177711000 > Query Type: QUERY > Query State: EXCEPTION > Query Status: Rejected query from pool default-pool: minimum memory > reservation is greater than memory available to the query for buffer > reservations. Memory reservation needed given the current plan: 128.00 KB. > Adjust either the mem_limit or the pool config (max-query-mem-limit, > min-query-mem-limit) for the query to allow the query memory limit to be at > least 32.12 MB. Note that changing the mem_limit may also change the plan. > See the query profile for more information about the per-node memory > requirements. > Impala Version: impalad version 3.1.0-SNAPSHOT DEBUG (build > 9f5c5e6df03824cba292fe5a619153462c11669c) > User: mikeb > Connected User: mikeb > Delegated User: > Network Address: :::127.0.0.1:46458 > Default Db: tpch_parquet > Sql Statement: SELECT COUNT(*) FROM customer > Coordinator: mikeb-ub162:22000 > Query Options (set by configuration): MEM_LIMIT=25165824,MT_DOP=4 > Query Options (set by configuration and planner): > MEM_LIMIT=25165824,NUM_SCANNER_THREADS=1,MT_DOP=4 > Plan: > > Max Per-Host Resource Reservation: Memory=512.00KB Threads=5 > Per-Host Resource Estimates: Memory=146MB > F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1 > | Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B > thread-reservation=1 > PLAN-ROOT SINK > | mem-estimate=0B mem-reservation=0B thread-reservation=0 > | > 03:AGGREGATE [FINALIZE] > | output: count:merge(*) > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB > thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 03(GETNEXT), 01(OPEN) > | > 02:EXCHANGE [UNPARTITIONED] > | mem-estimate=0B mem-reservation=0B thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 01(GETNEXT) > | > F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=4 > Per-Host Resources: mem-estimate=136.00MB mem-reservation=512.00KB > thread-reservation=4 > 01:AGGREGATE > | output: sum_init_zero(tpch_parquet.customer.parquet-stats: num_rows) > | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB > thread-reservation=0 > | tuple-ids=1 row-size=8B cardinality=1 > | in pipelines: 01(GETNEXT), 00(OPEN) > | > 00:SCAN HDFS [tpch_parquet.customer, RANDOM] >partitions=1/1 files=1 size=12.34MB >stored statistics: > table: rows=15 size=12.34MB > columns: all >extrapolated-rows=disabled max-scan-range-rows=15 >mem-estimate=24.00MB mem-reservation=128.00KB thread-reservation=0 >tuple-ids=0 row-size=8B cardinality=15 >in pipelines: 00(GETNEXT) > > Estimated Per-Host Mem: 153092096 > Per Host Min Memory Reservation: mikeb-ub162:22000(0) > mikeb-ub162:22001(128.00 KB) > Request Pool: default-pool > Admission result: Rejected > Query Compilation: 126.903ms >- Metadata of all 1 tables cached: 5.484ms (5.484ms) >- Analysis finished: 16.104ms (10.619ms) >- V