Hey Mostafa,
Here are two query profiles on two different tables where COMPUTE STATS
OOMed at different steps. The first one OOMed on the first stats query
(counts) and the second one OOMed on the second stats query (NDV, MAX, etc).
Don't be fooled by the corrupt parquet error, you can still see the
exceeded memory limit further down. I believe I am getting
https://issues.apache.org/jira/browse/IMPALA-5197
Thanks a lot in advance!
Thoralf
On Thu, 18 Jan 2018 at 22:30 Alexander Behm <[email protected]> wrote:
> The documentation has good overview of the limitations and caveats:
>
> https://impala.apache.org/docs/build/html/topics/impala_perf_stats.html#perf_stats_incremental
>
> On Thu, Jan 18, 2018 at 7:29 PM, Fawze Abujaber <[email protected]> wrote:
>
>> Hi,
>>
>> I didn’t in the documentation of the incremental compute stats any
>> limitations,
>>
>> Is it size limit or memory limit ( 200 MB)?
>>
>> Why should compute stats successes and incremental compute stats not?
>>
>> I’m upgrading my cluster at Sunday as the incremental compute stats was
>> one of the incentives :(
>>
>> On Fri, 19 Jan 2018 at 4:13 Mostafa Mokhtar <[email protected]>
>> wrote:
>>
>>> Hi,
>>>
>>> Do you mind sharing the query profile for the query that failed with
>>> OOM? there should be some clues on to why the OOM is happening.
>>>
>>> Thanks
>>> Mostafa
>>>
>>>
>>> On Thu, Jan 18, 2018 at 5:54 PM, Thoralf Gutierrez <
>>> [email protected]> wrote:
>>>
>>>> Hello everybody!
>>>>
>>>> (I am using Impala 2.8.0, out of Cloudera Express 5.11.1)
>>>>
>>>> I now understand that we are _highly_ recommended to compute stats for
>>>> our tables so I have decided to make sure we do.
>>>>
>>>> On my quest to do so, I started with a first `COMPUTE INCREMENTAL STATS
>>>> my_big_partitioned_parquet_table` and ran into :
>>>>
>>>> > HiveServer2Error: AnalysisException: Incremental stats size estimate
>>>> exceeds 200.00MB. Please try COMPUTE STATS instead.
>>>>
>>>> I found out that we could increase this limit, so I set
>>>> inc_stats_size_limit_bytes to 1073741824 (1GB)
>>>>
>>>> > HiveServer2Error: AnalysisException: Incremental stats size estimate
>>>> exceeds 1.00GB. Please try COMPUTE STATS instead.
>>>>
>>>> So I ended up trying to COMPUTE STATS for the whole table instead of
>>>> incrementally, but I still hit memory limits when computing counts with my
>>>> mem_limit at 34359738368 (32GB)
>>>>
>>>> > Process: memory limit exceeded. Limit=32.00 GB Total=48.87 GB
>>>> Peak=51.97 GB
>>>>
>>>> 1. Am I correct to assume that even if I did not have enough memory,
>>>> the query should spill to disk and just be slower instead of OOMing?
>>>> 2. Any other recommendation on how else I could go about computing some
>>>> stats on my big partitioned parquet table?
>>>>
>>>> Thanks a lot!
>>>> Thoralf
>>>>
>>>>
>>>
>
Query (id=2b4362f21d335595:1cd3083a00000000)
Summary
Session ID: be4c7e009242db0d:b5e0671a68ec8991
Session Type: HIVESERVER2
HiveServer2 Protocol Version: V6
Start Time: 2018-01-16 15:28:29.751259000
End Time: 2018-01-16 15:33:15.642148000
Query Type: QUERY
Query State: EXCEPTION
Query Status:
Corrupt Parquet file 'hdfs://xxxxxx': negative row count -2081643451 in file
metadata
Impala Version: impalad version 2.8.0-cdh5.11.1 RELEASE (build
3382c1c488dff12d5ca8d049d2b59babee605b4e)
User: thgutierrez
Connected User: thgutierrez
Delegated User:
Network Address: 10.33.147.93:49955
Default Db: default
Sql Statement: SELECT COUNT(*), sys_load_dt, event_year, event_month,
event_day, vin_bucket FROM big_partitioned_parquet_table_1 GROUP BY
sys_load_dt, event_year, event_month, event_day, vin_bucket
Coordinator: dw-wk-038.car.logs:22000
Query Options (non default): MT_DOP=4
Plan:
----------------
Estimated Per-Host Requirements: Memory=1.21GB VCores=2
PLAN-ROOT SINK
|
04:EXCHANGE [UNPARTITIONED]
| hosts=43 per-host-mem=unavailable
| tuple-ids=1 row-size=39B cardinality=30151680
|
03:AGGREGATE [FINALIZE]
| output: count:merge(*)
| group by: sys_load_dt, event_year, event_month, event_day, vin_bucket
| hosts=43 per-host-mem=10.00MB
| tuple-ids=1 row-size=39B cardinality=30151680
|
02:EXCHANGE [HASH(sys_load_dt,event_year,event_month,event_day,vin_bucket)]
| hosts=43 per-host-mem=0B
| tuple-ids=1 row-size=39B cardinality=30151680
|
01:AGGREGATE [STREAMING]
| output: count(*)
| group by: sys_load_dt, event_year, event_month, event_day, vin_bucket
| hosts=43 per-host-mem=1.20GB
| tuple-ids=1 row-size=39B cardinality=30151680
|
00:SCAN HDFS [big_partitioned_parquet_table_1, RANDOM]
partitions=313578/313578 files=321858 size=214.68TB
table stats: unavailable
column stats: all
hosts=43 per-host-mem=0B
tuple-ids=0 row-size=31B cardinality=42851887740586
----------------
Estimated Per-Host Mem: 1303992832
Estimated Per-Host VCores: 2
Request Pool: default-pool
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est.
Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------
04:EXCHANGE 1 0.000ns 0.000ns 0 30.15M 0
-1.00 B UNPARTITIONED
03:AGGREGATE 172 1.489ms 7.547ms 0 30.15M 2.29 MB
10.00 MB FINALIZE
02:EXCHANGE 172 0.000ns 0.000ns 0 30.15M 0
0 HASH(sys_load_dt,event_year...
01:AGGREGATE 172 0.000ns 0.000ns 0 30.15M 1.57 MB
1.20 GB STREAMING
00:SCAN HDFS 172 2s980ms 10s944ms 381.36B 42851.89B 1.25 MB
0 big_partitioned_parquet_table_1
Errors: Corrupt Parquet file 'xxxxxxxxxx': negative row count -2081643451
in file metadata
Memory Limit Exceeded by fragment: 2b4362f21d335595:81
Process: memory limit exceeded. Limit=32.00 GB Total=49.58 GB Peak=51.56 GB
RequestPool=default-pool: Total=19.61 MB Peak=21.97 GB
Query(2b4362f21d335595:1cd3083a00000000): Total=19.61 MB Peak=19.98 MB
Fragment 2b4362f21d335595:1cd3083a00000000: Total=8.00 KB Peak=8.00 KB
EXCHANGE_NODE (id=4): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Block Manager: Limit=25.60 GB Total=3.00 MB Peak=3.02 MB
Fragment 2b4362f21d335595:12f: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:12e: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:12d: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:130: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:82: Total=2.59 MB Peak=2.90 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=832.66 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:81: Total=2.59 MB Peak=2.97 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=896.81 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:83: Total=2.59 MB Peak=2.70 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=880.51 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:84: Total=2.59 MB Peak=2.96 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=893.17 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Memory Limit Exceeded by fragment: 2b4362f21d335595:82
Process: memory limit exceeded. Limit=32.00 GB Total=49.58 GB Peak=51.56 GB
RequestPool=default-pool: Total=19.61 MB Peak=21.97 GB
Query(2b4362f21d335595:1cd3083a00000000): Total=19.61 MB Peak=19.98 MB
Fragment 2b4362f21d335595:1cd3083a00000000: Total=8.00 KB Peak=8.00 KB
EXCHANGE_NODE (id=4): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Block Manager: Limit=25.60 GB Total=3.00 MB Peak=3.02 MB
Fragment 2b4362f21d335595:12f: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:12e: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:12d: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:130: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:82: Total=2.59 MB Peak=2.90 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=832.66 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:81: Total=2.59 MB Peak=2.97 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=896.81 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:83: Total=2.59 MB Peak=2.70 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=880.51 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:84: Total=2.59 MB Peak=2.96 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=893.17 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Memory Limit Exceeded by fragment: 2b4362f21d335595:83
Process: memory limit exceeded. Limit=32.00 GB Total=49.58 GB Peak=51.56 GB
RequestPool=default-pool: Total=19.61 MB Peak=21.97 GB
Query(2b4362f21d335595:1cd3083a00000000): Total=19.61 MB Peak=19.98 MB
Fragment 2b4362f21d335595:1cd3083a00000000: Total=8.00 KB Peak=8.00 KB
EXCHANGE_NODE (id=4): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Block Manager: Limit=25.60 GB Total=3.00 MB Peak=3.02 MB
Fragment 2b4362f21d335595:12f: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:12e: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:12d: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:130: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:82: Total=2.59 MB Peak=2.90 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=832.66 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:81: Total=2.59 MB Peak=2.97 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=896.81 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:83: Total=2.59 MB Peak=2.70 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=880.51 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:84: Total=2.59 MB Peak=2.96 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=893.17 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Memory Limit Exceeded by fragment: 2b4362f21d335595:84
Process: memory limit exceeded. Limit=32.00 GB Total=49.58 GB Peak=51.56 GB
RequestPool=default-pool: Total=19.61 MB Peak=21.97 GB
Query(2b4362f21d335595:1cd3083a00000000): Total=19.61 MB Peak=19.98 MB
Fragment 2b4362f21d335595:1cd3083a00000000: Total=8.00 KB Peak=8.00 KB
EXCHANGE_NODE (id=4): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Block Manager: Limit=25.60 GB Total=3.00 MB Peak=3.02 MB
Fragment 2b4362f21d335595:12f: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:12e: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:12d: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:130: Total=2.31 MB Peak=2.80 MB
AGGREGATION_NODE (id=3): Total=2.29 MB Peak=2.29 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=4): Total=3.12 KB Peak=3.12 KB
CodeGen: Total=2.35 KB Peak=503.50 KB
Fragment 2b4362f21d335595:82: Total=2.59 MB Peak=2.90 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=832.66 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:81: Total=2.59 MB Peak=2.97 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=896.81 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:83: Total=2.59 MB Peak=2.70 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=880.51 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
Fragment 2b4362f21d335595:84: Total=2.59 MB Peak=2.96 MB
AGGREGATION_NODE (id=1): Total=1.55 MB Peak=1.57 MB
Exprs: Total=4.00 KB Peak=4.00 KB
HDFS_SCAN_NODE (id=0): Total=520.00 KB Peak=893.17 KB
DataStreamSender (dst_id=2): Total=536.16 KB Peak=536.16 KB
CodeGen: Total=2.51 KB Peak=631.50 KB
- ComputeScanRangeAssignmentTimer: 3242320644
- InactiveTotalTime: 0
- TotalTime: 0
Planner Timeline
Analysis finished: 1247912
Equivalence classes computed: 1399988
Single node plan created: 8029731685
Runtime filters computed: 8029767329
Distributed plan created: 8029901665
Lineage info computed: 8029951857
Parallel plans created: 8030009199
Planning finished: 9235984882
Query Timeline
Query submitted: 59788
Planning finished: 31632432772
Ready to start 345 fragment instances: 41008263152
All 345 fragment instances started: 282971042988
Unregister query: 285890899552
ImpalaServer
- ClientFetchWaitTimer: 0
- InactiveTotalTime: 0
- RowMaterializationTimer: 0
- TotalTime: 0
Query (id=6f4f5f8030525ca6:9b03cc6400000000)
Summary
Session ID: 23484b7554082f95:c46e10774d9e0bb6
Session Type: HIVESERVER2
HiveServer2 Protocol Version: V6
Start Time: 2018-01-16 23:36:54.200511000
End Time: 2018-01-16 23:43:35.187699000
Query Type: QUERY
Query State: EXCEPTION
Query Status:
Memory limit exceeded
Impala Version: impalad version 2.8.0-cdh5.11.1 RELEASE (build
3382c1c488dff12d5ca8d049d2b59babee605b4e)
User: thgutierrez
Connected User: thgutierrez
Delegated User:
Network Address: 10.33.147.93:46358
Default Db: default
Sql Statement: SELECT NDV(vin) AS vin, CAST(-1 as BIGINT),
MAX(length(vin)), AVG(length(vin)), NDV(vehicle_id) AS vehicle_id, CAST(-1 as
BIGINT), 8, CAST(8 as DOUBLE), NDV(logset_id) AS logset_id, CAST(-1 as BIGINT),
8, CAST(8 as DOUBLE), NDV(git_hash) AS git_hash, CAST(-1 as BIGINT),
MAX(length(git_hash)), AVG(length(git_hash)), NDV(firmware_version) AS
firmware_version, CAST(-1 as BIGINT), MAX(length(firmware_version)),
AVG(length(firmware_version)), NDV(epoch_ms) AS epoch_ms, CAST(-1 as BIGINT),
8, CAST(8 as DOUBLE), NDV(sig_enum_id) AS sig_enum_id, CAST(-1 as BIGINT), 8,
CAST(8 as DOUBLE), NDV(sig_name) AS sig_name, CAST(-1 as BIGINT),
MAX(length(sig_name)), AVG(length(sig_name)), NDV(sig_value) AS sig_value,
CAST(-1 as BIGINT), 8, CAST(8 as DOUBLE), NDV(sig_text) AS sig_text, CAST(-1 as
BIGINT), MAX(length(sig_text)), AVG(length(sig_text)), NDV(flags) AS flags,
CAST(-1 as BIGINT), 8, CAST(8 as DOUBLE), NDV(parse_order) AS parse_order,
CAST(-1 as BIGINT), 8, CAST(8 as DOUBLE), NDV(file_offset) AS file_offset,
CAST(-1 as BIGINT), 8, CAST(8 as DOUBLE), NDV(agg_id) AS agg_id, CAST(-1 as
BIGINT), 4, CAST(4 as DOUBLE), NDV(platform) AS platform, CAST(-1 as BIGINT),
MAX(length(platform)), AVG(length(platform)) FROM
big_partitioned_parquet_table_2
Coordinator: dw-wk-015.car.logs:22000
Query Options (non default): MT_DOP=4
Plan:
----------------
Estimated Per-Host Requirements: Memory=98.00MB VCores=1
PLAN-ROOT SINK
|
03:AGGREGATE [FINALIZE]
| output: ndv:merge(vin), max:merge(length(vin)), avg:merge(length(vin)),
ndv:merge(vehicle_id), ndv:merge(logset_id), ndv:merge(git_hash),
max:merge(length(git_hash)), avg:merge(length(git_hash)),
ndv:merge(firmware_version), max:merge(length(firmware_version)),
avg:merge(length(firmware_version)), ndv:merge(epoch_ms),
ndv:merge(sig_enum_id), ndv:merge(sig_name), max:merge(length(sig_name)),
avg:merge(length(sig_name)), ndv:merge(sig_value), ndv:merge(sig_text),
max:merge(length(sig_text)), avg:merge(length(sig_text)), ndv:merge(flags),
ndv:merge(parse_order), ndv:merge(file_offset), ndv:merge(agg_id),
ndv:merge(platform), max:merge(length(platform)), avg:merge(length(platform))
| hosts=43 per-host-mem=unavailable
| tuple-ids=2 row-size=192B cardinality=1
|
02:EXCHANGE [UNPARTITIONED]
| hosts=43 per-host-mem=unavailable
| tuple-ids=1 row-size=192B cardinality=1
|
01:AGGREGATE
| output: ndv(vin), max(length(vin)), avg(length(vin)), ndv(vehicle_id),
ndv(logset_id), ndv(git_hash), max(length(git_hash)), avg(length(git_hash)),
ndv(firmware_version), max(length(firmware_version)),
avg(length(firmware_version)), ndv(epoch_ms), ndv(sig_enum_id), ndv(sig_name),
max(length(sig_name)), avg(length(sig_name)), ndv(sig_value), ndv(sig_text),
max(length(sig_text)), avg(length(sig_text)), ndv(flags), ndv(parse_order),
ndv(file_offset), ndv(agg_id), ndv(platform), max(length(platform)),
avg(length(platform))
| hosts=43 per-host-mem=10.00MB
| tuple-ids=1 row-size=192B cardinality=1
|
00:SCAN HDFS [big_partitioned_parquet_table_2, RANDOM]
partitions=194666/194666 files=198880 size=65.66TB
table stats: 7388488094719 rows total (321 partition(s) missing stats)
column stats: all
hosts=43 per-host-mem=88.00MB
tuple-ids=0 row-size=261B cardinality=13872760007741
----------------
Estimated Per-Host Mem: 102760448
Estimated Per-Host VCores: 1
Request Pool: default-pool
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est.
Peak Mem Detail
-----------------------------------------------------------------------------------------------------------------------
03:AGGREGATE 1 2.435ms 2.435ms 0 1 32.00 KB
-1.00 B FINALIZE
02:EXCHANGE 1 0.000ns 0.000ns 0 1 0
-1.00 B UNPARTITIONED
01:AGGREGATE 172 0.000ns 0.000ns 0 1 7.21 MB
10.00 MB
00:SCAN HDFS 172 1m35s 2m55s 42.33B 13872.76B 53.25 MB
88.00 MB big_partitioned_parquet_table_2
Errors: Memory Limit Exceeded by fragment: 6f4f5f8030525ca6:6d
Process: memory limit exceeded. Limit=32.00 GB Total=32.10 GB Peak=32.75 GB
RequestPool=default-pool: Total=269.48 MB Peak=22.50 GB
Query(6f4f5f8030525ca6:9b03cc6400000000): Total=160.62 MB Peak=165.98 MB
Fragment 6f4f5f8030525ca6:9b03cc6400000000: Total=40.00 KB Peak=40.00 KB
AGGREGATION_NODE (id=3): Total=32.00 KB Peak=32.00 KB
Exprs: Total=28.00 KB Peak=28.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Block Manager: Limit=25.60 GB Total=0 Peak=0
Fragment 6f4f5f8030525ca6:6d: Total=38.99 MB Peak=41.66 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=4.20 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=38.93 MB Peak=41.60 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:6e: Total=46.60 MB Peak=50.73 MB
AGGREGATION_NODE (id=1): Total=32.00 KB Peak=4.18 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=46.56 MB Peak=47.37 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:70: Total=35.01 MB Peak=42.60 MB
AGGREGATION_NODE (id=1): Total=32.00 KB Peak=4.02 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=34.96 MB Peak=42.55 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:6f: Total=39.98 MB Peak=42.96 MB
AGGREGATION_NODE (id=1): Total=32.00 KB Peak=3.20 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=39.94 MB Peak=42.91 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Query(1b48ed0567336818:b936ae4100000000): Total=108.87 MB Peak=345.98 MB
Fragment 1b48ed0567336818:21: Total=108.87 MB Peak=345.98 MB
UNION_NODE (id=0): Total=8.81 MB Peak=11.53 MB
HDFS_SCAN_NODE (id=1): Total=99.71 MB Peak=344.75 MB
Exprs: Total=20.00 KB Peak=20.00 KB
HDFS_SCAN_NODE (id=2): Total=0 Peak=0
DataStreamSender (dst_id=3): Total=1.12 KB Peak=1.12 KB
CodeGen: Total=4.53 KB Peak=681.00 KB
Block Manager: Limit=25.60 GB Total=0 Peak=0
Memory Limit Exceeded by fragment: 6f4f5f8030525ca6:6e
Process: memory limit exceeded. Limit=32.00 GB Total=32.19 GB Peak=32.75 GB
RequestPool=default-pool: Total=363.90 MB Peak=22.50 GB
Query(6f4f5f8030525ca6:9b03cc6400000000): Total=160.64 MB Peak=165.98 MB
Fragment 6f4f5f8030525ca6:9b03cc6400000000: Total=40.00 KB Peak=40.00 KB
AGGREGATION_NODE (id=3): Total=32.00 KB Peak=32.00 KB
Exprs: Total=28.00 KB Peak=28.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Block Manager: Limit=25.60 GB Total=0 Peak=0
Fragment 6f4f5f8030525ca6:6d: Total=38.99 MB Peak=41.66 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=4.20 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=38.93 MB Peak=41.60 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:6e: Total=46.61 MB Peak=50.73 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=4.18 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=46.56 MB Peak=47.37 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:70: Total=35.01 MB Peak=42.60 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=4.02 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=34.96 MB Peak=42.55 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:6f: Total=39.99 MB Peak=42.96 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=3.20 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=39.94 MB Peak=42.91 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Query(1b48ed0567336818:b936ae4100000000): Total=203.26 MB Peak=345.98 MB
Fragment 1b48ed0567336818:21: Total=203.26 MB Peak=345.98 MB
UNION_NODE (id=0): Total=8.81 MB Peak=11.53 MB
HDFS_SCAN_NODE (id=1): Total=194.10 MB Peak=344.75 MB
Exprs: Total=20.00 KB Peak=20.00 KB
HDFS_SCAN_NODE (id=2): Total=0 Peak=0
DataStreamSender (dst_id=3): Total=1.12 KB Peak=1.12 KB
CodeGen: Total=4.53 KB Peak=681.00 KB
Block Manager: Limit=25.60 GB Total=0 Peak=0
Memory Limit Exceeded by fragment: 6f4f5f8030525ca6:6f
Process: memory limit exceeded. Limit=32.00 GB Total=32.11 GB Peak=32.75 GB
Free Disk IO Buffers: Total=8.00 MB Peak=2.00 GB
RequestPool=default-pool: Total=269.49 MB Peak=22.50 GB
Query(6f4f5f8030525ca6:9b03cc6400000000): Total=160.62 MB Peak=165.98 MB
Fragment 6f4f5f8030525ca6:9b03cc6400000000: Total=40.00 KB Peak=40.00 KB
AGGREGATION_NODE (id=3): Total=32.00 KB Peak=32.00 KB
Exprs: Total=28.00 KB Peak=28.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Block Manager: Limit=25.60 GB Total=0 Peak=0
Fragment 6f4f5f8030525ca6:6d: Total=38.99 MB Peak=41.66 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=4.20 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=38.93 MB Peak=41.60 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:6e: Total=46.60 MB Peak=50.73 MB
AGGREGATION_NODE (id=1): Total=32.00 KB Peak=4.18 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=46.56 MB Peak=47.37 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:70: Total=35.01 MB Peak=42.60 MB
AGGREGATION_NODE (id=1): Total=32.00 KB Peak=4.02 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=34.96 MB Peak=42.55 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:6f: Total=39.99 MB Peak=42.96 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=3.20 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=39.94 MB Peak=42.91 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Query(1b48ed0567336818:b936ae4100000000): Total=108.87 MB Peak=345.98 MB
Fragment 1b48ed0567336818:21: Total=108.87 MB Peak=345.98 MB
UNION_NODE (id=0): Total=8.81 MB Peak=11.53 MB
HDFS_SCAN_NODE (id=1): Total=99.71 MB Peak=344.75 MB
Exprs: Total=20.00 KB Peak=20.00 KB
HDFS_SCAN_NODE (id=2): Total=0 Peak=0
DataStreamSender (dst_id=3): Total=1.12 KB Peak=1.12 KB
CodeGen: Total=4.53 KB Peak=681.00 KB
Block Manager: Limit=25.60 GB Total=0 Peak=0
Memory Limit Exceeded by fragment: 6f4f5f8030525ca6:70
Process: memory limit exceeded. Limit=32.00 GB Total=32.19 GB Peak=32.75 GB
RequestPool=default-pool: Total=363.89 MB Peak=22.50 GB
Query(6f4f5f8030525ca6:9b03cc6400000000): Total=160.63 MB Peak=165.98 MB
Fragment 6f4f5f8030525ca6:9b03cc6400000000: Total=40.00 KB Peak=40.00 KB
AGGREGATION_NODE (id=3): Total=32.00 KB Peak=32.00 KB
Exprs: Total=28.00 KB Peak=28.00 KB
EXCHANGE_NODE (id=2): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
PLAN_ROOT_SINK: Total=0 Peak=0
CodeGen: Total=0 Peak=0
Block Manager: Limit=25.60 GB Total=0 Peak=0
Fragment 6f4f5f8030525ca6:6d: Total=38.99 MB Peak=41.66 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=4.20 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=38.93 MB Peak=41.60 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:6e: Total=46.60 MB Peak=50.73 MB
AGGREGATION_NODE (id=1): Total=32.00 KB Peak=4.18 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=46.56 MB Peak=47.37 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:70: Total=35.01 MB Peak=42.60 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=4.02 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=34.96 MB Peak=42.55 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Fragment 6f4f5f8030525ca6:6f: Total=39.99 MB Peak=42.96 MB
AGGREGATION_NODE (id=1): Total=40.00 KB Peak=3.20 MB
Exprs: Total=28.00 KB Peak=28.00 KB
HDFS_SCAN_NODE (id=0): Total=39.94 MB Peak=42.91 MB
DataStreamSender (dst_id=2): Total=360.00 B Peak=360.00 B
CodeGen: Total=5.40 KB Peak=1.35 MB
Query(1b48ed0567336818:b936ae4100000000): Total=203.26 MB Peak=345.98 MB
Fragment 1b48ed0567336818:21: Total=203.26 MB Peak=345.98 MB
UNION_NODE (id=0): Total=8.81 MB Peak=11.53 MB
HDFS_SCAN_NODE (id=1): Total=194.10 MB Peak=344.75 MB
Exprs: Total=20.00 KB Peak=20.00 KB
HDFS_SCAN_NODE (id=2): Total=0 Peak=0
DataStreamSender (dst_id=3): Total=1.12 KB Peak=1.12 KB
CodeGen: Total=4.53 KB Peak=681.00 KB
Block Manager: Limit=25.60 GB Total=0 Peak=0
- ComputeScanRangeAssignmentTimer: 903803864
- InactiveTotalTime: 0
- TotalTime: 0
Planner Timeline
Analysis finished: 675647
Equivalence classes computed: 841406
Single node plan created: 400780087
Runtime filters computed: 400824162
Distributed plan created: 401114438
Lineage info computed: 401220772
Parallel plans created: 401267606
Planning finished: 1154304282
Query Timeline
Query submitted: 43664
Planning finished: 5324146120
Ready to start 173 fragment instances: 8907082984
All 173 fragment instances started: 370842994776
Unregister query: 400987201272
ImpalaServer
- ClientFetchWaitTimer: 0
- InactiveTotalTime: 0
- RowMaterializationTimer: 0
- TotalTime: 0
