Hi,
Thank you very much for the advice. I have attached the profile logs. Each
table is have 160 columns. each will only return one row and only difference
between the queries is order by. Here the difference in time is 2 times the
query without order by.
Thanks
On Thursday, October 31, 2019, 08:21:08 AM GMT+5:30, Tim Armstrong
<[email protected]> wrote:
> 1) Why is Impala slow with order by?Impala's sort implementation is
> generally very fast, but the bottleneck for a query can be in many different
> places.
Like Shant said, the query profile will contain all the info needed to
determine where time was spent in the query. We don't have enough information
now to suggest why your query is performing the way it is. I'm surprised that a
sort operation on a single row would make that much difference in query runtime.
> 2) Can order by be made faster in clustered mode, that mean made to be
> parallelized ? Yes, sorts are parallelised across the cluster, *except* for
> the final merge stage where the rows are collected and streamed back to the
> client.
If you want to play around with increasing the parallelism of the sort, there
is a query option mt_dop that switches to a different multithreading mode where
you can increase the number of threads executing the plan, including sort. E.g.
if you set mt_dop=4 then on each node you will get 4 threads doing the sort
(assuming there are enough input files that the work can be partitioned).
> 3) Is it a good idea to use order by with Impala? if so have any body use it
> with a larger data set with good performance.If you're running a SELECT with
> an ORDER BY that returns a lot of rows, the bottleneck is most likely the
> output - Impala can probably sort the rows much faster than the client can
> fetch them.
This actually makes it kinda hard to benchmark sort performance. We have some
targeted benchmarks that use analytic functions to test sort performance to
avoid the problem (it works because the query plans for analytic functions
involve a sort). See
https://github.com/apache/impala/blob/master/testdata/workloads/targeted-perf/queries/primitive_orderby_bigint.test
> 4) Is there any other solutions to do fast order by queries within few
>seconds. (Interactive query engines)This is exactly what Impala is good at.
On Wed, Oct 30, 2019 at 7:07 PM Shant Hovsepian <[email protected]> wrote:
Hi Dinushka!
That's awesome you're working with Apache Impala for your internship! You
should know the Impala website provides some incredible documentation. In
particular check out the performance section to learn about how to benchmark
queries.
https://impala.apache.org/docs/build/html/topics/impala_performance.html
As you mentioned since it's in "standalone mode" all sorts of things can be
going, however the easiest way to get some insights to do look at the explain
plan and profile for the queries you run. For examples take a look at this page
https://impala.apache.org/docs/build/html/topics/impala_explain_plan.html
Any chance you can share the EXPLAIN and PROFILE statement output of your
queries? You should also take a look at the output of the SUMMARY statement for
yourself in case anything obvious stands out to you.
Thanks!
On Wed, Oct 30, 2019 at 9:10 PM Dinushka <[email protected]> wrote:
Hi,
I'm a student doing an internship, I have been given a task to do DB
performance testing for kudu with Impala for our data and use case.
Sample dataset is about 150 million records with 150 columns and total size of
kudu is 55GB. composite primary key (X,Y,Z) and partitioning by hash (X
=4,Y=2,Z=2)
SQL 1= "select A from table where G="value""SQL 2= "select A from table
where G="value" order by Z"
I'm testing kudu and Impala in standalone mode and have 2 queries which will
only return one row. One with "order by" and other without "order by".
When I do testing, I found that Impala with order by is about 15% to 35% slow.
when you have order by in the SQL.
In large row counts queries, it's time can be about 2-20 times more.
1) Why is Impala slow with order by?
2) Can order by be made faster in clustered mode, that mean made to be
parallelized ?
3) Is it a good idea to use order by with Impala? if so have any body use it
with a larger data set with good performance.
4) Is there any other solutions to do fast order by queries within few seconds.
(Interactive query engines)
Thank you
Query (id=394a24077d5c40a3:4cf8c0e200000000):
Summary:
Session ID: 914d568889734906:57644ff50565a6a9
Session Type: BEESWAX
Start Time: 2019-11-01 02:45:54.653821000
End Time: 2019-11-01 02:45:54.759479000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 3.2.0-cdh6.3.0 RELEASE (build
495397101e5807c701df71ea288f4815d69c2c8a)
User: user-s
Connected User: user-s
Delegated User:
Network Address: ::ffff:10.10.20.10:57757
Default Db: default
Sql Statement: SELECT A_ID, B_ID from T_table where link_id='1XAAAAAAAA'
AND time>=1212121212121212121 AND time<=1515151515151515151 AND A_ID = 'RRR'
limit 100000
Coordinator: localhost:42420
Query Options (set by configuration): CLIENT_IDENTIFIER=Impala Shell
v3.2.0-cdh6.3.0 (4953971) built on Thu Jul 18 10:24:11 PDT 2019
Query Options (set by configuration and planner):
MT_DOP=0,CLIENT_IDENTIFIER=Impala Shell v3.2.0-cdh6.3.0 (4953971) built on Thu
Jul 18 10:24:11 PDT 2019
Plan:
----------------
Max Per-Host Resource Reservation: Memory=0B Threads=3
Per-Host Resource Estimates: Memory=10MB
WARNING: The following tables are missing relevant table and/or column
statistics.
default.T_table
Analyzed query: SELECT A_ID, B_ID FROM `default`.T_table
WHERE link_id = '1XAAAAAAAA' AND time >=
CAST(1212121212121212121 AS BIGINT) AND time <=
CAST(1515151515151515151 AS BIGINT) AND A_ID = 'RRR' LIMIT CAST(100000
AS INT)
F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=2.22MB mem-reservation=0B
thread-reservation=1
PLAN-ROOT SINK
| mem-estimate=0B mem-reservation=0B thread-reservation=0
|
01:EXCHANGE [UNPARTITIONED]
| limit: 100000
| mem-estimate=2.22MB mem-reservation=0B thread-reservation=0
| tuple-ids=0 row-size=23B cardinality=100.00K
| in pipelines: 00(GETNEXT)
|
F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1
Per-Host Resources: mem-estimate=7.50MB mem-reservation=0B thread-reservation=2
00:SCAN KUDU [default.T_table]
kudu predicates: time <= CAST(1515151515151515151 AS BIGINT), time >=
CAST(1212121212121212121 AS BIGINT), A_ID = 'RRR', link_id = '1XAAAAAAAA'
limit: 100000
mem-estimate=7.50MB mem-reservation=0B thread-reservation=1
tuple-ids=0 row-size=23B cardinality=100.00K
in pipelines: 00(GETNEXT)
----------------
Estimated Per-Host Mem: 10485760
Tables Missing Stats: default.T_table
Per Host Min Memory Reservation: localhost:42420(0)
Request Pool: default-pool
Admission result: Admitted immediately
Cluster Memory Admitted: 10.00 MB
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
Est. Peak Mem Detail
-------------------------------------------------------------------------------------------------------------------
F01:ROOT 1 14.333us 14.333us 0
0
01:EXCHANGE 1 10.582us 10.582us 1 100.00K 16.00 KB
2.22 MB UNPARTITIONED
F00:EXCHANGE SENDER 1 69.487us 69.487us 5.33 KB
0
00:SCAN KUDU 1 6.086ms 6.086ms 1 100.00K 80.00 KB
7.50 MB default.T_table
Errors:
Query Compilation: 3.100ms
- Metadata of all 1 tables cached: 324.164us (324.164us)
- Analysis finished: 809.585us (485.421us)
- Value transfer graph computed: 834.473us (24.888us)
- Single node plan created: 2.560ms (1.725ms)
- Runtime filters computed: 2.637ms (76.857us)
- Distributed plan created: 2.669ms (32.634us)
- Planning finished: 3.100ms (431.018us)
Query Timeline: 107.588ms
- Query submitted: 32.906us (32.906us)
- Planning finished: 3.650ms (3.617ms)
- Submit for admission: 3.935ms (284.484us)
- Completed admission: 3.965ms (30.188us)
- Ready to start on 1 backends: 4.148ms (182.693us)
- All 1 execution backends (2 fragment instances) started: 4.665ms
(517.104us)
- Rows available: 24.854ms (20.189ms)
- First row fetched: 104.808ms (79.953ms)
- Last row fetched: 104.834ms (26.333us)
- Released admission control resources: 105.652ms (817.212us)
- Unregister query: 106.494ms (842.656us)
- AdmissionControlTimeSinceLastUpdate: 12.000ms
- ComputeScanRangeAssignmentTimer: 16.085us
Frontend:
ImpalaServer:
- ClientFetchWaitTimer: 80.459ms
- RowMaterializationTimer: 1.179ms
Execution Profile 394a24077d5c40a3:4cf8c0e200000000:(Total: 22.013ms,
non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected)
First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------
Backend startup latencies: Count: 1, min / max: 1ms / 1ms, 25th %-ile: 1ms,
50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th
%-ile: 1ms
Slowest backend to start up: localhost:42420
Per Node Peak Memory Usage: localhost:42420(101.33 KB)
Per Node Bytes Read: localhost:42420(0)
Per Node User Time: localhost:42420(32.043ms)
Per Node System Time: localhost:42420(1.747ms)
- ExchangeScanRatio: 0.00
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
- InnerNodeSelectivityRatio: 0.00
- NumBackends: 1 (1)
- NumFragmentInstances: 2 (2)
- NumFragments: 2 (2)
- TotalBytesRead: 0
- TotalBytesSent: 26.00 B (26)
- TotalCpuTime: 33.791ms
- TotalInnerBytesSent: 26.00 B (26)
- TotalScanBytesSent: 0
Per Node Profiles:
localhost:42420:
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Averaged Fragment F01:(Total: 100.045ms, non-child: 246.499us, % non-child:
0.25%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:100.899ms max:100.899ms mean: 100.899ms
stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 24.00 KB (24576)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 101.33 KB (103760)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 5.087ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 99.854ms
- TotalThreadsSysTime: 1.010ms
- TotalThreadsUserTime: 14.145ms
- TotalThreadsVoluntaryContextSwitches: 4 (4)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 79.984ms
- ExecTreeExecTime: 81.612us
- OpenTime: 19.873ms
- ExecTreeOpenTime: 5.011ms
- PrepareTime: 160.018us
- ExecTreePrepareTime: 73.512us
PLAN_ROOT_SINK:(Total: 79.896ms, non-child: 79.896ms, % non-child:
100.00%)
- PeakMemoryUsage: 0
EXCHANGE_NODE (id=1):(Total: 5.091ms, non-child: 5.091ms, % non-child:
100.00%)
- ConvertRowBatchTime: 669.000ns
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- RowsReturnedRate: 196.00 /sec
Buffer pool:
- AllocTime: 7.986us
- CumulativeAllocationBytes: 16.00 KB (16384)
- CumulativeAllocations: 2 (2)
- PeakReservation: 16.00 KB (16384)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 16.00 KB (16384)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- SystemAllocTime: 3.818us
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
- FirstBatchWaitTime: 5.007ms
- TotalBytesDequeued: 35.00 B (35)
- TotalGetBatchTime: 5.086ms
- DataWaitTime: 5.081ms
Enqueue:
- DeserializeRowBatchTime: 13.619us
- TotalBatchesEnqueued: 1 (1)
- TotalBatchesReceived: 1 (1)
- TotalBytesReceived: 26.00 B (26)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 1 (1)
- TotalHasDeferredRPCsTime: 0.000ns
- TotalRPCsDeferred: 0 (0)
CodeGen:(Total: 14.809ms, non-child: 14.809ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 14.806ms
- CodegenSysTime: 971.000us
- CodegenUserTime: 13.607ms
- CodegenVoluntaryContextSwitches: 1 (1)
- CompileTime: 0.000ns
- IrGenerationTime: 1.554us
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.28 MB (2393536)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PeakMemoryUsage: 0
- PrepareTime: 14.803ms
Coordinator Fragment F01:
Instance 394a24077d5c40a3:4cf8c0e200000000 (host=localhost:42420):(Total:
100.045ms, non-child: 246.499us, % non-child: 0.25%)
Last report received time: 2019-11-01 02:45:54.759
Fragment Instance Lifecycle Event Timeline: 100.201ms
- Prepare Finished: 319.046us (319.046us)
- Open Finished: 20.198ms (19.879ms)
- First Batch Produced: 20.280ms (82.373us)
- First Batch Sent: 100.177ms (79.896ms)
- ExecInternal Finished: 100.201ms (23.726us)
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 24.00 KB (24576)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 101.33 KB (103760)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 5.087ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 99.854ms
- TotalThreadsSysTime: 1.010ms
- TotalThreadsUserTime: 14.145ms
- TotalThreadsVoluntaryContextSwitches: 4 (4)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 79.984ms
- ExecTreeExecTime: 81.612us
- OpenTime: 19.873ms
- ExecTreeOpenTime: 5.011ms
- PrepareTime: 160.018us
- ExecTreePrepareTime: 73.512us
PLAN_ROOT_SINK:(Total: 79.896ms, non-child: 14.333us, % non-child:
0.02%)
- PeakMemoryUsage: 0
EXCHANGE_NODE (id=1):(Total: 5.091ms, non-child: 10.582us, % non-child:
0.21%)
Node Lifecycle Event Timeline: 100.191ms
- Open Started: 15.177ms (15.177ms)
- Open Finished: 20.187ms (5.010ms)
- First Batch Requested: 20.199ms (11.750us)
- First Batch Returned: 20.279ms (80.189us)
- Last Batch Returned: 20.279ms (163.000ns)
- Closed: 100.191ms (79.911ms)
- ConvertRowBatchTime: 669.000ns
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- RowsReturnedRate: 196.00 /sec
Buffer pool:
- AllocTime: 7.986us
- CumulativeAllocationBytes: 16.00 KB (16384)
- CumulativeAllocations: 2 (2)
- PeakReservation: 16.00 KB (16384)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 16.00 KB (16384)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- SystemAllocTime: 3.818us
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
- FirstBatchWaitTime: 5.007ms
- TotalBytesDequeued: 35.00 B (35)
- TotalGetBatchTime: 5.086ms
- DataWaitTime: 5.081ms
Enqueue:
- DispatchTime: (Avg: 30.365us ; Min: 30.365us ; Max: 30.365us ;
Number of samples: 1)
- DeserializeRowBatchTime: 13.619us
- TotalBatchesEnqueued: 1 (1)
- TotalBatchesReceived: 1 (1)
- TotalBytesReceived: 26.00 B (26)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 1 (1)
- TotalHasDeferredRPCsTime: 0.000ns
- TotalRPCsDeferred: 0 (0)
CodeGen:(Total: 14.809ms, non-child: 14.809ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 14.806ms
- CodegenSysTime: 971.000us
- CodegenUserTime: 13.607ms
- CodegenVoluntaryContextSwitches: 1 (1)
- CompileTime: 0.000ns
- IrGenerationTime: 1.554us
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.28 MB (2393536)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PeakMemoryUsage: 0
- PrepareTime: 14.803ms
Averaged Fragment F00:(Total: 20.111ms, non-child: 0.000ns, % non-child:
0.00%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:100.797ms max:100.797ms mean: 100.797ms
stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 93.33 KB (95568)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 101.33 KB (103760)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 216.953us
- TotalStorageWaitTime: 3.215ms
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 30.579ms
- TotalThreadsSysTime: 737.000us
- TotalThreadsUserTime: 17.899ms
- TotalThreadsVoluntaryContextSwitches: 30 (30)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 3.202ms
- ExecTreeExecTime: 2.933ms
- OpenTime: 16.709ms
- ExecTreeOpenTime: 1.676ms
- PrepareTime: 167.001us
- ExecTreePrepareTime: 44.603us
KrpcDataStreamSender (dst_id=1):(Total: 286.725us, non-child: 286.725us,
% non-child: 100.00%)
- EosSent: 1 (1)
- PeakMemoryUsage: 5.33 KB (5456)
- RowsSent: 1 (1)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 11.138us
- TotalBytesSent: 26.00 B (26)
- UncompressedRowBatchSize: 35.00 B (35)
KUDU_SCAN_NODE (id=0):(Total: 6.086ms, non-child: 6.086ms, % non-child:
100.00%)
- AverageScannerThreadConcurrency: 0.00
- KuduClientTime: 3.215ms
- KuduRemoteScanTokens: 4 (4)
- MaterializeTupleTime(*): 223.445us
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 4 (4)
- PeakMemoryUsage: 80.00 KB (81920)
- PeakScannerThreadConcurrency: 4 (4)
- RowBatchBytesEnqueued: 144.00 KB (147456)
- RowBatchQueueGetWaitTime: 2.917ms
- RowBatchQueuePeakMemoryUsage: 80.00 KB (81920)
- RowBatchQueuePutWaitTime: 0.000ns
- RowBatchesEnqueued: 4 (4)
- RowsRead: 1 (1)
- RowsReturned: 1 (1)
- RowsReturnedRate: 164.00 /sec
- ScanRangesComplete: 4 (4)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 10.682ms
- ScannerThreadsSysTime: 737.000us
- ScannerThreadsUserTime: 2.128ms
- ScannerThreadsVoluntaryContextSwitches: 17 (17)
- TotalKuduScanRoundTrips: 1 (1)
CodeGen:(Total: 14.978ms, non-child: 14.978ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 14.974ms
- CodegenSysTime: 0.000ns
- CodegenUserTime: 14.489ms
- CodegenVoluntaryContextSwitches: 1 (1)
- CompileTime: 0.000ns
- IrGenerationTime: 4.806us
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.28 MB (2393536)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PeakMemoryUsage: 0
- PrepareTime: 14.968ms
Fragment F00:
Instance 394a24077d5c40a3:4cf8c0e200000001 (host=localhost:42420):(Total:
20.111ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2019-11-01 02:45:54.759
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 20.336ms
- Prepare Finished: 395.714us (395.714us)
- Open Finished: 17.107ms (16.711ms)
- First Batch Produced: 19.450ms (2.343ms)
- First Batch Sent: 19.452ms (1.409us)
- ExecInternal Finished: 20.336ms (884.129us)
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 93.33 KB (95568)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 101.33 KB (103760)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 216.953us
- TotalStorageWaitTime: 3.215ms
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 30.579ms
- TotalThreadsSysTime: 737.000us
- TotalThreadsUserTime: 17.899ms
- TotalThreadsVoluntaryContextSwitches: 30 (30)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 3.202ms
- ExecTreeExecTime: 2.933ms
- OpenTime: 16.709ms
- ExecTreeOpenTime: 1.676ms
- PrepareTime: 167.001us
- ExecTreePrepareTime: 44.603us
KrpcDataStreamSender (dst_id=1):(Total: 286.725us, non-child: 69.487us,
% non-child: 24.23%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- NetworkThroughput: (Avg: 224.04 KB/sec ; Min: 224.04 KB/sec ; Max:
224.04 KB/sec ; Number of samples: 1)
- EosSent: 1 (1)
- PeakMemoryUsage: 5.33 KB (5456)
- RowsSent: 1 (1)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 11.138us
- TotalBytesSent: 26.00 B (26)
- UncompressedRowBatchSize: 35.00 B (35)
KUDU_SCAN_NODE (id=0):(Total: 6.086ms, non-child: 6.086ms, % non-child:
100.00%)
Node Lifecycle Event Timeline: 20.326ms
- Open Started: 15.424ms (15.424ms)
- Open Finished: 17.099ms (1.675ms)
- First Batch Requested: 17.108ms (8.869us)
- First Batch Returned: 19.450ms (2.341ms)
- Last Batch Returned: 20.077ms (627.054us)
- Closed: 20.326ms (248.874us)
- AverageScannerThreadConcurrency: 0.00
- KuduClientTime: 3.215ms
- KuduRemoteScanTokens: 4 (4)
- MaterializeTupleTime(*): 223.445us
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 4 (4)
- PeakMemoryUsage: 80.00 KB (81920)
- PeakScannerThreadConcurrency: 4 (4)
- RowBatchBytesEnqueued: 144.00 KB (147456)
- RowBatchQueueGetWaitTime: 2.917ms
- RowBatchQueuePeakMemoryUsage: 80.00 KB (81920)
- RowBatchQueuePutWaitTime: 0.000ns
- RowBatchesEnqueued: 4 (4)
- RowsRead: 1 (1)
- RowsReturned: 1 (1)
- RowsReturnedRate: 164.00 /sec
- ScanRangesComplete: 4 (4)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 10.682ms
- ScannerThreadsSysTime: 737.000us
- ScannerThreadsUserTime: 2.128ms
- ScannerThreadsVoluntaryContextSwitches: 17 (17)
- TotalKuduScanRoundTrips: 1 (1)
CodeGen:(Total: 14.978ms, non-child: 14.978ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 14.974ms
- CodegenSysTime: 0.000ns
- CodegenUserTime: 14.489ms
- CodegenVoluntaryContextSwitches: 1 (1)
- CompileTime: 0.000ns
- IrGenerationTime: 4.806us
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.28 MB (2393536)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PeakMemoryUsage: 0
- PrepareTime: 14.968msQuery (id=e049f884f0a7034c:fe21359c00000000):
Summary:
Session ID: 914d568889734906:57644ff50565a6a9
Session Type: BEESWAX
Start Time: 2019-11-01 02:45:10.724970000
End Time: 2019-11-01 02:45:10.930521000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 3.2.0-cdh6.3.0 RELEASE (build
495397101e5807c701df71ea288f4815d69c2c8a)
User: user-s
Connected User: user-s
Delegated User:
Network Address: ::ffff:10.10.20.10:57757
Default Db: default
Sql Statement: SELECT A_ID, B_ID from T_table where link_id='1XAAAAAAAA'
AND time>=1212121212121212121 AND time<=1515151515151515151 AND A_ID = 'RRR'
order by B_ID limit 100000
Coordinator: localhost:42420
Query Options (set by configuration): CLIENT_IDENTIFIER=Impala Shell
v3.2.0-cdh6.3.0 (4953971) built on Thu Jul 18 10:24:11 PDT 2019
Query Options (set by configuration and planner):
MT_DOP=0,CLIENT_IDENTIFIER=Impala Shell v3.2.0-cdh6.3.0 (4953971) built on Thu
Jul 18 10:24:11 PDT 2019
Plan:
----------------
Max Per-Host Resource Reservation: Memory=0B Threads=3
Per-Host Resource Estimates: Memory=11MB
WARNING: The following tables are missing relevant table and/or column
statistics.
default.T_table
Analyzed query: SELECT A_ID, B_ID FROM `default`.T_table
WHERE link_id = '1XAAAAAAAA' AND time >=
CAST(1212121212121212121 AS BIGINT) AND time <=
CAST(1515151515151515151 AS BIGINT) AND A_ID = 'RRR' ORDER BY
B_ID ASC LIMIT CAST(100000 AS INT)
F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=1.83MB mem-reservation=0B
thread-reservation=1
PLAN-ROOT SINK
| mem-estimate=0B mem-reservation=0B thread-reservation=0
|
02:MERGING-EXCHANGE [UNPARTITIONED]
| order by: B_ID ASC
| limit: 100000
| mem-estimate=1.83MB mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=19B cardinality=100.00K
| in pipelines: 01(GETNEXT)
|
F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1
Per-Host Resources: mem-estimate=9.31MB mem-reservation=0B thread-reservation=2
01:TOP-N [LIMIT=100000]
| order by: B_ID ASC
| mem-estimate=1.81MB mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=19B cardinality=100.00K
| in pipelines: 01(GETNEXT), 00(OPEN)
|
00:SCAN KUDU [default.T_table]
kudu predicates: time <= CAST(1515151515151515151 AS BIGINT), time >=
CAST(1212121212121212121 AS BIGINT), A_ID = 'RRR', link_id = '1XAAAAAAAA'
mem-estimate=7.50MB mem-reservation=0B thread-reservation=1
tuple-ids=0 row-size=23B cardinality=unavailable
in pipelines: 00(GETNEXT)
----------------
Estimated Per-Host Mem: 11687872
Tables Missing Stats: default.T_table
Per Host Min Memory Reservation: localhost:42420(0)
Request Pool: default-pool
Admission result: Admitted immediately
Cluster Memory Admitted: 11.15 MB
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem
Est. Peak Mem Detail
-------------------------------------------------------------------------------------------------------------------
F01:ROOT 1 14.213us 14.213us 0
0
02:MERGING-EXCHANGE 1 15.398us 15.398us 1 100.00K 16.00 KB
1.83 MB UNPARTITIONED
F00:EXCHANGE SENDER 1 51.928us 51.928us 6.09 KB
0
01:TOP-N 1 0.000ns 0.000ns 1 100.00K 84.00 KB
1.81 MB
00:SCAN KUDU 1 5.477ms 5.477ms 1 -1 80.00 KB
7.50 MB default.T_table
Errors:
Query Compilation: 2.830ms
- Metadata of all 1 tables cached: 329.246us (329.246us)
- Analysis finished: 787.620us (458.374us)
- Value transfer graph computed: 813.109us (25.489us)
- Single node plan created: 2.315ms (1.502ms)
- Runtime filters computed: 2.350ms (35.055us)
- Distributed plan created: 2.364ms (14.319us)
- Planning finished: 2.830ms (465.834us)
Query Timeline: 207.484ms
- Query submitted: 29.624us (29.624us)
- Planning finished: 3.392ms (3.362ms)
- Submit for admission: 3.570ms (177.718us)
- Completed admission: 3.593ms (22.923us)
- Ready to start on 1 backends: 3.755ms (162.766us)
- All 1 execution backends (2 fragment instances) started: 4.351ms
(595.182us)
- Rows available: 105.027ms (100.676ms)
- First row fetched: 204.660ms (99.633ms)
- Last row fetched: 204.689ms (29.102us)
- Released admission control resources: 205.547ms (858.019us)
- Unregister query: 206.353ms (806.359us)
- AdmissionControlTimeSinceLastUpdate: 37.000ms
- ComputeScanRangeAssignmentTimer: 15.158us
Frontend:
ImpalaServer:
- ClientFetchWaitTimer: 100.121ms
- RowMaterializationTimer: 1.205ms
Execution Profile e049f884f0a7034c:fe21359c00000000:(Total: 102.592ms,
non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected)
First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------
Backend startup latencies: Count: 1, min / max: 1ms / 1ms, 25th %-ile: 1ms,
50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th
%-ile: 1ms
Slowest backend to start up: localhost:42420
Per Node Peak Memory Usage: localhost:42420(1.26 MB)
Per Node Bytes Read: localhost:42420(0)
Per Node User Time: localhost:42420(124.641ms)
Per Node System Time: localhost:42420(1.006ms)
- ExchangeScanRatio: 0.00
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
- InnerNodeSelectivityRatio: 0.00
- NumBackends: 1 (1)
- NumFragmentInstances: 2 (2)
- NumFragments: 2 (2)
- TotalBytesRead: 0
- TotalBytesSent: 27.00 B (27)
- TotalCpuTime: 125.647ms
- TotalInnerBytesSent: 27.00 B (27)
- TotalScanBytesSent: 0
Per Node Profiles:
localhost:42420:
- ScratchBytesRead: 0
- ScratchBytesWritten: 0
- ScratchFileUsedBytes: 0
- ScratchReads: 0 (0)
- ScratchWrites: 0 (0)
- TotalEncryptionTime: 0.000ns
- TotalReadBlockTime: 0.000ns
Averaged Fragment F01:(Total: 200.296ms, non-child: 229.515us, % non-child:
0.11%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:201.118ms max:201.118ms mean: 201.118ms
stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 340.00 KB (348160)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.26 MB (1318496)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 200.119ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 26.380ms
- TotalThreadsVoluntaryContextSwitches: 3 (3)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 99.664ms
- ExecTreeExecTime: 86.326us
- OpenTime: 100.459ms
- ExecTreeOpenTime: 74.326ms
- PrepareTime: 147.824us
- ExecTreePrepareTime: 67.024us
PLAN_ROOT_SINK:(Total: 99.572ms, non-child: 99.572ms, % non-child:
100.00%)
- PeakMemoryUsage: 0
EXCHANGE_NODE (id=2):(Total: 74.411ms, non-child: 74.411ms, % non-child:
100.00%)
- ConvertRowBatchTime: 0.000ns
- MergeGetNext: 83.212us
- MergeGetNextBatch: 81.981us
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- RowsReturnedRate: 13.00 /sec
Buffer pool:
- AllocTime: 2.025us
- CumulativeAllocationBytes: 16.00 KB (16384)
- CumulativeAllocations: 2 (2)
- PeakReservation: 16.00 KB (16384)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 16.00 KB (16384)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
- FirstBatchWaitTime: 74.316ms
- TotalBytesDequeued: 32.00 B (32)
- TotalGetBatchTime: 74.399ms
- DataWaitTime: 74.395ms
Enqueue:
- DeserializeRowBatchTime: 7.802us
- TotalBatchesEnqueued: 1 (1)
- TotalBatchesReceived: 1 (1)
- TotalBytesReceived: 27.00 B (27)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 1 (1)
- TotalHasDeferredRPCsTime: 0.000ns
- TotalRPCsDeferred: 0 (0)
CodeGen:(Total: 26.083ms, non-child: 26.083ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 26.077ms
- CodegenSysTime: 0.000ns
- CodegenUserTime: 25.795ms
- CodegenVoluntaryContextSwitches: 0 (0)
- CompileTime: 2.513ms
- IrGenerationTime: 716.377us
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.28 MB (2393536)
- NumFunctions: 18 (18)
- NumInstructions: 664 (664)
- OptimizationTime: 7.066ms
- PeakMemoryUsage: 332.00 KB (339968)
- PrepareTime: 15.036ms
Coordinator Fragment F01:
Instance e049f884f0a7034c:fe21359c00000000 (host=localhost:42420):(Total:
200.296ms, non-child: 229.515us, % non-child: 0.11%)
Last report received time: 2019-11-01 02:45:10.930
Fragment Instance Lifecycle Event Timeline: 200.450ms
- Prepare Finished: 241.905us (241.905us)
- Open Finished: 100.707ms (100.465ms)
- First Batch Produced: 100.794ms (87.008us)
- First Batch Sent: 200.367ms (99.572ms)
- ExecInternal Finished: 200.450ms (82.904us)
- MemoryUsage (500.000ms): 8.10 KB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 340.00 KB (348160)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.26 MB (1318496)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 200.119ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 26.380ms
- TotalThreadsVoluntaryContextSwitches: 3 (3)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 99.664ms
- ExecTreeExecTime: 86.326us
- OpenTime: 100.459ms
- ExecTreeOpenTime: 74.326ms
- PrepareTime: 147.824us
- ExecTreePrepareTime: 67.024us
PLAN_ROOT_SINK:(Total: 99.572ms, non-child: 14.213us, % non-child:
0.01%)
- PeakMemoryUsage: 0
EXCHANGE_NODE (id=2):(Total: 74.411ms, non-child: 15.398us, %
non-child: 0.02%)
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 200.382ms
- Open Started: 26.372ms (26.372ms)
- Open Finished: 100.697ms (74.325ms)
- First Batch Requested: 100.709ms (11.886us)
- First Batch Returned: 100.794ms (84.787us)
- Last Batch Returned: 100.794ms (171.000ns)
- Closed: 200.382ms (99.587ms)
- ConvertRowBatchTime: 0.000ns
- MergeGetNext: 83.212us
- MergeGetNextBatch: 81.981us
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- RowsReturnedRate: 13.00 /sec
Buffer pool:
- AllocTime: 2.025us
- CumulativeAllocationBytes: 16.00 KB (16384)
- CumulativeAllocations: 2 (2)
- PeakReservation: 16.00 KB (16384)
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 16.00 KB (16384)
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Dequeue:
- BytesDequeued (500.000ms): 0
- FirstBatchWaitTime: 74.316ms
- TotalBytesDequeued: 32.00 B (32)
- TotalGetBatchTime: 74.399ms
- DataWaitTime: 74.395ms
Enqueue:
- BytesReceived (500.000ms): 0
- DeferredQueueSize (500.000ms): 0
- DispatchTime: (Avg: 29.077us ; Min: 29.077us ; Max: 29.077us ;
Number of samples: 1)
- DeserializeRowBatchTime: 7.802us
- TotalBatchesEnqueued: 1 (1)
- TotalBatchesReceived: 1 (1)
- TotalBytesReceived: 27.00 B (27)
- TotalEarlySenders: 0 (0)
- TotalEosReceived: 1 (1)
- TotalHasDeferredRPCsTime: 0.000ns
- TotalRPCsDeferred: 0 (0)
CodeGen:(Total: 26.083ms, non-child: 26.083ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 26.077ms
- CodegenSysTime: 0.000ns
- CodegenUserTime: 25.795ms
- CodegenVoluntaryContextSwitches: 0 (0)
- CompileTime: 2.513ms
- IrGenerationTime: 716.377us
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.28 MB (2393536)
- NumFunctions: 18 (18)
- NumInstructions: 664 (664)
- OptimizationTime: 7.066ms
- PeakMemoryUsage: 332.00 KB (339968)
- PrepareTime: 15.036ms
Averaged Fragment F00:(Total: 100.670ms, non-child: 0.000ns, % non-child:
0.00%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:201.014ms max:201.014ms mean: 201.014ms
stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec
stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 947.59 KB (970336)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.26 MB (1318496)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 233.102us
- TotalStorageWaitTime: 2.676ms
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 110.214ms
- TotalThreadsSysTime: 1.006ms
- TotalThreadsUserTime: 98.261ms
- TotalThreadsVoluntaryContextSwitches: 27 (27)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 264.368us
- ExecTreeExecTime: 1.829us
- OpenTime: 100.223ms
- ExecTreeOpenTime: 4.192ms
- PrepareTime: 155.173us
- ExecTreePrepareTime: 48.381us
KrpcDataStreamSender (dst_id=2):(Total: 285.296us, non-child: 285.296us,
% non-child: 100.00%)
- EosSent: 1 (1)
- PeakMemoryUsage: 6.09 KB (6240)
- RowsSent: 1 (1)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 4.659us
- TotalBytesSent: 27.00 B (27)
- UncompressedRowBatchSize: 32.00 B (32)
SORT_NODE (id=1):(Total: 4.240ms, non-child: 0.000ns, % non-child: 0.00%)
- InsertBatchTime: 1.130us
- PeakMemoryUsage: 84.00 KB (86016)
- RowsReturned: 1 (1)
- RowsReturnedRate: 235.00 /sec
- TuplePoolReclamations: 0 (0)
KUDU_SCAN_NODE (id=0):(Total: 5.477ms, non-child: 5.477ms, % non-child:
100.00%)
- AverageScannerThreadConcurrency: 0.00
- KuduClientTime: 2.676ms
- KuduRemoteScanTokens: 4 (4)
- MaterializeTupleTime(*): 223.080us
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 4 (4)
- PeakMemoryUsage: 80.00 KB (81920)
- PeakScannerThreadConcurrency: 4 (4)
- RowBatchBytesEnqueued: 144.00 KB (147456)
- RowBatchQueueGetWaitTime: 2.624ms
- RowBatchQueuePeakMemoryUsage: 80.00 KB (81920)
- RowBatchQueuePutWaitTime: 0.000ns
- RowBatchesEnqueued: 4 (4)
- RowsRead: 1 (1)
- RowsReturned: 1 (1)
- RowsReturnedRate: 182.00 /sec
- ScanRangesComplete: 4 (4)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 9.744ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 2.606ms
- ScannerThreadsVoluntaryContextSwitches: 14 (14)
- TotalKuduScanRoundTrips: 1 (1)
CodeGen:(Total: 95.989ms, non-child: 95.989ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 95.983ms
- CodegenSysTime: 997.000us
- CodegenUserTime: 94.718ms
- CodegenVoluntaryContextSwitches: 0 (0)
- CompileTime: 18.700ms
- IrGenerationTime: 2.172ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.28 MB (2393536)
- NumFunctions: 53 (53)
- NumInstructions: 1.87K (1867)
- OptimizationTime: 59.780ms
- PeakMemoryUsage: 933.50 KB (955904)
- PrepareTime: 14.361ms
Fragment F00:
Instance e049f884f0a7034c:fe21359c00000001 (host=localhost:42420):(Total:
100.670ms, non-child: 0.000ns, % non-child: 0.00%)
Last report received time: 2019-11-01 02:45:10.930
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
Fragment Instance Lifecycle Event Timeline: 100.914ms
- Prepare Finished: 332.792us (332.792us)
- Open Finished: 100.560ms (100.227ms)
- First Batch Produced: 100.562ms (2.146us)
- First Batch Sent: 100.580ms (18.046us)
- ExecInternal Finished: 100.914ms (333.552us)
- MemoryUsage (500.000ms): 14.09 KB
- ThreadUsage (500.000ms): 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- ExchangeScanRatio: 0.00
- PeakMemoryUsage: 947.59 KB (970336)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 1.26 MB (1318496)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 233.102us
- TotalStorageWaitTime: 2.676ms
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 110.214ms
- TotalThreadsSysTime: 1.006ms
- TotalThreadsUserTime: 98.261ms
- TotalThreadsVoluntaryContextSwitches: 27 (27)
Buffer pool:
- AllocTime: 0.000ns
- CumulativeAllocationBytes: 0
- CumulativeAllocations: 0 (0)
- PeakReservation: 0
- PeakUnpinnedBytes: 0
- PeakUsedReservation: 0
- ReadIoBytes: 0
- ReadIoOps: 0 (0)
- ReadIoWaitTime: 0.000ns
- ReservationLimit: 0
- SystemAllocTime: 0.000ns
- WriteIoBytes: 0
- WriteIoOps: 0 (0)
- WriteIoWaitTime: 0.000ns
Fragment Instance Lifecycle Timings:
- ExecTime: 264.368us
- ExecTreeExecTime: 1.829us
- OpenTime: 100.223ms
- ExecTreeOpenTime: 4.192ms
- PrepareTime: 155.173us
- ExecTreePrepareTime: 48.381us
KrpcDataStreamSender (dst_id=2):(Total: 285.296us, non-child: 51.928us,
% non-child: 18.20%)
ExecOption: Unpartitioned Sender Codegen Disabled: not needed
- BytesSent (500.000ms): 0
- NetworkThroughput: (Avg: 214.76 KB/sec ; Min: 214.76 KB/sec ; Max:
214.76 KB/sec ; Number of samples: 1)
- EosSent: 1 (1)
- PeakMemoryUsage: 6.09 KB (6240)
- RowsSent: 1 (1)
- RpcFailure: 0 (0)
- RpcRetry: 0 (0)
- SerializeBatchTime: 4.659us
- TotalBytesSent: 27.00 B (27)
- UncompressedRowBatchSize: 32.00 B (32)
SORT_NODE (id=1):(Total: 4.240ms, non-child: 0.000ns, % non-child:
0.00%)
SortType: TopN
ExecOption: Codegen Enabled
Node Lifecycle Event Timeline: 100.835ms
- Open Started: 96.361ms (96.361ms)
- Open Finished: 100.552ms (4.190ms)
- First Batch Requested: 100.561ms (9.019us)
- First Batch Returned: 100.561ms (695.000ns)
- Last Batch Returned: 100.562ms (165.000ns)
- Closed: 100.835ms (273.718us)
- InsertBatchTime: 1.130us
- PeakMemoryUsage: 84.00 KB (86016)
- RowsReturned: 1 (1)
- RowsReturnedRate: 235.00 /sec
- TuplePoolReclamations: 0 (0)
KUDU_SCAN_NODE (id=0):(Total: 5.477ms, non-child: 5.477ms, % non-child:
100.00%)
Node Lifecycle Event Timeline: 100.551ms
- Open Started: 96.366ms (96.366ms)
- Open Finished: 97.873ms (1.506ms)
- First Batch Requested: 97.874ms (1.658us)
- First Batch Returned: 99.979ms (2.104ms)
- Last Batch Returned: 100.515ms (536.655us)
- Closed: 100.551ms (35.697us)
- AverageScannerThreadConcurrency: 0.00
- KuduClientTime: 2.676ms
- KuduRemoteScanTokens: 4 (4)
- MaterializeTupleTime(*): 223.080us
- NumScannerThreadMemUnavailable: 0 (0)
- NumScannerThreadsStarted: 4 (4)
- PeakMemoryUsage: 80.00 KB (81920)
- PeakScannerThreadConcurrency: 4 (4)
- RowBatchBytesEnqueued: 144.00 KB (147456)
- RowBatchQueueGetWaitTime: 2.624ms
- RowBatchQueuePeakMemoryUsage: 80.00 KB (81920)
- RowBatchQueuePutWaitTime: 0.000ns
- RowBatchesEnqueued: 4 (4)
- RowsRead: 1 (1)
- RowsReturned: 1 (1)
- RowsReturnedRate: 182.00 /sec
- ScanRangesComplete: 4 (4)
- ScannerThreadsInvoluntaryContextSwitches: 0 (0)
- ScannerThreadsTotalWallClockTime: 9.744ms
- ScannerThreadsSysTime: 0.000ns
- ScannerThreadsUserTime: 2.606ms
- ScannerThreadsVoluntaryContextSwitches: 14 (14)
- TotalKuduScanRoundTrips: 1 (1)
CodeGen:(Total: 95.989ms, non-child: 95.989ms, % non-child: 100.00%)
- CodegenInvoluntaryContextSwitches: 0 (0)
- CodegenTotalWallClockTime: 95.983ms
- CodegenSysTime: 997.000us
- CodegenUserTime: 94.718ms
- CodegenVoluntaryContextSwitches: 0 (0)
- CompileTime: 18.700ms
- IrGenerationTime: 2.172ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.28 MB (2393536)
- NumFunctions: 53 (53)
- NumInstructions: 1.87K (1867)
- OptimizationTime: 59.780ms
- PeakMemoryUsage: 933.50 KB (955904)
- PrepareTime: 14.361ms
