Also including the Kudu list in case someone there recognises the problem.
On Thu, Jul 18, 2019 at 8:05 AM lk_hadoop <lk_had...@163.com> wrote:
> I0718 18:42:22.677520 51139 coordinator.cc:357] starting execution on 5
> backends for query_id=2e4a3fbec0d7d721:2ec73c1c00000000
> I0718 18:42:22.679605 12873 impala-internal-service.cc:44]
> ExecQueryFInstances(): query_id=2e4a3fbec0d7d721:2ec73c1c00000000
> I0718 18:42:22.679620 12873 query-exec-mgr.cc:46] StartQueryFInstances()
> query_id=2e4a3fbec0d7d721:2ec73c1c00000000
> coord=realtimeanalysis-kudu-04-10-8-50-58:22000
> I0718 18:42:22.679625 12873 query-state.cc:178] Buffer pool limit for
> 2e4a3fbec0d7d721:2ec73c1c00000000: 17179869184
> I0718 18:42:22.679675 12873 initial-reservations.cc:60] Successfully
> claimed initial reservations (4.00 MB) for query
> 2e4a3fbec0d7d721:2ec73c1c00000000
> I0718 18:42:22.679769 51332 query-state.cc:309] StartFInstances():
> query_id=2e4a3fbec0d7d721:2ec73c1c00000000 #instances=2
> I0718 18:42:22.680196 51332 query-state.cc:322] descriptor table for
> query=2e4a3fbec0d7d721:2ec73c1c00000000
> tuples:
> Tuple(id=2 size=567 slots=[Slot(id=52 type=INT col_path=[] offset=464
> null=(offset=563 mask=20) slot_idx=29 field_idx=-1), Slot(id=53 type=STRING
> col_path=[] offset=0 null=(offset=560 mask=1) slot_idx=0 field_idx=-1),
> Slot(id=54 type=STRING col_path=[] offset=16 null=(offset=560 mask=2)
> slot_idx=1 field_idx=-1), Slot(id=55 type=STRING col_path=[] offset=32
> null=(offset=560 mask=4) slot_idx=2 field_idx=-1), Slot(id=56 type=STRING
> col_path=[] offset=48 null=(offset=560 mask=8) slot_idx=3 field_idx=-1),
> Slot(id=57 type=STRING col_path=[] offset=64 null=(offset=560 mask=10)
> slot_idx=4 field_idx=-1), Slot(id=58 type=STRING col_path=[] offset=80
> null=(offset=560 mask=20) slot_idx=5 field_idx=-1), Slot(id=59 type=STRING
> col_path=[] offset=96 null=(offset=560 mask=40) slot_idx=6 field_idx=-1),
> Slot(id=60 type=STRING col_path=[] offset=112 null=(offset=560 mask=80)
> slot_idx=7 field_idx=-1), Slot(id=61 type=STRING col_path=[] offset=128
> null=(offset=561 mask=1) slot_idx=8 field_idx=-1), Slot(id=62 type=STRING
> col_path=[] offset=144 null=(offset=561 mask=2) slot_idx=9 field_idx=-1),
> Slot(id=63 type=STRING col_path=[] offset=160 null=(offset=561 mask=4)
> slot_idx=10 field_idx=-1), Slot(id=64 type=INT col_path=[] offset=468
> null=(offset=563 mask=40) slot_idx=30 field_idx=-1), Slot(id=65 type=INT
> col_path=[] offset=472 null=(offset=563 mask=80) slot_idx=31 field_idx=-1),
> Slot(id=66 type=INT col_path=[] offset=476 null=(offset=564 mask=1)
> slot_idx=32 field_idx=-1), Slot(id=67 type=INT col_path=[] offset=480
> null=(offset=564 mask=2) slot_idx=33 field_idx=-1), Slot(id=68 type=STRING
> col_path=[] offset=176 null=(offset=561 mask=8) slot_idx=11 field_idx=-1),
> Slot(id=69 type=STRING col_path=[] offset=192 null=(offset=561 mask=10)
> slot_idx=12 field_idx=-1), Slot(id=70 type=STRING col_path=[] offset=208
> null=(offset=561 mask=20) slot_idx=13 field_idx=-1), Slot(id=71 type=STRING
> col_path=[] offset=224 null=(offset=561 mask=40) slot_idx=14 field_idx=-1),
> Slot(id=72 type=STRING col_path=[] offset=240 null=(offset=561 mask=80)
> slot_idx=15 field_idx=-1), Slot(id=73 type=STRING col_path=[] offset=256
> null=(offset=562 mask=1) slot_idx=16 field_idx=-1), Slot(id=74 type=INT
> col_path=[] offset=484 null=(offset=564 mask=4) slot_idx=34 field_idx=-1),
> Slot(id=75 type=INT col_path=[] offset=488 null=(offset=564 mask=8)
> slot_idx=35 field_idx=-1), Slot(id=76 type=INT col_path=[] offset=492
> null=(offset=564 mask=10) slot_idx=36 field_idx=-1), Slot(id=77 type=INT
> col_path=[] offset=496 null=(offset=564 mask=20) slot_idx=37 field_idx=-1),
> Slot(id=78 type=INT col_path=[] offset=500 null=(offset=564 mask=40)
> slot_idx=38 field_idx=-1), Slot(id=79 type=INT col_path=[] offset=504
> null=(offset=564 mask=80) slot_idx=39 field_idx=-1), Slot(id=80 type=INT
> col_path=[] offset=508 null=(offset=565 mask=1) slot_idx=40 field_idx=-1),
> Slot(id=81 type=STRING col_path=[] offset=272 null=(offset=562 mask=2)
> slot_idx=17 field_idx=-1), Slot(id=82 type=STRING col_path=[] offset=288
> null=(offset=562 mask=4) slot_idx=18 field_idx=-1), Slot(id=83 type=INT
> col_path=[] offset=512 null=(offset=565 mask=2) slot_idx=41 field_idx=-1),
> Slot(id=84 type=STRING col_path=[] offset=304 null=(offset=562 mask=8)
> slot_idx=19 field_idx=-1), Slot(id=85 type=STRING col_path=[] offset=320
> null=(offset=562 mask=10) slot_idx=20 field_idx=-1), Slot(id=86 type=STRING
> col_path=[] offset=336 null=(offset=562 mask=20) slot_idx=21 field_idx=-1),
> Slot(id=87 type=STRING col_path=[] offset=352 null=(offset=562 mask=40)
> slot_idx=22 field_idx=-1), Slot(id=88 type=STRING col_path=[] offset=368
> null=(offset=562 mask=80) slot_idx=23 field_idx=-1), Slot(id=89 type=INT
> col_path=[] offset=516 null=(offset=565 mask=4) slot_idx=42 field_idx=-1),
> Slot(id=90 type=INT col_path=[] offset=520 null=(offset=565 mask=8)
> slot_idx=43 field_idx=-1), Slot(id=91 type=INT col_path=[] offset=524
> null=(offset=565 mask=10) slot_idx=44 field_idx=-1), Slot(id=92 type=INT
> col_path=[] offset=528 null=(offset=565 mask=20) slot_idx=45 field_idx=-1),
> Slot(id=93 type=INT col_path=[] offset=532 null=(offset=565 mask=40)
> slot_idx=46 field_idx=-1), Slot(id=94 type=INT col_path=[] offset=536
> null=(offset=565 mask=80) slot_idx=47 field_idx=-1), Slot(id=95 type=INT
> col_path=[] offset=540 null=(offset=566 mask=1) slot_idx=48 field_idx=-1),
> Slot(id=96 type=INT col_path=[] offset=544 null=(offset=566 mask=2)
> slot_idx=49 field_idx=-1), Slot(id=97 type=INT col_path=[] offset=548
> null=(offset=566 mask=4) slot_idx=50 field_idx=-1), Slot(id=98 type=INT
> col_path=[] offset=552 null=(offset=566 mask=8) slot_idx=51 field_idx=-1),
> Slot(id=99 type=INT col_path=[] offset=556 null=(offset=566 mask=10)
> slot_idx=52 field_idx=-1), Slot(id=100 type=STRING col_path=[] offset=384
> null=(offset=563 mask=1) slot_idx=24 field_idx=-1), Slot(id=101 type=STRING
> col_path=[] offset=400 null=(offset=563 mask=2) slot_idx=25 field_idx=-1),
> Slot(id=102 type=STRING col_path=[] offset=416 null=(offset=563 mask=4)
> slot_idx=26 field_idx=-1), Slot(id=103 type=DECIMAL(30,6) col_path=[]
> offset=432 null=(offset=563 mask=8) slot_idx=27 field_idx=-1), Slot(id=104
> type=DECIMAL(30,6) col_path=[] offset=448 null=(offset=563 mask=10)
> slot_idx=28 field_idx=-1)] tuple_path=[])
> Tuple(id=1 size=0 slots=[] tuple_path=[])
> Tuple(id=0 size=563 slots=[Slot(id=0 type=STRING col_path=[0] offset=0
> null=(offset=556 mask=1) slot_idx=0 field_idx=-1), Slot(id=1 type=STRING
> col_path=[29] offset=16 null=(offset=556 mask=2) slot_idx=1 field_idx=-1),
> Slot(id=2 type=STRING col_path=[1] offset=32 null=(offset=556 mask=4)
> slot_idx=2 field_idx=-1), Slot(id=3 type=STRING col_path=[30] offset=48
> null=(offset=556 mask=8) slot_idx=3 field_idx=-1), Slot(id=4 type=STRING
> col_path=[51] offset=64 null=(offset=556 mask=10) slot_idx=4 field_idx=-1),
> Slot(id=5 type=STRING col_path=[7] offset=80 null=(offset=556 mask=20)
> slot_idx=5 field_idx=-1), Slot(id=6 type=STRING col_path=[2] offset=96
> null=(offset=556 mask=40) slot_idx=6 field_idx=-1), Slot(id=7 type=STRING
> col_path=[3] offset=112 null=(offset=556 mask=80) slot_idx=7 field_idx=-1),
> Slot(id=8 type=STRING col_path=[4] offset=128 null=(offset=557 mask=1)
> slot_idx=8 field_idx=-1), Slot(id=9 type=STRING col_path=[5] offset=144
> null=(offset=557 mask=2) slot_idx=9 field_idx=-1), Slot(id=10 type=STRING
> col_path=[6] offset=160 null=(offset=557 mask=4) slot_idx=10 field_idx=-1),
> Slot(id=11 type=INT col_path=[8] offset=464 null=(offset=559 mask=20)
> slot_idx=29 field_idx=-1), Slot(id=12 type=INT col_path=[9] offset=468
> null=(offset=559 mask=40) slot_idx=30 field_idx=-1), Slot(id=13 type=INT
> col_path=[10] offset=472 null=(offset=559 mask=80) slot_idx=31
> field_idx=-1), Slot(id=14 type=INT col_path=[11] offset=476
> null=(offset=560 mask=1) slot_idx=32 field_idx=-1), Slot(id=15 type=STRING
> col_path=[12] offset=176 null=(offset=557 mask=8) slot_idx=11
> field_idx=-1), Slot(id=16 type=STRING col_path=[13] offset=192
> null=(offset=557 mask=10) slot_idx=12 field_idx=-1), Slot(id=17 type=STRING
> col_path=[14] offset=208 null=(offset=557 mask=20) slot_idx=13
> field_idx=-1), Slot(id=18 type=STRING col_path=[15] offset=224
> null=(offset=557 mask=40) slot_idx=14 field_idx=-1), Slot(id=19 type=STRING
> col_path=[16] offset=240 null=(offset=557 mask=80) slot_idx=15
> field_idx=-1), Slot(id=20 type=STRING col_path=[17] offset=256
> null=(offset=558 mask=1) slot_idx=16 field_idx=-1), Slot(id=21 type=INT
> col_path=[18] offset=480 null=(offset=560 mask=2) slot_idx=33
> field_idx=-1), Slot(id=22 type=INT col_path=[19] offset=484
> null=(offset=560 mask=4) slot_idx=34 field_idx=-1), Slot(id=23 type=INT
> col_path=[20] offset=488 null=(offset=560 mask=8) slot_idx=35
> field_idx=-1), Slot(id=24 type=INT col_path=[21] offset=492
> null=(offset=560 mask=10) slot_idx=36 field_idx=-1), Slot(id=25 type=INT
> col_path=[22] offset=496 null=(offset=560 mask=20) slot_idx=37
> field_idx=-1), Slot(id=26 type=INT col_path=[23] offset=500
> null=(offset=560 mask=40) slot_idx=38 field_idx=-1), Slot(id=27 type=INT
> col_path=[24] offset=504 null=(offset=560 mask=80) slot_idx=39
> field_idx=-1), Slot(id=28 type=STRING col_path=[25] offset=272
> null=(offset=558 mask=2) slot_idx=17 field_idx=-1), Slot(id=29 type=STRING
> col_path=[26] offset=288 null=(offset=558 mask=4) slot_idx=18
> field_idx=-1), Slot(id=30 type=INT col_path=[27] offset=508
> null=(offset=561 mask=1) slot_idx=40 field_idx=-1), Slot(id=31 type=STRING
> col_path=[31] offset=304 null=(offset=558 mask=8) slot_idx=19
> field_idx=-1), Slot(id=32 type=STRING col_path=[32] offset=320
> null=(offset=558 mask=10) slot_idx=20 field_idx=-1), Slot(id=33 type=STRING
> col_path=[33] offset=336 null=(offset=558 mask=20) slot_idx=21
> field_idx=-1), Slot(id=34 type=STRING col_path=[34] offset=352
> null=(offset=558 mask=40) slot_idx=22 field_idx=-1), Slot(id=35 type=STRING
> col_path=[35] offset=368 null=(offset=558 mask=80) slot_idx=23
> field_idx=-1), Slot(id=36 type=INT col_path=[36] offset=512
> null=(offset=561 mask=2) slot_idx=41 field_idx=-1), Slot(id=37 type=INT
> col_path=[37] offset=516 null=(offset=561 mask=4) slot_idx=42
> field_idx=-1), Slot(id=38 type=INT col_path=[38] offset=520
> null=(offset=561 mask=8) slot_idx=43 field_idx=-1), Slot(id=39 type=INT
> col_path=[39] offset=524 null=(offset=561 mask=10) slot_idx=44
> field_idx=-1), Slot(id=40 type=INT col_path=[40] offset=528
> null=(offset=561 mask=20) slot_idx=45 field_idx=-1), Slot(id=41 type=INT
> col_path=[41] offset=532 null=(offset=561 mask=40) slot_idx=46
> field_idx=-1), Slot(id=42 type=INT col_path=[42] offset=536
> null=(offset=561 mask=80) slot_idx=47 field_idx=-1), Slot(id=43 type=INT
> col_path=[43] offset=540 null=(offset=562 mask=1) slot_idx=48
> field_idx=-1), Slot(id=44 type=INT col_path=[44] offset=544
> null=(offset=562 mask=2) slot_idx=49 field_idx=-1), Slot(id=45 type=INT
> col_path=[45] offset=548 null=(offset=562 mask=4) slot_idx=50
> field_idx=-1), Slot(id=46 type=INT col_path=[46] offset=552
> null=(offset=562 mask=8) slot_idx=51 field_idx=-1), Slot(id=47 type=STRING
> col_path=[47] offset=384 null=(offset=559 mask=1) slot_idx=24
> field_idx=-1), Slot(id=48 type=STRING col_path=[48] offset=400
> null=(offset=559 mask=2) slot_idx=25 field_idx=-1), Slot(id=49 type=STRING
> col_path=[28] offset=416 null=(offset=559 mask=4) slot_idx=26
> field_idx=-1), Slot(id=50 type=DECIMAL(30,6) col_path=[49] offset=432
> null=(offset=559 mask=8) slot_idx=27 field_idx=-1), Slot(id=51
> type=DECIMAL(30,6) col_path=[50] offset=448 null=(offset=559 mask=10)
> slot_idx=28 field_idx=-1)] tuple_path=[])
> I0718 18:42:22.680297 51333 query-state.cc:404] Executing instance.
> instance_id=2e4a3fbec0d7d721:2ec73c1c00000009 fragment_idx=0
> per_fragment_instance_idx=4 coord_state_idx=3 #in-flight=1
> I0718 18:42:22.680315 51139 coordinator.cc:370] started execution on 5
> backends for query_id=2e4a3fbec0d7d721:2ec73c1c00000000
> I0718 18:42:22.680457 51334 query-state.cc:404] Executing instance.
> instance_id=2e4a3fbec0d7d721:2ec73c1c00000004 fragment_idx=1
> per_fragment_instance_idx=4 coord_state_idx=3 #in-flight=2
> I0718 18:42:22.680583 51334 hdfs-scan-node.cc:190] Max row batch queue
> size for scan node '0' in fragment instance
> '2e4a3fbec0d7d721:2ec73c1c00000004': 40
> I0718 18:42:22.687943 51332 query-exec-mgr.cc:155] ReleaseQueryState():
> query_id=2e4a3fbec0d7d721:2ec73c1c00000000 refcnt=4
> I0718 18:42:22.689262 51338 coordinator.cc:789] Coordinator waiting for
> backends to finish, 5 remaining. query_id=2e4a3fbec0d7d721:2ec73c1c00000000
> I0718 18:51:07.797159 51597 webserver.cc:362] Webserver: error reading:
> Resource temporarily unavailable
> I0718 19:22:57.687764 180696 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
> blocked reactor thread for 47602.1us
> I0718 19:23:27.786094 180696 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
> blocked reactor thread for 49243.8us
> I0718 19:27:28.426831 180697 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050
> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}}
> blocked reactor thread for 47536.6us
> I0718 19:29:28.859411 232946 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request call
> id 5453218) took 518541ms. Request Metrics: {}
> I0718 19:29:28.859459 232949 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request call
> id 5432432) took 518541ms. Request Metrics: {}
> I0718 19:29:28.859504 232946 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request call
> id 5500254) took 518534ms. Request Metrics: {}
> I0718 19:29:28.859417 232945 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request call
> id 5513959) took 518536ms. Request Metrics: {}
> I0718 19:29:28.859589 232941 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request call
> id 5485666) took 518524ms. Request Metrics: {}
> I0718 20:09:09.275442 180696 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
> blocked reactor thread for 52958.3us
> I0718 20:09:29.338707 180696 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
> blocked reactor thread for 49366.4us
> I0718 20:14:50.273746 180697 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050
> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}}
> blocked reactor thread for 47568.8us
> I0718 20:15:20.342495 180697 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050
> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}}
> blocked reactor thread for 51994us
> I0718 20:15:56.003931 232949 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request call
> id 5647298) took 510394ms. Request Metrics: {}
> I0718 20:15:56.004006 232949 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request call
> id 5626571) took 510394ms. Request Metrics: {}
> I0718 20:15:56.004089 232949 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request call
> id 5661461) took 510386ms. Request Metrics: {}
> I0718 20:15:56.004143 232949 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request call
> id 5601664) took 510383ms. Request Metrics: {}
> I0718 20:15:56.004218 232949 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request call
> id 5577825) took 510328ms. Request Metrics: {}
> I0718 21:56:51.345875 180696 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
> blocked reactor thread for 61974.1us
> I0718 21:57:41.519013 180696 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
> blocked reactor thread for 48016.9us
> I0718 22:07:02.248558 180696 outbound_call.cc:286] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.56:7050
> (realtimeanalysis-kudu-02-10-8-50-56), user_credentials={real_user=hive}}
> blocked reactor thread for 61139.4us
> I0718 22:07:56.005770 232949 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request call
> id 5804566) took 839901ms. Request Metrics: {}
> I0718 22:07:56.005787 232937 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request call
> id 5749547) took 839899ms. Request Metrics: {}
> I0718 22:07:56.005823 232934 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request call
> id 5820240) took 839850ms. Request Metrics: {}
> I0718 22:07:56.005831 232940 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request call
> id 5712080) took 839898ms. Request Metrics: {}
> I0718 22:07:56.005977 232939 rpcz_store.cc:255] Call
> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request call
> id 5768794) took 839898ms. Request Metrics: {}
>
> *I will write 632 million rows to kudu, normally it's about 50 thousand/s
> , but today after write about 270 million , then write speed become very
> slow , I don't know what was wrong.*
>
> 2019-07-18
> ------------------------------
> lk_hadoop
>
