Also including the Kudu list in case someone there recognises the problem. On Thu, Jul 18, 2019 at 8:05 AM lk_hadoop <lk_had...@163.com> wrote:
> I0718 18:42:22.677520 51139 coordinator.cc:357] starting execution on 5 > backends for query_id=2e4a3fbec0d7d721:2ec73c1c00000000 > I0718 18:42:22.679605 12873 impala-internal-service.cc:44] > ExecQueryFInstances(): query_id=2e4a3fbec0d7d721:2ec73c1c00000000 > I0718 18:42:22.679620 12873 query-exec-mgr.cc:46] StartQueryFInstances() > query_id=2e4a3fbec0d7d721:2ec73c1c00000000 > coord=realtimeanalysis-kudu-04-10-8-50-58:22000 > I0718 18:42:22.679625 12873 query-state.cc:178] Buffer pool limit for > 2e4a3fbec0d7d721:2ec73c1c00000000: 17179869184 > I0718 18:42:22.679675 12873 initial-reservations.cc:60] Successfully > claimed initial reservations (4.00 MB) for query > 2e4a3fbec0d7d721:2ec73c1c00000000 > I0718 18:42:22.679769 51332 query-state.cc:309] StartFInstances(): > query_id=2e4a3fbec0d7d721:2ec73c1c00000000 #instances=2 > I0718 18:42:22.680196 51332 query-state.cc:322] descriptor table for > query=2e4a3fbec0d7d721:2ec73c1c00000000 > tuples: > Tuple(id=2 size=567 slots=[Slot(id=52 type=INT col_path=[] offset=464 > null=(offset=563 mask=20) slot_idx=29 field_idx=-1), Slot(id=53 type=STRING > col_path=[] offset=0 null=(offset=560 mask=1) slot_idx=0 field_idx=-1), > Slot(id=54 type=STRING col_path=[] offset=16 null=(offset=560 mask=2) > slot_idx=1 field_idx=-1), Slot(id=55 type=STRING col_path=[] offset=32 > null=(offset=560 mask=4) slot_idx=2 field_idx=-1), Slot(id=56 type=STRING > col_path=[] offset=48 null=(offset=560 mask=8) slot_idx=3 field_idx=-1), > Slot(id=57 type=STRING col_path=[] offset=64 null=(offset=560 mask=10) > slot_idx=4 field_idx=-1), Slot(id=58 type=STRING col_path=[] offset=80 > null=(offset=560 mask=20) slot_idx=5 field_idx=-1), Slot(id=59 type=STRING > col_path=[] offset=96 null=(offset=560 mask=40) slot_idx=6 field_idx=-1), > Slot(id=60 type=STRING col_path=[] offset=112 null=(offset=560 mask=80) > slot_idx=7 field_idx=-1), Slot(id=61 type=STRING col_path=[] offset=128 > null=(offset=561 mask=1) slot_idx=8 field_idx=-1), Slot(id=62 type=STRING > col_path=[] offset=144 null=(offset=561 mask=2) slot_idx=9 field_idx=-1), > Slot(id=63 type=STRING col_path=[] offset=160 null=(offset=561 mask=4) > slot_idx=10 field_idx=-1), Slot(id=64 type=INT col_path=[] offset=468 > null=(offset=563 mask=40) slot_idx=30 field_idx=-1), Slot(id=65 type=INT > col_path=[] offset=472 null=(offset=563 mask=80) slot_idx=31 field_idx=-1), > Slot(id=66 type=INT col_path=[] offset=476 null=(offset=564 mask=1) > slot_idx=32 field_idx=-1), Slot(id=67 type=INT col_path=[] offset=480 > null=(offset=564 mask=2) slot_idx=33 field_idx=-1), Slot(id=68 type=STRING > col_path=[] offset=176 null=(offset=561 mask=8) slot_idx=11 field_idx=-1), > Slot(id=69 type=STRING col_path=[] offset=192 null=(offset=561 mask=10) > slot_idx=12 field_idx=-1), Slot(id=70 type=STRING col_path=[] offset=208 > null=(offset=561 mask=20) slot_idx=13 field_idx=-1), Slot(id=71 type=STRING > col_path=[] offset=224 null=(offset=561 mask=40) slot_idx=14 field_idx=-1), > Slot(id=72 type=STRING col_path=[] offset=240 null=(offset=561 mask=80) > slot_idx=15 field_idx=-1), Slot(id=73 type=STRING col_path=[] offset=256 > null=(offset=562 mask=1) slot_idx=16 field_idx=-1), Slot(id=74 type=INT > col_path=[] offset=484 null=(offset=564 mask=4) slot_idx=34 field_idx=-1), > Slot(id=75 type=INT col_path=[] offset=488 null=(offset=564 mask=8) > slot_idx=35 field_idx=-1), Slot(id=76 type=INT col_path=[] offset=492 > null=(offset=564 mask=10) slot_idx=36 field_idx=-1), Slot(id=77 type=INT > col_path=[] offset=496 null=(offset=564 mask=20) slot_idx=37 field_idx=-1), > Slot(id=78 type=INT col_path=[] offset=500 null=(offset=564 mask=40) > slot_idx=38 field_idx=-1), Slot(id=79 type=INT col_path=[] offset=504 > null=(offset=564 mask=80) slot_idx=39 field_idx=-1), Slot(id=80 type=INT > col_path=[] offset=508 null=(offset=565 mask=1) slot_idx=40 field_idx=-1), > Slot(id=81 type=STRING col_path=[] offset=272 null=(offset=562 mask=2) > slot_idx=17 field_idx=-1), Slot(id=82 type=STRING col_path=[] offset=288 > null=(offset=562 mask=4) slot_idx=18 field_idx=-1), Slot(id=83 type=INT > col_path=[] offset=512 null=(offset=565 mask=2) slot_idx=41 field_idx=-1), > Slot(id=84 type=STRING col_path=[] offset=304 null=(offset=562 mask=8) > slot_idx=19 field_idx=-1), Slot(id=85 type=STRING col_path=[] offset=320 > null=(offset=562 mask=10) slot_idx=20 field_idx=-1), Slot(id=86 type=STRING > col_path=[] offset=336 null=(offset=562 mask=20) slot_idx=21 field_idx=-1), > Slot(id=87 type=STRING col_path=[] offset=352 null=(offset=562 mask=40) > slot_idx=22 field_idx=-1), Slot(id=88 type=STRING col_path=[] offset=368 > null=(offset=562 mask=80) slot_idx=23 field_idx=-1), Slot(id=89 type=INT > col_path=[] offset=516 null=(offset=565 mask=4) slot_idx=42 field_idx=-1), > Slot(id=90 type=INT col_path=[] offset=520 null=(offset=565 mask=8) > slot_idx=43 field_idx=-1), Slot(id=91 type=INT col_path=[] offset=524 > null=(offset=565 mask=10) slot_idx=44 field_idx=-1), Slot(id=92 type=INT > col_path=[] offset=528 null=(offset=565 mask=20) slot_idx=45 field_idx=-1), > Slot(id=93 type=INT col_path=[] offset=532 null=(offset=565 mask=40) > slot_idx=46 field_idx=-1), Slot(id=94 type=INT col_path=[] offset=536 > null=(offset=565 mask=80) slot_idx=47 field_idx=-1), Slot(id=95 type=INT > col_path=[] offset=540 null=(offset=566 mask=1) slot_idx=48 field_idx=-1), > Slot(id=96 type=INT col_path=[] offset=544 null=(offset=566 mask=2) > slot_idx=49 field_idx=-1), Slot(id=97 type=INT col_path=[] offset=548 > null=(offset=566 mask=4) slot_idx=50 field_idx=-1), Slot(id=98 type=INT > col_path=[] offset=552 null=(offset=566 mask=8) slot_idx=51 field_idx=-1), > Slot(id=99 type=INT col_path=[] offset=556 null=(offset=566 mask=10) > slot_idx=52 field_idx=-1), Slot(id=100 type=STRING col_path=[] offset=384 > null=(offset=563 mask=1) slot_idx=24 field_idx=-1), Slot(id=101 type=STRING > col_path=[] offset=400 null=(offset=563 mask=2) slot_idx=25 field_idx=-1), > Slot(id=102 type=STRING col_path=[] offset=416 null=(offset=563 mask=4) > slot_idx=26 field_idx=-1), Slot(id=103 type=DECIMAL(30,6) col_path=[] > offset=432 null=(offset=563 mask=8) slot_idx=27 field_idx=-1), Slot(id=104 > type=DECIMAL(30,6) col_path=[] offset=448 null=(offset=563 mask=10) > slot_idx=28 field_idx=-1)] tuple_path=[]) > Tuple(id=1 size=0 slots=[] tuple_path=[]) > Tuple(id=0 size=563 slots=[Slot(id=0 type=STRING col_path=[0] offset=0 > null=(offset=556 mask=1) slot_idx=0 field_idx=-1), Slot(id=1 type=STRING > col_path=[29] offset=16 null=(offset=556 mask=2) slot_idx=1 field_idx=-1), > Slot(id=2 type=STRING col_path=[1] offset=32 null=(offset=556 mask=4) > slot_idx=2 field_idx=-1), Slot(id=3 type=STRING col_path=[30] offset=48 > null=(offset=556 mask=8) slot_idx=3 field_idx=-1), Slot(id=4 type=STRING > col_path=[51] offset=64 null=(offset=556 mask=10) slot_idx=4 field_idx=-1), > Slot(id=5 type=STRING col_path=[7] offset=80 null=(offset=556 mask=20) > slot_idx=5 field_idx=-1), Slot(id=6 type=STRING col_path=[2] offset=96 > null=(offset=556 mask=40) slot_idx=6 field_idx=-1), Slot(id=7 type=STRING > col_path=[3] offset=112 null=(offset=556 mask=80) slot_idx=7 field_idx=-1), > Slot(id=8 type=STRING col_path=[4] offset=128 null=(offset=557 mask=1) > slot_idx=8 field_idx=-1), Slot(id=9 type=STRING col_path=[5] offset=144 > null=(offset=557 mask=2) slot_idx=9 field_idx=-1), Slot(id=10 type=STRING > col_path=[6] offset=160 null=(offset=557 mask=4) slot_idx=10 field_idx=-1), > Slot(id=11 type=INT col_path=[8] offset=464 null=(offset=559 mask=20) > slot_idx=29 field_idx=-1), Slot(id=12 type=INT col_path=[9] offset=468 > null=(offset=559 mask=40) slot_idx=30 field_idx=-1), Slot(id=13 type=INT > col_path=[10] offset=472 null=(offset=559 mask=80) slot_idx=31 > field_idx=-1), Slot(id=14 type=INT col_path=[11] offset=476 > null=(offset=560 mask=1) slot_idx=32 field_idx=-1), Slot(id=15 type=STRING > col_path=[12] offset=176 null=(offset=557 mask=8) slot_idx=11 > field_idx=-1), Slot(id=16 type=STRING col_path=[13] offset=192 > null=(offset=557 mask=10) slot_idx=12 field_idx=-1), Slot(id=17 type=STRING > col_path=[14] offset=208 null=(offset=557 mask=20) slot_idx=13 > field_idx=-1), Slot(id=18 type=STRING col_path=[15] offset=224 > null=(offset=557 mask=40) slot_idx=14 field_idx=-1), Slot(id=19 type=STRING > col_path=[16] offset=240 null=(offset=557 mask=80) slot_idx=15 > field_idx=-1), Slot(id=20 type=STRING col_path=[17] offset=256 > null=(offset=558 mask=1) slot_idx=16 field_idx=-1), Slot(id=21 type=INT > col_path=[18] offset=480 null=(offset=560 mask=2) slot_idx=33 > field_idx=-1), Slot(id=22 type=INT col_path=[19] offset=484 > null=(offset=560 mask=4) slot_idx=34 field_idx=-1), Slot(id=23 type=INT > col_path=[20] offset=488 null=(offset=560 mask=8) slot_idx=35 > field_idx=-1), Slot(id=24 type=INT col_path=[21] offset=492 > null=(offset=560 mask=10) slot_idx=36 field_idx=-1), Slot(id=25 type=INT > col_path=[22] offset=496 null=(offset=560 mask=20) slot_idx=37 > field_idx=-1), Slot(id=26 type=INT col_path=[23] offset=500 > null=(offset=560 mask=40) slot_idx=38 field_idx=-1), Slot(id=27 type=INT > col_path=[24] offset=504 null=(offset=560 mask=80) slot_idx=39 > field_idx=-1), Slot(id=28 type=STRING col_path=[25] offset=272 > null=(offset=558 mask=2) slot_idx=17 field_idx=-1), Slot(id=29 type=STRING > col_path=[26] offset=288 null=(offset=558 mask=4) slot_idx=18 > field_idx=-1), Slot(id=30 type=INT col_path=[27] offset=508 > null=(offset=561 mask=1) slot_idx=40 field_idx=-1), Slot(id=31 type=STRING > col_path=[31] offset=304 null=(offset=558 mask=8) slot_idx=19 > field_idx=-1), Slot(id=32 type=STRING col_path=[32] offset=320 > null=(offset=558 mask=10) slot_idx=20 field_idx=-1), Slot(id=33 type=STRING > col_path=[33] offset=336 null=(offset=558 mask=20) slot_idx=21 > field_idx=-1), Slot(id=34 type=STRING col_path=[34] offset=352 > null=(offset=558 mask=40) slot_idx=22 field_idx=-1), Slot(id=35 type=STRING > col_path=[35] offset=368 null=(offset=558 mask=80) slot_idx=23 > field_idx=-1), Slot(id=36 type=INT col_path=[36] offset=512 > null=(offset=561 mask=2) slot_idx=41 field_idx=-1), Slot(id=37 type=INT > col_path=[37] offset=516 null=(offset=561 mask=4) slot_idx=42 > field_idx=-1), Slot(id=38 type=INT col_path=[38] offset=520 > null=(offset=561 mask=8) slot_idx=43 field_idx=-1), Slot(id=39 type=INT > col_path=[39] offset=524 null=(offset=561 mask=10) slot_idx=44 > field_idx=-1), Slot(id=40 type=INT col_path=[40] offset=528 > null=(offset=561 mask=20) slot_idx=45 field_idx=-1), Slot(id=41 type=INT > col_path=[41] offset=532 null=(offset=561 mask=40) slot_idx=46 > field_idx=-1), Slot(id=42 type=INT col_path=[42] offset=536 > null=(offset=561 mask=80) slot_idx=47 field_idx=-1), Slot(id=43 type=INT > col_path=[43] offset=540 null=(offset=562 mask=1) slot_idx=48 > field_idx=-1), Slot(id=44 type=INT col_path=[44] offset=544 > null=(offset=562 mask=2) slot_idx=49 field_idx=-1), Slot(id=45 type=INT > col_path=[45] offset=548 null=(offset=562 mask=4) slot_idx=50 > field_idx=-1), Slot(id=46 type=INT col_path=[46] offset=552 > null=(offset=562 mask=8) slot_idx=51 field_idx=-1), Slot(id=47 type=STRING > col_path=[47] offset=384 null=(offset=559 mask=1) slot_idx=24 > field_idx=-1), Slot(id=48 type=STRING col_path=[48] offset=400 > null=(offset=559 mask=2) slot_idx=25 field_idx=-1), Slot(id=49 type=STRING > col_path=[28] offset=416 null=(offset=559 mask=4) slot_idx=26 > field_idx=-1), Slot(id=50 type=DECIMAL(30,6) col_path=[49] offset=432 > null=(offset=559 mask=8) slot_idx=27 field_idx=-1), Slot(id=51 > type=DECIMAL(30,6) col_path=[50] offset=448 null=(offset=559 mask=10) > slot_idx=28 field_idx=-1)] tuple_path=[]) > I0718 18:42:22.680297 51333 query-state.cc:404] Executing instance. > instance_id=2e4a3fbec0d7d721:2ec73c1c00000009 fragment_idx=0 > per_fragment_instance_idx=4 coord_state_idx=3 #in-flight=1 > I0718 18:42:22.680315 51139 coordinator.cc:370] started execution on 5 > backends for query_id=2e4a3fbec0d7d721:2ec73c1c00000000 > I0718 18:42:22.680457 51334 query-state.cc:404] Executing instance. > instance_id=2e4a3fbec0d7d721:2ec73c1c00000004 fragment_idx=1 > per_fragment_instance_idx=4 coord_state_idx=3 #in-flight=2 > I0718 18:42:22.680583 51334 hdfs-scan-node.cc:190] Max row batch queue > size for scan node '0' in fragment instance > '2e4a3fbec0d7d721:2ec73c1c00000004': 40 > I0718 18:42:22.687943 51332 query-exec-mgr.cc:155] ReleaseQueryState(): > query_id=2e4a3fbec0d7d721:2ec73c1c00000000 refcnt=4 > I0718 18:42:22.689262 51338 coordinator.cc:789] Coordinator waiting for > backends to finish, 5 remaining. query_id=2e4a3fbec0d7d721:2ec73c1c00000000 > I0718 18:51:07.797159 51597 webserver.cc:362] Webserver: error reading: > Resource temporarily unavailable > I0718 19:22:57.687764 180696 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 > (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} > blocked reactor thread for 47602.1us > I0718 19:23:27.786094 180696 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 > (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} > blocked reactor thread for 49243.8us > I0718 19:27:28.426831 180697 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050 > (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}} > blocked reactor thread for 47536.6us > I0718 19:29:28.859411 232946 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request call > id 5453218) took 518541ms. Request Metrics: {} > I0718 19:29:28.859459 232949 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request call > id 5432432) took 518541ms. Request Metrics: {} > I0718 19:29:28.859504 232946 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request call > id 5500254) took 518534ms. Request Metrics: {} > I0718 19:29:28.859417 232945 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request call > id 5513959) took 518536ms. Request Metrics: {} > I0718 19:29:28.859589 232941 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request call > id 5485666) took 518524ms. Request Metrics: {} > I0718 20:09:09.275442 180696 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 > (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} > blocked reactor thread for 52958.3us > I0718 20:09:29.338707 180696 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 > (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} > blocked reactor thread for 49366.4us > I0718 20:14:50.273746 180697 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050 > (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}} > blocked reactor thread for 47568.8us > I0718 20:15:20.342495 180697 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050 > (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}} > blocked reactor thread for 51994us > I0718 20:15:56.003931 232949 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request call > id 5647298) took 510394ms. Request Metrics: {} > I0718 20:15:56.004006 232949 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request call > id 5626571) took 510394ms. Request Metrics: {} > I0718 20:15:56.004089 232949 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request call > id 5661461) took 510386ms. Request Metrics: {} > I0718 20:15:56.004143 232949 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request call > id 5601664) took 510383ms. Request Metrics: {} > I0718 20:15:56.004218 232949 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request call > id 5577825) took 510328ms. Request Metrics: {} > I0718 21:56:51.345875 180696 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 > (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} > blocked reactor thread for 61974.1us > I0718 21:57:41.519013 180696 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050 > (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}} > blocked reactor thread for 48016.9us > I0718 22:07:02.248558 180696 outbound_call.cc:286] RPC callback for RPC > call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.56:7050 > (realtimeanalysis-kudu-02-10-8-50-56), user_credentials={real_user=hive}} > blocked reactor thread for 61139.4us > I0718 22:07:56.005770 232949 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request call > id 5804566) took 839901ms. Request Metrics: {} > I0718 22:07:56.005787 232937 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request call > id 5749547) took 839899ms. Request Metrics: {} > I0718 22:07:56.005823 232934 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request call > id 5820240) took 839850ms. Request Metrics: {} > I0718 22:07:56.005831 232940 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request call > id 5712080) took 839898ms. Request Metrics: {} > I0718 22:07:56.005977 232939 rpcz_store.cc:255] Call > impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request call > id 5768794) took 839898ms. Request Metrics: {} > > *I will write 632 million rows to kudu, normally it's about 50 thousand/s > , but today after write about 270 million , then write speed become very > slow , I don't know what was wrong.* > > 2019-07-18 > ------------------------------ > lk_hadoop >