Fwd: impala with kudu write become very slow
Hi, It's hard to say what might be the problem without additional information. Could you clarify on the following questions: 1. What was the rate of write operations for the 270M rows you mentioned? Was that regular 50K rows/sec or something else? 2. Do you still observe the slowness or it's already gone? (if that was just a spike in ops rate, Kudu tablet servers might be rejecting writes due to the memory pressure when not being able to flush the data on disk fast enough). 3. What was special about those 270M rows? Maybe, the order of keys changed somehow? Kudu ingests data much faster if row keys come in kind of sequential order: so if the prior write batches had that property, but the recent 270M chunk didn't, it might be the case. 4. Was there any additional concurrent activity on the nodes where Kudu tablet servers run? Thanks, Alexey On Thu, Jul 18, 2019 at 7:48 PM Tim Armstrong wrote: > Also including the Kudu list in case someone there recognises the problem. > > On Thu, Jul 18, 2019 at 8:05 AM lk_hadoop wrote: > >> I0718 18:42:22.677520 51139 coordinator.cc:357] starting execution on 5 >> backends for query_id=2e4a3fbec0d7d721:2ec73c1c >> I0718 18:42:22.679605 12873 impala-internal-service.cc:44] >> ExecQueryFInstances(): query_id=2e4a3fbec0d7d721:2ec73c1c >> I0718 18:42:22.679620 12873 query-exec-mgr.cc:46] StartQueryFInstances() >> query_id=2e4a3fbec0d7d721:2ec73c1c >> coord=realtimeanalysis-kudu-04-10-8-50-58:22000 >> I0718 18:42:22.679625 12873 query-state.cc:178] Buffer pool limit for >> 2e4a3fbec0d7d721:2ec73c1c: 17179869184 >> I0718 18:42:22.679675 12873 initial-reservations.cc:60] Successfully >> claimed initial reservations (4.00 MB) for query >> 2e4a3fbec0d7d721:2ec73c1c >> I0718 18:42:22.679769 51332 query-state.cc:309] StartFInstances(): >> query_id=2e4a3fbec0d7d721:2ec73c1c #instances=2 >> I0718 18:42:22.680196 51332 query-state.cc:322] descriptor table for >> query=2e4a3fbec0d7d721:2ec73c1c >> tuples: >> Tuple(id=2 size=567 slots=[Slot(id=52 type=INT col_path=[] offset=464 >> null=(offset=563 mask=20) slot_idx=29 field_idx=-1), Slot(id=53 type=STRING >> col_path=[] offset=0 null=(offset=560 mask=1) slot_idx=0 field_idx=-1), >> Slot(id=54 type=STRING col_path=[] offset=16 null=(offset=560 mask=2) >> slot_idx=1 field_idx=-1), Slot(id=55 type=STRING col_path=[] offset=32 >> null=(offset=560 mask=4) slot_idx=2 field_idx=-1), Slot(id=56 type=STRING >> col_path=[] offset=48 null=(offset=560 mask=8) slot_idx=3 field_idx=-1), >> Slot(id=57 type=STRING col_path=[] offset=64 null=(offset=560 mask=10) >> slot_idx=4 field_idx=-1), Slot(id=58 type=STRING col_path=[] offset=80 >> null=(offset=560 mask=20) slot_idx=5 field_idx=-1), Slot(id=59 type=STRING >> col_path=[] offset=96 null=(offset=560 mask=40) slot_idx=6 field_idx=-1), >> Slot(id=60 type=STRING col_path=[] offset=112 null=(offset=560 mask=80) >> slot_idx=7 field_idx=-1), Slot(id=61 type=STRING col_path=[] offset=128 >> null=(offset=561 mask=1) slot_idx=8 field_idx=-1), Slot(id=62 type=STRING >> col_path=[] offset=144 null=(offset=561 mask=2) slot_idx=9 field_idx=-1), >> Slot(id=63 type=STRING col_path=[] offset=160 null=(offset=561 mask=4) >> slot_idx=10 field_idx=-1), Slot(id=64 type=INT col_path=[] offset=468 >> null=(offset=563 mask=40) slot_idx=30 field_idx=-1), Slot(id=65 type=INT >> col_path=[] offset=472 null=(offset=563 mask=80) slot_idx=31 field_idx=-1), >> Slot(id=66 type=INT col_path=[] offset=476 null=(offset=564 mask=1) >> slot_idx=32 field_idx=-1), Slot(id=67 type=INT col_path=[] offset=480 >> null=(offset=564 mask=2) slot_idx=33 field_idx=-1), Slot(id=68 type=STRING >> col_path=[] offset=176 null=(offset=561 mask=8) slot_idx=11 field_idx=-1), >> Slot(id=69 type=STRING col_path=[] offset=192 null=(offset=561 mask=10) >> slot_idx=12 field_idx=-1), Slot(id=70 type=STRING col_path=[] offset=208 >> null=(offset=561 mask=20) slot_idx=13 field_idx=-1), Slot(id=71 type=STRING >> col_path=[] offset=224 null=(offset=561 mask=40) slot_idx=14 field_idx=-1), >> Slot(id=72 type=STRING col_path=[] offset=240 null=(offset=561 mask=80) >> slot_idx=15 field_idx=-1), Slot(id=73 type=STRING col_path=[] offset=256 >> null=(offset=562 mask=1) slot_idx=16 field_idx=-1), Slot(id=74 type=INT >> col_path=[] offset=484 null=(offset=564 mask=4) slot_idx=34 field_idx=-1), >> Slot(id=75 type=INT col_path=[] offset=488 null=(offset=564 mask=8) >> slot_idx=35 field_idx=-1), Slot(id=76 type=INT col_path=[] offset=492 >> null=(offset=564 mask=10) slot_idx=36 field_idx=-1), Slot(id=77 type=INT >> col_path=[] offset=496 null=(offset=564 mask=20) slot_idx=37 field_idx=-1), >> Slot(id=78 type=INT col_path=[] offset=500 null=(offset=564 mask=40) >> slot_idx=38 field_idx=-1), Slot(id=79 type=INT col_path=[] offset=504 >> null=(offset=564 mask=80) slot_idx=39 field_idx=-1), Slot(id=80 type=INT >> col_path=[] offset=508 null=(offset=565 mask=1) slot_idx=40 field_idx=-1), >> Slot(id=81 type=STR
Re: impala with kudu write become very slow
Hi, It's hard to say what might be the problem without additional information. Could you clarify on the following questions: 1. What was the rate of write operations for the 270M rows you mentioned? Was that regular 50K rows/sec or something else? 2. Do you still observe the slowness or it's already gone? (if that was just a spike in ops rate, Kudu tablet servers might be rejecting writes due to the memory pressure when not being able to flush the data on disk fast enough). 3. What was special about those 270M rows? Maybe, the order of keys changed somehow? Kudu ingests data much faster if row keys come in kind of sequential order: so if the prior write batches had that property, but the recent 270M chunk didn't, it might be the case. 4. Was there any additional concurrent activity on the nodes where Kudu tablet servers run? Thanks, Alexey On Thu, Jul 18, 2019 at 7:48 PM Tim Armstrong wrote: > Also including the Kudu list in case someone there recognises the problem. > > On Thu, Jul 18, 2019 at 8:05 AM lk_hadoop wrote: > >> I0718 18:42:22.677520 51139 coordinator.cc:357] starting execution on 5 >> backends for query_id=2e4a3fbec0d7d721:2ec73c1c >> I0718 18:42:22.679605 12873 impala-internal-service.cc:44] >> ExecQueryFInstances(): query_id=2e4a3fbec0d7d721:2ec73c1c >> I0718 18:42:22.679620 12873 query-exec-mgr.cc:46] StartQueryFInstances() >> query_id=2e4a3fbec0d7d721:2ec73c1c >> coord=realtimeanalysis-kudu-04-10-8-50-58:22000 >> I0718 18:42:22.679625 12873 query-state.cc:178] Buffer pool limit for >> 2e4a3fbec0d7d721:2ec73c1c: 17179869184 >> I0718 18:42:22.679675 12873 initial-reservations.cc:60] Successfully >> claimed initial reservations (4.00 MB) for query >> 2e4a3fbec0d7d721:2ec73c1c >> I0718 18:42:22.679769 51332 query-state.cc:309] StartFInstances(): >> query_id=2e4a3fbec0d7d721:2ec73c1c #instances=2 >> I0718 18:42:22.680196 51332 query-state.cc:322] descriptor table for >> query=2e4a3fbec0d7d721:2ec73c1c >> tuples: >> Tuple(id=2 size=567 slots=[Slot(id=52 type=INT col_path=[] offset=464 >> null=(offset=563 mask=20) slot_idx=29 field_idx=-1), Slot(id=53 type=STRING >> col_path=[] offset=0 null=(offset=560 mask=1) slot_idx=0 field_idx=-1), >> Slot(id=54 type=STRING col_path=[] offset=16 null=(offset=560 mask=2) >> slot_idx=1 field_idx=-1), Slot(id=55 type=STRING col_path=[] offset=32 >> null=(offset=560 mask=4) slot_idx=2 field_idx=-1), Slot(id=56 type=STRING >> col_path=[] offset=48 null=(offset=560 mask=8) slot_idx=3 field_idx=-1), >> Slot(id=57 type=STRING col_path=[] offset=64 null=(offset=560 mask=10) >> slot_idx=4 field_idx=-1), Slot(id=58 type=STRING col_path=[] offset=80 >> null=(offset=560 mask=20) slot_idx=5 field_idx=-1), Slot(id=59 type=STRING >> col_path=[] offset=96 null=(offset=560 mask=40) slot_idx=6 field_idx=-1), >> Slot(id=60 type=STRING col_path=[] offset=112 null=(offset=560 mask=80) >> slot_idx=7 field_idx=-1), Slot(id=61 type=STRING col_path=[] offset=128 >> null=(offset=561 mask=1) slot_idx=8 field_idx=-1), Slot(id=62 type=STRING >> col_path=[] offset=144 null=(offset=561 mask=2) slot_idx=9 field_idx=-1), >> Slot(id=63 type=STRING col_path=[] offset=160 null=(offset=561 mask=4) >> slot_idx=10 field_idx=-1), Slot(id=64 type=INT col_path=[] offset=468 >> null=(offset=563 mask=40) slot_idx=30 field_idx=-1), Slot(id=65 type=INT >> col_path=[] offset=472 null=(offset=563 mask=80) slot_idx=31 field_idx=-1), >> Slot(id=66 type=INT col_path=[] offset=476 null=(offset=564 mask=1) >> slot_idx=32 field_idx=-1), Slot(id=67 type=INT col_path=[] offset=480 >> null=(offset=564 mask=2) slot_idx=33 field_idx=-1), Slot(id=68 type=STRING >> col_path=[] offset=176 null=(offset=561 mask=8) slot_idx=11 field_idx=-1), >> Slot(id=69 type=STRING col_path=[] offset=192 null=(offset=561 mask=10) >> slot_idx=12 field_idx=-1), Slot(id=70 type=STRING col_path=[] offset=208 >> null=(offset=561 mask=20) slot_idx=13 field_idx=-1), Slot(id=71 type=STRING >> col_path=[] offset=224 null=(offset=561 mask=40) slot_idx=14 field_idx=-1), >> Slot(id=72 type=STRING col_path=[] offset=240 null=(offset=561 mask=80) >> slot_idx=15 field_idx=-1), Slot(id=73 type=STRING col_path=[] offset=256 >> null=(offset=562 mask=1) slot_idx=16 field_idx=-1), Slot(id=74 type=INT >> col_path=[] offset=484 null=(offset=564 mask=4) slot_idx=34 field_idx=-1), >> Slot(id=75 type=INT col_path=[] offset=488 null=(offset=564 mask=8) >> slot_idx=35 field_idx=-1), Slot(id=76 type=INT col_path=[] offset=492 >> null=(offset=564 mask=10) slot_idx=36 field_idx=-1), Slot(id=77 type=INT >> col_path=[] offset=496 null=(offset=564 mask=20) slot_idx=37 field_idx=-1), >> Slot(id=78 type=INT col_path=[] offset=500 null=(offset=564 mask=40) >> slot_idx=38 field_idx=-1), Slot(id=79 type=INT col_path=[] offset=504 >> null=(offset=564 mask=80) slot_idx=39 field_idx=-1), Slot(id=80 type=INT >> col_path=[] offset=508 null=(offset=565 mask=1) slot_idx=40 field_idx=-1), >> Slot(id=81 type=STR
