This is not indicative of an outright problem and can just be a result
of your data and the hardware which you are running HBase on.
Things to note from this data:
1. This RPC will return up to 1000 rows
2. The size of the data returned is not consistent (200KB for one, 18B
for the other)
3. The queuetimems was 0 which means that the full RPC time was spent
scanning data (not waiting to begin)
A general pattern of debugging is to identify one or a few Scans from
your application(s) which are "slow", and diagnose if they are always
slow or sometimes slow. Try to come up with a specific problem statement.
You would have to first make a decision if they _should_ be slow (i.e.
are they reading a significant amount of data and filtering out data?),
or if they are unexpectedly slow.
If they are unexpectedly slow, you would need to start collecting
additional logging from HBase and/or HDFS, and use other JDK-based
performance tracking tools (jvisualvm, flightrecorder, yourkit profiler,
async-profiler) and try to identify if there is a software explanation
for the slowness. Look at the obvious things and keep a list of things
you rule out as "not a problem" vs. "potentially a problem".
Good luck.
On 10/28/21 11:16 AM, Hamado Dene wrote:
Sorry i forgot to Specify my hbase version:
Hbase 2.2.6Hadoop: 2.8.5
Il giovedì 28 ottobre 2021, 16:55:40 CEST, Hamado Dene
<[email protected]> ha scritto:
Hi hbase community,
Lately during our nactivities we constantly receive the warn:
2021-10-28 16:45:00,854 WARN [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16020] ipc.RpcServer: (responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1635432272849","responsesize":"221799","method":"Scan","param":"scanner_id:
3011016724423115474 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: tr
\u003cTRUNCATED\u003e","processingtimems":28005,"client":"10.200.86.173:60806","queuetimems":0,"class":"HRegionServer","scandetails":"table: mn1_7491_hinvio region:
mn1_7491_hinvio.....}
2021-10-28 16:45:02,527 WARN [RpcServer.default.FPBQ.Fifo.handler=36,queue=1,port=16020] ipc.RpcServer: (responseTooSlow):
{"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1635432272520","responsesize":"18","method":"Scan","param":"scanner_id:
3011016724423115473 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: tr
\u003cTRUNCATED\u003e","processingtimems":30006,"client":"10.200.86.130:53476","queuetimems":0,"class":"HRegionServer","scandetails":"table: mn1_7491_hinvio region:
mn1_7491_hinvio ..........}
We are still trying to understand which improvements to implement in order to
be able to manage the problem.Has anyone ever had the same problem? AndWhat are
the configurations on which we can act to have better performance?
Thanks,
Hamado Dene
