Can you check server log pertaining to scanner ids 692411060 and 692411074 ?
Pastebin relevant snippet, please.
Thanks
On Fri, Apr 14, 2017 at 1:01 PM, Kohki Nishio <tarop...@icloud.com> wrote:
> Yes it is from hbase-client 1.3.0, strange thing is that this only happens
> under load.
> Thanks
> -Kohki
>
> On Apr 14, 2017, at 12:30 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> Was the trace obtained with 1.3 client ?
>
> Thanks
>
>
> On Apr 14, 2017, at 11:43 AM, Kohki Nishio <tarop...@icloud.com> wrote:
>
>
> Here I have trace output, the exact consecutive queries got two different
> results, not sure where to look about my region servers..
>
>
>
> 18:30:00,073 o.a.h.h.c.ClientScanner: Scan table=mytable,
> startRow=AAAAAZJRJgq702gz9Qq-bXvyvi5Vfwg-\x00\x00\x00\x00X\xF1\x06\xDC
>
> 18:30:00,075 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms
>
> 18:30:00,075 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
> scanner as id=692411060 associated with replica=0
>
> 18:30:00,076 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms
>
> 18:30:00,076 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
> scanner as id=692411060 associated with replica=0
>
> 18:30:00,077 o.a.h.h.c.ClientScanner: number results from RPC: 50,partial
> != null: false,number of partials so far: 0
>
> 18:30:00,078 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms
>
> 18:30:00,078 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
> scanner as id=692411060 associated with replica=0
>
> 18:30:00,078 o.a.h.h.c.ClientScanner: number results from RPC: 6,partial
> != null: false,number of partials so far: 0
>
> 18:30:00,078 o.a.h.h.c.ScannerCallableWithReplicas: Closing scanner
> id=692411060
>
> 18:30:00,079 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms
>
> 18:30:00,079 o.a.h.h.c.ClientScanner: Finished {ENCODED =>
> c6e51623c143984e36c21340b182d2c7, NAME => 'mytable,AAAAAZ-
> L4MCUr7LCj8ZzmWJdhOqML0fy\x00\x00\x00\x00X\xE8A\x8D,1492193926115.
> c6e51623c143984e36c21340b182d2c7.', STARTKEY => 'AAAAAZ-
> L4MCUr7LCj8ZzmWJdhOqML0fy\x00\x00\x00\x00X\xE8A\x8D', ENDKEY => '
> AAAAAZzZUufbQvQaVdhpt2wUk1CkRcNz\x00\x00\x00\x00X\xE9\xE71'}
>
>
>
> 18:30:00,458 o.a.h.h.c.ClientScanner: Scan table=mytable,
> startRow=AAAAAZJRJgq702gz9Qq-bXvyvi5Vfwg-\x00\x00\x00\x00X\xF1\x06\xDC
>
> 18:30:00,459 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms
>
> 18:30:00,459 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
> scanner as id=692411074 associated with replica=0
>
> 18:30:00,461 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 1ms
>
> 18:30:00,461 o.a.h.h.c.ScannerCallableWithReplicas: Setting current
> scanner as id=692411074 associated with replica=0
>
> 18:30:00,461 o.a.h.h.c.ClientScanner: number results from RPC: 44,partial
> != null: false,number of partials so far: 0
>
> 18:30:00,461 o.a.h.h.c.ScannerCallableWithReplicas: Closing scanner
> id=692411074
>
> 18:30:00,461 o.a.h.h.i.AbstractRpcClient: Call: Scan, callTime: 0ms
>
> 18:30:00,461 o.a.h.h.c.ClientScanner: Finished {ENCODED =>
> c6e51623c143984e36c21340b182d2c7, NAME => 'mytable,AAAAAZ-
> L4MCUr7LCj8ZzmWJdhOqML0fy\x00\x00\x00\x00X\xE8A\x8D,1492193926115.
> c6e51623c143984e36c21340b182d2c7.', STARTKEY => 'AAAAAZ-
> L4MCUr7LCj8ZzmWJdhOqML0fy\x00\x00\x00\x00X\xE8A\x8D', ENDKEY => '
> AAAAAZzZUufbQvQaVdhpt2wUk1CkRcNz\x00\x00\x00\x00X\xE9\xE71'}
>
>
>
>
> On Apr 13, 2017, at 08:09 PM, Kohki Nishio <tarop...@icloud.com> wrote:
>
>
>
> Actually I tried the one with HBASE-15378, but it didn’t fix the problem ..
>
>
>
> On Apr 12, 2017, at 8:14 PM, Kohki Nishio <tarop...@icloud.com> wrote:
>
>
> As far as I can see, the change is in ClientScanner class, which is a part
> of Hbase-client, does this work inside Server (region server?) ?
>
>
> Thanks
>
> -Kohki
>
>
>
> On Apr 12, 2017, at 6:03 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>
> Does the release (deployed on servers) contain HBASE-15378 ?
>
>
> Cheers
>
>
> On Wed, Apr 12, 2017 at 5:34 PM, Kohki Nishio <tarop...@icloud.com> wrote:
>
>
> Hello HBase users !
>
>
> I’m seeing very strange behavior with HBase (1.1.2) and I’d like to ask
>
> some help here. I’m scanning rows but sometimes it returns incomplete
>
> results. Let’s say I’m expecting 60 rows to come back, most of times I get
>
> all of them. However sometimes (1 in 50), I get only 1 or 2 rows.
>
>
> According to my test, this happens only when there are other activities
>
> (lots of PUT / GET are happening at the same time), I couldn’t reproduce
>
> with a simple standalone scan test case. First we saw this problem with
>
> AsyncHBase client then I tried v1.3 HBase client, both showed the exact
>
> same behavior.
>
>
> Based on the trace log message, it seems this is not a problem at client
>
> side since two different client implementations showed the same behavior.
>
> However HBase is an established storage solution, it is so strange to see
>
> this problem at this frequency, that leads me to believe that I might be
>
> doing something very strange. But I’m not sure what to do next.
>
>
> The key symptom is that it returns the correct result for most of times,
>
> but sometimes it returns very small set of results. Any thoughts would be
>
> appreciated (RegionServer is at 1.1.2.x.x)
>
>
> Thanks
>
> -Kohki
>
>
>
