w.r.t. "Unable to find cached index metadata" error, have you seen this ?
http://search-hadoop.com/m/Phoenix/9UY0h2YBSOhgbflB1?subj=Re+Global+Secondary+Index+ERROR+2008+INT10+Unable+to+find+cached+index+metadata+PHOENIX+1718+
Cheers
On Wednesday, November 30, 2016 11:59 PM, Neelesh <neele...@gmail.com>
wrote:
Ted, we use HDP 2.3.4 (HBase 1.1.2, phoenix 4.4 - but with a lot of backports
from later versions)
The key of the data table is <customerid- 11 bytes><userid- 36 bytes- but it
really is a right padded long due to historic data><event type
long><timestamp><random long>
The two global indexes are <customerid-11 bytes><event type long><timestamp>
<user id> and <customerid-11 bytes><campaign id long><timestamp> Around 100B
rows in the main table. The main issues we see are
# Sudden spikes in queueSize - going all the way to 1G limit and staying there,
without any correlated client traffic# Boatloads of these errors 2016-11-30
11:28:54,907 INFO [RW.default.writeRpcServer.handler=43,queue=9,port=16020]
util.IndexManagementUtil: Rethrowing
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR 2008
(INT10): Unable to find cached index metadata. key=120521194876100862
region=<region-key>. Index update failed
We have cross datacenter WAL replication enabled.We saw PHOENIX-1718, and
changed all recommended timeouts to 1 hour. Our HBase version has HBase-11705.
We also discovered that the queuesize is global (across
general/replication/priority queues) and if it reaches the 1GB limit, calls to
all queues will drop. That was interesting because even though the replication
handlers have a different queue, the size is counted globally, affecting
others. Please correct me on this. I hope I'm wrong on this one :)
Our challenge has been to understand what's HBase doing under various
scenarios. We monitor call queue lengths, sizes and latencies as the primary
alerting mechanism to tell us something is going on with HBase.
Thanks!-neelesh
On Wed, Nov 30, 2016 at 1:15 PM, Ted Yu <ted...@yahoo.com.invalid> wrote:
Neelesh:Can you share more details about the sluggish cluster performance (such
as version of hbase / phoenix, your schema, region server log snippet, stack
traces, etc) ?
As hbase / phoenix evolve, I hope the performance keeps getting better for your
use case.
Cheers
On Wednesday, November 30, 2016 10:07 AM, Neelesh <neele...@gmail.com>
wrote:
We use both, in different capacities. Cassandra is an x-DC archive store
with mostly batch writes and occasional key based reads. Hbase is for
real-time event ingestion. Our experience so far on hbase + phoenix is that
when it works, it is fast and scales like crazy. But if you ever hit a snag
around data patterns, you will have a VERY hard time figuring out what's
going on. A combination of global phoenix indexes and heavy writes leave an
entire cluster sluggish, if there is a hint of hotspotting.
On the other hand, we had a big struggle getting Cassandra when a node
recovery was in progress. What with twice the amount of disk requirements
during recovery etc. Other than that, it is quiet.
But the access patterns are not the same.
I think the old rule still stays. If you are already on hadoop , or
interested in using/analysing data in several different ways, go with hbase
. If you just need a big data store with a few predefined query patterns,
Cassandra is good
Of course, I'm biased towards HBase.
On Nov 30, 2016 7:02 AM, "Mich Talebzadeh" <mich.talebza...@gmail.com>
wrote:
> Hi Guys,
>
> Used Hbase on HDFS reasonably well. Happy to to stick with it and more with
> Hive/Phoenix views and Phoenix indexes where I can.
>
> I have a bunch of users now vocal about the use case for Cassandra and
> whether it can do a better job than Hbase.
>
> Unfortunately I am no expert on Cassandra. However, some use case fit would
> be very valuable.
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/ profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCd OABUrV8Pw
> <https://www.linkedin.com/ profile/view?id= AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
>
>
>
> http://talebzadehmich. wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
