Re: Salting based on partial rowkeys

[Thomas D'Silva]

For the usage example that you provided when you write data how does the
values of id_1, id_2 and other_key vary?
I assume id_1 and id_2 remain the same while other_key is monotonically
increasing, and thats why the table is salted.
If you create the salt bucket only on id_2 then wouldn't you run into
region server hotspotting during writes?

On Thu, Sep 13, 2018 at 8:02 PM, Jaanai Zhang 
wrote:

> Sorry, I don't understander your purpose. According to your proposal, it
> seems that can't achieve.  You need a hash partition, However,  Some things
> need to clarify that HBase is a range partition engine and the salt buckets
> were used to avoid hotspot, in other words, HBase as a storage engine can't
> support hash partition.
>
> 
>Jaanai Zhang
>Best regards!
>
>
>
> Gerald Sangudi  于2018年9月13日周四 下午11:32写道：
>
>> Hi folks,
>>
>> Any thoughts or feedback on this?
>>
>> Thanks,
>> Gerald
>>
>> On Mon, Sep 10, 2018 at 1:56 PM, Gerald Sangudi 
>> wrote:
>>
>>> Hello folks,
>>>
>>> We have a requirement for salting based on partial, rather than full,
>>> rowkeys. My colleague Mike Polcari has identified the requirement and
>>> proposed an approach.
>>>
>>> I found an already-open JIRA ticket for the same issue:
>>> https://issues.apache.org/jira/browse/PHOENIX-4757. I can provide more
>>> details from the proposal.
>>>
>>> The JIRA proposes a syntax of SALT_BUCKETS(col, ...) = N, whereas Mike
>>> proposes SALT_COLUMN=col or SALT_COLUMNS=col, ... .
>>>
>>> The benefit at issue is that users gain more control over partitioning,
>>> and this can be used to push some additional aggregations and hash joins
>>> down to region servers.
>>>
>>> I would appreciate any go-ahead / thoughts / guidance / objections /
>>> feedback. I'd like to be sure that the concept at least is not
>>> objectionable. We would like to work on this and submit a patch down the
>>> road. I'll also add a note to the JIRA ticket.
>>>
>>> Thanks,
>>> Gerald
>>>
>>>
>>

Re: ABORTING region server and following HBase cluster "crash"

[Jonathan Leech]

This seems similar to a failure scenario I’ve seen a couple times. I believe 
after multiple restarts you got lucky and tables were brought up by Hbase in 
the correct order. 

What happens is some kind of semi-catastrophic failure where 1 or more region 
servers go down with edits that weren’t flushed, and are only in the WAL. These 
edits belong to regions whose tables have secondary indexes. Hbase wants to 
replay the WAL before bringing up the region server. Phoenix wants to talk to 
the index region during this, but can’t. It fails enough times then stops. 

The more region servers / tables / indexes affected, the more likely that a 
full restart will get stuck in a classic deadlock. A good old-fashioned data 
center outage is a great way to get started with this kind of problem. You 
might make some progress and get stuck again, or restart number N might get 
those index regions initialized before the main table. 

The sure fire way to recover a cluster in this condition is to strategically 
disable all the tables that are failing to come up. You can do this from the 
Hbase shell as long as the master is running. If I remember right, it’s a pain 
since the disable command will hang. You might need to disable a table, kill 
the shell, disable the next table, etc. Then restart. You’ll eventually have a 
cluster with all the region servers finally started, and a bunch of disabled 
regions. If you disabled index tables, enable one, wait for it to become 
available; eg its WAL edits will be replayed, then enable the associated main 
table and wait for it to come online. If Hbase did it’s job without error, and 
your failure didn’t include losing 4 disks at once, order will be restored. 
Lather, rinse, repeat until everything is enabled and online. 

 A big enough failure sprinkled with a little bit of bad luck and what 
seems to be a Phoenix flaw == deadlock trying to get HBASE to start up. Fix by 
forcing the order that Hbase brings regions online. Finally, never go full 
restart. 

> On Sep 10, 2018, at 7:30 PM, Batyrshin Alexander <0x62...@gmail.com> wrote:
> 
> After update web interface at Master show that every region server now 1.4.7 
> and no RITS.
> 
> Cluster recovered only when we restart all regions servers 4 times...
> 
>> On 11 Sep 2018, at 04:08, Josh Elser  wrote:
>> 
>> Did you update the HBase jars on all RegionServers?
>> 
>> Make sure that you have all of the Regions assigned (no RITs). There could 
>> be a pretty simple explanation as to why the index can't be written to.
>> 
>>> On 9/9/18 3:46 PM, Batyrshin Alexander wrote:
>>> Correct me if im wrong.
>>> But looks like if you have A and B region server that has index and primary 
>>> table then possible situation like this.
>>> A and B under writes on table with indexes
>>> A - crash
>>> B failed on index update because A is not operating then B starting aborting
>>> A after restart try to rebuild index from WAL but B at this time is 
>>> aborting then A starting aborting too
>>> From this moment nothing happens (0 requests to region servers) and A and B 
>>> is not responsible from Master-status web interface
 On 9 Sep 2018, at 04:38, Batyrshin Alexander <0x62...@gmail.com 
 > wrote:
 
 After update we still can't recover HBase cluster. Our region servers 
 ABORTING over and over:
 
 prod003:
 Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL 
 [RpcServer.default.FPBQ.Fifo.handler=92,queue=2,port=60020] 
 regionserver.HRegionServer: ABORTING region server 
 prod003,60020,1536446665703: Could not update the index table, killing 
 server region because couldn't write to an index table
 Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL 
 [RpcServer.default.FPBQ.Fifo.handler=77,queue=7,port=60020] 
 regionserver.HRegionServer: ABORTING region server 
 prod003,60020,1536446665703: Could not update the index table, killing 
 server region because couldn't write to an index table
 Sep 09 02:52:19 prod003 hbase[1440]: 2018-09-09 02:52:19,224 FATAL 
 [RpcServer.default.FPBQ.Fifo.handler=82,queue=2,port=60020] 
 regionserver.HRegionServer: ABORTING region server 
 prod003,60020,1536446665703: Could not update the index table, killing 
 server region because couldn't write to an index table
 Sep 09 02:52:28 prod003 hbase[1440]: 2018-09-09 02:52:28,922 FATAL 
 [RpcServer.default.FPBQ.Fifo.handler=94,queue=4,port=60020] 
 regionserver.HRegionServer: ABORTING region server 
 prod003,60020,1536446665703: Could not update the index table, killing 
 server region because couldn't write to an index table
 Sep 09 02:55:02 prod003 hbase[957]: 2018-09-09 02:55:02,096 FATAL 
 [RpcServer.default.FPBQ.Fifo.handler=95,queue=5,port=60020] 
 regionserver.HRegionServer: ABORTING region server 
 prod003,60020,1536450772841: Could not update the index table, killing 
 server 

Re: Salting based on partial rowkeys

[Jaanai Zhang]

Sorry, I don't understander your purpose. According to your proposal, it
seems that can't achieve.  You need a hash partition, However,  Some things
need to clarify that HBase is a range partition engine and the salt buckets
were used to avoid hotspot, in other words, HBase as a storage engine can't
support hash partition.


   Jaanai Zhang
   Best regards!



Gerald Sangudi  于2018年9月13日周四 下午11:32写道：

> Hi folks,
>
> Any thoughts or feedback on this?
>
> Thanks,
> Gerald
>
> On Mon, Sep 10, 2018 at 1:56 PM, Gerald Sangudi 
> wrote:
>
>> Hello folks,
>>
>> We have a requirement for salting based on partial, rather than full,
>> rowkeys. My colleague Mike Polcari has identified the requirement and
>> proposed an approach.
>>
>> I found an already-open JIRA ticket for the same issue:
>> https://issues.apache.org/jira/browse/PHOENIX-4757. I can provide more
>> details from the proposal.
>>
>> The JIRA proposes a syntax of SALT_BUCKETS(col, ...) = N, whereas Mike
>> proposes SALT_COLUMN=col or SALT_COLUMNS=col, ... .
>>
>> The benefit at issue is that users gain more control over partitioning,
>> and this can be used to push some additional aggregations and hash joins
>> down to region servers.
>>
>> I would appreciate any go-ahead / thoughts / guidance / objections /
>> feedback. I'd like to be sure that the concept at least is not
>> objectionable. We would like to work on this and submit a patch down the
>> road. I'll also add a note to the JIRA ticket.
>>
>> Thanks,
>> Gerald
>>
>>
>

Re: Salting based on partial rowkeys

[Gerald Sangudi]

Hi folks,

Any thoughts or feedback on this?

Thanks,
Gerald

On Mon, Sep 10, 2018 at 1:56 PM, Gerald Sangudi 
wrote:

> Hello folks,
>
> We have a requirement for salting based on partial, rather than full,
> rowkeys. My colleague Mike Polcari has identified the requirement and
> proposed an approach.
>
> I found an already-open JIRA ticket for the same issue:
> https://issues.apache.org/jira/browse/PHOENIX-4757. I can provide more
> details from the proposal.
>
> The JIRA proposes a syntax of SALT_BUCKETS(col, ...) = N, whereas Mike
> proposes SALT_COLUMN=col or SALT_COLUMNS=col, ... .
>
> The benefit at issue is that users gain more control over partitioning,
> and this can be used to push some additional aggregations and hash joins
> down to region servers.
>
> I would appreciate any go-ahead / thoughts / guidance / objections /
> feedback. I'd like to be sure that the concept at least is not
> objectionable. We would like to work on this and submit a patch down the
> road. I'll also add a note to the JIRA ticket.
>
> Thanks,
> Gerald
>
>

Re: Issue in upgrading phoenix : java.lang.ArrayIndexOutOfBoundsException: SYSTEM:CATALOG 63

[venk sham]

Did you check system.stats,. If it us empty, needs to be rebuilt by running
major compact on hbasr

On Tue, Sep 11, 2018, 11:33 AM Tanvi Bhandari 
wrote:

> Hi,
>
>
>
> I am trying to upgrade the phoenix binaries in my setup from phoenix-4.6
> (had optional concept of schema) to phoenix-4.14 (schema is a must in
> here).
>
> Earlier, I had the phoenix-4.6-hbase-1.1 binaries. When I try to run the
> phoenix-4.14-hbase-1.3 on the same data. Hbase comes up fine But when I try
> to connect to phoenix using sqline client,  I get the following error on
> *console*:
>
>
>
> 18/09/07 04:22:48 WARN ipc.CoprocessorRpcChannel: Call failed on
> IOException
>
> org.apache.hadoop.hbase.DoNotRetryIOException:
> org.apache.hadoop.hbase.DoNotRetryIOException: SYSTEM:CATALOG: 63
>
> at
> org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:120)
>
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getVersion(MetaDataEndpointImpl.java:3572)
>
> at
> org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:16422)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7435)
>
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1875)
>
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1857)
>
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32209)
>
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
>
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
>
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 63
>
> at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:517)
>
> at org.apache.phoenix.schema.PTableImpl.(PTableImpl.java:421)
>
> at
> org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:406)
>
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:1046)
>
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:587)
>
>at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.loadTable(MetaDataEndpointImpl.java:1305)
>
> at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getVersion(MetaDataEndpointImpl.java:3568)
>
> ... 10 more
>
>
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:326)
>
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1629)
>
> at
> org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:104)
>
> at
> org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:94)
>
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
>
> at
> org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:107)
>
> at
> org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
>
> at
> org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getVersion(MetaDataProtos.java:16739)
>
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$5.call(ConnectionQueryServicesImpl.java:1271)
>
> at
> org.apache.phoenix.query.ConnectionQueryServicesImpl$5.call(ConnectionQueryServicesImpl.java:1263)
>
> at org.apache.hadoop.hbase.client.HTable$15.call(HTable.java:1736)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> *Region-server logs are as follows: *
>
> 2018-09-07 03:23:36,170 ERROR
> [B.defaultRpcServer.handler=1,queue=1,port=29062]
> 

Re: Issue in upgrading phoenix : java.lang.ArrayIndexOutOfBoundsException: SYSTEM:CATALOG 63

[Tanvi Bhandari]

I think i found the issue :

I had the tables created in Phoenix 4.6 which did not have Column name
encoding feature(https://issues.apache.org/jira/browse/PHOENIX-1598). And
then now I have to move to phoenix 4.14 directly.
As far as I know Phoenix handles the upgrade for Column names encoding
using SYSTEM tables.Since, connecting to phoenix 4.14 was failing with
ArrayIndexOutOfBoundsException for SYSTEM:CATALOG: issue, I went ahead and
deleted all SYSTEM tables in from hbase shell. And then reconnecting from
sqlline re-created all Phoenix SYSTEM tables. Then I used my tables DDL to
recreate the Phoenix tables on existing hbase tables. Hbase was still
showing all column names of my table as it is (in english) .

*select count(*)* query was returning correct number of records because it
should be calculating it using row-keys, but *select ** query is not able
to map the column names since they are not encoded in Hbase table.

*Solution *:
I disabled the column name encoding by setting
*phoenix.default.column.encoded.bytes.attrib=0* in my hbase-site.xml
(globally). Although, I can set it at table create statement as well by
setting *COLUMN_ENCODED_BYTES = 0* in table create-statement.

*Steps for upgrade from phoenix-4.6 to phoenix-4.14 :*
1) After upgrade to phoenix 4.14, set
"phoenix.default.column.encoded.bytes.attrib=0" at hbase-site.xml in hbase
(if you want to disable column name encoding globally) or set it at table
level in DDL.
2) Delete all SYSTEM tables from hbase shell.
3) Again connect through phoenix-4.14 sqlline which will recreate all
SYSTEM tables.
4) Map all your data tables from hbase to phoenix by executing all DDLs
(add "COLUMN_ENCODED_BYTES = 0" in create-statement if you do not want to
disable column name encoding globally).

P.S. If you have immutable tables you might want to handle that as well
while disable column-name encoding.(
https://phoenix.apache.org/columnencoding.html). Got all data through
phoenix as well.

Thanks,
Tanvi

On Thu, Sep 13, 2018 at 2:06 AM Thomas D'Silva 
wrote:

> can you attach the schema of your table? and the explain plan for select *
> from mytable?
>
> On Tue, Sep 11, 2018 at 10:24 PM, Tanvi Bhandari  > wrote:
>
>> " mapped hbase tables to phoenix and created them explicitly from
>> phoenix sqlline client. I first created schema corresponding to namespace
>> and then tables." By this statement, I meant the same. I re-created my
>> tables since I had the DDLs with me.
>>
>> After that I tried getting the count of records in my table which gave me
>> 8 records (expected result). - *select count(*) from "myTable"*;
>> But when I performed the *select * from "myTable";* it is not returning
>> any result.
>>
>> On Wed, Sep 12, 2018 at 1:55 AM Thomas D'Silva 
>> wrote:
>>
>>> Since you dropped all the system tables, all the phoenix metadata was
>>> lost. If you have the ddl statements used to create your tables, you can
>>> try rerunning them.
>>>
>>> On Tue, Sep 11, 2018 at 9:32 AM, Tanvi Bhandari <
>>> tanvi.bhand...@gmail.com> wrote:
>>>
 Hi,



 I am trying to upgrade the phoenix binaries in my setup from
 phoenix-4.6 (had optional concept of schema) to phoenix-4.14 (schema is a
 must in here).

 Earlier, I had the phoenix-4.6-hbase-1.1 binaries. When I try to run
 the phoenix-4.14-hbase-1.3 on the same data. Hbase comes up fine But when I
 try to connect to phoenix using sqline client,  I get the following error
 on *console*:



 18/09/07 04:22:48 WARN ipc.CoprocessorRpcChannel: Call failed on
 IOException

 org.apache.hadoop.hbase.DoNotRetryIOException:
 org.apache.hadoop.hbase.DoNotRetryIOException: SYSTEM:CATALOG: 63

 at
 org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:120)

 at
 org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getVersion(MetaDataEndpointImpl.java:3572)

 at
 org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:16422)

 at
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7435)

 at
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1875)

 at
 org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1857)

 at
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32209)

 at
 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)

 at
 org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)

 at
 org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

 at
 org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)

 at java.lang.Thread.run(Thread.java:745)

Re: Missing content in phoenix after writing from Spark

[Josh Elser]

Pretty sure we ran tests with Spark 2.3 with Phoenix 5.0. Not sure if 
Spark has already moved beyond that.


On 9/12/18 11:00 PM, Saif Addin wrote:
Thanks, we'll try Spark Connector then. Thought it didn't support newest 
Spark Versions


On Wed, Sep 12, 2018 at 11:03 PM Jaanai Zhang > wrote:


It seems columns data missing mapping information of the schema. if
you want to use this way to write HBase table,  you can create an
HBase table and uses Phoenix mapping it.


    Jaanai Zhang
    Best regards!



Thomas D'Silva mailto:tdsi...@salesforce.com>> 于2018年9月13日周四 上午6:03写道：

Is there a reason you didn't use the spark-connector to
serialize your data?

On Wed, Sep 12, 2018 at 2:28 PM, Saif Addin mailto:saif1...@gmail.com>> wrote:

Thank you Josh! That was helpful. Indeed, there was a salt
bucket on the table, and the key-column now shows correctly.

However, the problem still persists in that the rest of the
columns show as completely empty on Phoenix (appear
correctly on Hbase). We'll be looking into this but if you
have any further advice, appreciated.

Saif

On Wed, Sep 12, 2018 at 5:50 PM Josh Elser
mailto:els...@apache.org>> wrote:

Reminder: Using Phoenix internals forces you to
understand exactly how
the version of Phoenix that you're using serializes
data. Is there a
reason you're not using SQL to interact with Phoenix?

Sounds to me that Phoenix is expecting more data at the
head of your
rowkey. Maybe a salt bucket that you've defined on the
table but not
created?

On 9/12/18 4:32 PM, Saif Addin wrote:
 > Hi all,
 >
 > We're trying to write tables with all string columns
from spark.
 > We are not using the Spark Connector, instead we are
directly writing
 > byte arrays from RDDs.
 >
 > The process works fine, and Hbase receives the data
correctly, and
 > content is consistent.
 >
 > However reading the table from Phoenix, we notice the
first character of
 > strings are missing. This sounds like it's a byte
encoding issue, but
 > we're at loss. We're using PVarchar to generate bytes.
 >
 > Here's the snippet of code creating the RDD:
 >
 > val tdd = pdd.flatMap(x => {
 >    val rowKey = PVarchar.INSTANCE.toBytes(x._1)
 >    for(i <- 0 until cols.length) yield {
 >      other stuff for other columns ...
 >      ...
 >      (rowKey, (column1, column2, column3))
 >    }
 > })
 >
 > ...
 >
 > We then create the following output to be written
down in Hbase
 >
 > val output = tdd.map(x => {
 >      val rowKeyByte: Array[Byte] = x._1
 >      val immutableRowKey = new
ImmutableBytesWritable(rowKeyByte)
 >
 >      val kv = new KeyValue(rowKeyByte,
 >          PVarchar.INSTANCE.toBytes(column1),
 >          PVarchar.INSTANCE.toBytes(column2),
 >        PVarchar.INSTANCE.toBytes(column3)
 >      )
 >      (immutableRowKey, kv)
 > })
 >
 > By the way, we are using *KryoSerializer* in order to
be able to
 > serialize all classes necessary for Hbase (KeyValue,
BytesWritable, etc).
 >
 > The key of this table is the one missing data when
queried from Phoenix.
 > So we guess something is wrong with the byte ser.
 >
 > Any ideas? Appreciated!
 > Saif