Re: Issue with Restoration on Phoenix version 4.12

[Sergey Soldatov]

If you exported tables from 4.8 and importing them to the preexisting
tables in 4.12, make sure that you created tables using COLUMN_ENCODED_BYTES
= 0 or have phoenix.default.column.encoded.bytes.attrib set to 0 in
hbase-site.xml.
I believe that the problem you see is the column name encoding that is
enabled by default. So your previous tables have full column name as the
column qualifier, but the newer version of Phoenix expects to see the
column index there. More details about column encoding can be found at
https://phoenix.apache.org/columnencoding.html

Thanks,
Sergey

On Fri, Sep 7, 2018 at 5:51 AM Azharuddin Shaikh <
azharuddin.sha...@resilinc.com> wrote:

> Hi All,
>
> We have upgraded the phoenix version from 4.8 to 4.12 to resolve duplicate
> count issue but we are now facing issue with restoration of tables on
> phoenix version 4.12. Our Hbase version is 1.2.3
>
> We are using Hbase EXPORT/IMPORT utilities to export and import data into
> Hbase tables but when we try to check the records(Rows) using phoenix tool
> it is returning '0' records but when we run count query it returns the
> count
> of records(Rows) in the table but not able to print the content of table.
>
> Is this a bug with Phoenix version 4.12 or is there any fix for this
> restoration issue. Request you to please advice, your help would be greatly
> appreciated.
>
>
>
>
>
> --
> Sent from: http://apache-phoenix-user-list.1124778.n5.nabble.com/
>

Re: ABORTING region server and following HBase cluster "crash"

[Sergey Soldatov]

Forgot to mention. That kind of problems can be mitigated by increasing the
number of threads for open regions. By default, it's 3 (?), but we haven't
seen any problems with increasing it up to several hundred for clusters
that have up to 2k regions per RS.
Thanks,
Sergey

On Fri, Sep 14, 2018 at 4:04 PM Sergey Soldatov 
wrote:

> That was the real problem quite a long time ago (couple years?). Can't say
> for sure in which version that was fixed, but now indexes has a priority
> over regular tables and their regions open first. So by the moment when we
> replay WALs for tables, all index regions are supposed to be online. If you
> see the problem on recent versions that usually means that cluster is not
> healthy and some of the index regions stuck in RiT state.
>
> Thanks,
> Sergey
>
> On Thu, Sep 13, 2018 at 8:12 PM Jonathan Leech  wrote:
>
>> This seems similar to a failure scenario I’ve seen a couple times. I
>> believe after multiple restarts you got lucky and tables were brought up by
>> Hbase in the correct order.
>>
>> What happens is some kind of semi-catastrophic failure where 1 or more
>> region servers go down with edits that weren’t flushed, and are only in the
>> WAL. These edits belong to regions whose tables have secondary indexes.
>> Hbase wants to replay the WAL before bringing up the region server. Phoenix
>> wants to talk to the index region during this, but can’t. It fails enough
>> times then stops.
>>
>> The more region servers / tables / indexes affected, the more likely that
>> a full restart will get stuck in a classic deadlock. A good old-fashioned
>> data center outage is a great way to get started with this kind of problem.
>> You might make some progress and get stuck again, or restart number N might
>> get those index regions initialized before the main table.
>>
>> The sure fire way to recover a cluster in this condition is to
>> strategically disable all the tables that are failing to come up. You can
>> do this from the Hbase shell as long as the master is running. If I
>> remember right, it’s a pain since the disable command will hang. You might
>> need to disable a table, kill the shell, disable the next table, etc. Then
>> restart. You’ll eventually have a cluster with all the region servers
>> finally started, and a bunch of disabled regions. If you disabled index
>> tables, enable one, wait for it to become available; eg its WAL edits will
>> be replayed, then enable the associated main table and wait for it to come
>> online. If Hbase did it’s job without error, and your failure didn’t
>> include losing 4 disks at once, order will be restored. Lather, rinse,
>> repeat until everything is enabled and online.
>>
>>  A big enough failure sprinkled with a little bit of bad luck and
>> what seems to be a Phoenix flaw == deadlock trying to get HBASE to start
>> up. Fix by forcing the order that Hbase brings regions online. Finally,
>> never go full restart. 
>>
>> > On Sep 10, 2018, at 7:30 PM, Batyrshin Alexander <0x62...@gmail.com>
>> wrote:
>> >
>> > After update web interface at Master show that every region server now
>> 1.4.7 and no RITS.
>> >
>> > Cluster recovered only when we restart all regions servers 4 times...
>> >
>> >> On 11 Sep 2018, at 04:08, Josh Elser  wrote:
>> >>
>> >> Did you update the HBase jars on all RegionServers?
>> >>
>> >> Make sure that you have all of the Regions assigned (no RITs). There
>> could be a pretty simple explanation as to why the index can't be written
>> to.
>> >>
>> >>> On 9/9/18 3:46 PM, Batyrshin Alexander wrote:
>> >>> Correct me if im wrong.
>> >>> But looks like if you have A and B region server that has index and
>> primary table then possible situation like this.
>> >>> A and B under writes on table with indexes
>> >>> A - crash
>> >>> B failed on index update because A is not operating then B starting
>> aborting
>> >>> A after restart try to rebuild index from WAL but B at this time is
>> aborting then A starting aborting too
>> >>> From this moment nothing happens (0 requests to region servers) and A
>> and B is not responsible from Master-status web interface
>>  On 9 Sep 2018, at 04:38, Batyrshin Alexander <0x62...@gmail.com
>> > wrote:
>> 
>>  After update we still can't recover HBase cluster. Our region
>> servers ABORTING over and over:
>> 
>>  prod003:
>>  Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL
>> [RpcServer.default.FPBQ.Fifo.handler=92,queue=2,port=60020]
>> regionserver.HRegionServer: ABORTING region server
>> prod003,60020,1536446665703: Could not update the index table, killing
>> server region because couldn't write to an index table
>>  Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL
>> [RpcServer.default.FPBQ.Fifo.handler=77,queue=7,port=60020]
>> regionserver.HRegionServer: ABORTING region server
>> prod003,60020,1536446665703: Could not update the index table, killing
>> server region because couldn't w

Re: ABORTING region server and following HBase cluster "crash"

[Sergey Soldatov]

That was the real problem quite a long time ago (couple years?). Can't say
for sure in which version that was fixed, but now indexes has a priority
over regular tables and their regions open first. So by the moment when we
replay WALs for tables, all index regions are supposed to be online. If you
see the problem on recent versions that usually means that cluster is not
healthy and some of the index regions stuck in RiT state.

Thanks,
Sergey

On Thu, Sep 13, 2018 at 8:12 PM Jonathan Leech  wrote:

> This seems similar to a failure scenario I’ve seen a couple times. I
> believe after multiple restarts you got lucky and tables were brought up by
> Hbase in the correct order.
>
> What happens is some kind of semi-catastrophic failure where 1 or more
> region servers go down with edits that weren’t flushed, and are only in the
> WAL. These edits belong to regions whose tables have secondary indexes.
> Hbase wants to replay the WAL before bringing up the region server. Phoenix
> wants to talk to the index region during this, but can’t. It fails enough
> times then stops.
>
> The more region servers / tables / indexes affected, the more likely that
> a full restart will get stuck in a classic deadlock. A good old-fashioned
> data center outage is a great way to get started with this kind of problem.
> You might make some progress and get stuck again, or restart number N might
> get those index regions initialized before the main table.
>
> The sure fire way to recover a cluster in this condition is to
> strategically disable all the tables that are failing to come up. You can
> do this from the Hbase shell as long as the master is running. If I
> remember right, it’s a pain since the disable command will hang. You might
> need to disable a table, kill the shell, disable the next table, etc. Then
> restart. You’ll eventually have a cluster with all the region servers
> finally started, and a bunch of disabled regions. If you disabled index
> tables, enable one, wait for it to become available; eg its WAL edits will
> be replayed, then enable the associated main table and wait for it to come
> online. If Hbase did it’s job without error, and your failure didn’t
> include losing 4 disks at once, order will be restored. Lather, rinse,
> repeat until everything is enabled and online.
>
>  A big enough failure sprinkled with a little bit of bad luck and
> what seems to be a Phoenix flaw == deadlock trying to get HBASE to start
> up. Fix by forcing the order that Hbase brings regions online. Finally,
> never go full restart. 
>
> > On Sep 10, 2018, at 7:30 PM, Batyrshin Alexander <0x62...@gmail.com>
> wrote:
> >
> > After update web interface at Master show that every region server now
> 1.4.7 and no RITS.
> >
> > Cluster recovered only when we restart all regions servers 4 times...
> >
> >> On 11 Sep 2018, at 04:08, Josh Elser  wrote:
> >>
> >> Did you update the HBase jars on all RegionServers?
> >>
> >> Make sure that you have all of the Regions assigned (no RITs). There
> could be a pretty simple explanation as to why the index can't be written
> to.
> >>
> >>> On 9/9/18 3:46 PM, Batyrshin Alexander wrote:
> >>> Correct me if im wrong.
> >>> But looks like if you have A and B region server that has index and
> primary table then possible situation like this.
> >>> A and B under writes on table with indexes
> >>> A - crash
> >>> B failed on index update because A is not operating then B starting
> aborting
> >>> A after restart try to rebuild index from WAL but B at this time is
> aborting then A starting aborting too
> >>> From this moment nothing happens (0 requests to region servers) and A
> and B is not responsible from Master-status web interface
>  On 9 Sep 2018, at 04:38, Batyrshin Alexander <0x62...@gmail.com
> > wrote:
> 
>  After update we still can't recover HBase cluster. Our region servers
> ABORTING over and over:
> 
>  prod003:
>  Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL
> [RpcServer.default.FPBQ.Fifo.handler=92,queue=2,port=60020]
> regionserver.HRegionServer: ABORTING region server
> prod003,60020,1536446665703: Could not update the index table, killing
> server region because couldn't write to an index table
>  Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL
> [RpcServer.default.FPBQ.Fifo.handler=77,queue=7,port=60020]
> regionserver.HRegionServer: ABORTING region server
> prod003,60020,1536446665703: Could not update the index table, killing
> server region because couldn't write to an index table
>  Sep 09 02:52:19 prod003 hbase[1440]: 2018-09-09 02:52:19,224 FATAL
> [RpcServer.default.FPBQ.Fifo.handler=82,queue=2,port=60020]
> regionserver.HRegionServer: ABORTING region server
> prod003,60020,1536446665703: Could not update the index table, killing
> server region because couldn't write to an index table
>  Sep 09 02:52:28 prod003 hbase[1440]: 2018-09-09 02:52:28,922 FATAL
> [RpcServer.def

Re: Salting based on partial rowkeys

[Sergey Soldatov]

Thomas is absolutely right that there will be a possibility of hotspotting.
Salting is the mechanism that should prevent that in all cases (because all
rowids are different). The partitioning described above actually can be
implemented by using id2 as a first column of the PK and using presplit by
values without salting. The only difference will be that in the suggested
approach we don't need to know the values range for that particular
column(s). If we want to implement that (well, I remember several cases
when people asked how to presplit the table without information about the
range of values for PK columns to improve bulk load to a new table without
performance lost for some queries due salting) it would be better to
separate it from salting and call it 'partitioning' or something like
that.

Thanks,
Sergey

On Thu, Sep 13, 2018 at 10:09 PM Thomas D'Silva 
wrote:

> For the usage example that you provided when you write data how does the
> values of id_1, id_2 and other_key vary?
> I assume id_1 and id_2 remain the same while other_key is monotonically
> increasing, and thats why the table is salted.
> If you create the salt bucket only on id_2 then wouldn't you run into
> region server hotspotting during writes?
>
> On Thu, Sep 13, 2018 at 8:02 PM, Jaanai Zhang 
> wrote:
>
>> Sorry, I don't understander your purpose. According to your proposal, it
>> seems that can't achieve.  You need a hash partition, However,  Some things
>> need to clarify that HBase is a range partition engine and the salt buckets
>> were used to avoid hotspot, in other words, HBase as a storage engine can't
>> support hash partition.
>>
>> 
>>Jaanai Zhang
>>Best regards!
>>
>>
>>
>> Gerald Sangudi  于2018年9月13日周四 下午11:32写道：
>>
>>> Hi folks,
>>>
>>> Any thoughts or feedback on this?
>>>
>>> Thanks,
>>> Gerald
>>>
>>> On Mon, Sep 10, 2018 at 1:56 PM, Gerald Sangudi 
>>> wrote:
>>>
 Hello folks,

 We have a requirement for salting based on partial, rather than full,
 rowkeys. My colleague Mike Polcari has identified the requirement and
 proposed an approach.

 I found an already-open JIRA ticket for the same issue:
 https://issues.apache.org/jira/browse/PHOENIX-4757. I can provide more
 details from the proposal.

 The JIRA proposes a syntax of SALT_BUCKETS(col, ...) = N, whereas Mike
 proposes SALT_COLUMN=col or SALT_COLUMNS=col, ... .

 The benefit at issue is that users gain more control over partitioning,
 and this can be used to push some additional aggregations and hash joins
 down to region servers.

 I would appreciate any go-ahead / thoughts / guidance / objections /
 feedback. I'd like to be sure that the concept at least is not
 objectionable. We would like to work on this and submit a patch down the
 road. I'll also add a note to the JIRA ticket.

 Thanks,
 Gerald


>>>
>

Re: Salting based on partial rowkeys

[Josh Elser]

Yeah, I think that's his point :)

For a fine-grained facet, the hotspotting is desirable to co-locate the 
data for query. To try to make an example to drive this point home:


Consider a primary key constraint(col1, col2, col3, col4);

If I defined the SALT_HASH based on "col1" alone, you'd get terrible 
hotspotting. However, the contrast is when we have SALT_HASH on col1, 
col2, col3, and col4, we have no row-oriented data locality (we have to 
check *all* salt buckets for every query).


If you define the SALT_HASH on col1, col2, and col3, all values for col4 
where col1-3 are fixed are co-located which would make faceted search 
queries much faster (num SALT_BUCKET RPCs down to 1 RPC).


Concretely: if I'm on Amazon searching for "water bottle" "1L size" 
"plastic composition" (col1, col2, and col3), it's really fast to give 
me "manufacturer" (col4) given my other three constraints.


Hopefully I'm getting this right too. Tell me to shut up, Gerald, if I'm 
not :)


On 9/14/18 1:01 AM, Thomas D'Silva wrote:
For the usage example that you provided when you write data how does the 
values of id_1, id_2 and other_key vary?
I assume id_1 and id_2 remain the same while other_key is monotonically 
increasing, and thats why the table is salted.
If you create the salt bucket only on id_2 then wouldn't you run into 
region server hotspotting during writes?


On Thu, Sep 13, 2018 at 8:02 PM, Jaanai Zhang > wrote:


Sorry, I don't understander your purpose. According to your
proposal, it seems that can't achieve.  You need a hash partition,
However,  Some things need to clarify that HBase is a range
partition engine and the salt buckets were used to avoid hotspot, in
other words, HBase as a storage engine can't support hash partition.


    Jaanai Zhang
    Best regards!



Gerald Sangudi mailto:gsang...@23andme.com>>
于2018年9月13日周四 下午11:32写道：

Hi folks,

Any thoughts or feedback on this?

Thanks,
Gerald

On Mon, Sep 10, 2018 at 1:56 PM, Gerald Sangudi
mailto:gsang...@23andme.com>> wrote:

Hello folks,

We have a requirement for salting based on partial, rather
than full, rowkeys. My colleague Mike Polcari has identified
the requirement and proposed an approach.

I found an already-open JIRA ticket for the same issue:
https://issues.apache.org/jira/browse/PHOENIX-4757
. I can
provide more details from the proposal.

The JIRA proposes a syntax of SALT_BUCKETS(col, ...) = N,
whereas Mike proposes SALT_COLUMN=col or SALT_COLUMNS=col, ... .

The benefit at issue is that users gain more control over
partitioning, and this can be used to push some additional
aggregations and hash joins down to region servers.

I would appreciate any go-ahead / thoughts / guidance /
objections / feedback. I'd like to be sure that the concept
at least is not objectionable. We would like to work on this
and submit a patch down the road. I'll also add a note to
the JIRA ticket.

Thanks,
Gerald

Re: Missing content in phoenix after writing from Spark

[Saif Addin]

Hi, I am attempting to make connection with Spark but no success so far.

For writing into Phoenix, I am trying this:

tdd.toDF("ID", "COL1", "COL2",
"COL3").write.format("org.apache.phoenix.spark").option("zkUrl",
"zookeper-host-url:2181").option("table",
htablename).mode("overwrite").save()

But getting:
*java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.*

For reading, on the other hand, attempting this:

val hbConf = HBaseConfiguration.create()
val hbaseSitePath = "/etc/hbase/conf/hbase-site.xml"
hbConf.addResource(new Path(hbaseSitePath))

spark.sqlContext.phoenixTableAsDataFrame("VISTA_409X68", Array("ID"), conf
= hbConf)

Gets me
*java.lang.NoClassDefFoundError: Could not initialize class
org.apache.phoenix.query.QueryServicesOptions*

I have added phoenix-queryserver-5.0.0-HBase-2.0.jar and
phoenix-queryserver-client-5.0.0-HBase-2.0.jar

Any thoughts? I have an hbase-site.xml file with more configuration but not
sure how to get it to be read in the saving instance.

Thanks

On Thu, Sep 13, 2018 at 11:38 AM Josh Elser  wrote:

> Pretty sure we ran tests with Spark 2.3 with Phoenix 5.0. Not sure if
> Spark has already moved beyond that.
>
> On 9/12/18 11:00 PM, Saif Addin wrote:
> > Thanks, we'll try Spark Connector then. Thought it didn't support newest
> > Spark Versions
> >
> > On Wed, Sep 12, 2018 at 11:03 PM Jaanai Zhang  > > wrote:
> >
> > It seems columns data missing mapping information of the schema. if
> > you want to use this way to write HBase table,  you can create an
> > HBase table and uses Phoenix mapping it.
> >
> > 
> > Jaanai Zhang
> > Best regards!
> >
> >
> >
> > Thomas D'Silva  > > 于2018年9月13日周四 上午6:03写道：
> >
> > Is there a reason you didn't use the spark-connector to
> > serialize your data?
> >
> > On Wed, Sep 12, 2018 at 2:28 PM, Saif Addin  > > wrote:
> >
> > Thank you Josh! That was helpful. Indeed, there was a salt
> > bucket on the table, and the key-column now shows correctly.
> >
> > However, the problem still persists in that the rest of the
> > columns show as completely empty on Phoenix (appear
> > correctly on Hbase). We'll be looking into this but if you
> > have any further advice, appreciated.
> >
> > Saif
> >
> > On Wed, Sep 12, 2018 at 5:50 PM Josh Elser
> > mailto:els...@apache.org>> wrote:
> >
> > Reminder: Using Phoenix internals forces you to
> > understand exactly how
> > the version of Phoenix that you're using serializes
> > data. Is there a
> > reason you're not using SQL to interact with Phoenix?
> >
> > Sounds to me that Phoenix is expecting more data at the
> > head of your
> > rowkey. Maybe a salt bucket that you've defined on the
> > table but not
> > created?
> >
> > On 9/12/18 4:32 PM, Saif Addin wrote:
> >  > Hi all,
> >  >
> >  > We're trying to write tables with all string columns
> > from spark.
> >  > We are not using the Spark Connector, instead we are
> > directly writing
> >  > byte arrays from RDDs.
> >  >
> >  > The process works fine, and Hbase receives the data
> > correctly, and
> >  > content is consistent.
> >  >
> >  > However reading the table from Phoenix, we notice the
> > first character of
> >  > strings are missing. This sounds like it's a byte
> > encoding issue, but
> >  > we're at loss. We're using PVarchar to generate bytes.
> >  >
> >  > Here's the snippet of code creating the RDD:
> >  >
> >  > val tdd = pdd.flatMap(x => {
> >  >val rowKey = PVarchar.INSTANCE.toBytes(x._1)
> >  >for(i <- 0 until cols.length) yield {
> >  >  other stuff for other columns ...
> >  >  ...
> >  >  (rowKey, (column1, column2, column3))
> >  >}
> >  > })
> >  >
> >  > ...
> >  >
> >  > We then create the following output to be written
> > down in Hbase
> >  >
> >  > val output = tdd.map(x => {
> >  >  val rowKeyByte: Array[Byte] = x._1
> >  >  val immutableRowKey = new
> > ImmutableBytesWritable(rowKeyByte)
> >