Re: [ANNOUNCE] Apache Phoenix 4.14.1 released

2018-11-21 Thread Batyrshin Alexander 
Looks like bin packed with some non standard options.

Extracting on Ubuntu-16.04 looks like this:
$ tar -xzf apache-phoenix-4.14.1-HBase-1.4-bin.tar.gz
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.creationtime'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
tar: Ignoring unknown extended header keyword 'SCHILY.nlink'
tar: Ignoring unknown extended header keyword 'SCHILY.dev'
tar: Ignoring unknown extended header keyword 'SCHILY.ino'
...

> On 15 Nov 2018, at 00:39, Vincent Poon  wrote:
> 
> The Apache Phoenix team is pleased to announce the immediate availability
> of the 4.14.1 patch release. Apache Phoenix enables SQL-based OLTP and
> operational analytics for Apache Hadoop using Apache HBase as its backing
> store and providing integration with other projects in the Apache ecosystem
> such as Spark, Hive, Pig, Flume, and MapReduce.
> 
> This patch release has feature parity with supported HBase versions and 
> includes critical bug fixes for secondary indexes.
> 
> Download source and binaries here [1].
> 
> Thanks,
> Vincent (on behalf of the Apache Phoenix team)
> 
> [1] http://phoenix.apache.org/download.html 
>

Re: HBase Compaction Fails for Phoenix Table

2018-11-21 Thread William Shen 
Hi Jaanai,

We are running Phoenix 4.13, and actually upon investigating the issue
more, we are not so sure if it is that row that's having issue. It seems
like the data issue might be on a different row than the one HBase
compaction is complaining about because when we do a point look up, the
data comes back, but it does not return if it is a range scan (#1). Not
totally sure why HBase complained about that row when the failure is
somewhere else...

I don't have the specific Phoenix DDL handy, but here is describe in HBase
(#2), and also selective info from SYSTEMS.CATALOGS (#3). Hopefully that's
helpful to make sense of the data types / keys, etc. Is there anything
specific that you want to see?



*#1: Range scan vs point look up*

0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
"cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
"cstId" = 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4
and "accId" = 1904409 and "cpgnId" = 4811421;

+--+-+++

| PLAN
| EST_BYTES_READ  | EST_ROWS_READ  |
EST_INFO_TS   |

+--+-+++

| CLIENT 1-CHUNK 1 ROWS 8052 BYTES PARALLEL 1-WAY ROUND ROBIN POINT LOOKUP
ON 1 KEY OVER qa2.ADGROUPS  | 8052| 1  |
1542827075237  |

+--+-+++

1 row selected (0.048 seconds)

0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
"cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "cstId"
= 2053276 and "cltId" = 1867077 and "id" = 30496531 and "pubId" = 4 and
"accId" = 1904409 and "cpgnId" = 4811421;

+-+--+--++--+--+---+

|  stts   |  cstId   |  cltId   | pubId  |  accId   |  cpgnId  |id |

+-+--+--++--+--+---+

| ACTIVE  | 2053276  | 1867077  | 4  | 1904409  | 4811421  | 30496531  |

+-+--+--++--+--+---+

1 row selected (0.072 seconds)

0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> explain select "stts",
"cstId", "cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where
"id" = 30496531;

+--+-+++

| PLAN
| EST_BYTES_READ  |
EST_ROWS_READ  |  EST_INFO_TS   |

+--+-+++

| CLIENT 364-CHUNK 36227938 ROWS 34037631025 BYTES PARALLEL 10-WAY ROUND
ROBIN RANGE SCAN OVER qa2.ADGROUPS [0] - [9]  | 34037631025 | 36227938
| 1542823263107  |

| SERVER FILTER BY "id" = 30496531
| 34037631025 | 36227938
| 1542823263107  |

+--+-+++

2 rows selected (0.063 seconds)

0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select "stts", "cstId",
"cltId", "pubId", "accId","cpgnId", "id" from "qa2".adgroups where "id" =
30496531;

*[Hangs]*

#2

hbase(main):001:0> describe "qa2.ADGROUPS"

Table qa2.ADGROUPS is ENABLED



qa2.ADGROUPS, {TABLE_ATTRIBUTES => {coprocessor$1 =>
'|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
coprocessor$2 =>
'|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
coprocessor$

3 =>
'|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
coprocessor$4 =>
'|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
coprocessor$5 => '|org.apache.phoenix.hbase.index.Index

er|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'}


COLUMN FAMILIES DESCRIPTION



{NAME => 'AG', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY =>
'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'FAST_DIFF',
TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE =>
'true'

, BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}



1 row(s) in 0.4210 seconds

#3

0: jdbc:phoenix:qa2-zod-journalnode-lv-101,qa> select KEY_SEQ,
ORDINAL_POSITION, COLUMN_QUALIFIER, DATA

Re: HBase Compaction Fails for Phoenix Table

2018-11-21 Thread Jaanai Zhang 
Can you please attach Create the table/indexes SQL？ is this Phoneix's
version?  Are you sure many rows data had been corrupted or only this one
row?


   Jaanai Zhang
   Best regards!



William Shen  于2018年11月21日周三 上午10:20写道：

> Hi there,
>
> We've encountered the following compaction failure(#1) for a Phoenix
> table, and are not sure how to make sense of it. Using the HBase row key
> from the error, we are able to query data directly from hbase shell, and by
> examining the data, there arent anything immediately obvious about the data
> as they seem to be stored consistent with the Phoenix data type (#2). When
> querying Phoenix for the given row through sqline, the row would return if
> only primary key columns are selected, and the query would not return if
> non primary key columns are selected (#3).
>
> Few questions hoping to find some help on:
>
> a. Are we correct in understanding the error message to indicate an issue
> with data for the row key (
> \x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13)?
> We are not sure what to make sense of the string "
> 1539019716378.3dcf2b1e057915feb74395d9711ba4ad." that is included with
> the row key...
>
> b. What is out of bound here? It's not apparently clear here what
> StatisticsScanner and FastDiffDeltaEncoder are tripping over...
>
> c. Is it normal for hbase shell to return some parts of the hex string as
> ascii characters? We are seeing that in the row key as well as column name
> encoding and value. We are not sure if that is causing any issues, or if
> that was just a display issue that we can safely ignore.
>
> *#1 Compaction Failure*
>
> Compaction failed Request = 
> regionName=qa2.ADGROUPS,\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13,1539019716378.3dcf2b1e057915feb74395d9711ba4ad.,
>  storeName=AG, fileCount=4, fileSize=316.0 M (315.8 M, 188.7 K, 6.8 K, 14.2 
> K), priority=1, time=40613533856170784
> java.lang.IndexOutOfBoundsException
>   at java.nio.Buffer.checkBounds(Buffer.java:567)
>   at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
>   at 
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
>   at 
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
>   at 
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
>   at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:458)
>   at 
> org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:69)
>   at 
> org.apache.phoenix.schema.stats.StatisticsScanner.next(StatisticsScanner.java:76)
>   at 
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:334)
>   at 
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:106)
>   at 
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:131)
>   at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1245)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1852)
>   at 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:529)
>   at 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:566)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
>
>
> *#2 Data on the HBase level*
>
> hbase(main):002:0> get 'qa2.ADGROUPS', 
> "\x05\x80\x00\x00\x00\x00\x1FT\x9C\x80\x00\x00\x00\x00\x1C}E\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00Ij\x9D\x80\x00\x00\x00\x01\xD1W\x13"
> COLUMN  CELL
>  AG:\x00\x00\x00\x00timestamp=1539019457506, value=x
>  AG:\x80\x0Ftimestamp=1539019457506, 
> value=D:USA_AK:Nightmute:1456903:hotel
>  AG:\x80\x12timestamp=1539019457506, value=ACTIVE
>  AG:\x80\x13timestamp=1539019457506, value=ADD
>  AG:\x80#