Re: Re: Performance options for doing Phoenix full table scans to complete some data statistics and summary collection work

2015-01-08 Thread su...@certusnet.com.cn
Hi,guys Thanks for all of your kindly advice. For #1, we are planning to retry that. Mujtaba, compression set to snappy already. Actually we are using only one column family name and we are good to utilize multiple column families. The table schema is high-narrow. For example, our table use one d

Re: Performance options for doing Phoenix full table scans to complete some data statistics and summary collection work

2015-01-08 Thread Mujtaba Chohan
With 100+ columns, using multiple column families will help a lot if your full scan uses only few columns. Also if columns are wide then turning on compression would help if you are seeing disk I/O contention on region servers. On Wednesday, January 7, 2015, James Taylor wrote: > Hi Sun, > Can

Re: Numbers low-level format in Phoenix

2015-01-08 Thread Gabriel Reid
Hi Lavrenty, Thanks for posting that hbase shell session info. You're currently inserting strings like "[B@1e0f477f" into HBase. It's actually not possible to insert binary (byte array) values into HBase using the HBase shell like you're doing. Internally, the HBase shell calls the equivalent to

Re: Re: Fwd: Phoenix in production

2015-01-08 Thread Josh Mahonin
On Wed, Jan 7, 2015 at 1:43 PM, anil gupta wrote: > Yup, I am aware of Spark HBase integration. Phoenix-Spark integration > would be more sweet. :) Hi Anil, I'm using Spark and Phoenix in production fairly successfully. There's very little required for integration, since Phoenix has Hadoop Inp

RE: Numbers low-level format in Phoenix

2015-01-08 Thread Lavrenty Eskin
There are full table scan from hbase shell. Table has been created from phoenix. Two first values (UL=Unsigned Long) has been upserted from phoenix via jdbc driver (each upsert generate two KV pairs). Another KV pairs has been inserted into the same table via hbase shell the way like this: put

Re: Numbers low-level format in Phoenix

2015-01-08 Thread Gabriel Reid
It sounds like you might be storing the toString() representation of a byte array of HBase. Could you post an example snippet of the code you're using to store things in HBase, as well as a snippet of how you're reading this data in the HBase shell (or wherever you're reading it). On Thu, Jan 8,

RE: Numbers low-level format in Phoenix

2015-01-08 Thread Lavrenty Eskin
Hi Gabriel, But why then I receive in HBase shell two different string representation of the byte array? For byte arrays stored from phoenix - \x00\x00\x00\x00I\x96\x02\xD2 and [B@13217cf6 for stored from HBase. The same time phoenix have wrong understanding of "[B@13217cf6" and receives -32383

Re: Numbers low-level format in Phoenix

2015-01-08 Thread Gabriel Reid
Hi Lavrenty, Phoenix actually does store numerical data using byte arrays, in a similar fashion to what the HBase bytes class does. There's more information on the various types and their underlying encoding available here: http://phoenix.apache.org/language/datatypes.html I'm guessing you got th

Numbers low-level format in Phoenix

2015-01-08 Thread Lavrenty Eskin
Helo all, I'm surprised that phoenix store numbers not in HBase 'Byte' format. Looks like a big overhead there, isn't it? Just takes 1234567890 value (0х499602D2): Phoenix stores that as string '\x00\x00\x00\x00I\x96\x02\xD2' But why it cannot store as in HBase format value=[B@499602d2 ? Another