Salting and time column

2015-09-29 Thread Sumit Nigam 
Hello,
I am planning on using to_number(current_time()) as my primary key along with 
salting enabled. 
However, multiple transactions can be upserted at the same current_time(). Will 
salting still be able to prevent overwriting one row with another? Or do I need 
to postfix another column into the primary key other than the current server 
time?
Thanks,Sumit

Re: [ANNOUNCE] New Apache Phoenix committer - Jan Fernando

2015-09-29 Thread Jan Fernando 
Thank you everyone! I am really excited to continue working on Phoenix!

Thanks again,
--Jan

On Tue, Sep 29, 2015 at 4:38 PM, rajeshb...@apache.org <
chrajeshbab...@gmail.com> wrote:

> Congratulations Jan!!!
>
> Thanks,
> Rajeshbabu.
> On Sep 30, 2015 4:37 AM, "James Taylor"  wrote:
>
>> Welcome, Jan. Great to have you onboard as a committer!
>>
>> James
>>
>> On Tuesday, September 29, 2015, Andrew Purtell 
>> wrote:
>>
>>> Congratulations Jan, and welcome!
>>>
>>>
>>> On Tue, Sep 29, 2015 at 11:23 AM, Eli Levine 
>>> wrote:
>>>
>>> > On behalf of the Apache Phoenix project I am happy to welcome Jan
>>> Fernando
>>> > as a committer. Jan has been an active user and contributor to Phoenix
>>> in
>>> > the last couple of years. Some of his major contributions are:
>>> > 1) Worked deeply in the sequence code including implementing Bulk
>>> Sequence
>>> > Allocation: PHOENIX-1954 and debugging and fixing several tricky
>>> Sequence
>>> > Bugs:
>>> > PHOENIX-2149, PHOENIX-1096.
>>> > 2) Implemented DROP TABLE...CASCADE to support tenant-specific views
>>> being
>>> > dropped: PHOENIX-1098.
>>> > 3) Worked closely with Cody and Mujtaba in the design of the interfaces
>>> > for Pherf and contributed patches to increase support for
>>> tenant-specific
>>> > use cases: PHOENIX-1791, PHOENIX-2227 Pioneered creating Pherf
>>> scenarios at
>>> > Salesforce.
>>> > 4) Worked closely with Samarth on requirements and API design and
>>> > validation for Phoenix global- and query-level metrics:
>>> > PHOENIX-1452, PHOENIX-1819 to get better visibility into Phoenix
>>> internals.
>>> >
>>> > Look forward to continuing working with Jan on Apache Phoenix!
>>> >
>>> > Thanks,
>>> >
>>> > Eli Levine
>>> > elilev...@apache.org
>>> >
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>>- Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
>>>
>>

Re: [ANNOUNCE] New Apache Phoenix committer - Jan Fernando

2015-09-29 Thread rajeshb...@apache.org 
Congratulations Jan!!!

Thanks,
Rajeshbabu.
On Sep 30, 2015 4:37 AM, "James Taylor"  wrote:

> Welcome, Jan. Great to have you onboard as a committer!
>
> James
>
> On Tuesday, September 29, 2015, Andrew Purtell 
> wrote:
>
>> Congratulations Jan, and welcome!
>>
>>
>> On Tue, Sep 29, 2015 at 11:23 AM, Eli Levine  wrote:
>>
>> > On behalf of the Apache Phoenix project I am happy to welcome Jan
>> Fernando
>> > as a committer. Jan has been an active user and contributor to Phoenix
>> in
>> > the last couple of years. Some of his major contributions are:
>> > 1) Worked deeply in the sequence code including implementing Bulk
>> Sequence
>> > Allocation: PHOENIX-1954 and debugging and fixing several tricky
>> Sequence
>> > Bugs:
>> > PHOENIX-2149, PHOENIX-1096.
>> > 2) Implemented DROP TABLE...CASCADE to support tenant-specific views
>> being
>> > dropped: PHOENIX-1098.
>> > 3) Worked closely with Cody and Mujtaba in the design of the interfaces
>> > for Pherf and contributed patches to increase support for
>> tenant-specific
>> > use cases: PHOENIX-1791, PHOENIX-2227 Pioneered creating Pherf
>> scenarios at
>> > Salesforce.
>> > 4) Worked closely with Samarth on requirements and API design and
>> > validation for Phoenix global- and query-level metrics:
>> > PHOENIX-1452, PHOENIX-1819 to get better visibility into Phoenix
>> internals.
>> >
>> > Look forward to continuing working with Jan on Apache Phoenix!
>> >
>> > Thanks,
>> >
>> > Eli Levine
>> > elilev...@apache.org
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>- Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>

Re: [ANNOUNCE] New Apache Phoenix committer - Jan Fernando

2015-09-29 Thread James Taylor 
Welcome, Jan. Great to have you onboard as a committer!

James

On Tuesday, September 29, 2015, Andrew Purtell  wrote:

> Congratulations Jan, and welcome!
>
>
> On Tue, Sep 29, 2015 at 11:23 AM, Eli Levine  > wrote:
>
> > On behalf of the Apache Phoenix project I am happy to welcome Jan
> Fernando
> > as a committer. Jan has been an active user and contributor to Phoenix in
> > the last couple of years. Some of his major contributions are:
> > 1) Worked deeply in the sequence code including implementing Bulk
> Sequence
> > Allocation: PHOENIX-1954 and debugging and fixing several tricky Sequence
> > Bugs:
> > PHOENIX-2149, PHOENIX-1096.
> > 2) Implemented DROP TABLE...CASCADE to support tenant-specific views
> being
> > dropped: PHOENIX-1098.
> > 3) Worked closely with Cody and Mujtaba in the design of the interfaces
> > for Pherf and contributed patches to increase support for tenant-specific
> > use cases: PHOENIX-1791, PHOENIX-2227 Pioneered creating Pherf scenarios
> at
> > Salesforce.
> > 4) Worked closely with Samarth on requirements and API design and
> > validation for Phoenix global- and query-level metrics:
> > PHOENIX-1452, PHOENIX-1819 to get better visibility into Phoenix
> internals.
> >
> > Look forward to continuing working with Jan on Apache Phoenix!
> >
> > Thanks,
> >
> > Eli Levine
> > elilev...@apache.org 
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: [ANNOUNCE] New Apache Phoenix committer - Jan Fernando

2015-09-29 Thread Andrew Purtell 
Congratulations Jan, and welcome!


On Tue, Sep 29, 2015 at 11:23 AM, Eli Levine  wrote:

> On behalf of the Apache Phoenix project I am happy to welcome Jan Fernando
> as a committer. Jan has been an active user and contributor to Phoenix in
> the last couple of years. Some of his major contributions are:
> 1) Worked deeply in the sequence code including implementing Bulk Sequence
> Allocation: PHOENIX-1954 and debugging and fixing several tricky Sequence
> Bugs:
> PHOENIX-2149, PHOENIX-1096.
> 2) Implemented DROP TABLE...CASCADE to support tenant-specific views being
> dropped: PHOENIX-1098.
> 3) Worked closely with Cody and Mujtaba in the design of the interfaces
> for Pherf and contributed patches to increase support for tenant-specific
> use cases: PHOENIX-1791, PHOENIX-2227 Pioneered creating Pherf scenarios at
> Salesforce.
> 4) Worked closely with Samarth on requirements and API design and
> validation for Phoenix global- and query-level metrics:
> PHOENIX-1452, PHOENIX-1819 to get better visibility into Phoenix internals.
>
> Look forward to continuing working with Jan on Apache Phoenix!
>
> Thanks,
>
> Eli Levine
> elilev...@apache.org
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: integration Phoenix and Spark

2015-09-29 Thread Josh Mahonin 
Make sure to double check your imports. Note the following from 
https://phoenix.apache.org/phoenix_spark.html


import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.apache.phoenix.spark._

There's also a sample repository here: 
https://github.com/jmahonin/spark-graphx-phoenix

From: Hardika Catur Sapta
Reply-To: "user@phoenix.apache.org"
Date: Tuesday, September 29, 2015 at 5:28 AM
To: "user@phoenix.apache.org"
Subject: Re: integration Phoenix and Spark

/spark/Project Spark$ scala SavingPhoenix.scala
/home/hduser/spark/Project Spark/SavingPhoenix.scala:1: error: object spark is 
not a member of package phoenix.org.apache
import phoenix.org.apache.spark.SparkContext
  ^
/home/hduser/spark/Project Spark/SavingPhoenix.scala:4: error: not found: type 
SparkContext
val sc = new SparkContext("local", "phoenix-test")
 ^
two errors found


2015-09-29 16:20 GMT+07:00 Konstantinos Kougios 
mailto:kostas.koug...@googlemail.com>>:
Hi,

Just to add that, at least for hadoop-2.7.1 and phoenix 4.5.2-HBase-1.1, hadoop 
guava lib has to be patched to 14.0.1 (under hadoop/share/hadoop/common/lib) 
otherwise spark tasks might fail due to missing guava methods.

Cheers


On 29/09/15 10:17, Hardika Catur Sapta wrote:
Spark setup

  1.  Ensure that all requisite Phoenix / HBase platform dependencies are 
available on the classpath for the Spark executors and drivers

  2.  One method is to add the phoenix-4.4.0-client.jar to ‘SPARK_CLASSPATH’ in 
spark-env.sh, or setting both ‘spark.executor.

  3.  To help your IDE, you may want to add the following ‘provided’ dependency


sorry for bad English.

intent to number 2 and 3 how ??

please explain step by step.


Thanks.

Re: Error with union all and a hint

2015-09-29 Thread Alicia Shu 
Great. Thanks.

Alicia

From: Gašper Metelko 
mailto:gasper.mete...@salviol.com>>
Reply-To: "user@phoenix.apache.org" 
mailto:user@phoenix.apache.org>>
Date: Tuesday, September 29, 2015 at 2:50 AM
To: "user@phoenix.apache.org" 
mailto:user@phoenix.apache.org>>
Subject: RE: Error with union all and a hint

I have created the jira 
PHOENIX-2295.

Running the query without the hint, executes the statment with a full table 
scan:

0: jdbc:phoenix:localhost> explain SELECT  col1, col2 FROM table1 WHERE 
col1='123';
+--+
|   PLAN   |
+--+
| CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER TABLE1 |
| SERVER FILTER BY COL1 = '123'|
+--+

Running the query wit the hint:

0: jdbc:phoenix:localhost> explain SELECT /*+ INDEX(table1 INDEX_TABLE1_COL) */ 
col1, col2 FROM table1 WHERE col1='123';
+--+
|   PLAN   |
+--+
| CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER TABLE1 |
| SKIP-SCAN-JOIN TABLE 0   |
| CLIENT 1-CHUNK PARALLEL 1-WAY RANGE SCAN OVER INDEX_TABLE1_COL 
['123'] |
| SERVER FILTER BY FIRST KEY ONLY |
| DYNAMIC SERVER FILTER BY "TABLE1.ID" IN ($72.$74) |
+--+

Regards, Gasper

From: Alicia Shu [mailto:a...@hortonworks.com]
Sent: Tuesday, September 29, 2015 9:24 AM
To: user@phoenix.apache.org
Subject: Re: Error with union all and a hint

By default if there is an index defined for a table, the index will be used if 
the plan with the index is better. Please file a Jira for the NPE.

Thanks,
Alicia

From: Gašper Metelko 
mailto:gasper.mete...@salviol.com>>
Reply-To: "user@phoenix.apache.org" 
mailto:user@phoenix.apache.org>>
Date: Friday, September 25, 2015 at 6:52 AM
To: "user@phoenix.apache.org" 
mailto:user@phoenix.apache.org>>
Subject: Error with union all and a hint

Hi

I have a problem with executing a UNION ALL statment with added hints. I need 
the hints to be able to use the index. For example:

explain SELECT /*+ INDEX(table1 INDEX_TABLE1_COL) */ col1, col2 FROM table1 
WHERE col1='123' UNION ALL SELECT /*+ INDEX(table2 INDEX_TABLE2_COL) */ col1, 
col2 FROM table2 WHERE col1='123';

I get an error:
java.lang.NullPointerException
at java.util.ArrayList.addAll(ArrayList.java:559)
at 
org.apache.phoenix.iterate.UnionResultIterators.(UnionResultIterators.java:51)
at org.apache.phoenix.execute.UnionPlan.iterator(UnionPlan.java:131)
at org.apache.phoenix.execute.UnionPlan.iterator(UnionPlan.java:127)
at 
org.apache.phoenix.execute.UnionPlan.getExplainPlan(UnionPlan.java:150)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableExplainStatement.compilePlan(PhoenixStatement.java:424)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableExplainStatement.compilePlan(PhoenixStatement.java:403)
at 
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:246)
at 
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
at 
org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
at sqlline.Commands.execute(Commands.java:822)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)

I am using phoenix-4.4.0-HBase-1.1, the tables and indexes are as follows. I 
will greatly appriciate any help.

Regards, Gasper

CREATE TABLE table1 (
  id BIGINT,
  col1 VARCHAR,
  col2 VARCHAR,
  CONSTRAINT pk PRIMARY KEY (id)) IMMUTABLE_ROWS=true;

CREATE TABLE table2 (
  id BIGINT,
  col1 VARCHAR,
  col2 VARCHAR,
  CONSTRAINT pk PRIMARY KEY (id)) IMMUTABLE_ROWS=true;

CREATE INDEX INDEX_TABLE1_COL ON table1 (col1);

CREATE INDEX INDEX_TABLE2_COL ON table2 (col1);

Re: [ANNOUNCE] New Apache Phoenix committer - Jan Fernando

2015-09-29 Thread Cody Marcel 
Sweet! Congrats Jan.

On Tue, Sep 29, 2015 at 11:23 AM, Eli Levine  wrote:

> On behalf of the Apache Phoenix project I am happy to welcome Jan Fernando
> as a committer. Jan has been an active user and contributor to Phoenix in
> the last couple of years. Some of his major contributions are:
> 1) Worked deeply in the sequence code including implementing Bulk Sequence
> Allocation: PHOENIX-1954 and debugging and fixing several tricky Sequence
> Bugs:
> PHOENIX-2149, PHOENIX-1096.
> 2) Implemented DROP TABLE...CASCADE to support tenant-specific views being
> dropped: PHOENIX-1098.
> 3) Worked closely with Cody and Mujtaba in the design of the interfaces
> for Pherf and contributed patches to increase support for tenant-specific
> use cases: PHOENIX-1791, PHOENIX-2227 Pioneered creating Pherf scenarios at
> Salesforce.
> 4) Worked closely with Samarth on requirements and API design and
> validation for Phoenix global- and query-level metrics:
> PHOENIX-1452, PHOENIX-1819 to get better visibility into Phoenix internals.
>
> Look forward to continuing working with Jan on Apache Phoenix!
>
> Thanks,
>
> Eli Levine
> elilev...@apache.org
>

Re: [ANNOUNCE] New Apache Phoenix committer - Jan Fernando

2015-09-29 Thread Thomas D'Silva 
Congrats Jan!

On Tue, Sep 29, 2015 at 11:23 AM, Eli Levine  wrote:
> On behalf of the Apache Phoenix project I am happy to welcome Jan Fernando
> as a committer. Jan has been an active user and contributor to Phoenix in
> the last couple of years. Some of his major contributions are:
> 1) Worked deeply in the sequence code including implementing Bulk Sequence
> Allocation: PHOENIX-1954 and debugging and fixing several tricky Sequence
> Bugs:
> PHOENIX-2149, PHOENIX-1096.
> 2) Implemented DROP TABLE...CASCADE to support tenant-specific views being
> dropped: PHOENIX-1098.
> 3) Worked closely with Cody and Mujtaba in the design of the interfaces for
> Pherf and contributed patches to increase support for tenant-specific use
> cases: PHOENIX-1791, PHOENIX-2227 Pioneered creating Pherf scenarios at
> Salesforce.
> 4) Worked closely with Samarth on requirements and API design and
> validation for Phoenix global- and query-level metrics:
> PHOENIX-1452, PHOENIX-1819 to get better visibility into Phoenix internals.
>
> Look forward to continuing working with Jan on Apache Phoenix!
>
> Thanks,
>
> Eli Levine
> elilev...@apache.org

[ANNOUNCE] New Apache Phoenix committer - Jan Fernando

2015-09-29 Thread Eli Levine 
On behalf of the Apache Phoenix project I am happy to welcome Jan Fernando
as a committer. Jan has been an active user and contributor to Phoenix in
the last couple of years. Some of his major contributions are:
1) Worked deeply in the sequence code including implementing Bulk Sequence
Allocation: PHOENIX-1954 and debugging and fixing several tricky Sequence
Bugs:
PHOENIX-2149, PHOENIX-1096.
2) Implemented DROP TABLE...CASCADE to support tenant-specific views being
dropped: PHOENIX-1098.
3) Worked closely with Cody and Mujtaba in the design of the interfaces for
Pherf and contributed patches to increase support for tenant-specific use
cases: PHOENIX-1791, PHOENIX-2227 Pioneered creating Pherf scenarios at
Salesforce.
4) Worked closely with Samarth on requirements and API design and
validation for Phoenix global- and query-level metrics:
PHOENIX-1452, PHOENIX-1819 to get better visibility into Phoenix internals.

Look forward to continuing working with Jan on Apache Phoenix!

Thanks,

Eli Levine
elilev...@apache.org

RE: Error with union all and a hint

2015-09-29 Thread Gašper Metelko 
I have created the jira 
PHOENIX-2295.

Running the query without the hint, executes the statment with a full table 
scan:

0: jdbc:phoenix:localhost> explain SELECT  col1, col2 FROM table1 WHERE 
col1='123';
+--+
|   PLAN   |
+--+
| CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER TABLE1 |
| SERVER FILTER BY COL1 = '123'|
+--+

Running the query wit the hint:

0: jdbc:phoenix:localhost> explain SELECT /*+ INDEX(table1 INDEX_TABLE1_COL) */ 
col1, col2 FROM table1 WHERE col1='123';
+--+
|   PLAN   |
+--+
| CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER TABLE1 |
| SKIP-SCAN-JOIN TABLE 0   |
| CLIENT 1-CHUNK PARALLEL 1-WAY RANGE SCAN OVER INDEX_TABLE1_COL 
['123'] |
| SERVER FILTER BY FIRST KEY ONLY |
| DYNAMIC SERVER FILTER BY "TABLE1.ID" IN ($72.$74) |
+--+

Regards, Gasper

From: Alicia Shu [mailto:a...@hortonworks.com]
Sent: Tuesday, September 29, 2015 9:24 AM
To: user@phoenix.apache.org
Subject: Re: Error with union all and a hint

By default if there is an index defined for a table, the index will be used if 
the plan with the index is better. Please file a Jira for the NPE.

Thanks,
Alicia

From: Gašper Metelko 
mailto:gasper.mete...@salviol.com>>
Reply-To: "user@phoenix.apache.org" 
mailto:user@phoenix.apache.org>>
Date: Friday, September 25, 2015 at 6:52 AM
To: "user@phoenix.apache.org" 
mailto:user@phoenix.apache.org>>
Subject: Error with union all and a hint

Hi

I have a problem with executing a UNION ALL statment with added hints. I need 
the hints to be able to use the index. For example:

explain SELECT /*+ INDEX(table1 INDEX_TABLE1_COL) */ col1, col2 FROM table1 
WHERE col1='123' UNION ALL SELECT /*+ INDEX(table2 INDEX_TABLE2_COL) */ col1, 
col2 FROM table2 WHERE col1='123';

I get an error:
java.lang.NullPointerException
at java.util.ArrayList.addAll(ArrayList.java:559)
at 
org.apache.phoenix.iterate.UnionResultIterators.(UnionResultIterators.java:51)
at org.apache.phoenix.execute.UnionPlan.iterator(UnionPlan.java:131)
at org.apache.phoenix.execute.UnionPlan.iterator(UnionPlan.java:127)
at 
org.apache.phoenix.execute.UnionPlan.getExplainPlan(UnionPlan.java:150)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableExplainStatement.compilePlan(PhoenixStatement.java:424)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableExplainStatement.compilePlan(PhoenixStatement.java:403)
at 
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:246)
at 
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
at 
org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
at sqlline.Commands.execute(Commands.java:822)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)

I am using phoenix-4.4.0-HBase-1.1, the tables and indexes are as follows. I 
will greatly appriciate any help.

Regards, Gasper

CREATE TABLE table1 (
  id BIGINT,
  col1 VARCHAR,
  col2 VARCHAR,
  CONSTRAINT pk PRIMARY KEY (id)) IMMUTABLE_ROWS=true;

CREATE TABLE table2 (
  id BIGINT,
  col1 VARCHAR,
  col2 VARCHAR,
  CONSTRAINT pk PRIMARY KEY (id)) IMMUTABLE_ROWS=true;

CREATE INDEX INDEX_TABLE1_COL ON table1 (col1);

CREATE INDEX INDEX_TABLE2_COL ON table2 (col1);

Re: integration Phoenix and Spark

2015-09-29 Thread Hardika Catur Sapta 
/spark/Project Spark$ scala SavingPhoenix.scala
/home/hduser/spark/Project Spark/SavingPhoenix.scala:1: error: object spark
is not a member of package phoenix.org.apache
import phoenix.org.apache.spark.SparkContext
  ^
/home/hduser/spark/Project Spark/SavingPhoenix.scala:4: error: not found:
type SparkContext
val sc = new SparkContext("local", "phoenix-test")
 ^
two errors found


2015-09-29 16:20 GMT+07:00 Konstantinos Kougios <
kostas.koug...@googlemail.com>:

> Hi,
>
> Just to add that, at least for hadoop-2.7.1 and phoenix 4.5.2-HBase-1.1,
> hadoop guava lib has to be patched to 14.0.1 (under
> hadoop/share/hadoop/common/lib) otherwise spark tasks might fail due to
> missing guava methods.
>
> Cheers
>
>
> On 29/09/15 10:17, Hardika Catur Sapta wrote:
>
> Spark setup
>
>1.
>
>Ensure that all requisite Phoenix / HBase platform dependencies are
>available on the classpath for the Spark executors and drivers
>2.
>
>One method is to add the phoenix-4.4.0-client.jar to ‘SPARK_CLASSPATH’
>in spark-env.sh, or setting both ‘spark.executor.
>3.
>
>To help your IDE, you may want to add the following ‘provided’
>dependency
>
>
> sorry for bad English.
>
> intent to number 2 and 3 how ??
>
> please explain step by step.
>
>
> Thanks.
>
>
>
>

Re: integration Phoenix and Spark

2015-09-29 Thread Konstantinos Kougios 

Hi,

Just to add that, at least for hadoop-2.7.1 and phoenix 4.5.2-HBase-1.1, 
hadoop guava lib has to be patched to 14.0.1 (under 
hadoop/share/hadoop/common/lib) otherwise spark tasks might fail due to 
missing guava methods.


Cheers

On 29/09/15 10:17, Hardika Catur Sapta wrote:



Spark setup


1.

Ensure that all requisite Phoenix / HBase platform
dependencies are available on the classpath for the Spark
executors and drivers

2.

One method is to add the phoenix-4.4.0-client.jar to
‘SPARK_CLASSPATH’ in spark-env.sh, or setting both
‘spark.executor.

3.

To help your IDE, you may want to add the following
‘provided’ dependency


sorry for bad English.

intent to number 2 and 3 how ??

please explain step by step.


Thanks.

a good hbase config to work with phoenix and very large tables

2015-09-29 Thread Konstantinos Kougios 
Hi, anyone wants to share a good config that will work for not-so-fast 
clusters (7200rpm disks) and very large tables?


I got a cluster of 3x region servers on lxc-virtual boxes on a 16-core 
machine with 3x7200rpm disks. Each region server runs on a virtual box 
using one of the disks and with Xmx4G.


I get various issues for tables that are above a few GB, i.e. I can't 
create indexes or do counts(*) and even regionserver crashes (due to 
outofmem) on aggregate queries & joins.


I believe my hardware to be adequate for a 500GB table or a 150GB one, I 
don't mind so much for slow responses but operations should eventually 
complete successfully.


Thanks,

Kostas

integration Phoenix and Spark

2015-09-29 Thread Hardika Catur Sapta 
Spark setup

   1.

   Ensure that all requisite Phoenix / HBase platform dependencies are
   available on the classpath for the Spark executors and drivers
   2.

   One method is to add the phoenix-4.4.0-client.jar to ‘SPARK_CLASSPATH’
   in spark-env.sh, or setting both ‘spark.executor.
   3.

   To help your IDE, you may want to add the following ‘provided’ dependency


sorry for bad English.

intent to number 2 and 3 how ??

please explain step by step.


Thanks.

issue when creating index for a large table

2015-09-29 Thread Konstantinos Kougios 
I got a table "words" which is 200GB on a not-so-fast cluster, 7200rpm 
disks.


0: jdbc:phoenix:nn.lan> create index IX_WORDS on words(word);
15/09/29 10:04:48 WARN client.AsyncProcess: #1, table=IX_WORDS, 
attempt=1/1 failed=80ops, last exception: null on 
d3.lan,16020,1443514600886, tracking started Tue Sep 29 10:04:48 BST 
2015; not retrying 80 - final failure
15/09/29 10:04:48 WARN client.AsyncProcess: #1, table=IX_WORDS, 
attempt=1/1 failed=142ops, last exception: null on 
d3.lan,16020,1443514600886, tracking started Tue Sep 29 10:04:48 BST 
2015; not retrying 142 - final failure
15/09/29 10:04:48 WARN client.AsyncProcess: #1, table=IX_WORDS, 
attempt=1/1 failed=141ops, last exception: null on 
d3.lan,16020,1443514600886, tracking started Tue Sep 29 10:04:48 BST 
2015; not retrying 141 - final failure
15/09/29 10:04:48 WARN client.AsyncProcess: #1, table=IX_WORDS, 
attempt=1/1 failed=61ops, last exception: null on 
d3.lan,16020,1443514600886, tracking started Tue Sep 29 10:04:48 BST 
2015; not retrying 61 - final failure


It eventually fails the creation of the index. Is there a config I 
should alter to fix this?




On the server log there is no exception:

2015-09-29 10:07:47,193 INFO  [MemStoreFlusher.1] 
regionserver.DefaultStoreFlusher: Flushed, sequenceid=846905, 
memsize=128.2 M, hasBloomFilter=true, into tmp file 
hdfs://nn.lan:8020/hbase/data/default/IX_WORDS/e4bcc5bcda8227204db81eced37b5546/.tmp/3fee1d78302146ccac035138a498a966
2015-09-29 10:07:47,204 INFO  [MemStoreFlusher.1] regionserver.HStore: 
Added 
hdfs://nn.lan:8020/hbase/data/default/IX_WORDS/e4bcc5bcda8227204db81eced37b5546/0/3fee1d78302146ccac035138a498a966, 
entries=786646, sequenceid=846905, filesize=13.4 M
2015-09-29 10:07:47,208 INFO  [MemStoreFlusher.1] regionserver.HRegion: 
Finished memstore flush of ~128.19 MB/134418456, currentsize=11.45 
MB/12001160 for region 
IX_WORDS,\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00,1443516250552.e4bcc5bcda8227204db81eced37b5546. 
in 1972ms, sequenceid=846905, compaction requested=true
2015-09-29 10:07:53,630 INFO  [sync.3] wal.FSHLog: Slow sync cost: 143 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:56,807 INFO  [sync.3] wal.FSHLog: Slow sync cost: 113 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:56,808 INFO  [sync.2] wal.FSHLog: Slow sync cost: 115 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:57,225 INFO  [sync.4] wal.FSHLog: Slow sync cost: 103 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:58,693 INFO  [sync.3] wal.FSHLog: Slow sync cost: 939 
ms, current pipeline: [192.168.0.31:50010]
*2015-09-29 10:07:58,693 INFO  [sync.4] wal.FSHLog: Slow sync cost: 939 
ms, current pipeline: [192.168.0.31:50010]**
*2015-09-29 10:07:58,693 INFO  [sync.0] wal.FSHLog: Slow sync cost: 938 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:58,693 INFO  [sync.2] wal.FSHLog: Slow sync cost: 940 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:58,694 INFO  [sync.1] wal.FSHLog: Slow sync cost: 938 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:07:59,484 INFO  [sync.4] wal.FSHLog: Slow sync cost: 147 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:00,116 INFO  [sync.3] wal.FSHLog: Slow sync cost: 131 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:00,117 INFO  [sync.2] wal.FSHLog: Slow sync cost: 133 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:00,654 INFO  [sync.2] wal.FSHLog: Slow sync cost: 127 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:01,128 INFO  [sync.0] wal.FSHLog: Slow sync cost: 117 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:01,128 INFO  [sync.1] wal.FSHLog: Slow sync cost: 117 
ms, current pipeline: [192.168.0.31:50010]
2015-09-29 10:08:02,988 INFO 
[regionserver/d3.lan/192.168.0.31:16020.logRoller] wal.FSHLog: Rolled 
WAL 
/hbase/WALs/d3.lan,16020,1443514600886/d3.lan%2C16020%2C1443514600886.default.1443517654445 
with entries=23584, filesize=121.87 MB; new WAL 
/hbase/WALs/d3.lan,16020,1443514600886/d3.lan%2C16020%2C1443514600886.default.1443517682939
2015-09-29 10:08:08,122 INFO  [MemStoreFlusher.1] regionserver.HRegion: 
Started memstore flush for 
IX_WORDS,\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00,1443516250552.e4bcc5bcda8227204db81eced37b5546., 
current region memstore size 128.00 MB, and 1/1 column families' 
memstores are being flushed.
2015-09-29 10:08:10,819 INFO  [MemStoreFlusher.1] 
regionserver.DefaultStoreFlusher: Flushed, sequenceid=859636, 
memsize=128.0 M, hasBloomFilter=true, into tmp file 
hdfs://nn.lan:8020/hbase/data/default/IX_WORDS/e4bcc5bcda8227204db81eced37b5546/.tmp/197a91b7e345448a9905576f0e66e411
2015-09-29 10:08:10,839 INFO  [MemStoreFlusher.1] regionserver.HStore: 
Added 
hdfs://nn.lan:8020/hbase/data/default/IX_WORDS/e4bcc5bcda8227204db81eced37b5546/0/197a91b7e345448a9905576f0e66e411, 
entries=785829, sequenceid=859636, filesize=13.4 M
2015-09-29 10:08:10,850 INFO  [MemStoreFlusher.1] regionserver.HR

client.ScannerCallable: Ignore, probably already closed, Name: 3085, already closed?

2015-09-29 Thread Konstantinos Kougios 
Hi, I am running a query that takes a long time, via sqlline.py, but I 
am repeatedly getting


15/09/29 08:54:27 WARN client.ScannerCallable: Ignore, probably already 
closed
org.apache.hadoop.hbase.UnknownScannerException: 
org.apache.hadoop.hbase.UnknownScannerException: Name: 3085, already closed?
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at 
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at 
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at 
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322)
at 
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:357)
at 
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:195)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:142)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
at 
org.apache.hadoop.hbase.client.StatsTrackingRpcRetryingCaller.callWithoutRetries(StatsTrackingRpcRetryingCaller.java:56)
at 
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at 
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:258)
at 
org.apache.hadoop.hbase.client.ClientScanner.possiblyNextScanner(ClientScanner.java:241)
at 
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:532)
at 
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
at 
org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:55)
at 
org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:107)
at 
org.apache.phoenix.iterate.SpoolingResultIterator.(SpoolingResultIterator.java:125)
at 
org.apache.phoenix.iterate.SpoolingResultIterator.(SpoolingResultIterator.java:83)
at 
org.apache.phoenix.iterate.SpoolingResultIterator.(SpoolingResultIterator.java:62)
at 
org.apache.phoenix.iterate.SpoolingResultIterator$SpoolingResultIteratorFactory.newIterator(SpoolingResultIterator.java:78)
at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:109)
at 
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): 
org.apache.hadoop.hbase.UnknownScannerException: *Name: 3085, already 
closed?*
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)

at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1196)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at 
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:355)

.

Re: Error with union all and a hint

2015-09-29 Thread Alicia Shu 
By default if there is an index defined for a table, the index will be used if 
the plan with the index is better. Please file a Jira for the NPE.

Thanks,
Alicia

From: Gašper Metelko 
mailto:gasper.mete...@salviol.com>>
Reply-To: "user@phoenix.apache.org" 
mailto:user@phoenix.apache.org>>
Date: Friday, September 25, 2015 at 6:52 AM
To: "user@phoenix.apache.org" 
mailto:user@phoenix.apache.org>>
Subject: Error with union all and a hint

Hi

I have a problem with executing a UNION ALL statment with added hints. I need 
the hints to be able to use the index. For example:

explain SELECT /*+ INDEX(table1 INDEX_TABLE1_COL) */ col1, col2 FROM table1 
WHERE col1='123' UNION ALL SELECT /*+ INDEX(table2 INDEX_TABLE2_COL) */ col1, 
col2 FROM table2 WHERE col1='123';

I get an error:
java.lang.NullPointerException
at java.util.ArrayList.addAll(ArrayList.java:559)
at 
org.apache.phoenix.iterate.UnionResultIterators.(UnionResultIterators.java:51)
at org.apache.phoenix.execute.UnionPlan.iterator(UnionPlan.java:131)
at org.apache.phoenix.execute.UnionPlan.iterator(UnionPlan.java:127)
at 
org.apache.phoenix.execute.UnionPlan.getExplainPlan(UnionPlan.java:150)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableExplainStatement.compilePlan(PhoenixStatement.java:424)
at 
org.apache.phoenix.jdbc.PhoenixStatement$ExecutableExplainStatement.compilePlan(PhoenixStatement.java:403)
at 
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:246)
at 
org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:241)
at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
at 
org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:240)
at 
org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1250)
at sqlline.Commands.execute(Commands.java:822)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)

I am using phoenix-4.4.0-HBase-1.1, the tables and indexes are as follows. I 
will greatly appriciate any help.

Regards, Gasper

CREATE TABLE table1 (
  id BIGINT,
  col1 VARCHAR,
  col2 VARCHAR,
  CONSTRAINT pk PRIMARY KEY (id)) IMMUTABLE_ROWS=true;

CREATE TABLE table2 (
  id BIGINT,
  col1 VARCHAR,
  col2 VARCHAR,
  CONSTRAINT pk PRIMARY KEY (id)) IMMUTABLE_ROWS=true;

CREATE INDEX INDEX_TABLE1_COL ON table1 (col1);

CREATE INDEX INDEX_TABLE2_COL ON table2 (col1);

Phoenix vs. Kudu performance

2015-09-29 Thread ALEX K 
interesting comparison of Impala/Kudu vs. Hbase/Phoenix (section 6.3):
http://getkudu.io/kudu.pdf