[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-03-01 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917102#comment-13917102
 ] 

Hive QA commented on HIVE-6356:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12631814/HIVE-6356.4.patch.txt

{color:green}SUCCESS:{color} +1 5185 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1566/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1566/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12631814

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.4.patch.txt, HIVE-6356.4.patch.txt, 
> HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-28 Thread Xuefu Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13916231#comment-13916231
 ] 

Xuefu Zhang commented on HIVE-6356:
---

The above test seems transient. I manually re-ran it and it passed. Reattach 
the same patch (v4) for another run.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.4.patch.txt, HIVE-6356.4.patch.txt, 
> HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-28 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915598#comment-13915598
 ] 

Hive QA commented on HIVE-6356:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12631340/HIVE-6356.4.patch.txt

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5180 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.hbase.snapshot.lock.TestWriteLock.testRun
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1551/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1551/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12631340

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.4.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-26 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914141#comment-13914141
 ] 

Navis commented on HIVE-6356:
-

+1

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.4.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-26 Thread Xuefu Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913427#comment-13913427
 ] 

Xuefu Zhang commented on HIVE-6356:
---

Thanks for the clarification, [~ashutoshc]. I will update the patch v3 with the 
necessary comments.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-26 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913347#comment-13913347
 ] 

Ashutosh Chauhan commented on HIVE-6356:


I think v3 is required in addition to v2. I am fine with committing v3 (with a 
comment in the code that problem is fixed in 0.96.1 and later, so those changes 
will not be required once Hive bumps up its hbase version).

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-26 Thread Xuefu Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913211#comment-13913211
 ] 

Xuefu Zhang commented on HIVE-6356:
---

If we all agree that patch v3 is better, should we revert the following commit 
(which is patch v2) and commit patch v3 instead?

commit 57467b4fd8cc5acb5250b635f86c3115691cf367
Author: Ashutosh Chauhan 
Date:   Tue Feb 4 23:42:30 2014 +

HIVE-6356 : Dependency injection in hbase storage handler is broken (Navis v


> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-25 Thread Xuefu Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912002#comment-13912002
 ] 

Xuefu Zhang commented on HIVE-6356:
---

[~ashutoshc] Could we put a close on this? I understood that patch v3 is 
committed to trunk. Do we still need addendum patch to be committed, in order 
to close this JIRA?

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-18 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904960#comment-13904960
 ] 

Nick Dimiduk commented on HIVE-6356:


Yes, of course; I now see the issue. My sincerest apologies for belaboring an 
invalid point :(

In the interest of supporting as many users as possible, patch v3 seems the 
most prudent approach, with a note about deferring back to HBase methods once 
Hive moves on to 0.98.0.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-18 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904913#comment-13904913
 ] 

Navis commented on HIVE-6356:
-

In hbase-0.96.0, mapred.TableMapReduceUtil#addDependencyJars(JobConf) does not 
call 
"org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addHBaseDependencyJars(job)"
 which is not exist.

{code}
public static void addDependencyJars(JobConf job) throws IOException {
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(
  job,
  org.apache.zookeeper.ZooKeeper.class,
  org.jboss.netty.channel.ChannelFactory.class,
  com.google.common.base.Function.class,
  com.google.protobuf.Message.class,
  job.getMapOutputKeyClass(),
  job.getMapOutputValueClass(),
  job.getOutputKeyClass(),
  job.getOutputValueClass(),
  job.getPartitionerClass(),
  job.getClass("mapred.input.format.class", TextInputFormat.class, 
InputFormat.class),
  job.getClass("mapred.output.format.class", TextOutputFormat.class, 
OutputFormat.class),
  job.getCombinerClass());
  }
{code}
Again, this only 
bq. adds zookeeper, netty, guava, hadoop, hive-exec, hive-hbase-handler to 
tmpJars, missing hbase related classes.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-18 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904910#comment-13904910
 ] 

Navis commented on HIVE-6356:
-

In hbase-0.96.0, mapred.TableMapReduceUtil#addDependencyJars(JobConf) does not 
call 
"org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addHBaseDependencyJars(job)"
 which is not exist.

{code}
public static void addDependencyJars(JobConf job) throws IOException {
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(
  job,
  org.apache.zookeeper.ZooKeeper.class,
  org.jboss.netty.channel.ChannelFactory.class,
  com.google.common.base.Function.class,
  com.google.protobuf.Message.class,
  job.getMapOutputKeyClass(),
  job.getMapOutputValueClass(),
  job.getOutputKeyClass(),
  job.getOutputValueClass(),
  job.getPartitionerClass(),
  job.getClass("mapred.input.format.class", TextInputFormat.class, 
InputFormat.class),
  job.getClass("mapred.output.format.class", TextOutputFormat.class, 
OutputFormat.class),
  job.getCombinerClass());
  }
{code}
Again, this only 
bq. adds zookeeper, netty, guava, hadoop, hive-exec, hive-hbase-handler to 
tmpJars, missing hbase related classes.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-18 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904894#comment-13904894
 ] 

Nick Dimiduk commented on HIVE-6356:


bq. all we got is JobConf, not Job

So why not call mapred.TableMapReduceUtil#addDependencyJars(JobConf) ? It does 
what you need; that's why it's there.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-18 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904874#comment-13904874
 ] 

Navis commented on HIVE-6356:
-

bq. Even on 0.96.0, both mapred.TableMapReduceUtil#addDependencyJars(JobConf) 
and mapreduce.TableMapReduceUtil#addDependencyJars(Job) handle the hbase 
dependencies for you

Really wish to call the method but all we got is JobConf, not Job and there is 
no way just to wrap existing JobConf(copies in it) with Job. So in patch3, I've 
create dummy Job instance to call it and copied values(in "tmpJars") in it to 
original JobConf. I've mentioned this once already and even quoted again as you 
requested explanation. 

bq. o.a.h.hbase.mapreduce.TableMapReduceUtil takes Job as parameter rather than 
JobConf
bq. "..or make new Job instance and copy values from it to original JobConf". 
You can see it in patch3.

I'm ok with fixing this in anyway (upgrade hbase to 0.96.1 or applying patch3), 
though I prefer the later way to support hbase-0.96.0, which is still in use by 
many companies.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-18 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904678#comment-13904678
 ] 

Ashutosh Chauhan commented on HIVE-6356:


I am also in favor of bumping HBase version. We already have code complexity 
supporting multiple versions of Hadoop, we don't worsen that by supporting 
multiple HBase version.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-18 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904676#comment-13904676
 ] 

Nick Dimiduk commented on HIVE-6356:


Even on 0.96.0, both 
[mapred.TableMapReduceUtil#addDependencyJars(JobConf)|https://github.com/apache/hbase/blob/hbase-0.96.0/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java#L258-L276]
 and 
[mapreduce.TableMapReduceUtil#addDependencyJars(Job)|https://github.com/apache/hbase/blob/hbase-0.96.0/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java#L514-L544]
 handle the hbase dependencies for you. I'm back to not understanding what 
issue you're having :)

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-18 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13904670#comment-13904670
 ] 

Nick Dimiduk commented on HIVE-6356:


With my HBase hat on, I'd say the upgrade to 0.96.1 us prudent for both the API 
improvements and the performance improvements. Not sure how Hive folks feel 
about this.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-13 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901173#comment-13901173
 ] 

Navis commented on HIVE-6356:
-

Ah, you are using hbase-0.96.1, which is applied HBASE-9165. I got it. Hive in 
trunk uses hbase-0.96.0 (addHBaseDependencyJars() does not exist in it).

Should we upgrade hbase to 0.96.1? I prefer to support both versions by 
applying patch3 but I'm good with any way.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-13 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13900695#comment-13900695
 ] 

Nick Dimiduk commented on HIVE-6356:


Which version of HBase are you testing against? 
[mapred.TableMapReduceUtil#addDependencyJars(JobConf)|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java#L298-L316]
 calls the [same 
method|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java#L586-L612]
 from the mapreduce package, which in turn adds all the hbase jars, plus the 
hbase dependencies you mention.

ClassNotFoundException thrown at the job client or in the job? At the client 
means your local classpath is missing something. In the job means it wasn't 
packaged properly. What's the missing class/jar?

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-12 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899984#comment-13899984
 ] 

Navis commented on HIVE-6356:
-

bq. Is the job failing due to missing dependencies
Yes, it fails with ClassNotFoundException. 
{code}
org.apache.hadoop.hbase.mapred.TableMapReduceUtil.addDependencyJars(jobConf)
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(jobConf, 
HBaseStorageHandler.class)
{code}
This adds zookeeper, netty, guava, hadoop, hive-exec, hive-hbase-handler to 
tmpJars, missing hbase related classes. 

bq. I think copying really is that bad
Agree. Just wanted quick, easy fix.

bq. What do you mean "second method"?
In my comment above, "Anyway, we can resolve this with patch1 (copy needed 
class list from TableMapReduceUtil) or make new Job instance and copy values 
from it to original JobConf". You can see it in patch3.


> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-12 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899906#comment-13899906
 ] 

Nick Dimiduk commented on HIVE-6356:


Patch 3 is not the addendum. My mistake, I thought patch2 had been committed.

bq. it's not fixing this problem

What exactly is the problem it's not fixing? Is the job failing due to missing 
dependencies? Which dependencies? Are they on the classpath of the job client?

I think copying really is that bad; it means this integration will break every 
time HBase changes it's dependencies. Not often, but it does happen. Instead, 
use the provided method and the correct dependencies will be added at runtime, 
everytime. Is this behavior undesirable for some reason?

What do you mean "second method"?

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.3.patch.txt, HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-12 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899842#comment-13899842
 ] 

Navis commented on HIVE-6356:
-

addendum is basically same with already applied patch(patch2) and it's not 
fixing this problem. If copying is that bad, I should use second method to fix 
this.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-12 Thread Sergey Shelukhin (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13899520#comment-13899520
 ] 

Sergey Shelukhin commented on HIVE-6356:


addendum looks good to me

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt, 
> HIVE-6356.addendum.00.patch
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-05 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13892899#comment-13892899
 ] 

Navis commented on HIVE-6356:
-

Sadly, patch2 cannot resolve this issue. I should I have done some tests before 
updating the patch (the problem looked so obvious, sorry). 
o.a.h.hbase.mapred.TableMapReduceUtil#addDependencyJars(JobConf) is only works 
with older version of hbase.

Now back to the problem, o.a.h.hbase.mapreduce.TableMapReduceUtil takes Job as 
parameter rather than JobConf, which seemed not good idea thinking that it's 
mere a wrapper of JobConf. Anyway, we can resolve this with patch1 (copy needed 
class list from TableMapReduceUtil) or make new Job instance and copy values 
from it to original JobConf. Which one is better?

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-04 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13891352#comment-13891352
 ] 

Ashutosh Chauhan commented on HIVE-6356:


+1

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-04 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890967#comment-13890967
 ] 

Nick Dimiduk commented on HIVE-6356:


Patch 2 looks better. You should be able to do without explicitly requesting 
org.apache.hadoop.hbase.HBaseConfiguration.class as well; this should be 
included by the underlying HBase method.

+1 (for what it's worth)

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-04 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890496#comment-13890496
 ] 

Hive QA commented on HIVE-6356:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12626785/HIVE-6356.2.patch.txt

{color:green}SUCCESS:{color} +1 4997 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1172/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1172/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12626785

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6356.1.patch.txt, HIVE-6356.2.patch.txt
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-03 Thread Navis (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890156#comment-13890156
 ] 

Navis commented on HIVE-6356:
-

Right. I've forgot there are two version of TableMapReduceUtil. HIVE-3603 
changed import clause of TableMapReduceUtil to mapred, which cause this 
problem. I'll fix this shortly after.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6356.1.patch.txt
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-03 Thread Nick Dimiduk (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889708#comment-13889708
 ] 

Nick Dimiduk commented on HIVE-6356:


I stumbled into this recently as well. HTrace is now a required runtime 
dependency, even when it's not used. This patch is incorrect, however. Because 
Hive is using the mapred namespace classes, the correct API is to invoke 
o.a.h.hbase.mapred.TableMapReduceUtil#addDependencyJars(JobConf). This will 
wire in all of HBase's runtime dependencies for you, and also attempt to 
auto-detect additional dependencies based on the JobConf (output classes, 
partitioners, formats, etc). If you want more fine-grained control over these 
dependencies (as Pig did, see PIG-3285), there are additional static methods in 
the o.a.h.hbase.mapreduce.TableMapReduceUtil class.

For Hive's purpose, I think you'll be fine with just calling 
mapred.TableMapReduceUtil#addDependencyJars(JobConf). Having a smoke test that 
runs in pseudo-distributed mode would be helpful in verifying all requirements 
are met.

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6356.1.patch.txt
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-03 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889639#comment-13889639
 ] 

Ashutosh Chauhan commented on HIVE-6356:


Is htrace strictly required ? If so, than don't we need to make sure its jar is 
available at run-time (currently it seems we don't have it in our lib/ dir of 
package?) [~ndimiduk] Can you also take a look?

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6356.1.patch.txt
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HIVE-6356) Dependency injection in hbase storage handler is broken

2014-02-03 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889605#comment-13889605
 ] 

Hive QA commented on HIVE-6356:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12626614/HIVE-6356.1.patch.txt

{color:green}SUCCESS:{color} +1 4997 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1165/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1165/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12626614

> Dependency injection in hbase storage handler is broken
> ---
>
> Key: HIVE-6356
> URL: https://issues.apache.org/jira/browse/HIVE-6356
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-6356.1.patch.txt
>
>
> Dependent jars for hbase is not added to tmpjars, which is caused by the 
> change of method signature(TableMapReduceUtil.addDependencyJars).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)