[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714652#comment-13714652
]
Hive QA commented on HIVE-4502:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593370/HIVE-4502.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 2647 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/117/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/117/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch,
smb_mapjoin_25.q, smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4502: --- Status: Open (was: Patch Available) NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[ANNOUNCE] New Hive Committer - Gunther Hagleitner
The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congratulations, Gunther. Well deserved! On Jul 21, 2013, at 10:00 AM, Carl Steinbach c...@apache.org wrote: The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
[jira] [Commented] (HIVE-3739) Hive auto convert join result error: java.lang.InstantiationException: org.antlr.runtime.CommonToken
[ https://issues.apache.org/jira/browse/HIVE-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714703#comment-13714703 ] fantasy commented on HIVE-3739: --- Thanks for all your help, I use a raw Hive0.9.0 to run HiveSQL with JDK 7.0, maybe is the problem cause by the version of JDK. Now I don't work on hive anymore. therefore, I can provide more information about the error, neither reproduce the error. Next time I work on hive and cause same problem, I will try to provide more details. Hive auto convert join result error: java.lang.InstantiationException: org.antlr.runtime.CommonToken Key: HIVE-3739 URL: https://issues.apache.org/jira/browse/HIVE-3739 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.9.0 Environment: hive.auto.convert.join=true Reporter: fantasy After I set hive.auto.convert.join=true. Any HiveQL with a join executed in hive result a error as this: - java.lang.InstantiationException: org.antlr.runtime.CommonToken Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... java.lang.InstantiationException: org.antlr.runtime.CommonToken Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... java.lang.InstantiationException: org.antlr.runtime.CommonToken Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... java.lang.InstantiationException: org.antlr.runtime.CommonToken Continuing ... java.lang.RuntimeException: failed to evaluate: unbound=Class.new(); Continuing ... --- can anyone tell why? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congratulations, Gunther! --Xuefu On Sun, Jul 21, 2013 at 4:09 AM, Alexander Alten-Lorenz wget.n...@gmail.com wrote: Congratulations, Gunther. Well deserved! On Jul 21, 2013, at 10:00 AM, Carl Steinbach c...@apache.org wrote: The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congratulations Gunther! Jarcec On Sun, Jul 21, 2013 at 01:00:31AM -0700, Carl Steinbach wrote: The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl signature.asc Description: Digital signature
[jira] [Updated] (HIVE-4876) Beeling help text do not contain -f and -e parameters
[
https://issues.apache.org/jira/browse/HIVE-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Jarcec Cecho updated HIVE-4876:
-
Attachment: HIVE-4876.patch
Beeling help text do not contain -f and -e parameters
-
Key: HIVE-4876
URL: https://issues.apache.org/jira/browse/HIVE-4876
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Priority: Trivial
Attachments: bugHIVE-4876.patch, HIVE-4876.patch
I've noticed that beeline do have support for {{-e}} and {{-f}} parameters,
however those are not documented in the help text that can be retrieved by
calling beeline with parameter {{-h}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4876) Beeling help text do not contain -f and -e parameters
[
https://issues.apache.org/jira/browse/HIVE-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714719#comment-13714719
]
Jarek Jarcec Cecho commented on HIVE-4876:
--
[~brocknoland], attached, thank you for your note!
Beeling help text do not contain -f and -e parameters
-
Key: HIVE-4876
URL: https://issues.apache.org/jira/browse/HIVE-4876
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Priority: Trivial
Attachments: bugHIVE-4876.patch, HIVE-4876.patch
I've noticed that beeline do have support for {{-e}} and {{-f}} parameters,
however those are not documented in the help text that can be retrieved by
calling beeline with parameter {{-h}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714725#comment-13714725 ] Xuefu Zhang commented on HIVE-4900: --- I'm not sure the test error is transient or having something to do with my change, given that fact that error happens at test cleanup time. Anyone has any thought? Thanks. Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714726#comment-13714726 ] Brock Noland commented on HIVE-4900: Pretty sure it's flaky. I have a similar test over in HIVE-4851. You could upload the patch again for a second run. Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4851) Fix flaky tests
[ https://issues.apache.org/jira/browse/HIVE-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4851: --- Description: I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestHCatLoader.testReadPartitionedBasic was: I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite Fix flaky tests --- Key: HIVE-4851 URL: https://issues.apache.org/jira/browse/HIVE-4851 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestHCatLoader.testReadPartitionedBasic -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4851) Fix flaky tests
[ https://issues.apache.org/jira/browse/HIVE-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4851: --- Description: I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat was: I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestHCatLoader.testReadPartitionedBasic Fix flaky tests --- Key: HIVE-4851 URL: https://issues.apache.org/jira/browse/HIVE-4851 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland I see the following tests fail quite often: * TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 * TestOrcHCatLoader.testReadDataBasic * TestMinimrCliDriver.testCliDriver_bucketmpjoin6 * TestNotificationListener.testAMQListener This one is less often, but still fails randomly: * TestMinimrCliDriver.testCliDriver_bucket4 * TestHCatHiveCompatibility.testUnpartedReadWrite * TestHCatLoader.testReadPartitionedBasic * TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714736#comment-13714736
]
Edward Capriolo commented on HIVE-4827:
---
Because two map reduce jobs are becoming a single one now does this mean that
there is a greater chance of the map task failing with oom conditions?
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congratulations Gunther! On Sun, Jul 21, 2013 at 7:55 AM, Jarek Jarcec Cecho jar...@apache.orgwrote: Congratulations Gunther! Jarcec On Sun, Jul 21, 2013 at 01:00:31AM -0700, Carl Steinbach wrote: The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl
[jira] [Commented] (HIVE-305) Port Hadoop streaming's counters/status reporters to Hive Transforms
[ https://issues.apache.org/jira/browse/HIVE-305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714741#comment-13714741 ] Brock Noland commented on HIVE-305: --- The latest patch HIVE-305.patch.txt looks good and that test which failed is flaky. The only question I have is, shouldn't the configuration variables start with hive? as opposed to stream? Port Hadoop streaming's counters/status reporters to Hive Transforms Key: HIVE-305 URL: https://issues.apache.org/jira/browse/HIVE-305 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Venky Iyer Assignee: Guo Hongjie Attachments: HIVE-305.1.patch, HIVE-305.2.patch, hive-305.3.diff.txt, HIVE-305.patch.txt https://issues.apache.org/jira/browse/HADOOP-1328 Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use reporter:counter:group,counter,amount to update a counter. Use reporter:status:message to update status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congratulations Gunther!! On Sun, Jul 21, 2013 at 11:51 AM, Yin Huai huaiyin@gmail.com wrote: Congratulations Gunther! On Sun, Jul 21, 2013 at 7:55 AM, Jarek Jarcec Cecho jar...@apache.org wrote: Congratulations Gunther! Jarcec On Sun, Jul 21, 2013 at 01:00:31AM -0700, Carl Steinbach wrote: The Apache Hive PMC has voted to make Gunther Hagleitner a committer on the Apache Hive project. Congratulations Gunther! Carl -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congratulations Gunther! 2013/7/22 Brock Noland br...@cloudera.com Congratulations Gunther!! Cheers, Zhuoluo (Clark) Yang
[jira] [Commented] (HIVE-305) Port Hadoop streaming's counters/status reporters to Hive Transforms
[
https://issues.apache.org/jira/browse/HIVE-305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714748#comment-13714748
]
Edward Capriolo commented on HIVE-305:
--
In this patch we are using the same variable as the upstream hadoop.
{code}
public MRErrorThread() {
+ this.reporterPrefix = job_.get(stream.stderr.reporter.prefix,
reporter:);
{code}
I am ok with this because the meaning is the same, and duplicating hadoop
variables is something we are trying to avoid. (like the ticket on deprecating
hive's speculative execution settings.
Port Hadoop streaming's counters/status reporters to Hive Transforms
Key: HIVE-305
URL: https://issues.apache.org/jira/browse/HIVE-305
Project: Hive
Issue Type: New Feature
Components: Query Processor
Reporter: Venky Iyer
Assignee: Guo Hongjie
Attachments: HIVE-305.1.patch, HIVE-305.2.patch, hive-305.3.diff.txt,
HIVE-305.patch.txt
https://issues.apache.org/jira/browse/HADOOP-1328
Introduced a way for a streaming process to update global counters and
status using stderr stream to emit information. Use
reporter:counter:group,counter,amount to update a counter. Use
reporter:status:message to update status.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-305) Port Hadoop streaming's counters/status reporters to Hive Transforms
[ https://issues.apache.org/jira/browse/HIVE-305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714749#comment-13714749 ] Brock Noland commented on HIVE-305: --- Gotcha. Sounds good :) Port Hadoop streaming's counters/status reporters to Hive Transforms Key: HIVE-305 URL: https://issues.apache.org/jira/browse/HIVE-305 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Venky Iyer Assignee: Guo Hongjie Attachments: HIVE-305.1.patch, HIVE-305.2.patch, hive-305.3.diff.txt, HIVE-305.patch.txt https://issues.apache.org/jira/browse/HADOOP-1328 Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use reporter:counter:group,counter,amount to update a counter. Use reporter:status:message to update status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congrats Gunther ! Great to see more bandwidth to get the patch available counts down ! On Jul 21, 2013 9:56 AM, Clark Yang (杨卓荦) yangzhuo...@gmail.com wrote: Congratulations Gunther! 2013/7/22 Brock Noland br...@cloudera.com Congratulations Gunther!! Cheers, Zhuoluo (Clark) Yang
[jira] [Updated] (HIVE-4876) Beeling help text do not contain -f and -e parameters
[
https://issues.apache.org/jira/browse/HIVE-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-4876:
---
Status: Patch Available (was: Open)
Marking Patch Available so the precommit tests will run.
Beeling help text do not contain -f and -e parameters
-
Key: HIVE-4876
URL: https://issues.apache.org/jira/browse/HIVE-4876
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Priority: Trivial
Attachments: bugHIVE-4876.patch, HIVE-4876.patch
I've noticed that beeline do have support for {{-e}} and {{-f}} parameters,
however those are not documented in the help text that can be retrieved by
calling beeline with parameter {{-h}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714754#comment-13714754 ] Edward Capriolo edited comment on HIVE-4900 at 7/21/13 5:32 PM: I have hive 10.0 My tables contains a table named SORT_COLS was (Author: appodictic): I have hive 10.0 My tables contains a table named Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714754#comment-13714754 ] Edward Capriolo commented on HIVE-4900: --- I have hive 10.0 My tables contains a table named Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714754#comment-13714754 ] Edward Capriolo edited comment on HIVE-4900 at 7/21/13 5:34 PM: I have hive 10.0 My tables contains a table named SORT_COLS not SORT_ORDER(but I dont have any tables sorted by columns so we do not use that feature) was (Author: appodictic): I have hive 10.0 My tables contains a table named SORT_COLS Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714754#comment-13714754 ] Edward Capriolo edited comment on HIVE-4900 at 7/21/13 5:35 PM: I have hive 10.0 My mysql database contains a table named SORT_COLS not SORT_ORDER(but I dont have any tables sorted by columns so we do not use that feature). was (Author: appodictic): I have hive 10.0 My tables contains a table named SORT_COLS not SORT_ORDER(but I dont have any tables sorted by columns so we do not use that feature) Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-305) Port Hadoop streaming's counters/status reporters to Hive Transforms
[ https://issues.apache.org/jira/browse/HIVE-305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714756#comment-13714756 ] Edward Capriolo commented on HIVE-305: -- If someone does the commit please make sure Guo Hongjie is listed as he did most of the work as the assignee. Port Hadoop streaming's counters/status reporters to Hive Transforms Key: HIVE-305 URL: https://issues.apache.org/jira/browse/HIVE-305 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Venky Iyer Assignee: Guo Hongjie Attachments: HIVE-305.1.patch, HIVE-305.2.patch, hive-305.3.diff.txt, HIVE-305.patch.txt https://issues.apache.org/jira/browse/HADOOP-1328 Introduced a way for a streaming process to update global counters and status using stderr stream to emit information. Use reporter:counter:group,counter,amount to update a counter. Use reporter:status:message to update status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2905) Desc table can't show non-ascii comments
[ https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-2905: -- Status: Patch Available (was: Open) Desc table can't show non-ascii comments Key: HIVE-2905 URL: https://issues.apache.org/jira/browse/HIVE-2905 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.10.0, 0.7.0 Environment: hive 0.7.0, mysql 5.1.45 hive 0.10.0, mysql 5.5.30 Reporter: Sheng Zhou Labels: patch Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch When desc a table with command line or hive jdbc way, the table's comment can't be read. 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml file. jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8 2. In mysql database, the comment field of COLUMNS table can be read normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2905) Desc table can't show non-ascii comments
[ https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714757#comment-13714757 ] Edward Capriolo commented on HIVE-2905: --- [~brocknoland] You do not necessarily have to cancel the patch over it, comments do not need retesting. It would be a nice to have not a blocker. Desc table can't show non-ascii comments Key: HIVE-2905 URL: https://issues.apache.org/jira/browse/HIVE-2905 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.0, 0.10.0 Environment: hive 0.7.0, mysql 5.1.45 hive 0.10.0, mysql 5.5.30 Reporter: Sheng Zhou Labels: patch Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch When desc a table with command line or hive jdbc way, the table's comment can't be read. 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml file. jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8 2. In mysql database, the comment field of COLUMNS table can be read normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2905) Desc table can't show non-ascii comments
[ https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714758#comment-13714758 ] Edward Capriolo commented on HIVE-2905: --- Also I do not always bounce back petty things like this or spelling mistakes to the original author. It's a big time suck I just fix them and commit. Desc table can't show non-ascii comments Key: HIVE-2905 URL: https://issues.apache.org/jira/browse/HIVE-2905 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.0, 0.10.0 Environment: hive 0.7.0, mysql 5.1.45 hive 0.10.0, mysql 5.5.30 Reporter: Sheng Zhou Labels: patch Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch When desc a table with command line or hive jdbc way, the table's comment can't be read. 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml file. jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8 2. In mysql database, the comment field of COLUMNS table can be read normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
VOTE: moving hive from forest to Apache CMS
http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward
Re: VOTE: moving hive from forest to Apache CMS
+1 On Sun, Jul 21, 2013 at 1:53 PM, Edward Capriolo edlinuxg...@gmail.comwrote: http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward
Re: VOTE: moving hive from forest to Apache CMS
+1 -- Brock Noland Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Sunday, July 21, 2013 at 12:55 PM, Edward Capriolo wrote: +1 On Sun, Jul 21, 2013 at 1:53 PM, Edward Capriolo edlinuxg...@gmail.comwrote: http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714762#comment-13714762
]
Yin Huai commented on HIVE-4502:
The failed query in correlationoptimizer3.q is ...
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT d.key AS key, d.cnt AS cnt, b.value as value
FROM (SELECT x.key as key, x.value as value FROM src1 x JOIN src y ON (x.key =
y.key)) b
JOIN (SELECT x.key as key, count(1) AS cnt FROM src1 x JOIN src y ON (x.key =
y.key) group by x.key) d
ON (b.key = d.key);
{\code}
and here is the stack trace ...
{code}
2013-07-21 10:14:04,440 INFO lazybinary.LazyBinaryStruct
(LazyBinaryStruct.java:parse(172)) - Missing fields! Expected 1 fields but only
got 0! Ignoring similar problems.
2013-07-21 10:14:04,444 FATAL ExecReducer (ExecReducer.java:reduce(269)) -
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=0) {key:{joinkey0:128},value:{_col1:}}
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:258)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
Caused by: java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.io.Text.set(Text.java:205)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:216)
at
org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:197)
at
org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:61)
at
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
at
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
at
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236)
at
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222)
at
org.apache.hadoop.hive.ql.exec.MuxOperator.processOp(MuxOperator.java:249)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.DemuxOperator.processOp(DemuxOperator.java:245)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:249)
... 3 more
{\code}
At here, DemuxOperator only replaces the tag (the tag seems correct at here)
and forward the row to MuxOperator. Then, MuxOperator will just forward the row
to the JoinOperator. Seems we cannot get the value in this case... Still
looking for the root cause... Seems related to MapJoin because the query with
hive.optimize.correlation=true and hive.auto.convert.join=false is fine.
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch,
smb_mapjoin_25.q, smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: VOTE: moving hive from forest to Apache CMS
+1 (non-binding) Jarcec On Sun, Jul 21, 2013 at 01:53:39PM -0400, Edward Capriolo wrote: http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward signature.asc Description: Digital signature
[jira] [Commented] (HIVE-2905) Desc table can't show non-ascii comments
[ https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714763#comment-13714763 ] Brock Noland commented on HIVE-2905: [~appodictic] Sounds good and thanks for the feedback! I am trying to help cleanup the PA queue and it's good to get feedback as to how aggressive I am being. Desc table can't show non-ascii comments Key: HIVE-2905 URL: https://issues.apache.org/jira/browse/HIVE-2905 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.0, 0.10.0 Environment: hive 0.7.0, mysql 5.1.45 hive 0.10.0, mysql 5.5.30 Reporter: Sheng Zhou Labels: patch Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch When desc a table with command line or hive jdbc way, the table's comment can't be read. 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml file. jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8 2. In mysql database, the comment field of COLUMNS table can be read normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: VOTE: moving hive from forest to Apache CMS
+1 (non-binding) On Sun, Jul 21, 2013 at 11:08 AM, Jarek Jarcec Cecho jar...@apache.org wrote: +1 (non-binding) Jarcec On Sun, Jul 21, 2013 at 01:53:39PM -0400, Edward Capriolo wrote: http://hive.apache.org is generated by forest, a rather cumbersome and confusing way to run a website. Forest is difficult to maintain and publish updates with. As a nail in the coffin forest does not even work well with recent versions of java. This vote is to move the site to: Apache CMShttps://www.apache.org/dev/cms.html and away from forest. Brock Noland has offered to move the site, and I am offering to help him and look it over. Vote +1 if you support the move to Apache CMS. (This is the one case where cutting down a forest is a very good idea :) Edward
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Many congratulations, Gunther! On Sun, Jul 21, 2013 at 10:55 AM, Shreepadma Venugopalan shreepa...@cloudera.com wrote: Congratulations, Gunther! On Sun, Jul 21, 2013 at 10:29 AM, Thejas Nair the...@hortonworks.comwrote: Congrats Gunther ! Great to see more bandwidth to get the patch available counts down ! On Jul 21, 2013 9:56 AM, Clark Yang (杨卓荦) yangzhuo...@gmail.com wrote: Congratulations Gunther! 2013/7/22 Brock Noland br...@cloudera.com Congratulations Gunther!! Cheers, Zhuoluo (Clark) Yang
[jira] [Commented] (HIVE-2905) Desc table can't show non-ascii comments
[ https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714767#comment-13714767 ] Edward Capriolo commented on HIVE-2905: --- It just depends how 'letter of the law' we want to be. Sometimes if you ask for something small and people get busy the issue gets delayed days or weeks, the the reviewer gets busy then it gets delayed again. I would not kick a patch batch for a typo or a checkstyle violation, I usually just fix it and re-upload the patch, just use your judgement. I am going to email the list on a few related items. Desc table can't show non-ascii comments Key: HIVE-2905 URL: https://issues.apache.org/jira/browse/HIVE-2905 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.0, 0.10.0 Environment: hive 0.7.0, mysql 5.1.45 hive 0.10.0, mysql 5.5.30 Reporter: Sheng Zhou Labels: patch Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch When desc a table with command line or hive jdbc way, the table's comment can't be read. 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml file. jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8 2. In mysql database, the comment field of COLUMNS table can be read normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Nice work Gunther on many issues BTW. On Sun, Jul 21, 2013 at 2:25 PM, Mark Grover grover.markgro...@gmail.comwrote: Many congratulations, Gunther! On Sun, Jul 21, 2013 at 10:55 AM, Shreepadma Venugopalan shreepa...@cloudera.com wrote: Congratulations, Gunther! On Sun, Jul 21, 2013 at 10:29 AM, Thejas Nair the...@hortonworks.com wrote: Congrats Gunther ! Great to see more bandwidth to get the patch available counts down ! On Jul 21, 2013 9:56 AM, Clark Yang (杨卓荦) yangzhuo...@gmail.com wrote: Congratulations Gunther! 2013/7/22 Brock Noland br...@cloudera.com Congratulations Gunther!! Cheers, Zhuoluo (Clark) Yang
[jira] [Commented] (HIVE-1169) Add Checkstyle Pre-commit Hook
[ https://issues.apache.org/jira/browse/HIVE-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714769#comment-13714769 ] Edward Capriolo commented on HIVE-1169: --- [~brocknoland]Maybe this is something we can make a part of the new auto-builds. Add Checkstyle Pre-commit Hook -- Key: HIVE-1169 URL: https://issues.apache.org/jira/browse/HIVE-1169 Project: Hive Issue Type: Task Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Add a pre-commit hook to the Hive svn repository that triggers Checkstyle and saves the results to a file in the checkstyle directory. Using this information we will be able to spot checkstyle errors that were caused by local changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1169) Add Checkstyle Pre-commit Hook
[ https://issues.apache.org/jira/browse/HIVE-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714774#comment-13714774 ] Brock Noland commented on HIVE-1169: Good call! Hadoop has 7-8 checks it does including checkstyle. I'd like to add many of those as well. Add Checkstyle Pre-commit Hook -- Key: HIVE-1169 URL: https://issues.apache.org/jira/browse/HIVE-1169 Project: Hive Issue Type: Task Components: Build Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach Add a pre-commit hook to the Hive svn repository that triggers Checkstyle and saves the results to a file in the checkstyle directory. Using this information we will be able to spot checkstyle errors that were caused by local changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4225) HiveServer2 does not support SASL QOP
[
https://issues.apache.org/jira/browse/HIVE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714780#comment-13714780
]
Hive QA commented on HIVE-4225:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12584948/HIVE-4225.D10959.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2646 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/118/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/118/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
HiveServer2 does not support SASL QOP
-
Key: HIVE-4225
URL: https://issues.apache.org/jira/browse/HIVE-4225
Project: Hive
Issue Type: Bug
Components: HiveServer2, Shims
Affects Versions: 0.11.0
Reporter: Chris Drome
Assignee: Chris Drome
Attachments: HIVE-4225-1.patch, HIVE-4225.D10959.1.patch,
HIVE-4225.patch
HiveServer2 implements Kerberos authentication through SASL framework, but
does not support setting QOP.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Great progress with the hive pre-commit hive build farm
We now have a system automatically building and testing patches. Thanks much to Brock and Cloudera for driving this through. https://builds.apache.org/user/brock/my-views/view/hive/job/PreCommit-HIVE-Build/ I can already see by the 30 day summary that this has greatly accelerate the test-and-commit process. We are getting one or sometimes two reviews and votes on issues! We are also creating more issues to now, but that is a happy problem :) There is a big swath of things on the backlog that we and the patch tester will have to chew through. If you have a feature you would like reviewed please read: 1) contributing rules https://cwiki.apache.org/confluence/display/Hive/HowToContribute and https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing and We are not yet running this automatically. You should run checkstyle before you send a patch. You can also setup your IDE to use apache conventions and your IDE will do the cleanup for you. 2) As Namit pointed out on the dev list in an earlier email, we are a thin on java-doc. I do believe that good code is self documenting, but the 'why' gets lost sometimes. I think Gunther did a great job here to document the fact that ':' is not an acceptable character as an example. https://issues.apache.org/jira/browse/HIVE-4865 As I mentioned, we have a backlog now and it will take some time to chop that down, but now might be a good time to rebase your patch and clean it up a bit. Edward
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714791#comment-13714791 ] Yin Huai commented on HIVE-4502: Problem found. It's because in 'splitTasks' the new patch (uploaded at 20/Jul/13 17:06) did not set needsTagging for the MR job which has DemuxOperator. Will upload a incremental patch later. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4123) The RLE encoding for ORC can be improved
[ https://issues.apache.org/jira/browse/HIVE-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714795#comment-13714795 ] Owen O'Malley commented on HIVE-4123: - More comments: * I don't see why bitpack reader/writer are more than static methods that read/write to the underlying stream. So I would have expected a method like writeInts(long[] data, int offset, int length, int numBits, OutputStream stream) and the corresponding one for reading. * Utils.bytesToLongBE should take an input stream rather than a byte[]. * In IntegerCompressionReader: ** I'd write a method to translate the int into an opcode rather than use ordinal. ** It is probably worth remembering that you are in a repeat, so that you don't need to copy the value N times in short repeat. ** It may be easier to loop through the base values and then run through the patches. You might even do three loops: unpack the main values, unpack the patches, add the base to each value. ** For patched based only the base is zigzag encoded. The rest of the values are always positive. ** For delta only the base and base delta are zigzag encoded. * In IntegerCompressionWriter: ** You should give more comments about the patched base encoding. ** Instead of sorting for the percentiles, you could keep a count of how many values use each number of bits. ** Replace the commented out printlns with LOG.debug surrounded by LOG.ifDebugEnabled ** flush should use if/then/else to prevent writing the data twice ** the constructor should probably call clear rather than risk having the default values be different ** in write, just copy the data with system.arraycopy instead of cloning the array ** write should track whether the values are monotonically increasing or decreasing so that we know if delta applies ** there is a lot of duplication of effort in determine encoding ** if the sequence is both increasing and decreasing, it is constant and we should either use short literal or delta depending on the length ** delta encoding should return before doing the percentile work ** * How much unit test coverage do you have of the new code? * Have you run the encoder/decoder round trip over the github data to test it? The RLE encoding for ORC can be improved Key: HIVE-4123 URL: https://issues.apache.org/jira/browse/HIVE-4123 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Prasanth J Attachments: HIVE-4123.1.git.patch.txt, HIVE-4123.2.git.patch.txt, ORC-Compression-Ratio-Comparison.xlsx The run length encoding of integers can be improved: * tighter bit packing * allow delta encoding * allow longer runs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4502: --- Attachment: HIVE-4502.incremental.patch NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4502: --- Attachment: HIVE-4502.1.patch I merged my incremental patch with Ashutosh's patch. Attaching it (HIVE-4502.1.patch) at here as a reference. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.1.patch, HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated HIVE-4502: --- Status: Patch Available (was: Open) NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.1.patch, HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714800#comment-13714800 ] Yin Huai commented on HIVE-4502: HIVE-4502.incremental.patch has extra changes I made based on HIVE-4502.patch (uploaded at 20/Jul/13 17:06). Since in GenMapRedUtils there are three places that we need to check if we need to need to do tagging, I add a utility method to check the class of the reducer. We will set needsTagging when the reducer is JoinOperator or DemuxOperator. Also, some small updates are needed for test results of those queries related to Correlation Optimizer. Those updates just replace ' A masked pattern was here ' with '$INTNAME'. NPE - subquery smb joins fails -- Key: HIVE-4502 URL: https://issues.apache.org/jira/browse/HIVE-4502 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Navis Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q Found this issue while running some SMB joins. Attaching test case that causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4876) Beeling help text do not contain -f and -e parameters
[
https://issues.apache.org/jira/browse/HIVE-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714804#comment-13714804
]
Hive QA commented on HIVE-4876:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593409/HIVE-4876.patch
{color:green}SUCCESS:{color} +1 2646 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/119/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/119/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Beeling help text do not contain -f and -e parameters
-
Key: HIVE-4876
URL: https://issues.apache.org/jira/browse/HIVE-4876
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Jarek Jarcec Cecho
Assignee: Jarek Jarcec Cecho
Priority: Trivial
Attachments: bugHIVE-4876.patch, HIVE-4876.patch
I've noticed that beeline do have support for {{-e}} and {{-f}} parameters,
however those are not documented in the help text that can be retrieved by
calling beeline with parameter {{-h}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714820#comment-13714820 ] Xuefu Zhang commented on HIVE-4900: --- [~appodictic] Mysql 0.10 schema file doesn't create SORT_ORDER either. I think MOrder-SORT_ORDER definition there is redundant, it can be probably removed safely. Also, it's declared as embedded-only=true, so there should be no standalone objects mapped to that table. I didn't spend much time on this as my effort is about HIVE-3632. I can create a different JIRA for that. I modified the column name just in order to be consistent with MOrder O-R mapping later defined in SDS, which I could have spared. Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4225) HiveServer2 does not support SASL QOP
[
https://issues.apache.org/jira/browse/HIVE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714824#comment-13714824
]
Hive QA commented on HIVE-4225:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12584948/HIVE-4225.D10959.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2646 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/120/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/120/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
HiveServer2 does not support SASL QOP
-
Key: HIVE-4225
URL: https://issues.apache.org/jira/browse/HIVE-4225
Project: Hive
Issue Type: Bug
Components: HiveServer2, Shims
Affects Versions: 0.11.0
Reporter: Chris Drome
Assignee: Chris Drome
Attachments: HIVE-4225-1.patch, HIVE-4225.D10959.1.patch,
HIVE-4225.patch
HiveServer2 implements Kerberos authentication through SASL framework, but
does not support setting QOP.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [ANNOUNCE] New Hive Committer - Gunther Hagleitner
Congratulations, Gunther! On Sun, Jul 21, 2013 at 11:43 AM, Edward Capriolo edlinuxg...@gmail.com wrote: Nice work Gunther on many issues BTW. On Sun, Jul 21, 2013 at 2:25 PM, Mark Grover grover.markgro...@gmail.comwrote: Many congratulations, Gunther! On Sun, Jul 21, 2013 at 10:55 AM, Shreepadma Venugopalan shreepa...@cloudera.com wrote: Congratulations, Gunther! On Sun, Jul 21, 2013 at 10:29 AM, Thejas Nair the...@hortonworks.com wrote: Congrats Gunther ! Great to see more bandwidth to get the patch available counts down ! On Jul 21, 2013 9:56 AM, Clark Yang (杨卓荦) yangzhuo...@gmail.com wrote: Congratulations Gunther! 2013/7/22 Brock Noland br...@cloudera.com Congratulations Gunther!! Cheers, Zhuoluo (Clark) Yang
Hive-trunk-hadoop2 - Build # 299 - Failure
Changes for Build #266 [hashutosh] HIVE-4618 : show create table creating unusable DDL when field delimiter is \001 (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4559 : hcatalog/webhcat scripts in tar.gz don't have execute permissions set (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4798 : NPE when we call isSame from an instance of ExprNodeConstantDesc with null value (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4781 : LEFT SEMI JOIN generates wrong results when the number of rows belonging to a single key of the right table exceed hive.join.emit.interval (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4647 : RetryingHMSHandler logs too many error messages (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4692 : Constant agg parameters will be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases (Navis via Ashutosh Chauhan) Changes for Build #267 [hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original commit. Changes for Build #268 [navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via Navis) [hashutosh] HIVE-4406 : Missing / or /dbname in hs2 jdbc uri switches mode to embedded mode(Anandha Ranganathan via Ashutosh Chauhan) [hashutosh] HIVE-4430 : Semantic analysis fails in presence of certain literals in on clause (Kevin Wilfong via Ashutosh Chauhan) [hashutosh] HIVE-4757 : LazyTimestamp goes into irretrievable NULL mode once inited with NULL once (Gopal V via Ashutosh Chauhan) [hashutosh] HIVE-4785 : Implement isCaseSensitive for Hive JDBC driver (Robert Roland via Ashutosh Chauhan) Changes for Build #269 [navis] HIVE-4436 : hive.exec.parallel=true doesn't work on hadoop-2 (Gopal V via Navis) Changes for Build #270 [hashutosh] HIVE-4689 : For outerjoins, joinEmitInterval might make wrong result (Navis via Ashutosh Chauhan) [hashutosh] HIVE-3253 : ArrayIndexOutOfBounds exception for deeply nested structs (Thejas Nair via Ashutosh Chauhan) Changes for Build #271 Changes for Build #272 Changes for Build #273 [hashutosh] HIVE-4089 : javax.jdo : jdo2-api dependency not in Maven Central (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-4804 parallel order by fails for small datasets (Navis via egc) Submitted by: Navis Reviewed by:Edward Capriolo Changes for Build #274 Changes for Build #275 [hashutosh] HIVE-4811 : (Slightly) break up the SemanticAnalyzer monstrosity (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4814 : Adjust WebHCat e2e tests until HIVE4703 is addressed (Eugene Koifman via Ashutosh Chauhan) Changes for Build #276 [hashutosh] HIVE-4251 : Indices can't be built on tables whose schema info comes from SerDe (Mark Wagner via Ashutosh Chauhan) [hashutosh] HIVE-4805 : Enhance coverage of package org.apache.hadoop.hive.ql.exec.errors (Ivan Veselovsky via Ashutosh Chauhan) Changes for Build #277 [hashutosh] HIVE-4733 : HiveLockObjectData is not compared properly (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-3475 INLINE UDTF does not convert types properly (Igor Kabiljo and Navis Ryu via egc) Submitted by: Navis Ryu and Igor Kabiljo Reviewed by:Edward Capriolo Changes for Build #278 [hashutosh] HIVE-4802 : Fix url check for missing / or /db after hostname in jdb uri (Thejas Nair via Ashutosh Chauhan) Changes for Build #279 [hashutosh] HIVE-3810 : HiveHistory.log need to replace \r with space before writing Entry.value to historyfile (Mark Grover via Ashutosh Chauhan) Changes for Build #280 [hashutosh] HIVE-4819 : Comments in CommonJoinOperator for aliasTag is not valid (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4813 : Improve test coverage of package org.apache.hadoop.hive.ql.optimizer.pcr (Ivan Veselovsky via Ashutosh Chauhan) [hashutosh] HIVE-4580 : Change DDLTask to report errors using canonical error messages rather than http status codes (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4796 : Increase coverage of package org.apache.hadoop.hive.common.metrics (Ivan Veselovsky via Ashutosh Chauhan) [navis] HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) Changes for Build #281 [hashutosh] HIVE-4833 : Fix eclipse template classpath to include the correct jdo lib (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4830 : Test clientnegative/nested_complex_neg.q got broken due to 4580 (Vikram Dixit via Ashutosh Chauhan) [hashutosh] HIVE-4810 [jira] Refactor exec package (Gunther Hagleitner via Ashutosh Chauhan) Summary: HIVE-4810 The exec package contains both operators and classes used to execute the job. Moving the latter into a sub package makes the package slightly more manageable and will make it easier to provide a tez-based implementation. Test Plan: Refactoring Reviewers: ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D11625 [hashutosh] HIVE-4829 : TestWebHCatE2e checkstyle violation causes all tests to fail (Eugene Koifman via Ashutosh Chauhan) Changes for
[jira] [Commented] (HIVE-4225) HiveServer2 does not support SASL QOP
[
https://issues.apache.org/jira/browse/HIVE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714849#comment-13714849
]
Hive QA commented on HIVE-4225:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12584948/HIVE-4225.D10959.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2646 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/121/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/121/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
HiveServer2 does not support SASL QOP
-
Key: HIVE-4225
URL: https://issues.apache.org/jira/browse/HIVE-4225
Project: Hive
Issue Type: Bug
Components: HiveServer2, Shims
Affects Versions: 0.11.0
Reporter: Chris Drome
Assignee: Chris Drome
Attachments: HIVE-4225-1.patch, HIVE-4225.D10959.1.patch,
HIVE-4225.patch
HiveServer2 implements Kerberos authentication through SASL framework, but
does not support setting QOP.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-hadoop2 - Build # 300 - Still Failing
Changes for Build #267 [hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original commit. Changes for Build #268 [navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via Navis) [hashutosh] HIVE-4406 : Missing / or /dbname in hs2 jdbc uri switches mode to embedded mode(Anandha Ranganathan via Ashutosh Chauhan) [hashutosh] HIVE-4430 : Semantic analysis fails in presence of certain literals in on clause (Kevin Wilfong via Ashutosh Chauhan) [hashutosh] HIVE-4757 : LazyTimestamp goes into irretrievable NULL mode once inited with NULL once (Gopal V via Ashutosh Chauhan) [hashutosh] HIVE-4785 : Implement isCaseSensitive for Hive JDBC driver (Robert Roland via Ashutosh Chauhan) Changes for Build #269 [navis] HIVE-4436 : hive.exec.parallel=true doesn't work on hadoop-2 (Gopal V via Navis) Changes for Build #270 [hashutosh] HIVE-4689 : For outerjoins, joinEmitInterval might make wrong result (Navis via Ashutosh Chauhan) [hashutosh] HIVE-3253 : ArrayIndexOutOfBounds exception for deeply nested structs (Thejas Nair via Ashutosh Chauhan) Changes for Build #271 Changes for Build #272 Changes for Build #273 [hashutosh] HIVE-4089 : javax.jdo : jdo2-api dependency not in Maven Central (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-4804 parallel order by fails for small datasets (Navis via egc) Submitted by: Navis Reviewed by:Edward Capriolo Changes for Build #274 Changes for Build #275 [hashutosh] HIVE-4811 : (Slightly) break up the SemanticAnalyzer monstrosity (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4814 : Adjust WebHCat e2e tests until HIVE4703 is addressed (Eugene Koifman via Ashutosh Chauhan) Changes for Build #276 [hashutosh] HIVE-4251 : Indices can't be built on tables whose schema info comes from SerDe (Mark Wagner via Ashutosh Chauhan) [hashutosh] HIVE-4805 : Enhance coverage of package org.apache.hadoop.hive.ql.exec.errors (Ivan Veselovsky via Ashutosh Chauhan) Changes for Build #277 [hashutosh] HIVE-4733 : HiveLockObjectData is not compared properly (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-3475 INLINE UDTF does not convert types properly (Igor Kabiljo and Navis Ryu via egc) Submitted by: Navis Ryu and Igor Kabiljo Reviewed by:Edward Capriolo Changes for Build #278 [hashutosh] HIVE-4802 : Fix url check for missing / or /db after hostname in jdb uri (Thejas Nair via Ashutosh Chauhan) Changes for Build #279 [hashutosh] HIVE-3810 : HiveHistory.log need to replace \r with space before writing Entry.value to historyfile (Mark Grover via Ashutosh Chauhan) Changes for Build #280 [hashutosh] HIVE-4819 : Comments in CommonJoinOperator for aliasTag is not valid (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4813 : Improve test coverage of package org.apache.hadoop.hive.ql.optimizer.pcr (Ivan Veselovsky via Ashutosh Chauhan) [hashutosh] HIVE-4580 : Change DDLTask to report errors using canonical error messages rather than http status codes (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4796 : Increase coverage of package org.apache.hadoop.hive.common.metrics (Ivan Veselovsky via Ashutosh Chauhan) [navis] HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) Changes for Build #281 [hashutosh] HIVE-4833 : Fix eclipse template classpath to include the correct jdo lib (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4830 : Test clientnegative/nested_complex_neg.q got broken due to 4580 (Vikram Dixit via Ashutosh Chauhan) [hashutosh] HIVE-4810 [jira] Refactor exec package (Gunther Hagleitner via Ashutosh Chauhan) Summary: HIVE-4810 The exec package contains both operators and classes used to execute the job. Moving the latter into a sub package makes the package slightly more manageable and will make it easier to provide a tez-based implementation. Test Plan: Refactoring Reviewers: ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D11625 [hashutosh] HIVE-4829 : TestWebHCatE2e checkstyle violation causes all tests to fail (Eugene Koifman via Ashutosh Chauhan) Changes for Build #282 [hashutosh] HIVE-3691 : TestDynamicSerDe failed with IBM JDK (Bing Li Renata Ghisloti via Ashutosh Chauhan) [hashutosh] HIVE-4807 : Hive metastore hangs (Sarvesh Sakalanaga via Ashutosh Chauhan) Changes for Build #283 Changes for Build #284 Changes for Build #285 [hashutosh] HIVE-4840 : Fix eclipse template classpath to include the BoneCP lib (Yin Huai via Ashutosh Chauhan) Changes for Build #286 [navis] HIVE-4290 : Build profiles: Partial builds for quicker dev (Gunther Hagleitner via Navis) [navis] HIVE-4658 : Make KW_OUTER optional in outer joins (Edward Capriolo via Navis) Changes for Build #287 Changes for Build #288 Changes for Build #289 [hashutosh] HIVE-4852 : -Dbuild.profile=core fails (Gunther Hagleitner via Ashutosh Chauhan) [hashutosh] HIVE-4854 : testCliDriver_load_hdfs_file_with_space_in_the_name fails on hadoop 2
Re: Review Request 9276: Add support for pulling HBase columns with prefixes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9276/ --- (Updated July 22, 2013, 1:20 a.m.) Review request for hive. Changes --- Updated diff with proposed changes. Bugs: HIVE-3725 https://issues.apache.org/jira/browse/HIVE-3725 Repository: hive-git Description --- Added support for pulling hbase columns just by providing prefixes and a wildcard. So a query now could look something like this: CREATE EXTERNAL TABLE hive_hbase_test ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,fam1:col*) TBLPROPERTIES (hbase.table.name = TEST_HBASE_TABLE); This would pull in all columns under column family fam1 which start with col. This gives a little more flexibility over pull all columns format. Diffs (updated) - hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 65c81bf hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java b550f45 hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 01938a7 hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseCellMap.java a8ba9d9 hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 10a9207 hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java e821282 Diff: https://reviews.apache.org/r/9276/diff/ Testing --- Added unit tests to demonstrate the new functionality. Also made sure that all existing unit tests passed. Thanks, Swarnim Kulkarni
[jira] [Updated] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swarnim Kulkarni updated HIVE-3725: --- Attachment: HIVE-3725.patch.3.txt Add support for pulling HBase columns with prefixes --- Key: HIVE-3725 URL: https://issues.apache.org/jira/browse/HIVE-3725 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.9.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt, HIVE-3725.patch.3.txt Current HBase Hive integration supports reading many values from the same row by specifying a column family. And specifying just the column family can pull in all qualifiers within the family. We should add in support to be able to specify a prefix for the qualifier and all columns that start with the prefix would automatically get pulled in. A wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714855#comment-13714855 ] Edward Capriolo commented on HIVE-4900: --- [~xuefuz] Thank you for answering that. I do not know much about this piece so I was just doing what I could to double-check to make sure the mappings were correct. If you do not think we need it we should remove it, if the embedded mode needs it the tests should tell us as such. Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4903) LIST { FILE[S] | JAR[S] | ARCHIVE[S] } not work in hive Cli remote mode
benhuachen created HIVE-4903:
Summary: LIST { FILE[S] | JAR[S] | ARCHIVE[S] } not work in hive
Cli remote mode
Key: HIVE-4903
URL: https://issues.apache.org/jira/browse/HIVE-4903
Project: Hive
Issue Type: Bug
Components: CLI
Environment: hive server have been started
Reporter: benhuachen
Priority: Minor
for example:
step 1:start hive cli in remote mode like this:
./hive -h localhost
step 2:add file or jar:
add jar /opt/add.jar
stp3:list files or jars:
list jars;
/opt/add.jar is what I wanted, but I got nothing!
if I start Cli like this:
./hive
then I could get what I want!
Any advice? thanks
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714876#comment-13714876
]
Hive QA commented on HIVE-4502:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593432/HIVE-4502.1.patch
{color:green}SUCCESS:{color} +1 2647 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/122/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/122/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
NPE - subquery smb joins fails
--
Key: HIVE-4502
URL: https://issues.apache.org/jira/browse/HIVE-4502
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Vikram Dixit K
Assignee: Navis
Attachments: HIVE-4502.1.patch, HIVE-4502.D10695.1.patch,
HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch,
HIVE-4502.incremental.patch, HIVE-4502.patch, smb_mapjoin_25.q,
smb_mapjoin_25.q
Found this issue while running some SMB joins. Attaching test case that
causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-3725: --- Status: Patch Available (was: Open) Marking Patch Available so the precommit tests will execute. Add support for pulling HBase columns with prefixes --- Key: HIVE-3725 URL: https://issues.apache.org/jira/browse/HIVE-3725 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.9.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt, HIVE-3725.3.patch.txt, HIVE-3725.patch.3.txt Current HBase Hive integration supports reading many values from the same row by specifying a column family. And specifying just the column family can pull in all qualifiers within the family. We should add in support to be able to specify a prefix for the qualifier and all columns that start with the prefix would automatically get pulled in. A wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3725) Add support for pulling HBase columns with prefixes
[ https://issues.apache.org/jira/browse/HIVE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-3725: --- Attachment: HIVE-3725.3.patch.txt Hi, I am re-uploading your patch as HIVE-3725.3.patch.txt so the [precommit tests|https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing] will execute. Add support for pulling HBase columns with prefixes --- Key: HIVE-3725 URL: https://issues.apache.org/jira/browse/HIVE-3725 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.9.0 Reporter: Swarnim Kulkarni Assignee: Swarnim Kulkarni Attachments: HIVE-3725.1.patch.txt, HIVE-3725.2.patch.txt, HIVE-3725.3.patch.txt, HIVE-3725.patch.3.txt Current HBase Hive integration supports reading many values from the same row by specifying a column family. And specifying just the column family can pull in all qualifiers within the family. We should add in support to be able to specify a prefix for the qualifier and all columns that start with the prefix would automatically get pulled in. A wildcard support would be ideal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4892) PTest2 cleanup after merge
[ https://issues.apache.org/jira/browse/HIVE-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714882#comment-13714882 ] Brock Noland commented on HIVE-4892: Friendly ping :) When we commit this I can move the build infra to the official source tree. PTest2 cleanup after merge -- Key: HIVE-4892 URL: https://issues.apache.org/jira/browse/HIVE-4892 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4892.patch HIVE-4675 was merged but there are still a few minor issues we need to cleanup: * README is out of date * Need to limit the number of failed source directories we copy back from the slaves * when looking for TEST-*.xml files we look at both the log directory (good) and the failed source directories (bad) therefore duplicating failures in jenkins report * We need to process bad hosts in the finally block of PTest.run (HIVE-4882) * Need a mechanism to clean the ivy and maven cache (HIVE-4882) * PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) * Now that PTest2 is committed to the source tree it's copying in our TEST-SomeTest*.xml files Test Properties: NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 2212 - Still Failing
Changes for Build #2171 [hashutosh] HIVE-4618 : show create table creating unusable DDL when field delimiter is \001 (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4559 : hcatalog/webhcat scripts in tar.gz don't have execute permissions set (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4798 : NPE when we call isSame from an instance of ExprNodeConstantDesc with null value (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4781 : LEFT SEMI JOIN generates wrong results when the number of rows belonging to a single key of the right table exceed hive.join.emit.interval (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4647 : RetryingHMSHandler logs too many error messages (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4692 : Constant agg parameters will be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases (Navis via Ashutosh Chauhan) Changes for Build #2172 [hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original commit. Changes for Build #2173 Changes for Build #2174 [navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via Navis) [hashutosh] HIVE-4406 : Missing / or /dbname in hs2 jdbc uri switches mode to embedded mode(Anandha Ranganathan via Ashutosh Chauhan) [hashutosh] HIVE-4430 : Semantic analysis fails in presence of certain literals in on clause (Kevin Wilfong via Ashutosh Chauhan) [hashutosh] HIVE-4757 : LazyTimestamp goes into irretrievable NULL mode once inited with NULL once (Gopal V via Ashutosh Chauhan) [hashutosh] HIVE-4785 : Implement isCaseSensitive for Hive JDBC driver (Robert Roland via Ashutosh Chauhan) Changes for Build #2175 [navis] HIVE-4436 : hive.exec.parallel=true doesn't work on hadoop-2 (Gopal V via Navis) Changes for Build #2176 Changes for Build #2177 [hashutosh] HIVE-4689 : For outerjoins, joinEmitInterval might make wrong result (Navis via Ashutosh Chauhan) [hashutosh] HIVE-3253 : ArrayIndexOutOfBounds exception for deeply nested structs (Thejas Nair via Ashutosh Chauhan) Changes for Build #2178 Changes for Build #2179 Changes for Build #2180 Changes for Build #2181 [hashutosh] HIVE-4089 : javax.jdo : jdo2-api dependency not in Maven Central (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-4804 parallel order by fails for small datasets (Navis via egc) Submitted by: Navis Reviewed by:Edward Capriolo Changes for Build #2182 Changes for Build #2183 [hashutosh] HIVE-4814 : Adjust WebHCat e2e tests until HIVE4703 is addressed (Eugene Koifman via Ashutosh Chauhan) Changes for Build #2184 [hashutosh] HIVE-4811 : (Slightly) break up the SemanticAnalyzer monstrosity (Gunther Hagleitner via Ashutosh Chauhan) Changes for Build #2185 [hashutosh] HIVE-4251 : Indices can't be built on tables whose schema info comes from SerDe (Mark Wagner via Ashutosh Chauhan) [hashutosh] HIVE-4805 : Enhance coverage of package org.apache.hadoop.hive.ql.exec.errors (Ivan Veselovsky via Ashutosh Chauhan) Changes for Build #2186 [hashutosh] HIVE-4733 : HiveLockObjectData is not compared properly (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-3475 INLINE UDTF does not convert types properly (Igor Kabiljo and Navis Ryu via egc) Submitted by: Navis Ryu and Igor Kabiljo Reviewed by:Edward Capriolo Changes for Build #2187 [hashutosh] HIVE-4802 : Fix url check for missing / or /db after hostname in jdb uri (Thejas Nair via Ashutosh Chauhan) Changes for Build #2188 [hashutosh] HIVE-4813 : Improve test coverage of package org.apache.hadoop.hive.ql.optimizer.pcr (Ivan Veselovsky via Ashutosh Chauhan) [hashutosh] HIVE-4580 : Change DDLTask to report errors using canonical error messages rather than http status codes (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4796 : Increase coverage of package org.apache.hadoop.hive.common.metrics (Ivan Veselovsky via Ashutosh Chauhan) [navis] HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) [hashutosh] HIVE-3810 : HiveHistory.log need to replace \r with space before writing Entry.value to historyfile (Mark Grover via Ashutosh Chauhan) Changes for Build #2189 [hashutosh] HIVE-4810 [jira] Refactor exec package (Gunther Hagleitner via Ashutosh Chauhan) Summary: HIVE-4810 The exec package contains both operators and classes used to execute the job. Moving the latter into a sub package makes the package slightly more manageable and will make it easier to provide a tez-based implementation. Test Plan: Refactoring Reviewers: ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D11625 [hashutosh] HIVE-4829 : TestWebHCatE2e checkstyle violation causes all tests to fail (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4819 : Comments in CommonJoinOperator for aliasTag is not valid (Navis via Ashutosh Chauhan) Changes for Build #2190 [hashutosh] HIVE-4807 : Hive metastore hangs (Sarvesh Sakalanaga via Ashutosh Chauhan) [hashutosh] HIVE-4833 : Fix eclipse template classpath
Re: Review Request 12795: [HIVE-4827] Merge a Map-only job to its following MapReduce job with multiple inputs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12795/ --- (Updated July 22, 2013, 4:19 a.m.) Review request for hive. Changes --- Addressed Edward's comment and update test results. Bugs: HIVE-4827 https://issues.apache.org/jira/browse/HIVE-4827 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-4827 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 66b84ff ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java f98878c ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 7cbb1ff ql/src/test/queries/clientpositive/correlationoptimizer7.q 9b18972 ql/src/test/queries/clientpositive/multiMapJoin2.q PRE-CREATION ql/src/test/results/clientpositive/auto_join33.q.out 8fc0e84 ql/src/test/results/clientpositive/correlationoptimizer1.q.out db3bd78 ql/src/test/results/clientpositive/correlationoptimizer3.q.out cebddff ql/src/test/results/clientpositive/correlationoptimizer4.q.out 285a54f ql/src/test/results/clientpositive/correlationoptimizer6.q.out c40a786 ql/src/test/results/clientpositive/correlationoptimizer7.q.out ea54431 ql/src/test/results/clientpositive/multiMapJoin1.q.out 3b3eb3f ql/src/test/results/clientpositive/multiMapJoin2.q.out PRE-CREATION Diff: https://reviews.apache.org/r/12795/diff/ Testing --- Running tests. Thanks, Yin Huai
[jira] [Updated] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-4827:
---
Attachment: HIVE-4827.2.patch
Addressed Edward's comment and update test results.
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-4827:
---
Status: Patch Available (was: Open)
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714888#comment-13714888
]
Yin Huai commented on HIVE-4827:
Review board: https://reviews.apache.org/r/12795/
Merge a Map-only job to its following MapReduce job with multiple inputs
Key: HIVE-4827
URL: https://issues.apache.org/jira/browse/HIVE-4827
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Yin Huai
Assignee: Yin Huai
Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver
to merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same
MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
MapReduce job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-4900: -- Attachment: HIVE-4900.1.patch Patch updated with SORT_ORDER table section removed. Fix the mismatched column names in package.jdo -- Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4900.1.patch, HIVE-4900.patch There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4293) Predicates following UDTF operator are removed by PPD
[
https://issues.apache.org/jira/browse/HIVE-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-4293:
--
Attachment: HIVE-4293.D9933.4.patch
navis updated the revision HIVE-4293 [jira] Predicates following UDTF operator
are removed by PPD.
Rebased to trunk
Reviewers: JIRA
REVISION DETAIL
https://reviews.facebook.net/D9933
CHANGE SINCE LAST DIFF
https://reviews.facebook.net/D9933?vs=33483id=35949#toc
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java
ql/src/java/org/apache/hadoop/hive/ql/ppd/ExprWalkerInfo.java
ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
ql/src/java/org/apache/hadoop/hive/ql/ppd/PredicatePushDown.java
ql/src/test/queries/clientpositive/lateral_view_ppd.q
ql/src/test/queries/clientpositive/ppd_udtf.q
ql/src/test/results/clientpositive/cluster.q.out
ql/src/test/results/clientpositive/ctas_colname.q.out
ql/src/test/results/clientpositive/lateral_view_ppd.q.out
ql/src/test/results/clientpositive/ppd2.q.out
ql/src/test/results/clientpositive/ppd_gby.q.out
ql/src/test/results/clientpositive/ppd_gby2.q.out
ql/src/test/results/clientpositive/ppd_udtf.q.out
ql/src/test/results/clientpositive/udtf_json_tuple.q.out
ql/src/test/results/clientpositive/udtf_parse_url_tuple.q.out
ql/src/test/results/compiler/plan/join1.q.xml
ql/src/test/results/compiler/plan/join2.q.xml
ql/src/test/results/compiler/plan/join3.q.xml
ql/src/test/results/compiler/plan/join4.q.xml
ql/src/test/results/compiler/plan/join5.q.xml
ql/src/test/results/compiler/plan/join6.q.xml
ql/src/test/results/compiler/plan/join7.q.xml
ql/src/test/results/compiler/plan/join8.q.xml
To: JIRA, navis
Predicates following UDTF operator are removed by PPD
-
Key: HIVE-4293
URL: https://issues.apache.org/jira/browse/HIVE-4293
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Attachments: HIVE-4293.D9933.1.patch, HIVE-4293.D9933.2.patch,
HIVE-4293.D9933.3.patch, HIVE-4293.D9933.4.patch
For example,
{noformat}
explain SELECT value from (
select explode(array(key, value)) as (value) from (
select * FROM src WHERE key 200
) A
) B WHERE value 300
;
{noformat}
Makes plan like this, removing last predicates
{noformat}
TableScan
alias: src
Filter Operator
predicate:
expr: (key 200.0)
type: boolean
Select Operator
expressions:
expr: array(key,value)
type: arraystring
outputColumnNames: _col0
UDTF Operator
function name: explode
Select Operator
expressions:
expr: col
type: string
outputColumnNames: _col0
File Output Operator
compressed: false
GlobalTableId: 0
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
