[jira] [Commented] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714360#comment-13714360 ] Carl Steinbach commented on HIVE-4739: -- [~brocknoland] Ditto. Thanks! > Create public parallel test environment > --- > > Key: HIVE-4739 > URL: https://issues.apache.org/jira/browse/HIVE-4739 > Project: Hive > Issue Type: Improvement >Reporter: Brock Noland >Assignee: Brock Noland > > See discussion here > http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4871) Apache builds fail with Target "make-pom" does not exist in the project "hcatalog".
[
https://issues.apache.org/jira/browse/HIVE-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714377#comment-13714377
]
Hive QA commented on HIVE-4871:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593314/HIVE-4871.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2646 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/105/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/105/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
> Apache builds fail with Target "make-pom" does not exist in the project
> "hcatalog".
> ---
>
> Key: HIVE-4871
> URL: https://issues.apache.org/jira/browse/HIVE-4871
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.12.0
>
> Attachments: HIVE-4871.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> For example,
> https://builds.apache.org/job/Hive-trunk-h0.21/2192/console.
> All unit tests pass, but deployment of build artifacts fails.
> HIVE-4387 provided a bandaid for 0.11. Need to figure out long term fix for
> this for 0.12.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4670) Authentication module should pass the instance part of the Kerberos principle
[
https://issues.apache.org/jira/browse/HIVE-4670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714393#comment-13714393
]
Hive QA commented on HIVE-4670:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12586817/HIVE-4670.3.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2646 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/106/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/106/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
> Authentication module should pass the instance part of the Kerberos principle
> -
>
> Key: HIVE-4670
> URL: https://issues.apache.org/jira/browse/HIVE-4670
> Project: Hive
> Issue Type: Bug
> Components: Authentication, HiveServer2
>Affects Versions: 0.11.0
>Reporter: Shreepadma Venugopalan
>Assignee: Shreepadma Venugopalan
> Attachments: HIVE-4670.2.patch, HIVE-4670.3.patch
>
>
> When Kerberos authentication is enabled for HiveServer2, the thrift SASL
> layer passes instance@realm from the principal. It should instead strip the
> realm and pass just the instance part of the principal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2905) Desc table can't show non-ascii comments
[
https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714404#comment-13714404
]
Hive QA commented on HIVE-2905:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12589572/HIVE-2905.D11487.1.patch
{color:green}SUCCESS:{color} +1 2647 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/109/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/109/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
> Desc table can't show non-ascii comments
>
>
> Key: HIVE-2905
> URL: https://issues.apache.org/jira/browse/HIVE-2905
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 0.7.0, 0.10.0
> Environment: hive 0.7.0, mysql 5.1.45
> hive 0.10.0, mysql 5.5.30
>Reporter: Sheng Zhou
> Labels: patch
> Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch
>
>
> When desc a table with command line or hive jdbc way, the table's comment
> can't be read.
> 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml
> file.
>jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8
> 2. In mysql database, the comment field of COLUMNS table can be read normally.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3632) Upgrade datanucleus to support JDK7
[ https://issues.apache.org/jira/browse/HIVE-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-3632: --- Attachment: HIVE-3632.1.patch The patch "HIVE-3632.patch.1" is not in the correct format for the precommit tests and I'd like to see it run. Therefore I am re-uploading the exact same patch named as "HIVE-3632.1.patch". > Upgrade datanucleus to support JDK7 > --- > > Key: HIVE-3632 > URL: https://issues.apache.org/jira/browse/HIVE-3632 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.9.1, 0.10.0, 0.11.0 >Reporter: Chris Drome >Assignee: Xuefu Zhang >Priority: Critical > Fix For: 0.12.0 > > Attachments: HIVE-3632.1.patch, HIVE-3632.patch, HIVE-3632.patch.1 > > > I found serious problems with datanucleus code when using JDK7, resulting in > some sort of exception being thrown when datanucleus code is entered. > I tried source=1.7, target=1.7 with JDK7 as well as source=1.6, target=1.6 > with JDK7 and there was no visible difference in that the same unit tests > failed. > I tried upgrading datanucleus to 3.0.1, as per HIVE-2084.patch, which did not > fix the failing tests. > I tried upgrading datanucleus to 3.1-release, as per the advise of > http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-86, which suggests > using ASMv4 will allow datanucleus to work with JDK7. I was not successful > with this either. > I tried upgrading datanucleus to 3.1.2. I was not successful with this either. > Regarding datanucleus support for JDK7+, there is the following JIRA > http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-81 > which suggests that they don't plan to actively support JDK7+ bytecode any > time soon. > I also tested the following JVM parameters found on > http://veerasundar.com/blog/2012/01/java-lang-verifyerror-expecting-a-stackmap-frame-at-branch-target-jdk-7/ > with no success either. > This will become a more serious problem as people move to newer JVMs. If > there are other who have solved this issue, please post how this was done. > Otherwise, it is a topic that I would like to raise for discussion. > Test Properties: > CLEAR LIBRARY CACHE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4739) Create public parallel test environment
[ https://issues.apache.org/jira/browse/HIVE-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland resolved HIVE-4739. Resolution: Fixed I am going to mark this resolved. Two PMC members have accounts and a third has expressed interest. Thanks guys! > Create public parallel test environment > --- > > Key: HIVE-4739 > URL: https://issues.apache.org/jira/browse/HIVE-4739 > Project: Hive > Issue Type: Improvement >Reporter: Brock Noland >Assignee: Brock Noland > > See discussion here > http://mail-archives.apache.org/mod_mbox/hive-dev/201306.mbox/%3CCAENxBwx%3Dc4VFLhR1zXZ3uWbP2BUFZc%2BZuiqF4SZY5zUZ9%2BUcxQ%40mail.gmail.com%3E -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3632) Upgrade datanucleus to support JDK7
[
https://issues.apache.org/jira/browse/HIVE-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714427#comment-13714427
]
Hive QA commented on HIVE-3632:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593335/HIVE-3632.1.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/110/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/110/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: InterruptedException: null
{noformat}
This message is automatically generated.
> Upgrade datanucleus to support JDK7
> ---
>
> Key: HIVE-3632
> URL: https://issues.apache.org/jira/browse/HIVE-3632
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.9.1, 0.10.0, 0.11.0
>Reporter: Chris Drome
>Assignee: Xuefu Zhang
>Priority: Critical
> Fix For: 0.12.0
>
> Attachments: HIVE-3632.1.patch, HIVE-3632.patch, HIVE-3632.patch.1
>
>
> I found serious problems with datanucleus code when using JDK7, resulting in
> some sort of exception being thrown when datanucleus code is entered.
> I tried source=1.7, target=1.7 with JDK7 as well as source=1.6, target=1.6
> with JDK7 and there was no visible difference in that the same unit tests
> failed.
> I tried upgrading datanucleus to 3.0.1, as per HIVE-2084.patch, which did not
> fix the failing tests.
> I tried upgrading datanucleus to 3.1-release, as per the advise of
> http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-86, which suggests
> using ASMv4 will allow datanucleus to work with JDK7. I was not successful
> with this either.
> I tried upgrading datanucleus to 3.1.2. I was not successful with this either.
> Regarding datanucleus support for JDK7+, there is the following JIRA
> http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-81
> which suggests that they don't plan to actively support JDK7+ bytecode any
> time soon.
> I also tested the following JVM parameters found on
> http://veerasundar.com/blog/2012/01/java-lang-verifyerror-expecting-a-stackmap-frame-at-branch-target-jdk-7/
> with no success either.
> This will become a more serious problem as people move to newer JVMs. If
> there are other who have solved this issue, please post how this was done.
> Otherwise, it is a topic that I would like to raise for discussion.
> Test Properties:
> CLEAR LIBRARY CACHE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Tez branch and tez based patches
I agree we are getting into grey area with the term disruptive. For reference ( I have not been doing this all the time bad on me) we are supposed to +1 and wait a day. >> I am not familiar with these other engines, but the short answer is that >> Tez is built to work on YARN, which works well for Hive since it is tied >> to Hadoop I understand what you are saying here yarn support is a plus. However the rest of the answer is something relevant to the discussion. There are already frameworks like spark that are semi popular. http://www.slideshare.net/jetlore/spark-and-shark-lightningfast-analytics-over-hadoop-and-hive-data. There are also other framworks like s4 http://incubator.apache.org/s4/, or storm. A big part of making a design decision is doing a competitive analysis. Usually asking yourself "What else for this is already out there?" or "Can this be done other ways?" I do want to be convinced we do not lock into tez too early with tunnel vision. Possibly we should be thinking on how to build hive in such a way that many different frameworks could plug in. In other words convincing that tez is the best choice, since many people are claiming an mrr type solution. I will watch the video you posted and study the material myself as well. On Wed, Jul 17, 2013 at 8:43 PM, Ashutosh Chauhan wrote: > On Wed, Jul 17, 2013 at 1:41 PM, Edward Capriolo >wrote: > > > > > "In my opinion we should limit the amount of tez related optimizations to > > and trunk" Refactoring that cleans up code is good, but as you have > pointed > > out there wont be a tez release until sometime this fall, and this branch > > will be open for an extended period of time. Thus code cleanups and other > > tez related refactoring does not need to be disruptive to trunk. > > > I agree Tez specific changes need not to go in trunk. But general > refactoring and code cleanup needs to happen on trunk as and when someone > is willing to work on those. We have to continually improve our code > quality. Code maintainability and readability is a priority. Without that > code quality suffers and discourages new contributors to contribute because > code is unnecessarily complicated. SemanticAnalyzer is 11K line class. We > need to simplify it. Patch like HIVE-4811 is a welcome change which tackled > it. Exec package is all convoluted which mixes up runtime operators and > drivers for runtime. Thats a welcome patch because it makes it much more > easy to read and reason about that piece of code. HIVE-4825 is another > example which improves modularity of code. For contributors who are exposed > to Hive first time it will be easier for them to follow the code. > > Rather than disruptive to trunk, they are constructive for trunk and I am > glad people are choosing to work on that. Tez or no Tez Hive is better off > with these patches. > > Thanks, > Ashutosh > > > > > On Wed, Jul 17, 2013 at 3:35 PM, Alan Gates > > wrote: > > > > > Answers to some of your questions inlined. > > > > > > Alan. > > > > > > On Jul 16, 2013, at 10:20 PM, Edward Capriolo wrote: > > > > > > > There are some points I want to bring up. First, I am on the PMC. > Here > > is > > > > something I find relevant: > > > > > > > > http://www.apache.org/foundation/how-it-works.html > > > > > > > > -- > > > > > > > > The role of the PMC from a Foundation perspective is oversight. The > > main > > > > role of the PMC is not code and not coding - but to ensure that all > > legal > > > > issues are addressed, that procedure is followed, and that each and > > every > > > > release is the product of the community as a whole. That is key to > our > > > > litigation protection mechanisms. > > > > > > > > Secondly the role of the PMC is to further the long term development > > and > > > > health of the community as a whole, and to ensure that balanced and > > wide > > > > scale peer review and collaboration does happen. Within the ASF we > > worry > > > > about any community which centers around a few individuals who are > > > working > > > > virtually uncontested. We believe that this is detrimental to > quality, > > > > stability, and robustness of both code and long term social > structures. > > > > > > > > > > > > > > > > > > > > > > https://blogs.apache.org/comdev/entry/what_makes_apache_projects_different > > > > > > > > - > > > > > > > > All other decisions happen on the dev list, discussions on the > private > > > list > > > > are kept to a minimum. > > > > > > > > "If it didn't happen on the dev list, it didn't happen" - which leads > > to: > > > > > > > > a) Elections of committers and PMC members are published on the dev > > list > > > > once finalized. > > > > > > > > b) Out-of-band discussions (IRC etc.) are summarized on the dev list > as > > > > soon as they have impact on the project, code or community. > > > > - > > > > > > > > https://issues.
[jira] [Commented] (HIVE-2905) Desc table can't show non-ascii comments
[ https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714453#comment-13714453 ] Edward Capriolo commented on HIVE-2905: --- Can we add some comments stating that unicode comments are ok bit not columns / tables. It might br nice to add to a faq language manual as well. > Desc table can't show non-ascii comments > > > Key: HIVE-2905 > URL: https://issues.apache.org/jira/browse/HIVE-2905 > Project: Hive > Issue Type: Bug > Components: CLI >Affects Versions: 0.7.0, 0.10.0 > Environment: hive 0.7.0, mysql 5.1.45 > hive 0.10.0, mysql 5.5.30 >Reporter: Sheng Zhou > Labels: patch > Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch > > > When desc a table with command line or hive jdbc way, the table's comment > can't be read. > 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml > file. >jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8 > 2. In mysql database, the comment field of COLUMNS table can be read normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [DISCUSS] moving website to Apache CMS
The hive site is not very big how long do you think it will take to move? Does anyone want to take point on this or help out? If so i say we vote on it and then get it done. On Thursday, July 18, 2013, Owen O'Malley wrote: > +1 to replacing forrest with markdown. > > > On Thu, Jul 18, 2013 at 7:39 AM, Alexander Alten-Lorenz < wget.n...@gmail.com >> wrote: > >> +1 (non - binding) >> >> :) >> >> Agree with Ed. Thanks to bringing this to attention, Brock. >> >> On Jul 18, 2013, at 7:37 AM, Edward Capriolo >> wrote: >> >> > I like the idea. markdown is easy to learn. The correct system uses >> forest >> > which is a big PITA. >> >> >
[jira] [Commented] (HIVE-3632) Upgrade datanucleus to support JDK7
[
https://issues.apache.org/jira/browse/HIVE-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714457#comment-13714457
]
Hive QA commented on HIVE-3632:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593335/HIVE-3632.1.patch
{color:green}SUCCESS:{color} +1 2646 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/111/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/111/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
> Upgrade datanucleus to support JDK7
> ---
>
> Key: HIVE-3632
> URL: https://issues.apache.org/jira/browse/HIVE-3632
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.9.1, 0.10.0, 0.11.0
>Reporter: Chris Drome
>Assignee: Xuefu Zhang
>Priority: Critical
> Fix For: 0.12.0
>
> Attachments: HIVE-3632.1.patch, HIVE-3632.patch, HIVE-3632.patch.1
>
>
> I found serious problems with datanucleus code when using JDK7, resulting in
> some sort of exception being thrown when datanucleus code is entered.
> I tried source=1.7, target=1.7 with JDK7 as well as source=1.6, target=1.6
> with JDK7 and there was no visible difference in that the same unit tests
> failed.
> I tried upgrading datanucleus to 3.0.1, as per HIVE-2084.patch, which did not
> fix the failing tests.
> I tried upgrading datanucleus to 3.1-release, as per the advise of
> http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-86, which suggests
> using ASMv4 will allow datanucleus to work with JDK7. I was not successful
> with this either.
> I tried upgrading datanucleus to 3.1.2. I was not successful with this either.
> Regarding datanucleus support for JDK7+, there is the following JIRA
> http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-81
> which suggests that they don't plan to actively support JDK7+ bytecode any
> time soon.
> I also tested the following JVM parameters found on
> http://veerasundar.com/blog/2012/01/java-lang-verifyerror-expecting-a-stackmap-frame-at-branch-target-jdk-7/
> with no success either.
> This will become a more serious problem as people move to newer JVMs. If
> there are other who have solved this issue, please post how this was done.
> Otherwise, it is a topic that I would like to raise for discussion.
> Test Properties:
> CLEAR LIBRARY CACHE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3632) Upgrade datanucleus to support JDK7
[ https://issues.apache.org/jira/browse/HIVE-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714463#comment-13714463 ] Edward Capriolo commented on HIVE-3632: --- Can we open a separate patch to fix the mapping file and trest and commit that first. > Upgrade datanucleus to support JDK7 > --- > > Key: HIVE-3632 > URL: https://issues.apache.org/jira/browse/HIVE-3632 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.9.1, 0.10.0, 0.11.0 >Reporter: Chris Drome >Assignee: Xuefu Zhang >Priority: Critical > Fix For: 0.12.0 > > Attachments: HIVE-3632.1.patch, HIVE-3632.patch, HIVE-3632.patch.1 > > > I found serious problems with datanucleus code when using JDK7, resulting in > some sort of exception being thrown when datanucleus code is entered. > I tried source=1.7, target=1.7 with JDK7 as well as source=1.6, target=1.6 > with JDK7 and there was no visible difference in that the same unit tests > failed. > I tried upgrading datanucleus to 3.0.1, as per HIVE-2084.patch, which did not > fix the failing tests. > I tried upgrading datanucleus to 3.1-release, as per the advise of > http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-86, which suggests > using ASMv4 will allow datanucleus to work with JDK7. I was not successful > with this either. > I tried upgrading datanucleus to 3.1.2. I was not successful with this either. > Regarding datanucleus support for JDK7+, there is the following JIRA > http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-81 > which suggests that they don't plan to actively support JDK7+ bytecode any > time soon. > I also tested the following JVM parameters found on > http://veerasundar.com/blog/2012/01/java-lang-verifyerror-expecting-a-stackmap-frame-at-branch-target-jdk-7/ > with no success either. > This will become a more serious problem as people move to newer JVMs. If > there are other who have solved this issue, please post how this was done. > Otherwise, it is a topic that I would like to raise for discussion. > Test Properties: > CLEAR LIBRARY CACHE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2659) SHOW FUNCTIONS still returns internal operators
[
https://issues.apache.org/jira/browse/HIVE-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714465#comment-13714465
]
Hive QA commented on HIVE-2659:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12508389/HIVE-2659.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/112/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/112/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-112/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'conf/hive-default.xml.template'
Reverted 'metastore/ivy.xml'
Reverted 'metastore/src/model/package.jdo'
Reverted 'metastore/build.xml'
Reverted 'ivy/libraries.properties'
Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
Reverted 'ql/ivy.xml'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build
hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build
hcatalog/hcatalog-pig-adapter/build common/src/gen
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1505155.
At revision 1505155.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
> SHOW FUNCTIONS still returns internal operators
> ---
>
> Key: HIVE-2659
> URL: https://issues.apache.org/jira/browse/HIVE-2659
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Carl Steinbach
>Assignee: Priyadarshini
> Attachments: HIVE-2659.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [DISCUSS] moving website to Apache CMS
Hi, I agree the site is small so I think it'd only take a few hours of actual work. Most the time would be waiting for infra to get the staging site setup and then do the cut over. I'd be willing to do the work. Could you call the vote? I think it has to be a PMC member. Cheers! Brock On Sat, Jul 20, 2013 at 10:33 AM, Edward Capriolo wrote: > The hive site is not very big how long do you think it will take to move? > Does anyone want to take point on this or help out? If so i say we vote on > it and then get it done. > > On Thursday, July 18, 2013, Owen O'Malley wrote: > > +1 to replacing forrest with markdown. > > > > > > On Thu, Jul 18, 2013 at 7:39 AM, Alexander Alten-Lorenz < > wget.n...@gmail.com > >> wrote: > > > >> +1 (non - binding) > >> > >> :) > >> > >> Agree with Ed. Thanks to bringing this to attention, Brock. > >> > >> On Jul 18, 2013, at 7:37 AM, Edward Capriolo > >> wrote: > >> > >> > I like the idea. markdown is easy to learn. The correct system uses > >> forest > >> > which is a big PITA. > >> > >> > > > -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
[jira] [Commented] (HIVE-4876) Beeling help text do not contain -f and -e parameters
[
https://issues.apache.org/jira/browse/HIVE-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714468#comment-13714468
]
Brock Noland commented on HIVE-4876:
[~jarcec] I suggest you rename the patch so the precommit tests will run. More
details here
https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing
> Beeling help text do not contain -f and -e parameters
> -
>
> Key: HIVE-4876
> URL: https://issues.apache.org/jira/browse/HIVE-4876
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
>Priority: Trivial
> Attachments: bugHIVE-4876.patch
>
>
> I've noticed that beeline do have support for {{-e}} and {{-f}} parameters,
> however those are not documented in the help text that can be retrieved by
> calling beeline with parameter {{-h}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4876) Beeling help text do not contain -f and -e parameters
[
https://issues.apache.org/jira/browse/HIVE-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-4876:
---
Status: Open (was: Patch Available)
> Beeling help text do not contain -f and -e parameters
> -
>
> Key: HIVE-4876
> URL: https://issues.apache.org/jira/browse/HIVE-4876
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Jarek Jarcec Cecho
>Assignee: Jarek Jarcec Cecho
>Priority: Trivial
> Attachments: bugHIVE-4876.patch
>
>
> I've noticed that beeline do have support for {{-e}} and {{-f}} parameters,
> however those are not documented in the help text that can be retrieved by
> calling beeline with parameter {{-h}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2659) SHOW FUNCTIONS still returns internal operators
[ https://issues.apache.org/jira/browse/HIVE-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-2659: --- Status: Open (was: Patch Available) Cancelling patch since it no longer applies. > SHOW FUNCTIONS still returns internal operators > --- > > Key: HIVE-2659 > URL: https://issues.apache.org/jira/browse/HIVE-2659 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Carl Steinbach >Assignee: Priyadarshini > Attachments: HIVE-2659.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3632) Upgrade datanucleus to support JDK7
[ https://issues.apache.org/jira/browse/HIVE-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714470#comment-13714470 ] Xuefu Zhang commented on HIVE-3632: --- I think that's a good idea. It avoids confusions. Will do that. > Upgrade datanucleus to support JDK7 > --- > > Key: HIVE-3632 > URL: https://issues.apache.org/jira/browse/HIVE-3632 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.9.1, 0.10.0, 0.11.0 >Reporter: Chris Drome >Assignee: Xuefu Zhang >Priority: Critical > Fix For: 0.12.0 > > Attachments: HIVE-3632.1.patch, HIVE-3632.patch, HIVE-3632.patch.1 > > > I found serious problems with datanucleus code when using JDK7, resulting in > some sort of exception being thrown when datanucleus code is entered. > I tried source=1.7, target=1.7 with JDK7 as well as source=1.6, target=1.6 > with JDK7 and there was no visible difference in that the same unit tests > failed. > I tried upgrading datanucleus to 3.0.1, as per HIVE-2084.patch, which did not > fix the failing tests. > I tried upgrading datanucleus to 3.1-release, as per the advise of > http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-86, which suggests > using ASMv4 will allow datanucleus to work with JDK7. I was not successful > with this either. > I tried upgrading datanucleus to 3.1.2. I was not successful with this either. > Regarding datanucleus support for JDK7+, there is the following JIRA > http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-81 > which suggests that they don't plan to actively support JDK7+ bytecode any > time soon. > I also tested the following JVM parameters found on > http://veerasundar.com/blog/2012/01/java-lang-verifyerror-expecting-a-stackmap-frame-at-branch-target-jdk-7/ > with no success either. > This will become a more serious problem as people move to newer JVMs. If > there are other who have solved this issue, please post how this was done. > Otherwise, it is a topic that I would like to raise for discussion. > Test Properties: > CLEAR LIBRARY CACHE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 2209 - Still Failing
Changes for Build #2169 Changes for Build #2170 [hashutosh] HIVE-4740 : HIVE2379 is missing hbase.jar itself (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4756 : Upgrade Hadoop 0.23 profile to 2.0.5-alpha (Brock Noland via Ashutosh Chauhan) [hashutosh] HIVE-4791 : improve test coverage of package org.apache.hadoop.hive.ql.udf.xml (Ivan Veselovsky via Ashutosh Chauhan) Changes for Build #2171 [hashutosh] HIVE-4618 : show create table creating unusable DDL when field delimiter is \001 (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4559 : hcatalog/webhcat scripts in tar.gz don't have execute permissions set (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4798 : NPE when we call isSame from an instance of ExprNodeConstantDesc with null value (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4781 : LEFT SEMI JOIN generates wrong results when the number of rows belonging to a single key of the right table exceed hive.join.emit.interval (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4647 : RetryingHMSHandler logs too many error messages (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4692 : Constant agg parameters will be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases (Navis via Ashutosh Chauhan) Changes for Build #2172 [hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original commit. Changes for Build #2173 Changes for Build #2174 [navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via Navis) [hashutosh] HIVE-4406 : Missing / or / in hs2 jdbc uri switches mode to embedded mode(Anandha Ranganathan via Ashutosh Chauhan) [hashutosh] HIVE-4430 : Semantic analysis fails in presence of certain literals in on clause (Kevin Wilfong via Ashutosh Chauhan) [hashutosh] HIVE-4757 : LazyTimestamp goes into irretrievable NULL mode once inited with NULL once (Gopal V via Ashutosh Chauhan) [hashutosh] HIVE-4785 : Implement isCaseSensitive for Hive JDBC driver (Robert Roland via Ashutosh Chauhan) Changes for Build #2175 [navis] HIVE-4436 : hive.exec.parallel=true doesn't work on hadoop-2 (Gopal V via Navis) Changes for Build #2176 Changes for Build #2177 [hashutosh] HIVE-4689 : For outerjoins, joinEmitInterval might make wrong result (Navis via Ashutosh Chauhan) [hashutosh] HIVE-3253 : ArrayIndexOutOfBounds exception for deeply nested structs (Thejas Nair via Ashutosh Chauhan) Changes for Build #2178 Changes for Build #2179 Changes for Build #2180 Changes for Build #2181 [hashutosh] HIVE-4089 : javax.jdo : jdo2-api dependency not in Maven Central (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-4804 parallel order by fails for small datasets (Navis via egc) Submitted by: Navis Reviewed by:Edward Capriolo Changes for Build #2182 Changes for Build #2183 [hashutosh] HIVE-4814 : Adjust WebHCat e2e tests until HIVE4703 is addressed (Eugene Koifman via Ashutosh Chauhan) Changes for Build #2184 [hashutosh] HIVE-4811 : (Slightly) break up the SemanticAnalyzer monstrosity (Gunther Hagleitner via Ashutosh Chauhan) Changes for Build #2185 [hashutosh] HIVE-4251 : Indices can't be built on tables whose schema info comes from SerDe (Mark Wagner via Ashutosh Chauhan) [hashutosh] HIVE-4805 : Enhance coverage of package org.apache.hadoop.hive.ql.exec.errors (Ivan Veselovsky via Ashutosh Chauhan) Changes for Build #2186 [hashutosh] HIVE-4733 : HiveLockObjectData is not compared properly (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-3475 INLINE UDTF does not convert types properly (Igor Kabiljo and Navis Ryu via egc) Submitted by: Navis Ryu and Igor Kabiljo Reviewed by:Edward Capriolo Changes for Build #2187 [hashutosh] HIVE-4802 : Fix url check for missing / or / after hostname in jdb uri (Thejas Nair via Ashutosh Chauhan) Changes for Build #2188 [hashutosh] HIVE-4813 : Improve test coverage of package org.apache.hadoop.hive.ql.optimizer.pcr (Ivan Veselovsky via Ashutosh Chauhan) [hashutosh] HIVE-4580 : Change DDLTask to report errors using canonical error messages rather than http status codes (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4796 : Increase coverage of package org.apache.hadoop.hive.common.metrics (Ivan Veselovsky via Ashutosh Chauhan) [navis] HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) [hashutosh] HIVE-3810 : HiveHistory.log need to replace \r with space before writing Entry.value to historyfile (Mark Grover via Ashutosh Chauhan) Changes for Build #2189 [hashutosh] HIVE-4810 [jira] Refactor exec package (Gunther Hagleitner via Ashutosh Chauhan) Summary: HIVE-4810 The exec package contains both operators and classes used to execute the job. Moving the latter into a sub package makes the package slightly more manageable and will make it easier to provide a tez-based implementation. Test Plan: Refactoring Reviewers: ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D11625 [hashutosh] HIVE-4829 : TestWebHCatE2
[jira] [Updated] (HIVE-4892) PTest2 cleanup after merge
[ https://issues.apache.org/jira/browse/HIVE-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4892: --- Description: HIVE-4675 was merged but there are still a few minor issues we need to cleanup: * README is out of date * Need to limit the number of failed source directories we copy back from the slaves * when looking for TEST-*.xml files we look at both the log directory (good) and the failed source directories (bad) therefore duplicating failures in jenkins report * We need to process bad hosts in the finally block of PTest.run (HIVE-4882) * Need a mechanism to clean the ivy and maven cache (HIVE-4882) * PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) * Now that PTest2 is committed to the source tree it's copying in our TEST-SomeTest*.xml files Test Properties: NO PRECOMMIT TESTS was: HIVE-4675 was merged but there are still a few minor issues we need to cleanup: * README is out of date * Need to limit the number of failed source directories we copy back from the slaves * when looking for TEST-*.xml files we look at both the log directory (good) and the failed source directories (bad) therefore duplicating failures in jenkins report * We need to process bad hosts in the finally block of PTest.run (HIVE-4882) * Need a mechanism to clean the ivy and maven cache (HIVE-4882) * PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) * Now that PTest2 is committed to the source tree it's copying in our TEST-SomeTest*.xml files > PTest2 cleanup after merge > -- > > Key: HIVE-4892 > URL: https://issues.apache.org/jira/browse/HIVE-4892 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Brock Noland > > HIVE-4675 was merged but there are still a few minor issues we need to > cleanup: > * README is out of date > * Need to limit the number of failed source directories we copy back from the > slaves > * when looking for TEST-*.xml files we look at both the log directory (good) > and the failed source directories (bad) therefore duplicating failures in > jenkins report > * We need to process bad hosts in the finally block of PTest.run (HIVE-4882) > * Need a mechanism to clean the ivy and maven cache (HIVE-4882) > * PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) > * Now that PTest2 is committed to the source tree it's copying in our > TEST-SomeTest*.xml files > Test Properties: > NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request 12790: HIVE-4892: PTest2 cleanup after merge
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12790/ --- Review request for hive. Bugs: HIVE-4892 https://issues.apache.org/jira/browse/HIVE-4892 Repository: hive-git Description --- Fixes the following issues: README is out of date Need to limit the number of failed source directories we copy back from the slaves when looking for TEST-*.xml files we look at both the log directory (good) and the failed source directories (bad) therefore duplicating failures in jenkins report We need to process bad hosts in the finally block of PTest.run (HIVE-4882) Need a mechanism to clean the ivy and maven cache (HIVE-4882) PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) Now that PTest2 is committed to the source tree it's copying in our TEST-SomeTest*.xml files Diffs - testutils/ptest2/README.md a483851 testutils/ptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java d6e9ce3 testutils/ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java b41726a testutils/ptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java 9b2125c testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/ExecutionPhase.java b5d7909 testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/HostExecutor.java 34a087c testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JIRAService.java 09c6d66 testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/JUnitReportParser.java 50cce25 testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java f4f5076 testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/ReportingPhase.java 023c26b testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/QFileTestBatch.java 722312a testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java fef2a3e testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/UnitTestBatch.java e911304 testutils/ptest2/src/main/resources/batch-exec.vm a35fcec testutils/ptest2/src/main/resources/source-prep.vm 7a92272 testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.java eec5560 testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.java 6fc649a testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestReportingPhase.testExecute.approved.txt 19bc3e7 testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.java 0631996 testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testBatch.approved.txt 9b4bb2f testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepGit.approved.txt 2decd3e testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepNone.approved.txt 7d8719d testutils/ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepSvn.approved.txt 0af317e testutils/ptest2/src/test/resources/TEST-SomeTest-success.xml PRE-CREATION Diff: https://reviews.apache.org/r/12790/diff/ Testing --- This code is currently running in our ptest environment. Thanks, Brock Noland
[jira] [Updated] (HIVE-4892) PTest2 cleanup after merge
[ https://issues.apache.org/jira/browse/HIVE-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4892: --- Attachment: HIVE-4892.patch https://reviews.apache.org/r/12790/ > PTest2 cleanup after merge > -- > > Key: HIVE-4892 > URL: https://issues.apache.org/jira/browse/HIVE-4892 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Brock Noland > Attachments: HIVE-4892.patch > > > HIVE-4675 was merged but there are still a few minor issues we need to > cleanup: > * README is out of date > * Need to limit the number of failed source directories we copy back from the > slaves > * when looking for TEST-*.xml files we look at both the log directory (good) > and the failed source directories (bad) therefore duplicating failures in > jenkins report > * We need to process bad hosts in the finally block of PTest.run (HIVE-4882) > * Need a mechanism to clean the ivy and maven cache (HIVE-4882) > * PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) > * Now that PTest2 is committed to the source tree it's copying in our > TEST-SomeTest*.xml files > Test Properties: > NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4892) PTest2 cleanup after merge
[ https://issues.apache.org/jira/browse/HIVE-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4892: --- Status: Patch Available (was: Open) > PTest2 cleanup after merge > -- > > Key: HIVE-4892 > URL: https://issues.apache.org/jira/browse/HIVE-4892 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Brock Noland > Attachments: HIVE-4892.patch > > > HIVE-4675 was merged but there are still a few minor issues we need to > cleanup: > * README is out of date > * Need to limit the number of failed source directories we copy back from the > slaves > * when looking for TEST-*.xml files we look at both the log directory (good) > and the failed source directories (bad) therefore duplicating failures in > jenkins report > * We need to process bad hosts in the finally block of PTest.run (HIVE-4882) > * Need a mechanism to clean the ivy and maven cache (HIVE-4882) > * PTest2 fails to publish a comment to a JIRA sometimes (HIVE-4889) > * Now that PTest2 is committed to the source tree it's copying in our > TEST-SomeTest*.xml files > Test Properties: > NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4900) Fix the mismatched column names in package.jdo
Xuefu Zhang created HIVE-4900: - Summary: Fix the mismatched column names in package.jdo Key: HIVE-4900 URL: https://issues.apache.org/jira/browse/HIVE-4900 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.11.0, 0.10.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 There are several errors in DataNucleus O-R mapping file, package.jdo, which are not complained by the existing DN version. These errors may be subject to future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it is still better if we fix these errors as it also creates some confusion in the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4901) Connection should be closed when Statement#execute() failed byTTransportException
Azrael Park created HIVE-4901:
-
Summary: Connection should be closed when Statement#execute()
failed byTTransportException
Key: HIVE-4901
URL: https://issues.apache.org/jira/browse/HIVE-4901
Project: Hive
Issue Type: Improvement
Components: JDBC
Affects Versions: 0.11.0
Environment: Ubuntu 11.10
Reporter: Azrael Park
Priority: Trivial
Currently, Connection is not closed when Statement#execute() failed by
TTransportException due to shutdown hiveserver. It means this connection can't
be used but connection status is still 'not closed'. It might be clear that
query execution got 'Connection Closed' exception if try to use this
connection.
{noformat}
Scenario is as following:
1. Client connect hiveserver.
2. Hiveserver shutdown.
3. Client execute query: throw 'Socket' exception and close connection.
4. Hiveserver start.
5. Client execute query: throw 'Connection Closed' exception.
6. May be reconnect.
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4901) Connection should be closed when Statement#execute() failed byTTransportException
[
https://issues.apache.org/jira/browse/HIVE-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Azrael Park updated HIVE-4901:
--
Attachment: HIVE-4901.patch
> Connection should be closed when Statement#execute() failed
> byTTransportException
> -
>
> Key: HIVE-4901
> URL: https://issues.apache.org/jira/browse/HIVE-4901
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 0.11.0
> Environment: Ubuntu 11.10
>Reporter: Azrael Park
>Priority: Trivial
> Attachments: HIVE-4901.patch
>
>
> Currently, Connection is not closed when Statement#execute() failed by
> TTransportException due to shutdown hiveserver. It means this connection
> can't be used but connection status is still 'not closed'. It might be clear
> that query execution got 'Connection Closed' exception if try to use this
> connection.
> {noformat}
> Scenario is as following:
> 1. Client connect hiveserver.
> 2. Hiveserver shutdown.
> 3. Client execute query: throw 'Socket' exception and close connection.
> 4. Hiveserver start.
> 5. Client execute query: throw 'Connection Closed' exception.
> 6. May be reconnect.
> {noformat}
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-4900: -- Attachment: HIVE-4900.patch > Fix the mismatched column names in package.jdo > -- > > Key: HIVE-4900 > URL: https://issues.apache.org/jira/browse/HIVE-4900 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0, 0.11.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang >Priority: Minor > Fix For: 0.12.0 > > Attachments: HIVE-4900.patch > > > There are several errors in DataNucleus O-R mapping file, package.jdo, which > are not complained by the existing DN version. These errors may be subject to > future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it > is still better if we fix these errors as it also creates some confusion in > the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4724) ORC readers should have a better error detection for non-ORC files
[ https://issues.apache.org/jira/browse/HIVE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-4724: Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. > ORC readers should have a better error detection for non-ORC files > -- > > Key: HIVE-4724 > URL: https://issues.apache.org/jira/browse/HIVE-4724 > Project: Hive > Issue Type: Bug > Components: File Formats >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-4724.D11529.2.patch, HIVE-4724.D11529.3.patch > > > A customer loaded a text file into a table that is stored as ORC. The error > message was very unfriendly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4724) ORC readers should have a better error detection for non-ORC files
[ https://issues.apache.org/jira/browse/HIVE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-4724: Fix Version/s: 0.12.0 0.11.1 > ORC readers should have a better error detection for non-ORC files > -- > > Key: HIVE-4724 > URL: https://issues.apache.org/jira/browse/HIVE-4724 > Project: Hive > Issue Type: Bug > Components: File Formats >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.11.1, 0.12.0 > > Attachments: HIVE-4724.D11529.2.patch, HIVE-4724.D11529.3.patch > > > A customer loaded a text file into a table that is stored as ORC. The error > message was very unfriendly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4062) use column statistics for ORC to evaluate predicates for ORC
[ https://issues.apache.org/jira/browse/HIVE-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HIVE-4062. - Resolution: Duplicate > use column statistics for ORC to evaluate predicates for ORC > > > Key: HIVE-4062 > URL: https://issues.apache.org/jira/browse/HIVE-4062 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Namit Jain > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4246) Implement predicate pushdown for ORC
[
https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714520#comment-13714520
]
Hive QA commented on HIVE-4246:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12588764/HIVE-4246.D11415.1.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/115/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/115/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-115/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ svn status --no-ignore
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
+ rm -rf build hcatalog/build hcatalog/core/build
hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build
hcatalog/hcatalog-pig-adapter/build common/src/gen
+ svn update
Uql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java
Uql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto
Uql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
Uql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java
Uql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
Uql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
U .
Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1505191.
Updated to revision 1505191.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
> Implement predicate pushdown for ORC
>
>
> Key: HIVE-4246
> URL: https://issues.apache.org/jira/browse/HIVE-4246
> Project: Hive
> Issue Type: New Feature
> Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-4246.D11415.1.patch
>
>
> By using the push down predicates from the table scan operator, ORC can skip
> over 10,000 rows at a time that won't satisfy the predicate. This will help a
> lot, especially if the file is sorted by the column that is used in the
> predicate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2055) Hive HBase Integration issue
[ https://issues.apache.org/jira/browse/HIVE-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714536#comment-13714536 ] Nick Dimiduk commented on HIVE-2055: Fair enough. I lack privileges to change this attribute. > Hive HBase Integration issue > > > Key: HIVE-2055 > URL: https://issues.apache.org/jira/browse/HIVE-2055 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.10.0 >Reporter: sajith v > Attachments: > 0001-HIVE-2055-include-hbase-dependencies-in-launch-scrip.patch, > HIVE-2055.patch > > > Created an external table in hive , which points to the HBase table. When > tried to query a column using the column name in select clause got the > following exception : ( java.lang.ClassNotFoundException: > org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat), errorCode:12, > SQLState:42000) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4900) Fix the mismatched column names in package.jdo
[ https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-4900: -- Status: Patch Available (was: Open) > Fix the mismatched column names in package.jdo > -- > > Key: HIVE-4900 > URL: https://issues.apache.org/jira/browse/HIVE-4900 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.11.0, 0.10.0 >Reporter: Xuefu Zhang >Assignee: Xuefu Zhang >Priority: Minor > Fix For: 0.12.0 > > Attachments: HIVE-4900.patch > > > There are several errors in DataNucleus O-R mapping file, package.jdo, which > are not complained by the existing DN version. These errors may be subject to > future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it > is still better if we fix these errors as it also creates some confusion in > the community. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4877) In ExecReducer, remove tag from the row which will be passed to the first Operator at the Reduce-side
[
https://issues.apache.org/jira/browse/HIVE-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-4877:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Yin!
> In ExecReducer, remove tag from the row which will be passed to the first
> Operator at the Reduce-side
> -
>
> Key: HIVE-4877
> URL: https://issues.apache.org/jira/browse/HIVE-4877
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Yin Huai
>Assignee: Yin Huai
> Fix For: 0.12.0
>
> Attachments: HIVE-4877.1.patch, HIVE-4877.patch
>
>
> In ExecReducer, we can see ...
> {code}
> row.clear();
> row.add(keyObject);
> row.add(valueObject[tag.get()]);
> // The tag is not used any more, we should remove it.
> row.add(tag);
> {\code}
> I have seen this for a long time. Actually, we are using
> {code}
> reducer.process(row, tag.get());
> {\code}
> to pass the tag.
> If we do not need to add the tag to a row, we should remove it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714573#comment-13714573 ] Phabricator commented on HIVE-4502: --- ashutoshc has accepted the revision "HIVE-4502 [jira] NPE - subquery smb joins fails". +1 REVISION DETAIL https://reviews.facebook.net/D10695 BRANCH HIVE-4502 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Cc: brock, vikram > NPE - subquery smb joins fails > -- > > Key: HIVE-4502 > URL: https://issues.apache.org/jira/browse/HIVE-4502 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: Vikram Dixit K >Assignee: Navis > Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, > HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, smb_mapjoin_25.q, > smb_mapjoin_25.q > > > Found this issue while running some SMB joins. Attaching test case that > causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4502: --- Status: Open (was: Patch Available) Canceling patch to trigger pre-commit test. > NPE - subquery smb joins fails > -- > > Key: HIVE-4502 > URL: https://issues.apache.org/jira/browse/HIVE-4502 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: Vikram Dixit K >Assignee: Navis > Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, > HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, > smb_mapjoin_25.q, smb_mapjoin_25.q > > > Found this issue while running some SMB joins. Attaching test case that > causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4502: --- Status: Patch Available (was: Open) > NPE - subquery smb joins fails > -- > > Key: HIVE-4502 > URL: https://issues.apache.org/jira/browse/HIVE-4502 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: Vikram Dixit K >Assignee: Navis > Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, > HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, > smb_mapjoin_25.q, smb_mapjoin_25.q > > > Found this issue while running some SMB joins. Attaching test case that > causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4502) NPE - subquery smb joins fails
[ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4502: --- Attachment: HIVE-4502.patch [~navis] I rebased your patch on trunk. Attaching here for reference. > NPE - subquery smb joins fails > -- > > Key: HIVE-4502 > URL: https://issues.apache.org/jira/browse/HIVE-4502 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.11.0 >Reporter: Vikram Dixit K >Assignee: Navis > Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, > HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch, > smb_mapjoin_25.q, smb_mapjoin_25.q > > > Found this issue while running some SMB joins. Attaching test case that > causes this error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4902) Figure out what to do about build/ql/tmp/hive.log
Edward Capriolo created HIVE-4902: - Summary: Figure out what to do about build/ql/tmp/hive.log Key: HIVE-4902 URL: https://issues.apache.org/jira/browse/HIVE-4902 Project: Hive Issue Type: Wish Reporter: Edward Capriolo Unit test tell me 'something failed look at this'. Nothing in that file seems relevant to anything. Seems like just a bunch of zookeeper mumbo jumbo that has nothing to do with why the test failed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3325) serde-reported partition cols should not be persisted in metastore
[ https://issues.apache.org/jira/browse/HIVE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714582#comment-13714582 ] Edward Capriolo commented on HIVE-3325: --- +1 Travis can you look at show create table as well. I noticed if you show create table on a table that dynamically gets all its columns from the serde the create table statement is not correct. Maybe we can handle that as a follow on. > serde-reported partition cols should not be persisted in metastore > -- > > Key: HIVE-3325 > URL: https://issues.apache.org/jira/browse/HIVE-3325 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Travis Crawford >Assignee: Travis Crawford > Attachments: HIVE-3325.patch.txt, > HIVE-3325_serde_reported_part_cols.1.patch > > > In HIVE-3279 an issue was fixed where serde-reported columns were not added > to partitions. However, the fix in that issue caused serde-reported columns > to be stored in the partition storage descriptor. > Serde-reported columns should be dynamic and only reported at runtime (not > stored in the partition storage descriptor). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2055) Hive HBase Integration issue
[ https://issues.apache.org/jira/browse/HIVE-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-2055: --- Status: Open (was: Patch Available) > Hive HBase Integration issue > > > Key: HIVE-2055 > URL: https://issues.apache.org/jira/browse/HIVE-2055 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.10.0 >Reporter: sajith v > Attachments: > 0001-HIVE-2055-include-hbase-dependencies-in-launch-scrip.patch, > HIVE-2055.patch > > > Created an external table in hive , which points to the HBase table. When > tried to query a column using the column name in select clause got the > following exception : ( java.lang.ClassNotFoundException: > org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat), errorCode:12, > SQLState:42000) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3632) Upgrade datanucleus to support JDK7
[ https://issues.apache.org/jira/browse/HIVE-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714593#comment-13714593 ] Brock Noland commented on HIVE-3632: These new libraries were causing most tests to fail. Until we finish this upgrade I have enabled deletion of the library cache on all builds. I assume this versioning issue with the libraries is an ivy problem? > Upgrade datanucleus to support JDK7 > --- > > Key: HIVE-3632 > URL: https://issues.apache.org/jira/browse/HIVE-3632 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.9.1, 0.10.0, 0.11.0 >Reporter: Chris Drome >Assignee: Xuefu Zhang >Priority: Critical > Fix For: 0.12.0 > > Attachments: HIVE-3632.1.patch, HIVE-3632.patch, HIVE-3632.patch.1 > > > I found serious problems with datanucleus code when using JDK7, resulting in > some sort of exception being thrown when datanucleus code is entered. > I tried source=1.7, target=1.7 with JDK7 as well as source=1.6, target=1.6 > with JDK7 and there was no visible difference in that the same unit tests > failed. > I tried upgrading datanucleus to 3.0.1, as per HIVE-2084.patch, which did not > fix the failing tests. > I tried upgrading datanucleus to 3.1-release, as per the advise of > http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-86, which suggests > using ASMv4 will allow datanucleus to work with JDK7. I was not successful > with this either. > I tried upgrading datanucleus to 3.1.2. I was not successful with this either. > Regarding datanucleus support for JDK7+, there is the following JIRA > http://www.datanucleus.org/servlet/jira/browse/NUCENHANCER-81 > which suggests that they don't plan to actively support JDK7+ bytecode any > time soon. > I also tested the following JVM parameters found on > http://veerasundar.com/blog/2012/01/java-lang-verifyerror-expecting-a-stackmap-frame-at-branch-target-jdk-7/ > with no success either. > This will become a more serious problem as people move to newer JVMs. If > there are other who have solved this issue, please post how this was done. > Otherwise, it is a topic that I would like to raise for discussion. > Test Properties: > CLEAR LIBRARY CACHE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-4827:
---
Description:
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver to
merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same MapReduce
job (Reduce side). Since this MapReduce job has two inputs, right now,
CommonJoinResolver cannot attach two MapJoins to the Map side of a MapReduce
job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
was:
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver to
merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key2) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same MapReduce
job (Reduce side). Since this MapReduce job has two inputs, right now,
CommonJoinResolver cannot attach two MapJoins to the Map side of a MapReduce
job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
> Merge a Map-only job to its following MapReduce job with multiple inputs
>
>
> Key: HIVE-4827
> URL: https://issues.apache.org/jira/browse/HIVE-4827
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Yin Huai
>Assignee: Yin Huai
>
> When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
> Map-only job (MapJoin) to its following MapReduce job. But this merge only
> happens when the MapReduce job has a single input. With Correlation Optimizer
> (HIVE-2206), it is possible that the MapReduce job can have multiple inputs
> (for multiple operation paths). It is desired to improve CommonJoinResolver
> to merge a Map-only job to the corresponding Map task of the MapReduce job.
> Example:
> {code:sql}
> set hive.optimize.correlation=true;
> set hive.auto.convert.join=true;
> set hive.optimize.mapjoin.mapreduce=true;
> SELECT tmp1.key, count(*)
> FROM (SELECT x1.key1 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> GROUP BY x1.key1) tmp1
> JOIN (SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2
[jira] [Updated] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-4827:
---
Description:
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver to
merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same MapReduce
job (Reduce side). Since this MapReduce job has two inputs, right now,
CommonJoinResolver cannot attach two MapJoins to the Map side of a MapReduce
job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
was:
When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
Map-only job (MapJoin) to its following MapReduce job. But this merge only
happens when the MapReduce job has a single input. With Correlation Optimizer
(HIVE-2206), it is possible that the MapReduce job can have multiple inputs
(for multiple operation paths). It is desired to improve CommonJoinResolver to
merge a Map-only job to the corresponding Map task of the MapReduce job.
Example:
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;
SELECT tmp1.key, count(*)
FROM (SELECT x1.key1 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
GROUP BY x1.key1) tmp1
JOIN (SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)
GROUP BY x2.key2) tmp2
ON (tmp1.key = tmp2.key)
GROUP BY tmp1.key;
{\code}
In this query, join operations inside tmp1 and tmp2 will be converted to two
MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
tmp1 and tmp2, and the last aggregation will be executed in the same MapReduce
job (Reduce side). Since this MapReduce job has two inputs, right now,
CommonJoinResolver cannot attach two MapJoins to the Map side of a MapReduce
job.
Another example:
{code:sql}
SELECT tmp1.key
FROM (SELECT x1.key2 AS key
FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
UNION ALL
SELECT x2.key2 AS key
FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
{\code}
For this case, we will have three Map-only jobs (two for MapJoins and one for
Union). It will be good to use a single Map-only job to execute this query.
> Merge a Map-only job to its following MapReduce job with multiple inputs
>
>
> Key: HIVE-4827
> URL: https://issues.apache.org/jira/browse/HIVE-4827
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Yin Huai
>Assignee: Yin Huai
>
> When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
> Map-only job (MapJoin) to its following MapReduce job. But this merge only
> happens when the MapReduce job has a single input. With Correlation Optimizer
> (HIVE-2206), it is possible that the MapReduce job can have multiple inputs
> (for multiple operation paths). It is desired to improve CommonJoinResolver
> to merge a Map-only job to the corresponding Map task of the MapReduce job.
> Example:
> {code:sql}
> set hive.optimize.correlation=true;
> set hive.auto.convert.join=true;
> set hive.optimize.mapjoin.mapreduce=true;
> SELECT tmp1.key, count(*)
> FROM (SELECT x1.key1 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> GROUP BY x1.key1) tmp1
> JOIN (SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2
Hive-trunk-h0.21 - Build # 2210 - Still Failing
Changes for Build #2169 Changes for Build #2170 [hashutosh] HIVE-4740 : HIVE2379 is missing hbase.jar itself (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4756 : Upgrade Hadoop 0.23 profile to 2.0.5-alpha (Brock Noland via Ashutosh Chauhan) [hashutosh] HIVE-4791 : improve test coverage of package org.apache.hadoop.hive.ql.udf.xml (Ivan Veselovsky via Ashutosh Chauhan) Changes for Build #2171 [hashutosh] HIVE-4618 : show create table creating unusable DDL when field delimiter is \001 (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4559 : hcatalog/webhcat scripts in tar.gz don't have execute permissions set (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4798 : NPE when we call isSame from an instance of ExprNodeConstantDesc with null value (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4781 : LEFT SEMI JOIN generates wrong results when the number of rows belonging to a single key of the right table exceed hive.join.emit.interval (Yin Huai via Ashutosh Chauhan) [hashutosh] HIVE-4647 : RetryingHMSHandler logs too many error messages (Navis via Ashutosh Chauhan) [hashutosh] HIVE-4692 : Constant agg parameters will be replaced by ExprNodeColumnDesc with single-sourced multi-gby cases (Navis via Ashutosh Chauhan) Changes for Build #2172 [hashutosh] HIVE-4781 : Adding new data files for tests. Missed in original commit. Changes for Build #2173 Changes for Build #2174 [navis] HIVE-2517 : Support group by on struct type (Ashutosh Chauhan via Navis) [hashutosh] HIVE-4406 : Missing / or / in hs2 jdbc uri switches mode to embedded mode(Anandha Ranganathan via Ashutosh Chauhan) [hashutosh] HIVE-4430 : Semantic analysis fails in presence of certain literals in on clause (Kevin Wilfong via Ashutosh Chauhan) [hashutosh] HIVE-4757 : LazyTimestamp goes into irretrievable NULL mode once inited with NULL once (Gopal V via Ashutosh Chauhan) [hashutosh] HIVE-4785 : Implement isCaseSensitive for Hive JDBC driver (Robert Roland via Ashutosh Chauhan) Changes for Build #2175 [navis] HIVE-4436 : hive.exec.parallel=true doesn't work on hadoop-2 (Gopal V via Navis) Changes for Build #2176 Changes for Build #2177 [hashutosh] HIVE-4689 : For outerjoins, joinEmitInterval might make wrong result (Navis via Ashutosh Chauhan) [hashutosh] HIVE-3253 : ArrayIndexOutOfBounds exception for deeply nested structs (Thejas Nair via Ashutosh Chauhan) Changes for Build #2178 Changes for Build #2179 Changes for Build #2180 Changes for Build #2181 [hashutosh] HIVE-4089 : javax.jdo : jdo2-api dependency not in Maven Central (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-4804 parallel order by fails for small datasets (Navis via egc) Submitted by: Navis Reviewed by:Edward Capriolo Changes for Build #2182 Changes for Build #2183 [hashutosh] HIVE-4814 : Adjust WebHCat e2e tests until HIVE4703 is addressed (Eugene Koifman via Ashutosh Chauhan) Changes for Build #2184 [hashutosh] HIVE-4811 : (Slightly) break up the SemanticAnalyzer monstrosity (Gunther Hagleitner via Ashutosh Chauhan) Changes for Build #2185 [hashutosh] HIVE-4251 : Indices can't be built on tables whose schema info comes from SerDe (Mark Wagner via Ashutosh Chauhan) [hashutosh] HIVE-4805 : Enhance coverage of package org.apache.hadoop.hive.ql.exec.errors (Ivan Veselovsky via Ashutosh Chauhan) Changes for Build #2186 [hashutosh] HIVE-4733 : HiveLockObjectData is not compared properly (Navis via Ashutosh Chauhan) [ecapriolo] HIVE-3475 INLINE UDTF does not convert types properly (Igor Kabiljo and Navis Ryu via egc) Submitted by: Navis Ryu and Igor Kabiljo Reviewed by:Edward Capriolo Changes for Build #2187 [hashutosh] HIVE-4802 : Fix url check for missing / or / after hostname in jdb uri (Thejas Nair via Ashutosh Chauhan) Changes for Build #2188 [hashutosh] HIVE-4813 : Improve test coverage of package org.apache.hadoop.hive.ql.optimizer.pcr (Ivan Veselovsky via Ashutosh Chauhan) [hashutosh] HIVE-4580 : Change DDLTask to report errors using canonical error messages rather than http status codes (Eugene Koifman via Ashutosh Chauhan) [hashutosh] HIVE-4796 : Increase coverage of package org.apache.hadoop.hive.common.metrics (Ivan Veselovsky via Ashutosh Chauhan) [navis] HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) [hashutosh] HIVE-3810 : HiveHistory.log need to replace \r with space before writing Entry.value to historyfile (Mark Grover via Ashutosh Chauhan) Changes for Build #2189 [hashutosh] HIVE-4810 [jira] Refactor exec package (Gunther Hagleitner via Ashutosh Chauhan) Summary: HIVE-4810 The exec package contains both operators and classes used to execute the job. Moving the latter into a sub package makes the package slightly more manageable and will make it easier to provide a tez-based implementation. Test Plan: Refactoring Reviewers: ashutoshc Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D11625 [hashutosh] HIVE-4829 : TestWebHCatE2
[jira] [Commented] (HIVE-4900) Fix the mismatched column names in package.jdo
[
https://issues.apache.org/jira/browse/HIVE-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714626#comment-13714626
]
Hive QA commented on HIVE-4900:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593356/HIVE-4900.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2646 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/116/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/116/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
> Fix the mismatched column names in package.jdo
> --
>
> Key: HIVE-4900
> URL: https://issues.apache.org/jira/browse/HIVE-4900
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.10.0, 0.11.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Minor
> Fix For: 0.12.0
>
> Attachments: HIVE-4900.patch
>
>
> There are several errors in DataNucleus O-R mapping file, package.jdo, which
> are not complained by the existing DN version. These errors may be subject to
> future DN complaint (as experienced in HIVE-3632 and HIVE-2084). However, it
> is still better if we fix these errors as it also creates some confusion in
> the community.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request 12795: [HIVE-4827] Merge a Map-only job to its following MapReduce job with multiple inputs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12795/ --- Review request for hive. Bugs: HIVE-4827 https://issues.apache.org/jira/browse/HIVE-4827 Repository: hive-git Description --- https://issues.apache.org/jira/browse/HIVE-4827 Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 66b84ff ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java f98878c ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 7cbb1ff ql/src/test/queries/clientpositive/multiMapJoin2.q PRE-CREATION ql/src/test/results/clientpositive/multiMapJoin2.q.out PRE-CREATION Diff: https://reviews.apache.org/r/12795/diff/ Testing --- Running tests. Thanks, Yin Huai
[jira] [Updated] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated HIVE-4827:
---
Attachment: HIVE-4827.1.patch
> Merge a Map-only job to its following MapReduce job with multiple inputs
>
>
> Key: HIVE-4827
> URL: https://issues.apache.org/jira/browse/HIVE-4827
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Yin Huai
>Assignee: Yin Huai
> Attachments: HIVE-4827.1.patch
>
>
> When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
> Map-only job (MapJoin) to its following MapReduce job. But this merge only
> happens when the MapReduce job has a single input. With Correlation Optimizer
> (HIVE-2206), it is possible that the MapReduce job can have multiple inputs
> (for multiple operation paths). It is desired to improve CommonJoinResolver
> to merge a Map-only job to the corresponding Map task of the MapReduce job.
> Example:
> {code:sql}
> set hive.optimize.correlation=true;
> set hive.auto.convert.join=true;
> set hive.optimize.mapjoin.mapreduce=true;
> SELECT tmp1.key, count(*)
> FROM (SELECT x1.key1 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> GROUP BY x1.key1) tmp1
> JOIN (SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
> GROUP BY x2.key2) tmp2
> ON (tmp1.key = tmp2.key)
> GROUP BY tmp1.key;
> {\code}
> In this query, join operations inside tmp1 and tmp2 will be converted to two
> MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
> tmp1 and tmp2, and the last aggregation will be executed in the same
> MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
> now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
> MapReduce job.
> Another example:
> {code:sql}
> SELECT tmp1.key
> FROM (SELECT x1.key2 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> UNION ALL
> SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
> {\code}
> For this case, we will have three Map-only jobs (two for MapJoins and one for
> Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4827) Merge a Map-only job to its following MapReduce job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714648#comment-13714648
]
Yin Huai commented on HIVE-4827:
I am running tests. Will see if there is any result which needs to be updated
or any bug in my patch.
> Merge a Map-only job to its following MapReduce job with multiple inputs
>
>
> Key: HIVE-4827
> URL: https://issues.apache.org/jira/browse/HIVE-4827
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Yin Huai
>Assignee: Yin Huai
> Attachments: HIVE-4827.1.patch
>
>
> When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
> Map-only job (MapJoin) to its following MapReduce job. But this merge only
> happens when the MapReduce job has a single input. With Correlation Optimizer
> (HIVE-2206), it is possible that the MapReduce job can have multiple inputs
> (for multiple operation paths). It is desired to improve CommonJoinResolver
> to merge a Map-only job to the corresponding Map task of the MapReduce job.
> Example:
> {code:sql}
> set hive.optimize.correlation=true;
> set hive.auto.convert.join=true;
> set hive.optimize.mapjoin.mapreduce=true;
> SELECT tmp1.key, count(*)
> FROM (SELECT x1.key1 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> GROUP BY x1.key1) tmp1
> JOIN (SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
> GROUP BY x2.key2) tmp2
> ON (tmp1.key = tmp2.key)
> GROUP BY tmp1.key;
> {\code}
> In this query, join operations inside tmp1 and tmp2 will be converted to two
> MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
> tmp1 and tmp2, and the last aggregation will be executed in the same
> MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
> now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
> MapReduce job.
> Another example:
> {code:sql}
> SELECT tmp1.key
> FROM (SELECT x1.key2 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> UNION ALL
> SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
> {\code}
> For this case, we will have three Map-only jobs (two for MapJoins and one for
> Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
[
https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714652#comment-13714652
]
Hive QA commented on HIVE-4502:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12593370/HIVE-4502.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 2647 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/117/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/117/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
> NPE - subquery smb joins fails
> --
>
> Key: HIVE-4502
> URL: https://issues.apache.org/jira/browse/HIVE-4502
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: Vikram Dixit K
>Assignee: Navis
> Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch,
> HIVE-4502.D10695.3.patch, HIVE-4502.D10695.4.patch, HIVE-4502.patch,
> smb_mapjoin_25.q, smb_mapjoin_25.q
>
>
> Found this issue while running some SMB joins. Attaching test case that
> causes this error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
