[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736090#comment-13736090 ] Hudson commented on HIVE-4930: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4930 : Classes of metastore should not be included MR-task (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511541) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Fix For: 0.12.0 Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736139#comment-13736139 ] Hudson commented on HIVE-4930: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-4930 : Classes of metastore should not be included MR-task (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511541) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Fix For: 0.12.0 Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733527#comment-13733527 ] Hudson commented on HIVE-4930: -- SUCCESS: Integrated in Hive-trunk-h0.21 #2251 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2251/]) HIVE-4930 : Classes of metastore should not be included MR-task (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511541) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Fix For: 0.12.0 Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[
https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720013#comment-13720013
]
Hive QA commented on HIVE-4930:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594111/HIVE-4930.D11817.1.patch
{color:green}SUCCESS:{color} +1 2651 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/184/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/184/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Classes of metastore should not be included MR-task
---
Key: HIVE-4930
URL: https://issues.apache.org/jira/browse/HIVE-4930
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Attachments: HIVE-4930.D11817.1.patch
metastore.api.SkewedValueList is included in FileSinkOperator, which should
not be.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720201#comment-13720201 ] Ashutosh Chauhan commented on HIVE-4930: Patch looks benign, though I am wondering whats the motivation? Is it that we dont want to refer to metastore classes from backend ? Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720277#comment-13720277 ] Navis commented on HIVE-4930: - I remember classes in metastore does not included in job.jar for MR tasks. Isn't it? Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720287#comment-13720287 ] Ashutosh Chauhan commented on HIVE-4930: yes.. thats correct, thats why I am confused. Whats the motivation for the jira? Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720296#comment-13720296 ] Navis commented on HIVE-4930: - FileSinkOperator refs SkewedValueList which is in metastore. I wonder why ClassNotFoundException is not thrown in MR task. Or, am I misunderstanding something? Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720313#comment-13720313 ] Ashutosh Chauhan commented on HIVE-4930: Correct and thats confusing to me as well. I didnt understand the issue of this jira. Than I guessed it is that because FSOp refers to metastore class it must be resulting in CNF exception from backend which is what you are trying to fix, but clearly we have not seeing that happening. So, I am double confused now : ) 1) Why Classcast exception not happening? 2) If thats true (though not understood) what you are trying to fix here anyways? Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[
https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720332#comment-13720332
]
Navis commented on HIVE-4930:
-
Confirmed it's happening in real hadoop.
{noformat}
java.lang.RuntimeException: java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/metastore/api/SkewedValueList
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/metastore/api/SkewedValueList
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.generateListBucketingDirName(FileSinkOperator.java:737)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:564)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:848)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:848)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:501)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:848)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:657)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.metastore.api.SkewedValueList
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 20 more
{noformat}
I thinks metastore.jar is on classpath for tests(TestCliDriver or
TestMinimrCliDriver, either)
Classes of metastore should not be included MR-task
---
Key: HIVE-4930
URL: https://issues.apache.org/jira/browse/HIVE-4930
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Attachments: HIVE-4930.D11817.1.patch
metastore.api.SkewedValueList is included in FileSinkOperator, which should
not be.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13720383#comment-13720383 ] Phabricator commented on HIVE-4930: --- ashutoshc has accepted the revision HIVE-4930 [jira] Classes of metastore should not be included MR-task. +1 INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:996 It will be good to add a comment here saying that we do this so that we need not to ship metastore jars to backend. REVISION DETAIL https://reviews.facebook.net/D11817 BRANCH HIVE-4930 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
