[jira] [Commented] (HIVE-12687) LLAP Workdirs need to default to YARN local
[
https://issues.apache.org/jira/browse/HIVE-12687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069534#comment-15069534
]
Hive QA commented on HIVE-12687:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779100/HIVE-12687.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9976 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6455/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6455/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6455/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779100 - PreCommit-HIVE-TRUNK-Build
> LLAP Workdirs need to default to YARN local
> ---
>
> Key: HIVE-12687
> URL: https://issues.apache.org/jira/browse/HIVE-12687
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12687.01.patch, HIVE-12687.01.patch,
> HIVE-12687.patch
>
>
> {code}
>LLAP_DAEMON_WORK_DIRS("hive.llap.daemon.work.dirs", ""
> {code}
> is a bad default & fails at startup if not overridden.
> A better default would be to fall back onto YARN local dirs if this is not
> configured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12736) It seems that result of Hive on Spark be mistaken and result of Hive and Hive on Spark are not the same
[ https://issues.apache.org/jira/browse/HIVE-12736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JoneZhang updated HIVE-12736: - Summary: It seems that result of Hive on Spark be mistaken and result of Hive and Hive on Spark are not the same (was: It seems that result of Hive on Spark is mistake And result of Hive and Hive on Spark are not the same) > It seems that result of Hive on Spark be mistaken and result of Hive and Hive > on Spark are not the same > --- > > Key: HIVE-12736 > URL: https://issues.apache.org/jira/browse/HIVE-12736 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.1, 1.2.1 >Reporter: JoneZhang > > select * from staff; > 1 jone22 1 > 2 lucy21 1 > 3 hmm 22 2 > 4 james 24 3 > 5 xiaoliu 23 3 > select id,date_ from trade union all select id,"test" from trade ; > 1 201510210908 > 2 201509080234 > 2 201509080235 > 1 test > 2 test > 2 test > set hive.execution.engine=spark; > set spark.master=local; > select /*+mapjoin(t)*/ * from staff s join > (select id,date_ from trade union all select id,"test" from trade ) t on > s.id=t.id; > 1 jone22 1 1 201510210908 > 2 lucy21 1 2 201509080234 > 2 lucy21 1 2 201509080235 > set hive.execution.engine=mr; > select /*+mapjoin(t)*/ * from staff s join > (select id,date_ from trade union all select id,"test" from trade ) t on > s.id=t.id; > FAILED: SemanticException [Error 10227]: Not all clauses are supported with > mapjoin hint. Please remove mapjoin hint. > I have two questions > 1.Why result of hive on spark not include the following record? > 1 jone22 1 1 test > 2 lucy21 1 2 test > 2 lucy21 1 2 test > 2.Why there are two different ways of dealing same query? > explain 1: > set hive.execution.engine=spark; > set spark.master=local; > explain > select id,date_ from trade union all select id,"test" from trade; > OK > STAGE DEPENDENCIES: > Stage-1 is a root stage > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-1 > Spark > DagName: jonezhang_20151222191643_5301d90a-caf0-4934-8092-d165c87a4190:1 > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: trade > Statistics: Num rows: 6 Data size: 48 Basic stats: COMPLETE > Column stats: NONE > Select Operator > expressions: id (type: int), date_ (type: string) > outputColumnNames: _col0, _col1 > Statistics: Num rows: 6 Data size: 48 Basic stats: > COMPLETE Column stats: NONE > File Output Operator > compressed: false > Statistics: Num rows: 12 Data size: 96 Basic stats: > COMPLETE Column stats: NONE > table: > input format: > org.apache.hadoop.mapred.TextInputFormat > output format: > org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat > serde: > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe > Map 2 > Map Operator Tree: > TableScan > alias: trade > Statistics: Num rows: 6 Data size: 48 Basic stats: COMPLETE > Column stats: NONE > Select Operator > expressions: id (type: int), 'test' (type: string) > outputColumnNames: _col0, _col1 > Statistics: Num rows: 6 Data size: 48 Basic stats: > COMPLETE Column stats: NONE > File Output Operator > compressed: false > Statistics: Num rows: 12 Data size: 96 Basic stats: > COMPLETE Column stats: NONE > table: > input format: > org.apache.hadoop.mapred.TextInputFormat > output format: > org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat > serde: > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe > Stage: Stage-0 > Fetch Operator > limit: -1 > Processor Tree: > ListSink > explain 2: > set hive.execution.engine=spark; > set spark.master=local; > explain > select /*+mapjoin(t)*/ * from staff s join > (select id,date_ from trade union all select id,"test" from trade ) t on > s.id=t.id; > OK > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > DagName:
[jira] [Commented] (HIVE-6113) Upgrade DataNucleus [was: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient]
[
https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069683#comment-15069683
]
Hive QA commented on HIVE-6113:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779112/HIVE-6113.10.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9961 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6457/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6457/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6457/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779112 - PreCommit-HIVE-TRUNK-Build
> Upgrade DataNucleus [was: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient]
> -
>
> Key: HIVE-6113
> URL: https://issues.apache.org/jira/browse/HIVE-6113
> Project: Hive
> Issue Type: Bug
> Components: Database/Schema
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1
> Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0
>Reporter: William Stone
>Assignee: Oleksiy Sayankin
>Priority: Critical
> Labels: HiveMetaStoreClient, metastore, unable_instantiate
> Attachments: HIVE-6113-2.patch, HIVE-6113.10.patch,
> HIVE-6113.3.patch, HIVE-6113.4.patch, HIVE-6113.5.patch, HIVE-6113.6.patch,
> HIVE-6113.7.patch, HIVE-6113.8.patch, HIVE-6113.9.patch, HIVE-6113.patch,
> HIVE-6113.with.reflection.patch
>
>
> CLEAR LIBRARY CACHE
> When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw
> Error as followed.
> but when I tryit again , It will success.
> 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) -
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException:
> Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260)
> at
[jira] [Resolved] (HIVE-7181) Beginner User On Apache Jira
[ https://issues.apache.org/jira/browse/HIVE-7181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Dossett resolved HIVE-7181. - Resolution: Not A Problem > Beginner User On Apache Jira > > > Key: HIVE-7181 > URL: https://issues.apache.org/jira/browse/HIVE-7181 > Project: Hive > Issue Type: Wish >Reporter: Nishant Kelkar >Priority: Minor > Labels: documentation, newbie > > Hi All! > I've just started to use Apache's Jira board (I registered today). I've used > Jira for my work before, so I know how to navigate within Jira. But my main > question, was understanding how issues are handled in the open source > community (to which I want to contribute, but I'm a noob here too). So > basically, a person comes up with a ticket when he/she thinks that the issue > they are facing, is a bug/improvement. > Questions: > 1. Whom am I supposed to assign the ticket to? (myself?) > 2. Who would be the QA assignee? > 3. If addressing the issue requires looking at the code, how am I supposed to > change the code and bring into effect those changes? (At work, we maintain a > Git repo on our private server. So everyone always has access to the latest > code). > 4. Where can I find a list of all the people who are active on this project > (Hive)? It would be nice if I could tag people by their names in my ticket > comments. > 5. Where can I find well formatted documentation about how to take issues > from discovery to fixture on Apache Jira? > I apologize in advance, if my questions are too simple. > Thanks, and any/all help is appreciated! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12577) NPE in LlapTaskCommunicator when unregistering containers
[
https://issues.apache.org/jira/browse/HIVE-12577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069929#comment-15069929
]
Hive QA commented on HIVE-12577:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779119/HIVE-12577.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9962 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6458/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6458/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6458/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779119 - PreCommit-HIVE-TRUNK-Build
> NPE in LlapTaskCommunicator when unregistering containers
> -
>
> Key: HIVE-12577
> URL: https://issues.apache.org/jira/browse/HIVE-12577
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.0.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
>Priority: Critical
> Attachments: HIVE-12577.03.patch, HIVE-12577.1.review.txt,
> HIVE-12577.1.txt, HIVE-12577.1.wip.txt, HIVE-12577.2.review.txt,
> HIVE-12577.2.txt, HIVE-12577.3.patch, HIVE-12577.3.txt, HIVE-12577.4.patch
>
>
> {code}
> 2015-12-02 13:29:00,160 [ERROR] [Dispatcher thread {Central}]
> |common.AsyncDispatcher|: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$EntityTracker.unregisterContainer(LlapTaskCommunicator.java:586)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.registerContainerEnd(LlapTaskCommunicator.java:188)
> at
> org.apache.tez.dag.app.TaskCommunicatorManager.unregisterRunningContainer(TaskCommunicatorManager.java:389)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl.unregisterFromTAListener(AMContainerImpl.java:1121)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtLaunchingTransition.transition(AMContainerImpl.java:699)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtIdleTransition.transition(AMContainerImpl.java:805)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:892)
> at
> org.apache.tez.dag.app.rm.container.AMContainerImpl$StopRequestAtRunningTransition.transition(AMContainerImpl.java:887)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
>
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070080#comment-15070080
]
Gunther Hagleitner commented on HIVE-12738:
---
The fix assumes that there are two parents... This could be an n-way join. In
general, isn't there a more robust way to fix this? There are other ways of
having multi parents.
I see two fixes: One where the function just searches all branches, probably
depth first. The other where we avoid backtracking and just terminate the walk
when we find a vectorized reduce that produces rows. That should be sufficient
no?
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12075) add analyze command to explictly cache file metadata in HBase metastore
[ https://issues.apache.org/jira/browse/HIVE-12075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070038#comment-15070038 ] Thejas M Nair commented on HIVE-12075: -- The org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch test failure is caused by this patch. It was not an 'unrelated' failure. > add analyze command to explictly cache file metadata in HBase metastore > --- > > Key: HIVE-12075 > URL: https://issues.apache.org/jira/browse/HIVE-12075 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-12075.01.nogen.patch, HIVE-12075.01.patch, > HIVE-12075.02.patch, HIVE-12075.03.patch, HIVE-12075.04.patch, > HIVE-12075.nogen.patch, HIVE-12075.patch > > > ANALYZE TABLE (spec as usual) CACHE METADATA -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070074#comment-15070074
]
Gunther Hagleitner commented on HIVE-12738:
---
This should be reproducible w/o adding large new tables to the ut framework.
The query shape is pretty simple.
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11470) NPE in DynamicPartFileRecordWriterContainer on null part-keys.
[
https://issues.apache.org/jira/browse/HIVE-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070173#comment-15070173
]
Hive QA commented on HIVE-11470:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779124/HIVE-11470.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9961 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6459/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6459/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6459/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779124 - PreCommit-HIVE-TRUNK-Build
> NPE in DynamicPartFileRecordWriterContainer on null part-keys.
> --
>
> Key: HIVE-11470
> URL: https://issues.apache.org/jira/browse/HIVE-11470
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 1.2.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-11470.1.patch, HIVE-11470.2.patch
>
>
> When partitioning data using {{HCatStorer}}, one sees the following NPE, if
> the dyn-part-key is of null-value:
> {noformat}
> 2015-07-30 23:59:59,627 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.io.IOException: java.lang.NullPointerException
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:473)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:436)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:416)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hive.hcatalog.mapreduce.DynamicPartitionFileRecordWriterContainer.getLocalFileWriter(DynamicPartitionFileRecordWriterContainer.java:141)
> at
> org.apache.hive.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:110)
> at
>
[jira] [Commented] (HIVE-12674) HS2 Tez sessions should have maximum age
[ https://issues.apache.org/jira/browse/HIVE-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070184#comment-15070184 ] Sergey Shelukhin commented on HIVE-12674: - Test failures are unrelated. > HS2 Tez sessions should have maximum age > > > Key: HIVE-12674 > URL: https://issues.apache.org/jira/browse/HIVE-12674 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12674.01.patch, HIVE-12674.01.patch, > HIVE-12674.02.patch, HIVE-12674.03.patch, HIVE-12674.patch > > > Certain tokens passed to AM by clients (e.g. an HDFS token) have maximum > lifetime beyond which they cannot be renewed. We should cycle long-lived > session AMs after a configurable period to avoid problems with these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070338#comment-15070338
]
Matt McCline commented on HIVE-12738:
-
Thanks for producing a better test case.
In a few minutes I'll post a new patch that stops validating or vectorizing
after a vectorized GROUP BY that outputs rows. It does make the major
assumption we don't have multiple table scans in vectorization and that we are
validating / vectorizing a single GROUP BY operator.
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch, HIVE-12738.2.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-12738:
Attachment: HIVE-12738.3.patch
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch, HIVE-12738.2.patch,
> HIVE-12738.3.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12741) HS2 ShutdownHookManager holds extra of Driver instance in master/branch-2.0
[ https://issues.apache.org/jira/browse/HIVE-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12741: - Attachment: HIVE-12741.1.patch > HS2 ShutdownHookManager holds extra of Driver instance in master/branch-2.0 > --- > > Key: HIVE-12741 > URL: https://issues.apache.org/jira/browse/HIVE-12741 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.0.0, 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-12741.1.patch > > > HIVE-12187 was meant to fix the described memory leak, however because of > interaction with HIVE-12187 in branch-2.0/master, the fix fails to take > effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12741) HS2 ShutdownHookManager holds extra of Driver instance in master/branch-2.0
[ https://issues.apache.org/jira/browse/HIVE-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-12741: - Description: HIVE-12187 was meant to fix the described memory leak, however because of interaction with HIVE-12187 in branch-2.0/master, the fix fails to take effect. was: HIVE-12187 was meant to fix the described memory leak, however because of interaction with HIVE-11488 in branch-2.0/master, the fix fails to take effect. > HS2 ShutdownHookManager holds extra of Driver instance in master/branch-2.0 > --- > > Key: HIVE-12741 > URL: https://issues.apache.org/jira/browse/HIVE-12741 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.0.0, 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > > HIVE-12187 was meant to fix the described memory leak, however because of > interaction with HIVE-12187 in branch-2.0/master, the fix fails to take > effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[
https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070490#comment-15070490
]
Pengcheng Xiong commented on HIVE-12707:
I got the following error when I compile it locally
{code}
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/pxiong/Projects/cbo/hive/ql/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveParser_IdentifiersParser.java:[13450,27]
code too large
[INFO] 1 error
{code}
I guess the grammar after adding interval becomes too complicated.
> ParseException for column name 'INTERVAL', even set
> hive.support.sql11.reserved.keywords = false
> -
>
> Key: HIVE-12707
> URL: https://issues.apache.org/jira/browse/HIVE-12707
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Rajit Saha
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12707.01.patch
>
>
> Any column name INTERVAL is causing ParseException , even we set
> hive.support.sql11.reserved.keywords = false
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12743) RCFileInputFormat needs to be registered with kryo
[
https://issues.apache.org/jira/browse/HIVE-12743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070563#comment-15070563
]
Prasanth Jayachandran commented on HIVE-12743:
--
The cleaner solution is to add auto-registration support in kryo. But that's
more involved. Mostly kryo will work even without registration using FQCN but
sometimes exceptions like these happen. Not sure why.
> RCFileInputFormat needs to be registered with kryo
> --
>
> Key: HIVE-12743
> URL: https://issues.apache.org/jira/browse/HIVE-12743
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12743.1.patch
>
>
> Ran into an issue with union distinct query that uses RCFile table with the
> following exception
> {code}
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12429) Switch default Hive authorization to SQLStandardAuth in 2.0
[
https://issues.apache.org/jira/browse/HIVE-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070564#comment-15070564
]
Hive QA commented on HIVE-12429:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779218/HIVE-12429.8.patch
{color:green}SUCCESS:{color} +1 due to 55 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 9976 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_binary
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_external_table_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries_prefix
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_bulk
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_joins
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_null_first_col
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_join
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_scan_params
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_timestamp
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_timestamp_format
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_generatehfiles_require_family_path
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_selectDistinctStar
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_noscan_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_unionDistinct_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_multi_insert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_timestamp_funcs
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_join_nonexistent_part
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats5
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
[jira] [Commented] (HIVE-12674) HS2 Tez sessions should have maximum age
[ https://issues.apache.org/jira/browse/HIVE-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070335#comment-15070335 ] Vikram Dixit K commented on HIVE-12674: --- Left a few comments on the review board. Nothing major for now. We may want to simplify this logic at some point in time. > HS2 Tez sessions should have maximum age > > > Key: HIVE-12674 > URL: https://issues.apache.org/jira/browse/HIVE-12674 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12674.01.patch, HIVE-12674.01.patch, > HIVE-12674.02.patch, HIVE-12674.03.patch, HIVE-12674.patch > > > Certain tokens passed to AM by clients (e.g. an HDFS token) have maximum > lifetime beyond which they cannot be renewed. We should cycle long-lived > session AMs after a configurable period to avoid problems with these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12735) Constant folding for WHEN/CASE expression does not set return type correctly
[
https://issues.apache.org/jira/browse/HIVE-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070507#comment-15070507
]
Hive QA commented on HIVE-12735:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779164/HIVE-12735.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9963 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fold_eq_with_case_when
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6462/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6462/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6462/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779164 - PreCommit-HIVE-TRUNK-Build
> Constant folding for WHEN/CASE expression does not set return type correctly
>
>
> Key: HIVE-12735
> URL: https://issues.apache.org/jira/browse/HIVE-12735
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12735.1.patch
>
>
> For the following query
> {code}
> SELECT IF ( ( (CASE WHEN bool0 THEN 1 WHEN NOT bool0 THEN 0 END) = (CASE WHEN
> TRUE THEN 1 WHEN NOT TRUE THEN 0 END) ), key0, IF ( ( (CASE WHEN bool0 THEN 1
> WHEN NOT bool0 THEN 0 END) = (CASE WHEN FALSE THEN 1 WHEN NOT FALSE THEN 0
> END) ), key1, key2 ) ) FROM src_orc;
> {code}
> the expression gets constant folded to
> {code}
> if(CASE WHEN (bool0) THEN (true) WHEN ((not bool0)) THEN (false) END, key0,
> if(CASE WHEN (bool0) THEN (false) WHEN ((not bool0)) THEN (true) END, key1,
> key2)) (type: string)
> {code}
> however, the GenericUDFWhen and GenericUDFCase expression retain the original
> return type (int) instead of the folded return type (boolean). This can cause
> ClassCastException for the above query when vectorization is enabled.
> Following is the exception
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating
> if(CASE WHEN (bool0) THEN (true) WHEN ((not bool0)) THEN (false) END, key0,
> if(CASE WHEN (bool0) THEN (false) WHEN ((not bool0)) THEN (true) END, key1,
> key2))
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:126)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:852)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:114)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:168)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> ... 18 more
> Caused by: java.lang.ClassCastException: org.apache.hadoop.io.BooleanWritable
> cannot be cast to org.apache.hadoop.io.IntWritable
>
[jira] [Commented] (HIVE-12502) to_date UDF cannot accept NULLs of VOID type
[ https://issues.apache.org/jira/browse/HIVE-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070540#comment-15070540 ] Aaron Tokhy commented on HIVE-12502: Sure, will do. > to_date UDF cannot accept NULLs of VOID type > > > Key: HIVE-12502 > URL: https://issues.apache.org/jira/browse/HIVE-12502 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 1.0.0 >Reporter: Aaron Tokhy >Assignee: Aaron Tokhy >Priority: Trivial > Attachments: HIVE-12502-branch-1.patch, HIVE-12502.1.patch, > HIVE-12502.patch > > > The to_date method behaves differently based off the 'data type' of null > passed in. > hive> select to_date(null); > FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments 'TOK_NULL': > TO_DATE() only takes STRING/TIMESTAMP/DATEWRITABLE types, got VOID > hive> select to_date(cast(null as timestamp)); > OK > NULL > Time taken: 0.031 seconds, Fetched: 1 row(s) > This appears to be a regression introduced in HIVE-5731. The previous > version of to_date would not check the type: > https://github.com/apache/hive/commit/09b6553214d6db5ec7049b88bbe8ff640a7fef72#diff-204f5588c0767cf372a5ca7e3fb964afL56 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12743) RCFileInputFormat needs to be registered with kryo
[
https://issues.apache.org/jira/browse/HIVE-12743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-12743:
-
Attachment: HIVE-12743.1.patch
> RCFileInputFormat needs to be registered with kryo
> --
>
> Key: HIVE-12743
> URL: https://issues.apache.org/jira/browse/HIVE-12743
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12743.1.patch
>
>
> Ran into an issue with union distinct query that uses RCFile table with the
> following exception
> {code}
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12743) RCFileInputFormat needs to be registered with kryo
[
https://issues.apache.org/jira/browse/HIVE-12743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070561#comment-15070561
]
Sergey Shelukhin commented on HIVE-12743:
-
+1. Any other input formats need that?
> RCFileInputFormat needs to be registered with kryo
> --
>
> Key: HIVE-12743
> URL: https://issues.apache.org/jira/browse/HIVE-12743
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12743.1.patch
>
>
> Ran into an issue with union distinct query that uses RCFile table with the
> following exception
> {code}
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.io.RCFileInputFormat
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> at
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
> ~[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12741) HS2 ShutdownHookManager holds extra of Driver instance in master/branch-2.0
[ https://issues.apache.org/jira/browse/HIVE-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070356#comment-15070356 ] Thejas M Nair commented on HIVE-12741: -- [~daijy] Can you please review this patch ? > HS2 ShutdownHookManager holds extra of Driver instance in master/branch-2.0 > --- > > Key: HIVE-12741 > URL: https://issues.apache.org/jira/browse/HIVE-12741 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.0.0, 2.1.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-12741.1.patch > > > HIVE-12187 was meant to fix the described memory leak, however because of > interaction with HIVE-12187 in branch-2.0/master, the fix fails to take > effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12742) NULL table comparison within CASE does not work as previous hive versions
[ https://issues.apache.org/jira/browse/HIVE-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070371#comment-15070371 ] Pengcheng Xiong commented on HIVE-12742: [~gopalv], exactly. Could you please review the patch? Thanks. > NULL table comparison within CASE does not work as previous hive versions > - > > Key: HIVE-12742 > URL: https://issues.apache.org/jira/browse/HIVE-12742 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12742.01.patch > > > drop table test_1; > create table test_1 (id int, id2 int); > insert into table test_1 values (123, NULL); > SELECT cast(CASE WHEN id = id2 THEN FALSE ELSE TRUE END AS BOOLEAN) AS b > FROM test_1; > --NULL > But the output should be true (confirmed with postgres.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12742) NULL table comparison within CASE does not work as previous hive versions
[ https://issues.apache.org/jira/browse/HIVE-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12742: --- Attachment: HIVE-12742.01.patch > NULL table comparison within CASE does not work as previous hive versions > - > > Key: HIVE-12742 > URL: https://issues.apache.org/jira/browse/HIVE-12742 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12742.01.patch > > > drop table test_1; > create table test_1 (id int, id2 int); > insert into table test_1 values (123, NULL); > SELECT cast(CASE WHEN id = id2 THEN FALSE ELSE TRUE END AS BOOLEAN) AS b > FROM test_1; > --NULL > But the output should be true (confirmed with postgres.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12707: --- Attachment: HIVE-12707.01.patch > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > Attachments: HIVE-12707.01.patch > > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12742) NULL table comparison within CASE does not work as previous hive versions
[
https://issues.apache.org/jira/browse/HIVE-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070486#comment-15070486
]
Gopal V commented on HIVE-12742:
The rewrite is important for a number of tableau workloads. I would rewrite it
as {{NOT(NVL(val, false))}}
> NULL table comparison within CASE does not work as previous hive versions
> -
>
> Key: HIVE-12742
> URL: https://issues.apache.org/jira/browse/HIVE-12742
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12742.01.patch
>
>
> drop table test_1;
> create table test_1 (id int, id2 int);
> insert into table test_1 values (123, NULL);
> SELECT cast(CASE WHEN id = id2 THEN FALSE ELSE TRUE END AS BOOLEAN) AS b
> FROM test_1;
> --NULL
> But the output should be true (confirmed with postgres.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12674) HS2 Tez sessions should have maximum age
[ https://issues.apache.org/jira/browse/HIVE-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12674: Attachment: HIVE-12674.04.patch Added a couple of comments > HS2 Tez sessions should have maximum age > > > Key: HIVE-12674 > URL: https://issues.apache.org/jira/browse/HIVE-12674 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12674.01.patch, HIVE-12674.01.patch, > HIVE-12674.02.patch, HIVE-12674.03.patch, HIVE-12674.04.patch, > HIVE-12674.patch > > > Certain tokens passed to AM by clients (e.g. an HDFS token) have maximum > lifetime beyond which they cannot be renewed. We should cycle long-lived > session AMs after a configurable period to avoid problems with these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[ https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070547#comment-15070547 ] Prasanth Jayachandran commented on HIVE-12728: -- Fixes test failures. > Apply DDL restrictions for ORC schema evolution > --- > > Key: HIVE-12728 > URL: https://issues.apache.org/jira/browse/HIVE-12728 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-12728.1.patch, HIVE-12728.2.patch, > HIVE-12728.3.patch > > > HIVE-11981 added schema evolution for ORC. However, it does not enforce any > restrictions in DDL that can break schema evolution. Following changes have > to be enforced in DDL to support the assumptions in schema evolution (that > columns will only be added). > 1) Restrict changing the file format of the table > 2) Restrict changing the serde of the table > 3) Restrict replacing columns to not drop columns or do unsupported type > widening > 4) Restrict reordering columns > 5) Restrict unsupported type promotions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[ https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12728: - Attachment: HIVE-12728.3.patch > Apply DDL restrictions for ORC schema evolution > --- > > Key: HIVE-12728 > URL: https://issues.apache.org/jira/browse/HIVE-12728 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-12728.1.patch, HIVE-12728.2.patch, > HIVE-12728.3.patch > > > HIVE-11981 added schema evolution for ORC. However, it does not enforce any > restrictions in DDL that can break schema evolution. Following changes have > to be enforced in DDL to support the assumptions in schema evolution (that > columns will only be added). > 1) Restrict changing the file format of the table > 2) Restrict changing the serde of the table > 3) Restrict replacing columns to not drop columns or do unsupported type > widening > 4) Restrict reordering columns > 5) Restrict unsupported type promotions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12742) NULL table comparison within CASE does not work as previous hive versions
[ https://issues.apache.org/jira/browse/HIVE-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070368#comment-15070368 ] Gopal V commented on HIVE-12742: [~pxiong]: try disabling constant propogation. > NULL table comparison within CASE does not work as previous hive versions > - > > Key: HIVE-12742 > URL: https://issues.apache.org/jira/browse/HIVE-12742 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > > drop table test_1; > create table test_1 (id int, id2 int); > insert into table test_1 values (123, NULL); > SELECT cast(CASE WHEN id = id2 THEN FALSE ELSE TRUE END AS BOOLEAN) AS b > FROM test_1; > --NULL > But the output should be true (confirmed with postgres.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12739) Log4j2.x needs HADOOP_USER_CLASSPATH_FIRST to be set
[ https://issues.apache.org/jira/browse/HIVE-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070195#comment-15070195 ] Prasanth Jayachandran commented on HIVE-12739: -- I don't think this needs precommit test as this just changes the hive script. > Log4j2.x needs HADOOP_USER_CLASSPATH_FIRST to be set > > > Key: HIVE-12739 > URL: https://issues.apache.org/jira/browse/HIVE-12739 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.0.0 > > Attachments: HIVE-12739.1.patch > > > To load log4j2 jars properly HADOOP_USER_CLASSPATH_FIRST needs to set to > true. We should set this in hive script so that users don't have to do it > explicitly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-12707: -- Assignee: Pengcheng Xiong > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12707: --- Fix Version/s: 1.2.1 > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > Fix For: 1.2.1 > > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong resolved HIVE-12707. Resolution: Won't Fix > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070213#comment-15070213 ] Pengcheng Xiong commented on HIVE-12707: I also edit the document for reserved key words. "Note, set hive.support.sql11.reserved.keywords=false will only enable part of the reserved keywords to be used as identifier without backticks. Some of the reserved keywords, such as INTERVAL was always a reserved key word even before hive.support.sql11.reserved.keywords was introduced. Thus, the only way to use it as an identifier is through backticks." > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > Fix For: 1.2.1 > > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12516) HS2 log4j printing query about to be compiled
[ https://issues.apache.org/jira/browse/HIVE-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang resolved HIVE-12516. Resolution: Fixed > HS2 log4j printing query about to be compiled > -- > > Key: HIVE-12516 > URL: https://issues.apache.org/jira/browse/HIVE-12516 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Richard Walshe >Assignee: Chaoyu Tang > > Requesting HiveServer2 would use log4j to print out the query about to be > compiled to help identify badly written queries which take a lot of time to > compile and cause good queries to be queued. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12733) UX improvements for HIVE-12499
[
https://issues.apache.org/jira/browse/HIVE-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070310#comment-15070310
]
Hive QA commented on HIVE-12733:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779150/HIVE-12733.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6461/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6461/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6461/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult
[localFile=/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6461/succeeded/TestMinimrCliDriver-join1.q-file_with_header_footer.q-bucketmapjoin7.q-and-1-more,
remoteFile=/home/hiveptest/50.16.49.62-hiveptest-2/logs/, getExitCode()=12,
getException()=null, getUser()=hiveptest, getHost()=50.16.49.62,
getInstance()=2]: 'ssh_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
ssh_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
ssh_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
ssh_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
ssh_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.6]
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779150 - PreCommit-HIVE-TRUNK-Build
> UX improvements for HIVE-12499
> --
>
> Key: HIVE-12733
> URL: https://issues.apache.org/jira/browse/HIVE-12733
> Project: Hive
> Issue Type: Sub-task
> Components: Diagnosability
>Affects Versions: 2.1.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Fix For: 1.3.0, 2.0.0
>
> Attachments: HIVE-12733.patch
>
>
> From some early user feedback, the concept of delta metadata is a bit
> confusing, and should be separated into the 'created' and 'deleted' metadata
> metrics. Delta can be inferred from those.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070320#comment-15070320
]
Gunther Hagleitner commented on HIVE-12738:
---
(in case you're wondering - if produced the golden files with the original
patch, without any patch the query will fail on tez)
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch, HIVE-12738.2.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-12738:
--
Attachment: HIVE-12738.2.patch
.2 doesn't actually have a fix, but it does contain the same test case based on
"src" (i.e.: existing tables). I can reproduce the same issue with this query.
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch, HIVE-12738.2.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070208#comment-15070208 ] Pengcheng Xiong commented on HIVE-12707: [~rajsaha], INTERVAL is one of the reserved key words. It is there even before HIVE-6617. The only way to use it as an identifier is through backticks. Thanks. > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong resolved HIVE-12707. Resolution: Fixed > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reopened HIVE-12707: > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > Fix For: 1.2.1 > > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[
https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070272#comment-15070272
]
Rajit Saha commented on HIVE-12707:
---
Actually in hive 0.14.0.2.2.6.0-2800 , I could create the table with INTERVAL
column name without backtic
{noformat}
$ hive
Logging initialized using configuration in
file:/etc/hive/conf/hive-log4j.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/hdp/2.2.6.0-2800/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/hdp/2.2.6.0-2800/hive/lib/hive-jdbc-0.14.0.2.2.6.0-2800-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
hive> create table testinterval_test ( INTERVAL int);
OK
Time taken: 1.084 seconds
{noformat}
But in Hive 1.2.1 it fails
{noformat}
hive> set hive.support.sql11.reserved.keywords = false;
hive> create table testinterval ( INTERVAL int);
NoViableAltException(142@[])
at
org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:11576)
...
FAILED: ParseException line 1:28 cannot recognize input near 'INTERVAL' 'int'
')' in column specification
hive> set hive.support.sql11.reserved.keywords = true;
hive> create table testinterval ( INTERVAL int);
NoViableAltException(142@[])
...
FAILED: ParseException line 1:28 cannot recognize input near 'INTERVAL' 'int'
')' in column specification
{noformat}
> ParseException for column name 'INTERVAL', even set
> hive.support.sql11.reserved.keywords = false
> -
>
> Key: HIVE-12707
> URL: https://issues.apache.org/jira/browse/HIVE-12707
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Rajit Saha
>Assignee: Pengcheng Xiong
> Fix For: 1.2.1
>
>
> Any column name INTERVAL is causing ParseException , even we set
> hive.support.sql11.reserved.keywords = false
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070273#comment-15070273
]
Matt McCline commented on HIVE-12738:
-
I tried to reduce the size of the query files before submitting patch and the
query plan changed and it stop repro'ing. I'll try again.
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070278#comment-15070278
]
Matt McCline commented on HIVE-12738:
-
Vectorized vertices (Map and Reduce) can by mixed -- that is the root and some
number of children are vectorized but if there is a GroupBy with AVG that
generated STRUCT output (count,sum) then by the old design we have
VectorGroupBy generate rows and all down stream operators are non-vectorized...
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11072) Add data validation between Hive metastore upgrades tests
[ https://issues.apache.org/jira/browse/HIVE-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070293#comment-15070293 ] Alan Gates commented on HIVE-11072: --- +1, though I have to say I think it would have been easier to write metastore-validation-test.sh in Java. But it's fine in bash now that it's done. Would this test be run on every patch or only ones that change the DB schema? > Add data validation between Hive metastore upgrades tests > - > > Key: HIVE-11072 > URL: https://issues.apache.org/jira/browse/HIVE-11072 > Project: Hive > Issue Type: Task > Components: Tests >Reporter: Sergio Peña >Assignee: Naveen Gangam > Attachments: HIVE-11072.1.patch, HIVE-11072.2.patch, > HIVE-11072.3.patch > > > An existing Hive metastore upgrade test is running on Hive jenkins. However, > these scripts do test only database schema upgrade, not data validation > between upgrades. > We should validate data between metastore version upgrades. Using data > validation, we may ensure that data won't be damaged, or corrupted when > upgrading the Hive metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070312#comment-15070312 ] Pengcheng Xiong commented on HIVE-12707: cc'ing [~hagleitn] > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > Fix For: 1.2.1 > > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12724) ACID: Major compaction fails to include the original bucket files into MR job
[
https://issues.apache.org/jira/browse/HIVE-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12724:
-
Attachment: HIVE-12724.2.patch
> ACID: Major compaction fails to include the original bucket files into MR job
> -
>
> Key: HIVE-12724
> URL: https://issues.apache.org/jira/browse/HIVE-12724
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12724.1.patch, HIVE-12724.2.patch
>
>
> How the problem happens:
> * Create a non-ACID table
> * Before non-ACID to ACID table conversion, we inserted row one
> * After non-ACID to ACID table conversion, we inserted row two
> * Both rows can be retrieved before MAJOR compaction
> * After MAJOR compaction, row one is lost
> {code}
> hive> USE acidtest;
> OK
> Time taken: 0.77 seconds
> hive> CREATE TABLE t1 (nationkey INT, name STRING, regionkey INT, comment
> STRING)
> > CLUSTERED BY (regionkey) INTO 2 BUCKETS
> > STORED AS ORC;
> OK
> Time taken: 0.179 seconds
> hive> DESC FORMATTED t1;
> OK
> # col_namedata_type comment
> nationkey int
> name string
> regionkey int
> comment string
> # Detailed Table Information
> Database: acidtest
> Owner:wzheng
> CreateTime: Mon Dec 14 15:50:40 PST 2015
> LastAccessTime: UNKNOWN
> Retention:0
> Location: file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> Table Type: MANAGED_TABLE
> Table Parameters:
> transient_lastDdlTime 1450137040
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.ql.io.orc.OrcSerde
> InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
> Compressed: No
> Num Buckets: 2
> Bucket Columns: [regionkey]
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.198 seconds, Fetched: 28 row(s)
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db;
> Found 1 items
> drwxr-xr-x - wzheng staff 68 2015-12-14 15:50
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
> hive> INSERT INTO TABLE t1 VALUES (1, 'USA', 1, 'united states');
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. tez,
> spark) or using Hive 1.X releases.
> Query ID = wzheng_20151214155028_630098c6-605f-4e7e-a797-6b49fb48360d
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 2
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> Job running in-process (local Hadoop)
> 2015-12-14 15:51:58,070 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_local73977356_0001
> Loading data to table acidtest.t1
> MapReduce Jobs Launched:
> Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> Time taken: 2.825 seconds
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
> Found 2 items
> -rwxr-xr-x 1 wzheng staff112 2015-12-14 15:51
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1/00_0
> -rwxr-xr-x 1 wzheng staff472 2015-12-14 15:51
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1/01_0
> hive> SELECT * FROM t1;
> OK
> 1 USA 1 united states
> Time taken: 0.434 seconds, Fetched: 1 row(s)
> hive> ALTER TABLE t1 SET TBLPROPERTIES ('transactional' = 'true');
> OK
> Time taken: 0.071 seconds
> hive> DESC FORMATTED t1;
> OK
> # col_namedata_type comment
> nationkey int
> name string
> regionkey int
> comment string
> # Detailed Table Information
> Database: acidtest
> Owner:wzheng
> CreateTime: Mon Dec 14 15:50:40 PST 2015
> LastAccessTime: UNKNOWN
> Retention:0
> Location: file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_bywzheng
> last_modified_time 1450137141
> numFiles2
> numRows -1
> rawDataSize -1
> totalSize 584
> transactional true
[jira] [Updated] (HIVE-12739) Log4j2.x needs HADOOP_USER_CLASSPATH_FIRST to be set
[ https://issues.apache.org/jira/browse/HIVE-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12739: - Attachment: HIVE-12739.1.patch > Log4j2.x needs HADOOP_USER_CLASSPATH_FIRST to be set > > > Key: HIVE-12739 > URL: https://issues.apache.org/jira/browse/HIVE-12739 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.0.0 > > Attachments: HIVE-12739.1.patch > > > To load log4j2 jars properly HADOOP_USER_CLASSPATH_FIRST needs to set to > true. We should set this in hive script so that users don't have to do it > explicitly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6285) DESCRIBE EXTENDED shows numRows as zero
[
https://issues.apache.org/jira/browse/HIVE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070275#comment-15070275
]
Pengcheng Xiong commented on HIVE-6285:
---
[~navis] and [~haroon], are you guys still able to reproduce the problem? In my
case. both CLI and beeline works well.
Here is the results for beeline:
{code}
CREATE TABLE `colors`(`c` char(20));
insert into colors values ('green');
desc extended colors;
It shows:
parameters:{numFiles=1, transient_lastDdlTime=1450910401,
COLUMN_STATS_ACCURATE=true, totalSize=21, numRows=1, rawDataSize=20},
{code}
> DESCRIBE EXTENDED shows numRows as zero
> ---
>
> Key: HIVE-6285
> URL: https://issues.apache.org/jira/browse/HIVE-6285
> Project: Hive
> Issue Type: Bug
> Components: CLI, HiveServer2
>Affects Versions: 0.12.0
>Reporter: Haroon
>
> Running DESCRIBE EXTENDED always shows numRows as zero.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070276#comment-15070276
]
Matt McCline commented on HIVE-12738:
-
Some vectorized classes are marked as having new VectorizationContext but not
all vectorized classes are currently marked. So, it isn't easy to determine if
a class is generating rows.
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070309#comment-15070309 ] Rajit Saha commented on HIVE-12707: --- [~pxiong] Thanks for the help. Actual this backward compatibility missing is making trouble for our platform users as they have use backtics for every query they write , even after we added backtic in schema . To me it would be nice if we can have consistency in these reserved keywords and hive.support.sql11.reserved.keywords is a nice way to deal with backward compatibility. So to me it would be very helpful for users if we can add INTERVAL in hive.support.sql11.reserved.keywords list. > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > Fix For: 1.2.1 > > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6113) Upgrade DataNucleus [was: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient]
[ https://issues.apache.org/jira/browse/HIVE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070171#comment-15070171 ] Sergey Shelukhin commented on HIVE-6113: None of the tests have age 1. +1. Will commit later today unless there are objections > Upgrade DataNucleus [was: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient] > - > > Key: HIVE-6113 > URL: https://issues.apache.org/jira/browse/HIVE-6113 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 0.12.0, 0.13.0, 0.14.0, 1.0.0, 1.2.1 > Environment: hadoop-0.20.2-cdh3u3,hive-0.12.0 >Reporter: William Stone >Assignee: Oleksiy Sayankin >Priority: Critical > Labels: HiveMetaStoreClient, metastore, unable_instantiate > Attachments: HIVE-6113-2.patch, HIVE-6113.10.patch, > HIVE-6113.3.patch, HIVE-6113.4.patch, HIVE-6113.5.patch, HIVE-6113.6.patch, > HIVE-6113.7.patch, HIVE-6113.8.patch, HIVE-6113.9.patch, HIVE-6113.patch, > HIVE-6113.with.reflection.patch > > > CLEAR LIBRARY CACHE > When I exccute SQL "use fdm; desc formatted fdm.tableName;" in python, throw > Error as followed. > but when I tryit again , It will success. > 2013-12-25 03:01:32,290 ERROR exec.DDLTask (DDLTask.java:execute(435)) - > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1143) > at > org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1128) > at > org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:3479) > at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:237) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:260) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217) > at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:875) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:769) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:708) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > Caused by: java.lang.RuntimeException: Unable to instantiate > org.apache.hadoop.hive.metastore.HiveMetaStoreClient > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1217) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62) > at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) > at > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) > at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) > ... 20 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) > ... 25 more > Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes > to datastore > NestedThrowables: > java.sql.BatchUpdateException: Duplicate entry 'default' for key > 'UNIQUE_DATABASE' > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) > at > org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) >
[jira] [Updated] (HIVE-12740) NPE with HS2 when using null input format
[
https://issues.apache.org/jira/browse/HIVE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-12740:
--
Attachment: HIVE-12740.2.patch
.2 addresses [~sershe]'s concerns.
> NPE with HS2 when using null input format
> -
>
> Key: HIVE-12740
> URL: https://issues.apache.org/jira/browse/HIVE-12740
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Tez
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-12740.1.patch, HIVE-12740.2.patch
>
>
> When we have a query that returns empty rows and when using tez with hs2, we
> hit NPE:
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:490)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.writeOldSplits(MRInputHelpers.java:559)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplits(MRInputHelpers.java:619)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:109)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1816)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1561)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1338)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1147)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:234)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:247)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isVectorMode(Utilities.java:3241)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.wrapForLlap(HiveInputFormat.java:208)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputFormatFromCache(HiveInputFormat.java:267)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:80)
> ... 4 more
> 15/12/17 18:59:06 INFO log.PerfLogger: start=1450378746335 end=1450378746433 duration=98
> from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
> 15/12/17 18:59:06 ERROR exec.Task: Failed to execute tez graph.
> org.apache.tez.dag.api.TezUncheckedException: Failed to generate InputSplits
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:124)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
>
[jira] [Commented] (HIVE-12740) NPE with HS2 when using null input format
[
https://issues.apache.org/jira/browse/HIVE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070268#comment-15070268
]
Vikram Dixit K commented on HIVE-12740:
---
+1
> NPE with HS2 when using null input format
> -
>
> Key: HIVE-12740
> URL: https://issues.apache.org/jira/browse/HIVE-12740
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Tez
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-12740.1.patch, HIVE-12740.2.patch
>
>
> When we have a query that returns empty rows and when using tez with hs2, we
> hit NPE:
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:490)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.writeOldSplits(MRInputHelpers.java:559)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplits(MRInputHelpers.java:619)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:109)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1816)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1561)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1338)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1147)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:234)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:247)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isVectorMode(Utilities.java:3241)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.wrapForLlap(HiveInputFormat.java:208)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputFormatFromCache(HiveInputFormat.java:267)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:80)
> ... 4 more
> 15/12/17 18:59:06 INFO log.PerfLogger: start=1450378746335 end=1450378746433 duration=98
> from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
> 15/12/17 18:59:06 ERROR exec.Task: Failed to execute tez graph.
> org.apache.tez.dag.api.TezUncheckedException: Failed to generate InputSplits
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:124)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
>
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070297#comment-15070297
]
Gunther Hagleitner commented on HIVE-12738:
---
If I understand correctly:
When you're looking at a reduce vertex, at some point you might transition from
vector to row based processing. However, there's no way to come back from row
to vector, right?
So, when converting reduce work you just need to bail when the transition
happens, don't you?
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reopened HIVE-12707: > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > Fix For: 1.2.1 > > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12707: --- Fix Version/s: (was: 1.2.1) > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12739) Log4j2.x needs HADOOP_USER_CLASSPATH_FIRST to be set
[ https://issues.apache.org/jira/browse/HIVE-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070118#comment-15070118 ] Sergey Shelukhin commented on HIVE-12739: - +1 > Log4j2.x needs HADOOP_USER_CLASSPATH_FIRST to be set > > > Key: HIVE-12739 > URL: https://issues.apache.org/jira/browse/HIVE-12739 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.0.0 > > Attachments: HIVE-12739.1.patch > > > To load log4j2 jars properly HADOOP_USER_CLASSPATH_FIRST needs to set to > true. We should set this in hive script so that users don't have to do it > explicitly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12740) NPE with HS2 when using null input format
[
https://issues.apache.org/jira/browse/HIVE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070241#comment-15070241
]
Sergey Shelukhin commented on HIVE-12740:
-
+1
> NPE with HS2 when using null input format
> -
>
> Key: HIVE-12740
> URL: https://issues.apache.org/jira/browse/HIVE-12740
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Tez
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-12740.1.patch, HIVE-12740.2.patch
>
>
> When we have a query that returns empty rows and when using tez with hs2, we
> hit NPE:
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:490)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.writeOldSplits(MRInputHelpers.java:559)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplits(MRInputHelpers.java:619)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:109)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1816)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1561)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1338)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1147)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:234)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:247)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isVectorMode(Utilities.java:3241)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.wrapForLlap(HiveInputFormat.java:208)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputFormatFromCache(HiveInputFormat.java:267)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:80)
> ... 4 more
> 15/12/17 18:59:06 INFO log.PerfLogger: start=1450378746335 end=1450378746433 duration=98
> from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
> 15/12/17 18:59:06 ERROR exec.Task: Failed to execute tez graph.
> org.apache.tez.dag.api.TezUncheckedException: Failed to generate InputSplits
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:124)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
>
[jira] [Commented] (HIVE-12727) allow full table queries in strict mode
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070249#comment-15070249
]
Hive QA commented on HIVE-12727:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779141/HIVE-12727.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 538 failed/errored test(s), 9977 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_project
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_create_temp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_SortUnionTransposeRule
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby_empty
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_gby_empty
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_lineage2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udf_udaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_windowing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_windowing_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_not_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_udaf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_windowing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_date_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_1_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_trailing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_rdd_cache
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_merge
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape_orderby1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_rearrange
[jira] [Updated] (HIVE-12713) Miscellaneous improvements in driver compile and execute logging
[ https://issues.apache.org/jira/browse/HIVE-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12713: --- Attachment: HIVE-12713-branch-1.patch > Miscellaneous improvements in driver compile and execute logging > > > Key: HIVE-12713 > URL: https://issues.apache.org/jira/browse/HIVE-12713 > Project: Hive > Issue Type: Improvement > Components: Logging >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-12713-branch-1.patch, HIVE-12713-branch-2.0.patch, > HIVE-12713.1.patch, HIVE-12713.patch > > > Miscellaneous compile and execute logging improvements include: > 1. ensuring that only the redacted query to be logged out > 2. removing redundant variable substitution in HS2 SQLOperation > 3. logging out the query and its compilation time without having to enable > PerfLogger debug, to help identify badly written queries which take a lot of > time to compile and probably cause other good queries to be queued > (HIVE-12516) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12707) ParseException for column name 'INTERVAL', even set hive.support.sql11.reserved.keywords = false
[ https://issues.apache.org/jira/browse/HIVE-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070283#comment-15070283 ] Pengcheng Xiong commented on HIVE-12707: [~rajsaha], let me double check. > ParseException for column name 'INTERVAL', even set > hive.support.sql11.reserved.keywords = false > - > > Key: HIVE-12707 > URL: https://issues.apache.org/jira/browse/HIVE-12707 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Rajit Saha >Assignee: Pengcheng Xiong > Fix For: 1.2.1 > > > Any column name INTERVAL is causing ParseException , even we set > hive.support.sql11.reserved.keywords = false -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070311#comment-15070311
]
Matt McCline commented on HIVE-12738:
-
Yes, it can happen for Map or Reduce. We could detect when we have
transitioned and just stop trying to vectorized below...
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12724) ACID: Major compaction fails to include the original bucket files into MR job
[
https://issues.apache.org/jira/browse/HIVE-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12724:
-
Attachment: (was: HIVE-12724.2.patch)
> ACID: Major compaction fails to include the original bucket files into MR job
> -
>
> Key: HIVE-12724
> URL: https://issues.apache.org/jira/browse/HIVE-12724
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12724.1.patch
>
>
> How the problem happens:
> * Create a non-ACID table
> * Before non-ACID to ACID table conversion, we inserted row one
> * After non-ACID to ACID table conversion, we inserted row two
> * Both rows can be retrieved before MAJOR compaction
> * After MAJOR compaction, row one is lost
> {code}
> hive> USE acidtest;
> OK
> Time taken: 0.77 seconds
> hive> CREATE TABLE t1 (nationkey INT, name STRING, regionkey INT, comment
> STRING)
> > CLUSTERED BY (regionkey) INTO 2 BUCKETS
> > STORED AS ORC;
> OK
> Time taken: 0.179 seconds
> hive> DESC FORMATTED t1;
> OK
> # col_namedata_type comment
> nationkey int
> name string
> regionkey int
> comment string
> # Detailed Table Information
> Database: acidtest
> Owner:wzheng
> CreateTime: Mon Dec 14 15:50:40 PST 2015
> LastAccessTime: UNKNOWN
> Retention:0
> Location: file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> Table Type: MANAGED_TABLE
> Table Parameters:
> transient_lastDdlTime 1450137040
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.ql.io.orc.OrcSerde
> InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
> Compressed: No
> Num Buckets: 2
> Bucket Columns: [regionkey]
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.198 seconds, Fetched: 28 row(s)
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db;
> Found 1 items
> drwxr-xr-x - wzheng staff 68 2015-12-14 15:50
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
> hive> INSERT INTO TABLE t1 VALUES (1, 'USA', 1, 'united states');
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. tez,
> spark) or using Hive 1.X releases.
> Query ID = wzheng_20151214155028_630098c6-605f-4e7e-a797-6b49fb48360d
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 2
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> Job running in-process (local Hadoop)
> 2015-12-14 15:51:58,070 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_local73977356_0001
> Loading data to table acidtest.t1
> MapReduce Jobs Launched:
> Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> Time taken: 2.825 seconds
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
> Found 2 items
> -rwxr-xr-x 1 wzheng staff112 2015-12-14 15:51
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1/00_0
> -rwxr-xr-x 1 wzheng staff472 2015-12-14 15:51
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1/01_0
> hive> SELECT * FROM t1;
> OK
> 1 USA 1 united states
> Time taken: 0.434 seconds, Fetched: 1 row(s)
> hive> ALTER TABLE t1 SET TBLPROPERTIES ('transactional' = 'true');
> OK
> Time taken: 0.071 seconds
> hive> DESC FORMATTED t1;
> OK
> # col_namedata_type comment
> nationkey int
> name string
> regionkey int
> comment string
> # Detailed Table Information
> Database: acidtest
> Owner:wzheng
> CreateTime: Mon Dec 14 15:50:40 PST 2015
> LastAccessTime: UNKNOWN
> Retention:0
> Location: file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_bywzheng
> last_modified_time 1450137141
> numFiles2
> numRows -1
> rawDataSize -1
> totalSize 584
> transactional true
>
[jira] [Commented] (HIVE-12724) ACID: Major compaction fails to include the original bucket files into MR job
[
https://issues.apache.org/jira/browse/HIVE-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070322#comment-15070322
]
Wei Zheng commented on HIVE-12724:
--
[~damien.carol] Please take a look at here:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL
> ACID: Major compaction fails to include the original bucket files into MR job
> -
>
> Key: HIVE-12724
> URL: https://issues.apache.org/jira/browse/HIVE-12724
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12724.1.patch, HIVE-12724.2.patch
>
>
> How the problem happens:
> * Create a non-ACID table
> * Before non-ACID to ACID table conversion, we inserted row one
> * After non-ACID to ACID table conversion, we inserted row two
> * Both rows can be retrieved before MAJOR compaction
> * After MAJOR compaction, row one is lost
> {code}
> hive> USE acidtest;
> OK
> Time taken: 0.77 seconds
> hive> CREATE TABLE t1 (nationkey INT, name STRING, regionkey INT, comment
> STRING)
> > CLUSTERED BY (regionkey) INTO 2 BUCKETS
> > STORED AS ORC;
> OK
> Time taken: 0.179 seconds
> hive> DESC FORMATTED t1;
> OK
> # col_namedata_type comment
> nationkey int
> name string
> regionkey int
> comment string
> # Detailed Table Information
> Database: acidtest
> Owner:wzheng
> CreateTime: Mon Dec 14 15:50:40 PST 2015
> LastAccessTime: UNKNOWN
> Retention:0
> Location: file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> Table Type: MANAGED_TABLE
> Table Parameters:
> transient_lastDdlTime 1450137040
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.ql.io.orc.OrcSerde
> InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
> Compressed: No
> Num Buckets: 2
> Bucket Columns: [regionkey]
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.198 seconds, Fetched: 28 row(s)
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db;
> Found 1 items
> drwxr-xr-x - wzheng staff 68 2015-12-14 15:50
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
> hive> INSERT INTO TABLE t1 VALUES (1, 'USA', 1, 'united states');
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. tez,
> spark) or using Hive 1.X releases.
> Query ID = wzheng_20151214155028_630098c6-605f-4e7e-a797-6b49fb48360d
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 2
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> Job running in-process (local Hadoop)
> 2015-12-14 15:51:58,070 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_local73977356_0001
> Loading data to table acidtest.t1
> MapReduce Jobs Launched:
> Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> Time taken: 2.825 seconds
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
> Found 2 items
> -rwxr-xr-x 1 wzheng staff112 2015-12-14 15:51
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1/00_0
> -rwxr-xr-x 1 wzheng staff472 2015-12-14 15:51
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1/01_0
> hive> SELECT * FROM t1;
> OK
> 1 USA 1 united states
> Time taken: 0.434 seconds, Fetched: 1 row(s)
> hive> ALTER TABLE t1 SET TBLPROPERTIES ('transactional' = 'true');
> OK
> Time taken: 0.071 seconds
> hive> DESC FORMATTED t1;
> OK
> # col_namedata_type comment
> nationkey int
> name string
> regionkey int
> comment string
> # Detailed Table Information
> Database: acidtest
> Owner:wzheng
> CreateTime: Mon Dec 14 15:50:40 PST 2015
> LastAccessTime: UNKNOWN
> Retention:0
> Location: file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_bywzheng
> last_modified_time 1450137141
> numFiles2
> numRows
[jira] [Commented] (HIVE-12740) NPE with HS2 when using null input format
[
https://issues.apache.org/jira/browse/HIVE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070121#comment-15070121
]
Vikram Dixit K commented on HIVE-12740:
---
Sample query:
{code}
SELECT SUM(key) AS sum_profit_ok FROM src WHERE false GROUP BY 1 HAVING
(COUNT(1) > 0);
{code}
> NPE with HS2 when using null input format
> -
>
> Key: HIVE-12740
> URL: https://issues.apache.org/jira/browse/HIVE-12740
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Tez
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
>
> When we have a query that returns empty rows and when using tez with hs2, we
> hit NPE:
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:490)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.writeOldSplits(MRInputHelpers.java:559)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplits(MRInputHelpers.java:619)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:109)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1816)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1561)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1338)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1147)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:234)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:247)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isVectorMode(Utilities.java:3241)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.wrapForLlap(HiveInputFormat.java:208)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputFormatFromCache(HiveInputFormat.java:267)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:80)
> ... 4 more
> 15/12/17 18:59:06 INFO log.PerfLogger: start=1450378746335 end=1450378746433 duration=98
> from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
> 15/12/17 18:59:06 ERROR exec.Task: Failed to execute tez graph.
> org.apache.tez.dag.api.TezUncheckedException: Failed to generate InputSplits
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:124)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
>
[jira] [Updated] (HIVE-12740) NPE with HS2 when using null input format
[
https://issues.apache.org/jira/browse/HIVE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikram Dixit K updated HIVE-12740:
--
Attachment: HIVE-12740.1.patch
[~hagleitn] short patch for your review.
> NPE with HS2 when using null input format
> -
>
> Key: HIVE-12740
> URL: https://issues.apache.org/jira/browse/HIVE-12740
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Tez
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-12740.1.patch
>
>
> When we have a query that returns empty rows and when using tez with hs2, we
> hit NPE:
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:490)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.writeOldSplits(MRInputHelpers.java:559)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplits(MRInputHelpers.java:619)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:109)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1816)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1561)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1338)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1147)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:234)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:247)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isVectorMode(Utilities.java:3241)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.wrapForLlap(HiveInputFormat.java:208)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputFormatFromCache(HiveInputFormat.java:267)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:103)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CheckNonCombinablePathCallable.call(CombineHiveInputFormat.java:80)
> ... 4 more
> 15/12/17 18:59:06 INFO log.PerfLogger: start=1450378746335 end=1450378746433 duration=98
> from=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat>
> 15/12/17 18:59:06 ERROR exec.Task: Failed to execute tez graph.
> org.apache.tez.dag.api.TezUncheckedException: Failed to generate InputSplits
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:124)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
>
[jira] [Commented] (HIVE-12740) NPE with HS2 when using null input format
[
https://issues.apache.org/jira/browse/HIVE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070607#comment-15070607
]
Hive QA commented on HIVE-12740:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779322/HIVE-12740.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9978 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6464/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6464/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6464/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779322 - PreCommit-HIVE-TRUNK-Build
> NPE with HS2 when using null input format
> -
>
> Key: HIVE-12740
> URL: https://issues.apache.org/jira/browse/HIVE-12740
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Tez
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-12740.1.patch, HIVE-12740.2.patch
>
>
> When we have a query that returns empty rows and when using tez with hs2, we
> hit NPE:
> {code}
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:490)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:447)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.writeOldSplits(MRInputHelpers.java:559)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplits(MRInputHelpers.java:619)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.configureMRInputWithLegacySplitGeneration(MRInputHelpers.java:109)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:617)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:1103)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:386)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:175)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:156)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1816)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1561)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1338)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1147)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73)
> at
>
[jira] [Commented] (HIVE-12674) HS2 Tez sessions should have maximum age
[ https://issues.apache.org/jira/browse/HIVE-12674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070722#comment-15070722 ] Vikram Dixit K commented on HIVE-12674: --- +1 > HS2 Tez sessions should have maximum age > > > Key: HIVE-12674 > URL: https://issues.apache.org/jira/browse/HIVE-12674 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12674.01.patch, HIVE-12674.01.patch, > HIVE-12674.02.patch, HIVE-12674.03.patch, HIVE-12674.04.patch, > HIVE-12674.patch > > > Certain tokens passed to AM by clients (e.g. an HDFS token) have maximum > lifetime beyond which they cannot be renewed. We should cycle long-lived > session AMs after a configurable period to avoid problems with these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-12738:
Attachment: HIVE-12738.4.patch
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch, HIVE-12738.2.patch,
> HIVE-12738.3.patch, HIVE-12738.4.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12738) subquery with NOT IN failing due to ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-12738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070653#comment-15070653
]
Matt McCline commented on HIVE-12738:
-
Used a set to avoid making any assumptions about multiple table scans,
operators with multiple children, etc.
> subquery with NOT IN failing due to ClassCastException
> --
>
> Key: HIVE-12738
> URL: https://issues.apache.org/jira/browse/HIVE-12738
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12738.01.patch, HIVE-12738.2.patch,
> HIVE-12738.3.patch, HIVE-12738.4.patch
>
>
> {code}
> create table student(
> name string,
> age int,
> gpa double)
> create table voter(
> name string,
> age int,
> registration string,
> contributions float)
> select *
> from student
> where not student.age in
> (select age from voter)
> order by age;
> {code}
> Fails in Tez -- Vectorized Reduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12742) NULL table comparison within CASE does not work as previous hive versions
[ https://issues.apache.org/jira/browse/HIVE-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070704#comment-15070704 ] Pengcheng Xiong commented on HIVE-12742: [~gopalv], thanks for pointing out NVL. i have modified the patch accordingly. Could you take another look? Thanks. > NULL table comparison within CASE does not work as previous hive versions > - > > Key: HIVE-12742 > URL: https://issues.apache.org/jira/browse/HIVE-12742 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12742.01.patch, HIVE-12742.02.patch > > > drop table test_1; > create table test_1 (id int, id2 int); > insert into table test_1 values (123, NULL); > SELECT cast(CASE WHEN id = id2 THEN FALSE ELSE TRUE END AS BOOLEAN) AS b > FROM test_1; > --NULL > But the output should be true (confirmed with postgres.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12742) NULL table comparison within CASE does not work as previous hive versions
[ https://issues.apache.org/jira/browse/HIVE-12742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-12742: --- Attachment: HIVE-12742.02.patch > NULL table comparison within CASE does not work as previous hive versions > - > > Key: HIVE-12742 > URL: https://issues.apache.org/jira/browse/HIVE-12742 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-12742.01.patch, HIVE-12742.02.patch > > > drop table test_1; > create table test_1 (id int, id2 int); > insert into table test_1 values (123, NULL); > SELECT cast(CASE WHEN id = id2 THEN FALSE ELSE TRUE END AS BOOLEAN) AS b > FROM test_1; > --NULL > But the output should be true (confirmed with postgres.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12429) Switch default Hive authorization to SQLStandardAuth in 2.0
[ https://issues.apache.org/jira/browse/HIVE-12429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-12429: -- Attachment: HIVE-12429.8.patch > Switch default Hive authorization to SQLStandardAuth in 2.0 > --- > > Key: HIVE-12429 > URL: https://issues.apache.org/jira/browse/HIVE-12429 > Project: Hive > Issue Type: Task > Components: Authorization, Security >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Daniel Dai > Attachments: HIVE-12429.1.patch, HIVE-12429.2.patch, > HIVE-12429.3.patch, HIVE-12429.4.patch, HIVE-12429.5.patch, > HIVE-12429.6.patch, HIVE-12429.7.patch, HIVE-12429.8.patch > > > Hive's default authorization is not real security, as it does not secure a > number of features and anyone can grant access to any object to any user. We > should switch the default to SQLStandardAuth, which provides real > authentication. > As this is a backwards incompatible change this was hard to do previously, > but 2.0 gives us a place to do this type of change. > By default authorization will still be off, as there are a few other things > to set when turning on authorization (such as the list of admin users). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12623) Add an option to force allocation of fragments on requested nodes
[ https://issues.apache.org/jira/browse/HIVE-12623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069363#comment-15069363 ] Lefty Leverenz commented on HIVE-12623: --- Doc note: This adds *hive.llap.task.scheduler.locality.delay* to HiveConf.java so it needs to be documented in the wiki for 2.0.0 (in a new section for LLAP parameters, to be created for HIVE-9850 and HIVE-11908). * [Hive Configuration Properties | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveConfigurationProperties] > Add an option to force allocation of fragments on requested nodes > - > > Key: HIVE-12623 > URL: https://issues.apache.org/jira/browse/HIVE-12623 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12623.1.patch, HIVE-12623.1.txt, > HIVE-12623.1.wip.txt > > > Currently, fragments are sent to random nodes if the requested node does not > have capacity. In certain situations there's more to be gained by sending the > fragments to the requested node only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12724) ACID: Major compaction fails to include the original bucket files into MR job
[
https://issues.apache.org/jira/browse/HIVE-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069368#comment-15069368
]
Damien Carol commented on HIVE-12724:
-
[~wzheng] I don't see *transactional* TBLPROPERTIES
> ACID: Major compaction fails to include the original bucket files into MR job
> -
>
> Key: HIVE-12724
> URL: https://issues.apache.org/jira/browse/HIVE-12724
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12724.1.patch, HIVE-12724.2.patch
>
>
> How the problem happens:
> * Create a non-ACID table
> * Before non-ACID to ACID table conversion, we inserted row one
> * After non-ACID to ACID table conversion, we inserted row two
> * Both rows can be retrieved before MAJOR compaction
> * After MAJOR compaction, row one is lost
> {code}
> hive> USE acidtest;
> OK
> Time taken: 0.77 seconds
> hive> CREATE TABLE t1 (nationkey INT, name STRING, regionkey INT, comment
> STRING)
> > CLUSTERED BY (regionkey) INTO 2 BUCKETS
> > STORED AS ORC;
> OK
> Time taken: 0.179 seconds
> hive> DESC FORMATTED t1;
> OK
> # col_namedata_type comment
> nationkey int
> name string
> regionkey int
> comment string
> # Detailed Table Information
> Database: acidtest
> Owner:wzheng
> CreateTime: Mon Dec 14 15:50:40 PST 2015
> LastAccessTime: UNKNOWN
> Retention:0
> Location: file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> Table Type: MANAGED_TABLE
> Table Parameters:
> transient_lastDdlTime 1450137040
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.ql.io.orc.OrcSerde
> InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
> OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
> Compressed: No
> Num Buckets: 2
> Bucket Columns: [regionkey]
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.198 seconds, Fetched: 28 row(s)
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db;
> Found 1 items
> drwxr-xr-x - wzheng staff 68 2015-12-14 15:50
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
> hive> INSERT INTO TABLE t1 VALUES (1, 'USA', 1, 'united states');
> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. tez,
> spark) or using Hive 1.X releases.
> Query ID = wzheng_20151214155028_630098c6-605f-4e7e-a797-6b49fb48360d
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 2
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> Job running in-process (local Hadoop)
> 2015-12-14 15:51:58,070 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_local73977356_0001
> Loading data to table acidtest.t1
> MapReduce Jobs Launched:
> Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> OK
> Time taken: 2.825 seconds
> hive> dfs -ls /Users/wzheng/hivetmp/warehouse/acidtest.db/t1;
> Found 2 items
> -rwxr-xr-x 1 wzheng staff112 2015-12-14 15:51
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1/00_0
> -rwxr-xr-x 1 wzheng staff472 2015-12-14 15:51
> /Users/wzheng/hivetmp/warehouse/acidtest.db/t1/01_0
> hive> SELECT * FROM t1;
> OK
> 1 USA 1 united states
> Time taken: 0.434 seconds, Fetched: 1 row(s)
> hive> ALTER TABLE t1 SET TBLPROPERTIES ('transactional' = 'true');
> OK
> Time taken: 0.071 seconds
> hive> DESC FORMATTED t1;
> OK
> # col_namedata_type comment
> nationkey int
> name string
> regionkey int
> comment string
> # Detailed Table Information
> Database: acidtest
> Owner:wzheng
> CreateTime: Mon Dec 14 15:50:40 PST 2015
> LastAccessTime: UNKNOWN
> Retention:0
> Location: file:/Users/wzheng/hivetmp/warehouse/acidtest.db/t1
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_bywzheng
> last_modified_time 1450137141
> numFiles2
> numRows -1
> rawDataSize -1
>
[jira] [Updated] (HIVE-12395) Turn off CBO for hive.support.special.characters.tablename tests until feature is complete
[ https://issues.apache.org/jira/browse/HIVE-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12395: -- Labels: TODOC2.1 (was: ) > Turn off CBO for hive.support.special.characters.tablename tests until > feature is complete > -- > > Key: HIVE-12395 > URL: https://issues.apache.org/jira/browse/HIVE-12395 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-12395.01.patch, HIVE-12395.02.patch > > > Due to recent stats issue found in HIVE-12381, we need to turn off CBO for > hive.support.special.characters.tablename tests until feature is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12395) Turn off CBO for hive.support.special.characters.tablename tests until feature is complete
[ https://issues.apache.org/jira/browse/HIVE-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-12395: -- Labels: (was: TODOC2.1) > Turn off CBO for hive.support.special.characters.tablename tests until > feature is complete > -- > > Key: HIVE-12395 > URL: https://issues.apache.org/jira/browse/HIVE-12395 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-12395.01.patch, HIVE-12395.02.patch > > > Due to recent stats issue found in HIVE-12381, we need to turn off CBO for > hive.support.special.characters.tablename tests until feature is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12728) Apply DDL restrictions for ORC schema evolution
[
https://issues.apache.org/jira/browse/HIVE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069372#comment-15069372
]
Hive QA commented on HIVE-12728:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12779122/HIVE-12728.2.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 9987 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_char_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_date_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dbtxnmgr_ddl1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_orc_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_int_type_promotion
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_options1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_varchar_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_multiple_part_clause
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6453/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6453/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6453/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12779122 - PreCommit-HIVE-TRUNK-Build
> Apply DDL restrictions for ORC schema evolution
> ---
>
> Key: HIVE-12728
> URL: https://issues.apache.org/jira/browse/HIVE-12728
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-12728.1.patch, HIVE-12728.2.patch
>
>
> HIVE-11981 added schema evolution for ORC. However, it does not enforce any
> restrictions in DDL that can break schema evolution. Following changes have
> to be enforced in DDL to support the assumptions in schema evolution (that
> columns will only be added).
> 1) Restrict changing the file format of the table
> 2) Restrict changing the serde of the table
> 3) Restrict replacing columns to not drop columns or do unsupported type
> widening
> 4) Restrict reordering columns
> 5) Restrict unsupported type promotions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12395) Turn off CBO for hive.support.special.characters.tablename tests until feature is complete
[ https://issues.apache.org/jira/browse/HIVE-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069378#comment-15069378 ] Lefty Leverenz commented on HIVE-12395: --- Doc note: This changes the default value of *hive.support.special.characters.tablename* in release 2.1.0, so it needs to be documented in the wiki. The parameter was created by HIVE-11699 in release 2.0.0 -- see the doc note on that issue. > Turn off CBO for hive.support.special.characters.tablename tests until > feature is complete > -- > > Key: HIVE-12395 > URL: https://issues.apache.org/jira/browse/HIVE-12395 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-12395.01.patch, HIVE-12395.02.patch > > > Due to recent stats issue found in HIVE-12381, we need to turn off CBO for > hive.support.special.characters.tablename tests until feature is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
