[jira] [Commented] (HIVE-6893) out of sequence error in HiveMetastore server

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977861#comment-13977861
 ] 

Szehon Ho commented on HIVE-6893:
-

As FYI, these are known test errors due to recent switch to hadoop-2.  Ref for 
a clean hadoop-2 build run two days back, see bottom of: 
[http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-15/execution.txt|http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-15/execution.txt]

> out of sequence error in HiveMetastore server
> -
>
> Key: HIVE-6893
> URL: https://issues.apache.org/jira/browse/HIVE-6893
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Romain Rigaux
>Assignee: Naveen Gangam
> Fix For: 0.13.0
>
> Attachments: HIVE-6893.1.patch
>
>
> Calls listing databases or tables fail. It seems to be a concurrency problem.
> {code}
> 014-03-06 05:34:00,785 ERROR hive.log: 
> org.apache.thrift.TApplicationException: get_databases failed: out of 
> sequence response
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
> at 
> org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
> at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
> at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
> at 
> org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
> at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
> at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

[Amareshwari Sriramadasu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated HIVE-6953:
--

Attachment: TEST-org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.xml

Attaching TestInitiator.xml

> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments: 
> TEST-org.apache.hadoop.hive.ql.txn.compactor.TestInitiator.xml, nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install 
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner 
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
> transaction database java.sql.SQLSyntaxErrorException: Table/View 
> 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
> 
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
>  Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

[Amareshwari Sriramadasu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977849#comment-13977849
 ] 

Amareshwari Sriramadasu commented on HIVE-6953:
---

The machine on which I'm running is a Linux machine. Same thing happens on Mac 
as well.

{noformat}
uname -a
Linux  2.6.32-21-server #32-Ubuntu SMP Fri Apr 16 09:17:34 UTC 2010 
x86_64 GNU/Linux

$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
{noformat}

> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments: nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install 
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner 
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
> transaction database java.sql.SQLSyntaxErrorException: Table/View 
> 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
> 
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
>  Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

[Amareshwari Sriramadasu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977847#comment-13977847
 ] 

Amareshwari Sriramadasu commented on HIVE-6953:
---

Thanks [~alangates] and [~rhbutani] for trying. The tests are passing when i 
run them individually. When all the tests are run together, they are failing.

Here is what i have done :
{noformat}
git clone https://github.com/apache/hive apache-hive
git checkout branch-0.13
nohup mvn clean install -Phadoop-1 &
{noformat}

Attaching nohup output for reference.

bq. Is there anything in your logs indicating it tried to create the tables and 
failed?
Will check and update.

bq. Are you doing anything in your build to turn off the hive.in.test config 
value?
No. 

I'm thinking the test db or conf is getting cleaned up by some-other means, 
when all the tests are run together.



> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments: nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install 
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner 
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
> transaction database java.sql.SQLSyntaxErrorException: Table/View 
> 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
> 
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
>  Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

[Amareshwari Sriramadasu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amareshwari Sriramadasu updated HIVE-6953:
--

Attachment: nohup.out.gz

The nohup test output

> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
> Attachments: nohup.out.gz
>
>
> When I'm running all tests through the command 'mvn clean install 
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner 
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
> transaction database java.sql.SQLSyntaxErrorException: Table/View 
> 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
> 
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
>  Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6953) All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977841#comment-13977841
 ] 

Harish Butani commented on HIVE-6953:
-

Just ran the tests you mention: TestInitiator, TestWorker, TestCleaner on 0.13
I cannot reproduce the errors either.

> All CompactorTest failing with Table/View 'NEXT_TXN_ID' does not exist
> --
>
> Key: HIVE-6953
> URL: https://issues.apache.org/jira/browse/HIVE-6953
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Alan Gates
>
> When I'm running all tests through the command 'mvn clean install 
> -Phadoop-1', all CompactorTest classes TestInitiator, TestWorker, TestCleaner 
> fail with following exception :
> {noformat}
> org.apache.hadoop.hive.metastore.api.MetaException: Unable to select from 
> transaction database java.sql.SQLSyntaxErrorException: Table/View 
> 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown 
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown 
> Source)
> at 
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown 
> Source)
> 
> Caused by: java.sql.SQLException: Table/View 'NEXT_TXN_ID' does not exist.
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at 
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
>  Source)
> {noformat}
> This is happening on branch-0.13. Has anyone faced this problem?
> [~owen.omalley] or someone else help me solve this. Do i have to set anything?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6893) out of sequence error in HiveMetastore server

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977836#comment-13977836
 ] 

Hive QA commented on HIVE-6893:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641303/HIVE-6893.1.patch

{color:red}ERROR:{color} -1 due to 41 failed/errored test(s), 5417 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/9/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/9/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 41 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12641303

> out of sequence error in HiveMetastore server
> -
>
> Key: HIVE-6893
> URL: https://issues.apache.org/jira/browse/HIVE-6893
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Romain Rigaux
>Assignee: Naveen Gangam
> Fix For: 0.13.0
>
> Attachments: HIVE-6893.1.patch
>
>
> Calls listing databases or tables fail. It seems to be a concurrency problem.
> {code}
> 014-03-06 05:34:00,785 ERROR hive.log: 
> org.apache.thrift.TApplicationException: get_databases failed: out of 
> sequence response
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Cli

[jira] [Commented] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows

[shanyu zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977818#comment-13977818
 ] 

shanyu zhao commented on HIVE-5092:
---

There is a logic error in this patch, if HIVE_HADOOP_CLASSPATH is null, then it 
will go to else statement and try to read HIVE_HADOOP_CLASSPATH from HiveConf, 
resulting in error.

The correct statements should be:
{code}
if (HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_HADOOP_CLASSPATH)!= null) {
  if (variables.containsKey("HADOOP_CLASSPATH"))  {
variables.put("HADOOP_CLASSPATH", variables.get("HADOOP_CLASSPATH") + ";" + 
HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_HADOOP_CLASSPATH));
  } else {
variables.put("HADOOP_CLASSPATH", HiveConf.getVar(conf, 
HiveConf.ConfVars.HIVE_HADOOP_CLASSPATH));
  }
}
{code}

> Fix hiveserver2 mapreduce local job on Windows
> --
>
> Key: HIVE-5092
> URL: https://issues.apache.org/jira/browse/HIVE-5092
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Windows
>Reporter: Daniel Dai
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-5092-1.patch, HIVE-5092.2.patch
>
>
> Hiveserver2 fail on Mapreduce local job fail. For example:
> {code}
> select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v 
> on (s.name = v.name);
> {code}
> The root cause is "class not found" in the local hadoop job 
> (MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib. 
> Set HADOOP_CLASSPATH correctly will fix the issue.
> However, there is one complexity in Windows. We start Hiveserver2 using 
> Windows service console (services.msc), which takes hiveserver2.xml generated 
> by hive.cmd. There is no way to pass environment variable in hiveserver2.xml 
> (weird but reality). I attach a patch which pass it through command line 
> arguments and relay to HADOOP_CLASSPATH in Hive code. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6952) Hive 0.13 HiveOutputFormat breaks backwards compatibility

[Prasanth J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977785#comment-13977785
 ] 

Prasanth J commented on HIVE-6952:
--

LGTM. +1 (non-binding)

> Hive 0.13 HiveOutputFormat breaks backwards compatibility
> -
>
> Key: HIVE-6952
> URL: https://issues.apache.org/jira/browse/HIVE-6952
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Costin Leau
>Assignee: Ashutosh Chauhan
>Priority: Blocker
> Attachments: HIVE-6952.patch
>
>
> Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
> breaking backwards compatibility with previous releases; the return type of 
> getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.
> FSRecordWriter introduces one new method on top of RecordWriter however it 
> does not extend the previous interface and it lives in a completely new 
> package.
> Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
> code running on HIve 0.13, will break on anything lower than this.
> This could have easily been avoided by extending the existing interface or 
> introducing a new one that RecordWriter could have extended going forward. By 
> changing the signature, the existing contract (and compatibility) has been 
> voided.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20598: Make Hive Output Format backward compatible.

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20598/#review41114
---

Ship it!


Ship It!

- Prasanth_J


On April 23, 2014, 2:06 a.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20598/
> ---
> 
> (Updated April 23, 2014, 2:06 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Bugs: HIVE-6952
> https://issues.apache.org/jira/browse/HIVE-6952
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Make Hive Output Format backward compatible.
> 
> 
> Diffs
> -
> 
>   
> contrib/src/java/org/apache/hadoop/hive/contrib/fileformat/base64/Base64TextOutputFormat.java
>  bc601c4 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  6d383b5 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java
>  40927f6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 6af6b2d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java c52a093 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/PTFRowContainer.java 
> 4ab5516 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java 
> 768467e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 551f3aa 
>   ql/src/java/org/apache/hadoop/hive/ql/io/FSRecordWriter.java 83ac010 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveBinaryOutputFormat.java 
> 6768292 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 95db96b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveIgnoreKeyTextOutputFormat.java 
> ad6e4ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
>  ef6a982 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java e5ac805 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 
> 1fb5898 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java 
> 2186944 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java 
> 0cf00e9 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java e21f9fe 
>   ql/src/java/org/apache/hadoop/hive/ql/io/StatsProvidingRecordWriter.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroContainerOutputFormat.java 
> 8d75b44 
>   ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordWriter.java 
> 73e1cdd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 46c3bcc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java
>  b87c673 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java
>  cd603c2 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> de4867a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 5664f3f 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
> eaabc71 
> 
> Diff: https://reviews.apache.org/r/20598/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

Re: Review Request 20598: Make Hive Output Format backward compatible.

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20598/#review41113
---

Ship it!


LGTM.

- Prasanth_J


On April 23, 2014, 2:06 a.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20598/
> ---
> 
> (Updated April 23, 2014, 2:06 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Bugs: HIVE-6952
> https://issues.apache.org/jira/browse/HIVE-6952
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Make Hive Output Format backward compatible.
> 
> 
> Diffs
> -
> 
>   
> contrib/src/java/org/apache/hadoop/hive/contrib/fileformat/base64/Base64TextOutputFormat.java
>  bc601c4 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
>  6d383b5 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java
>  40927f6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 6af6b2d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java c52a093 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/PTFRowContainer.java 
> 4ab5516 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java 
> 768467e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 551f3aa 
>   ql/src/java/org/apache/hadoop/hive/ql/io/FSRecordWriter.java 83ac010 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveBinaryOutputFormat.java 
> 6768292 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 95db96b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveIgnoreKeyTextOutputFormat.java 
> ad6e4ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
>  ef6a982 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java e5ac805 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 
> 1fb5898 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java 
> 2186944 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java 
> 0cf00e9 
>   ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java e21f9fe 
>   ql/src/java/org/apache/hadoop/hive/ql/io/StatsProvidingRecordWriter.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroContainerOutputFormat.java 
> 8d75b44 
>   ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordWriter.java 
> 73e1cdd 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 46c3bcc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java
>  b87c673 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java
>  cd603c2 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> de4867a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 5664f3f 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
> eaabc71 
> 
> Diff: https://reviews.apache.org/r/20598/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

[jira] [Commented] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977756#comment-13977756
 ] 

Lefty Leverenz commented on HIVE-5092:
--

Could we have a description for *hive.hadoop.classpath* in either a 
HiveConf.java comment, hive-default.xml.template, or a release comment?

> Fix hiveserver2 mapreduce local job on Windows
> --
>
> Key: HIVE-5092
> URL: https://issues.apache.org/jira/browse/HIVE-5092
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Windows
>Reporter: Daniel Dai
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-5092-1.patch, HIVE-5092.2.patch
>
>
> Hiveserver2 fail on Mapreduce local job fail. For example:
> {code}
> select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v 
> on (s.name = v.name);
> {code}
> The root cause is "class not found" in the local hadoop job 
> (MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib. 
> Set HADOOP_CLASSPATH correctly will fix the issue.
> However, there is one complexity in Windows. We start Hiveserver2 using 
> Windows service console (services.msc), which takes hiveserver2.xml generated 
> by hive.cmd. There is no way to pass environment variable in hiveserver2.xml 
> (weird but reality). I attach a patch which pass it through command line 
> arguments and relay to HADOOP_CLASSPATH in Hive code. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6945:
---

Attachment: HIVE-6945.02.patch

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6945.01.patch, HIVE-6945.02.patch, HIVE-6945.patch
>
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20565: HIVE-6945 issues with dropping partitions on Oracle

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20565/
---

(Updated April 23, 2014, 2:37 a.m.)


Review request for hive, Ashutosh Chauhan and Thejas Nair.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
a5c7daf 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 6a0eabe 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java fe15101 

Diff: https://reviews.apache.org/r/20565/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-6962) NoSuchMethodError exception when using HIVE 0.13 with Hadoop 1.0.4

[Sun Rui (JIRA)]

Sun Rui created HIVE-6962:
-

 Summary: NoSuchMethodError exception when using HIVE 0.13 with 
Hadoop 1.0.4
 Key: HIVE-6962
 URL: https://issues.apache.org/jira/browse/HIVE-6962
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 0.13.0
 Environment: Hadoop 1.0.4
Reporter: Sun Rui


Use HIVE 0.13 with Hadoop 1.0.4. When executing a query, get the following 
exception:
Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.hadoop.conf.Configuration.unset(Ljava/lang/String;)V
at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3416)
at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3401)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214)
at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:265)
at org.apache.hadoop.hive.ql.Context.getMRTmpPath(Context.java:323)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1442)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1192)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9209)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:391)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:291)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:944)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1009)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:880)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:870)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

This is because there is no unset() method in some versions of Hadoop.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6962) NoSuchMethodError exception when using HIVE 0.13 with Hadoop 1.0.4

[Sun Rui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sun Rui updated HIVE-6962:
--

Description: 
Use HIVE 0.13 with Hadoop 1.0.4. When executing a query, get the following 
exception:

{noformat}
Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.hadoop.conf.Configuration.unset(Ljava/lang/String;)V
at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3416)
at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3401)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214)
at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:265)
at org.apache.hadoop.hive.ql.Context.getMRTmpPath(Context.java:323)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1442)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1192)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9209)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:391)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:291)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:944)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1009)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:880)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:870)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
{noformat}

This is because there is no unset() method in some versions of Hadoop.

  was:
Use HIVE 0.13 with Hadoop 1.0.4. When executing a query, get the following 
exception:
Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.hadoop.conf.Configuration.unset(Ljava/lang/String;)V
at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3416)
at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3401)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214)
at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:265)
at org.apache.hadoop.hive.ql.Context.getMRTmpPath(Context.java:323)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1442)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1192)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9209)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:391)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:291)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:944)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1009)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:880)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:870)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

This is because there is no unset() method in some 

[jira] [Commented] (HIVE-6932) hive README needs update

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977741#comment-13977741
 ] 

Lefty Leverenz commented on HIVE-6932:
--

bq. I actually prefer "SQL" or "Hive SQL"

A general discussion would be good, but let's not hold up this patch for it.

bq. Also, it talks about average latency being in minutes, which is very 
misleading.

Did you mean to revise the latency part too?

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6921) index creation fails with sql std auth turned on

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977737#comment-13977737
 ] 

Thejas M Nair commented on HIVE-6921:
-

+1

> index creation fails with sql std auth turned on 
> -
>
> Key: HIVE-6921
> URL: https://issues.apache.org/jira/browse/HIVE-6921
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Indexing, Security
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6921.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20426: index creation fails with std sql auth turned on

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20426/#review41110
---

Ship it!


Ship It!

- Thejas Nair


On April 16, 2014, 9:09 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/20426/
> ---
> 
> (Updated April 16, 2014, 9:09 p.m.)
> 
> 
> Review request for hive and Thejas Nair.
> 
> 
> Bugs: HIVE-6921
> https://issues.apache.org/jira/browse/HIVE-6921
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Issue was owner and default grants were not set for underlying table of index.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 254e2b0 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ae05f04 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java ae3c11b 
>   ql/src/test/queries/clientpositive/authorization_index.q PRE-CREATION 
>   ql/src/test/results/clientpositive/authorization_index.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/20426/diff/
> 
> 
> Testing
> ---
> 
> Added new test and ran existing auth tests.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

[jira] [Updated] (HIVE-6952) Hive 0.13 HiveOutputFormat breaks backwards compatibility

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6952:
---

Assignee: Ashutosh Chauhan
  Status: Patch Available  (was: Open)

> Hive 0.13 HiveOutputFormat breaks backwards compatibility
> -
>
> Key: HIVE-6952
> URL: https://issues.apache.org/jira/browse/HIVE-6952
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Costin Leau
>Assignee: Ashutosh Chauhan
>Priority: Blocker
> Attachments: HIVE-6952.patch
>
>
> Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
> breaking backwards compatibility with previous releases; the return type of 
> getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.
> FSRecordWriter introduces one new method on top of RecordWriter however it 
> does not extend the previous interface and it lives in a completely new 
> package.
> Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
> code running on HIve 0.13, will break on anything lower than this.
> This could have easily been avoided by extending the existing interface or 
> introducing a new one that RecordWriter could have extended going forward. By 
> changing the signature, the existing contract (and compatibility) has been 
> voided.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6952) Hive 0.13 HiveOutputFormat breaks backwards compatibility

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6952:
---

Attachment: HIVE-6952.patch

[~costin] Sorry for inconvenience. I have attached a patch. Can you try it out 
and see if it fixes the issue for you. If it does, we can aim to have it in 
0.13.1 

> Hive 0.13 HiveOutputFormat breaks backwards compatibility
> -
>
> Key: HIVE-6952
> URL: https://issues.apache.org/jira/browse/HIVE-6952
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Costin Leau
>Priority: Blocker
> Attachments: HIVE-6952.patch
>
>
> Hive 0.13 changed the signature of HiveOutputFormat (through commit r1527149) 
> breaking backwards compatibility with previous releases; the return type of 
> getHiveRecordWriter has been changed from RecordWriter to FSRecordWriter.
> FSRecordWriter introduces one new method on top of RecordWriter however it 
> does not extend the previous interface and it lives in a completely new 
> package.
> Thus code running fine on Hive 0.12 breaks on Hive 0.13. After the upgrade, 
> code running on HIve 0.13, will break on anything lower than this.
> This could have easily been avoided by extending the existing interface or 
> introducing a new one that RecordWriter could have extended going forward. By 
> changing the signature, the existing contract (and compatibility) has been 
> voided.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6924) MapJoinKeyBytes::hashCode() should use Murmur hash

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6924:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

committed to trunk

> MapJoinKeyBytes::hashCode() should use Murmur hash
> --
>
> Key: HIVE-6924
> URL: https://issues.apache.org/jira/browse/HIVE-6924
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.14.0
>
> Attachments: HIVE-6924.01.patch, HIVE-6924.patch
>
>
> Existing hashCode is bad, causes HashMap to cluster



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 20598: Make Hive Output Format backward compatible.

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20598/
---

Review request for hive and Prasanth_J.


Bugs: HIVE-6952
https://issues.apache.org/jira/browse/HIVE-6952


Repository: hive-git


Description
---

Make Hive Output Format backward compatible.


Diffs
-

  
contrib/src/java/org/apache/hadoop/hive/contrib/fileformat/base64/Base64TextOutputFormat.java
 bc601c4 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 
6d383b5 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java
 40927f6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 6af6b2d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java c52a093 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/PTFRowContainer.java 
4ab5516 
  ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java 
768467e 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 551f3aa 
  ql/src/java/org/apache/hadoop/hive/ql/io/FSRecordWriter.java 83ac010 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveBinaryOutputFormat.java 6768292 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 95db96b 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveIgnoreKeyTextOutputFormat.java 
ad6e4ba 
  
ql/src/java/org/apache/hadoop/hive/ql/io/HiveNullValueSequenceFileOutputFormat.java
 ef6a982 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveOutputFormat.java e5ac805 
  ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughOutputFormat.java 
1fb5898 
  ql/src/java/org/apache/hadoop/hive/ql/io/HivePassThroughRecordWriter.java 
2186944 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveSequenceFileOutputFormat.java 
0cf00e9 
  ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java e21f9fe 
  ql/src/java/org/apache/hadoop/hive/ql/io/StatsProvidingRecordWriter.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroContainerOutputFormat.java 
8d75b44 
  ql/src/java/org/apache/hadoop/hive/ql/io/avro/AvroGenericRecordWriter.java 
73e1cdd 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java 46c3bcc 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java 
b87c673 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java
 cd603c2 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java de4867a 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
5664f3f 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java 
eaabc71 

Diff: https://reviews.apache.org/r/20598/diff/


Testing
---


Thanks,

Ashutosh Chauhan

[jira] [Updated] (HIVE-6901) Explain plan doesn't show operator tree for the fetch operator

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6901:
--

Attachment: HIVE-6901.2.patch

> Explain plan doesn't show operator tree for the fetch operator
> --
>
> Key: HIVE-6901
> URL: https://issues.apache.org/jira/browse/HIVE-6901
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>Priority: Minor
> Attachments: HIVE-6901.1.patch, HIVE-6901.2.patch, HIVE-6901.2.patch, 
> HIVE-6901.2.patch, HIVE-6901.2.patch, HIVE-6901.patch
>
>
> Explaining a simple select query that involves a MR phase doesn't show 
> processor tree for the fetch operator.
> {code}
> hive> explain select d from test;
> OK
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 is a root stage
> STAGE PLANS:
>   Stage: Stage-1
> Map Reduce
>   Map Operator Tree:
> ...
>   Stage: Stage-0
> Fetch Operator
>   limit: -1
> {code}
> It would be nice if the operator tree is shown even if there is only one node.
> Please note that in local execution, the operator tree is complete:
> {code}
> hive> explain select * from test;
> OK
> STAGE DEPENDENCIES:
>   Stage-0 is a root stage
> STAGE PLANS:
>   Stage: Stage-0
> Fetch Operator
>   limit: -1
>   Processor Tree:
> TableScan
>   alias: test
>   Statistics: Num rows: 8 Data size: 34 Basic stats: COMPLETE Column 
> stats: NONE
>   Select Operator
> expressions: d (type: int)
> outputColumnNames: _col0
> Statistics: Num rows: 8 Data size: 34 Basic stats: COMPLETE 
> Column stats: NONE
> ListSink
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977718#comment-13977718
 ] 

Xuefu Zhang commented on HIVE-6835:
---

bq. I feel that prepending "table." might be a cleaner and less confusing 
approach. What are your thoughts on this?

This doesn't seem to be a viable approach due to its hacky/problematic nature.

bq.  many places just for this new method doesn't seem very clean to me.

You could have a utility method somewhere so that you need to call instanceof 
only once. Something like this:

{code}
  public static void initializeSerde(SerDe serde, Properties tblProps, 
Properties partProps) {
if (serde instanceof AbstractSerde) {
   ...
} else {
  ...
}
  }
{code}
Then, each caller just needs to switch to this method.

bq.  If we leave partProps null, then the default new initialize() method 
implementation will just pass null to the old initialize() method.

This sounds good to me.

> Reading of partitioned Avro data fails if partition schema does not match 
> table schema
> --
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch, HIVE-6835.3.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde 
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties 
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type": 
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"} 
> } ] }')  STORED as INPUTFORMAT  
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'  OUTPUTFORMAT  
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde 
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with 
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
>  "record", "fields": [ {"name":"intfield","type":"int","default":0},{ 
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector 
> cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6932) hive README needs update

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977717#comment-13977717
 ] 

Ashutosh Chauhan commented on HIVE-6932:


+1 LGTM

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6932) hive README needs update

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977713#comment-13977713
 ] 

Thejas M Nair commented on HIVE-6932:
-

bq. Also a question: Is "QL" the proper name for the language or "HiveQL"? The 
README says HiveQL twice in the Getting Started section. The wiki is 
inconsistent with QL, HiveQL, and Hive QL.
I am not sure. I actually prefer "SQL" or "Hive SQL" , but not everyone was 
happy with that. Maybe we should have a discussion on mailing list to see if we 
have an agreement now.


> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6958) update union_remove_*, other tests for hadoop-2

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6958:
-

Status: Open  (was: Patch Available)

Cancelling the patch.

> update union_remove_*, other tests for hadoop-2
> ---
>
> Key: HIVE-6958
> URL: https://issues.apache.org/jira/browse/HIVE-6958
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6958.1.patch
>
>
> Update q.out files to match totalSize for Linux platform.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6947) More fixes for tests on hadoop-2

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6947:
---

   Resolution: Fixed
Fix Version/s: 0.14.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Jason & Prashant!

> More fixes for tests on hadoop-2 
> -
>
> Key: HIVE-6947
> URL: https://issues.apache.org/jira/browse/HIVE-6947
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.14.0
>
> Attachments: HIVE-6947.1.patch, HIVE-6947.patch
>
>
> Few more fixes for test cases on hadoop-2



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema

[Anthony Hsu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977710#comment-13977710
 ] 

Anthony Hsu commented on HIVE-6835:
---

Yes, this is possible, but I would have to add these "instanceof AbstractSerde" 
checks and then cast the Deserializer as an AbstractSerde before I can use the 
new initialize() method.  There are dozens of usages of .initialize() and 
adding all this type checking/casting code in so many places just for this new 
method doesn't seem very clean to me.

Also, if we add the new initialize() method, what should we do for table-level 
serde initialization?  When dealing with the table, there are no partition 
properties, so are we supposed to pass the table properties for both the 
tblProps and partProps arguments? If we leave partProps null, then the default 
new initialize() method implementation will just pass null to the old 
initialize() method.

There doesn't seem to be a very clean way of adding a new initialize() method 
without creating a lot of redundant boilerplate code and creating confusion 
which initialize() method to use and what values to pass in.  Given these 
concerns, I feel that prepending "table." might be a cleaner and less confusing 
approach.  What are your thoughts on this?

> Reading of partitioned Avro data fails if partition schema does not match 
> table schema
> --
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch, HIVE-6835.3.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde 
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties 
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type": 
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"} 
> } ] }')  STORED as INPUTFORMAT  
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'  OUTPUTFORMAT  
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde 
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with 
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
>  "record", "fields": [ {"name":"intfield","type":"int","default":0},{ 
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector 
> cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6958) update union_remove_*, other tests for hadoop-2

[Prasanth J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977699#comment-13977699
 ] 

Prasanth J commented on HIVE-6958:
--

The reason for this failure, is related to the behaviour of UNION. INSERT 
queries with UNION ALL will create sub-directories under table/partition 
directory. For example:
{code}
insert overwrite table outputTbl1
SELECT *
FROM (
  SELECT key, count(1) as values from inputTbl1 group by key
  UNION ALL
  SELECT key, count(1) as values from inputTbl1 group by key
) a;
{code}

for the above query, the warehouse/outputTbl1 directory will have 2 
sub-directories corresponding to each SELECT queries like
warehouse/outputTbl1/15/, warehouse/outputTbl1/16/. Here 15 and 16 are operator 
identifiers 
https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java#L223

This special case (having directory under table) happens only for union insert. 
All other cases will have files underneath the table directory for 
unpartitioned tables. But the metastore utils for updating the fast stats are 
not aware of this directory structure (it expects files underneath table 
directory).  The Warehouse.getFileStatusesForUnpartitionedTable() recurses only 
one level under table directory if it is unpartitioned table 
https://github.com/apache/hive/blob/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java#L540.
 For union insert, if only 1 level is recursed you will get only the folder 
sizes and not the actual file sizes. Folder sizes are different for different 
OSes. It looks like original diff was generated using Mac OS X and the new diff 
was generated using Centos. Both the diffs are *wrong* as they return folder 
size as opposed to file sizes. 

1) One way to fix this is to change the recurse level to a value greater than 
1. 
2) Another way would be to fix UNION to create files instead of directories. To 
resolve filename conflict it can append the operator id to filename.

[~ashutoshc]/[~jdere] do you guys have any thoughts about this?

> update union_remove_*, other tests for hadoop-2
> ---
>
> Key: HIVE-6958
> URL: https://issues.apache.org/jira/browse/HIVE-6958
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6958.1.patch
>
>
> Update q.out files to match totalSize for Linux platform.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6948) HiveServer2 doesn't respect HIVE_AUX_JARS_PATH

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977696#comment-13977696
 ] 

Hive QA commented on HIVE-6948:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641179/HIVE-6948.patch

{color:red}ERROR:{color} -1 due to 42 failed/errored test(s), 5417 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/7/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/7/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 42 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12641179

> HiveServer2 doesn't respect HIVE_AUX_JARS_PATH
> --
>
> Key: HIVE-6948
> URL: https://issues.apache.org/jira/browse/HIVE-6948
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Peng Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-6948.patch
>
>
> HiveServer2 ignores HIVE_AUX_JARS_PATH.
> This will cause aux jars not distributed to Yarn cluster, and job will fail 
> without dependent jars.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5538) Turn on vectorization by default.

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5538:
---

Status: Patch Available  (was: Open)

> Turn on vectorization by default.
> -
>
> Key: HIVE-5538
> URL: https://issues.apache.org/jira/browse/HIVE-5538
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5538.1.patch, HIVE-5538.2.patch, HIVE-5538.3.patch, 
> HIVE-5538.4.patch
>
>
>   Vectorization should be turned on by default, so that users don't have to 
> specifically enable vectorization. 
>   Vectorization code validates and ensures that a query falls back to row 
> mode if it is not supported on vectorized code path. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5538) Turn on vectorization by default.

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5538:
---

Attachment: HIVE-5538.4.patch

Updated patch fixes some tests.

> Turn on vectorization by default.
> -
>
> Key: HIVE-5538
> URL: https://issues.apache.org/jira/browse/HIVE-5538
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5538.1.patch, HIVE-5538.2.patch, HIVE-5538.3.patch, 
> HIVE-5538.4.patch
>
>
>   Vectorization should be turned on by default, so that users don't have to 
> specifically enable vectorization. 
>   Vectorization code validates and ensures that a query falls back to row 
> mode if it is not supported on vectorized code path. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5538) Turn on vectorization by default.

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5538:
---

Status: Open  (was: Patch Available)

> Turn on vectorization by default.
> -
>
> Key: HIVE-5538
> URL: https://issues.apache.org/jira/browse/HIVE-5538
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5538.1.patch, HIVE-5538.2.patch, HIVE-5538.3.patch, 
> HIVE-5538.4.patch
>
>
>   Vectorization should be turned on by default, so that users don't have to 
> specifically enable vectorization. 
>   Vectorization code validates and ensures that a query falls back to row 
> mode if it is not supported on vectorized code path. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6469) skipTrash option in hive command line

[Edward Capriolo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977685#comment-13977685
 ] 

Edward Capriolo commented on HIVE-6469:
---

If a user is willing to commit an optional syntax that does not cause a 
language ambiguity I think we should allow the user to add the feature.

Rational : currently dfs -rm allows an optional --skip trash.  Normal users are 
able to control if a delete skips trash or not, regardless of how admins set 
the trash feature.

A natual extension is to extend this functionality to drop table.



> skipTrash option in hive command line
> -
>
> Key: HIVE-6469
> URL: https://issues.apache.org/jira/browse/HIVE-6469
> Project: Hive
>  Issue Type: New Feature
>  Components: CLI
>Affects Versions: 0.12.0
>Reporter: Jayesh
> Fix For: 0.12.1
>
> Attachments: HIVE-6469.patch
>
>
> hive drop table command deletes the data from HDFS warehouse and puts it into 
> Trash.
> Currently there is no way to provide flag to tell warehouse to skip trash 
> while deleting table data.
> This ticket is to add skipTrash feature in hive command-line, that looks as 
> following. 
> hive -e "drop table skipTrash testTable"
> This would be good feature to add, so that user can specify when not to put 
> data into trash directory and thus not to fill hdfs space instead of relying 
> on trash interval and policy configuration to take care of disk filling issue.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6379) webhcat is not operational with Hadoop 1 due to Slf4j conflicts

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-6379:
-

Priority: Minor  (was: Major)

> webhcat is not operational with Hadoop 1 due to Slf4j conflicts
> ---
>
> Key: HIVE-6379
> URL: https://issues.apache.org/jira/browse/HIVE-6379
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Minor
>
> 1. build current trunk: mvn clean install -Pdist,hadoop-1 -DskipTests
> 2. 
> /Users/ekoifman/dev/hive/packaging/target/apache-hive-0.13.0-SNAPSHOT-bin/apache-hive-0.13.0-SNAPSHOT-bin/hcatalog
> 3. ./sbin/webhcat_server.sh start
> 4. http://localhost:50111/templeton/v1/ddl/database?user.name=ekoifman
> webhcat-console-error.log has
> log4j:WARN No appenders could be found for logger 
> (org.apache.hive.hcatalog.templeton.AppConfig).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:613)
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:200)
> at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:94)
> at org.eclipse.jetty.util.component.Container.add(Container.java:201)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:164)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:106)
> at org.eclipse.jetty.server.Server.setConnectors(Server.java:184)
> at org.eclipse.jetty.server.Server.(Server.java:97)
> at org.apache.hive.hcatalog.templeton.Main.runServer(Main.java:162)
> at org.apache.hive.hcatalog.templeton.Main.run(Main.java:114)
> at org.apache.hive.hcatalog.templeton.Main.main(Main.java:259)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> webhcat-console-error.log (END) 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6961) Drop partitions treats partition columns as strings

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977650#comment-13977650
 ] 

Jitendra Nath Pandey commented on HIVE-6961:


+1, lgtm

> Drop partitions treats partition columns as strings
> ---
>
> Key: HIVE-6961
> URL: https://issues.apache.org/jira/browse/HIVE-6961
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6961.patch
>
>
> Discovered just now while testing HIVE-6945



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6961) Drop partitions treats partition columns as strings

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977629#comment-13977629
 ] 

Sergey Shelukhin commented on HIVE-6961:


[~jnp] can you take a look?

> Drop partitions treats partition columns as strings
> ---
>
> Key: HIVE-6961
> URL: https://issues.apache.org/jira/browse/HIVE-6961
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6961.patch
>
>
> Discovered just now while testing HIVE-6945



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6961) Drop partitions treats partition columns as strings

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6961:
---

Attachment: HIVE-6961.patch

> Drop partitions treats partition columns as strings
> ---
>
> Key: HIVE-6961
> URL: https://issues.apache.org/jira/browse/HIVE-6961
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6961.patch
>
>
> Discovered just now while testing HIVE-6945



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6961) Drop partitions treats partition columns as strings

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6961:
---

Status: Patch Available  (was: Open)

> Drop partitions treats partition columns as strings
> ---
>
> Key: HIVE-6961
> URL: https://issues.apache.org/jira/browse/HIVE-6961
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6961.patch
>
>
> Discovered just now while testing HIVE-6945



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6961) Drop partitions treats partition columns as strings

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-6961:
--

 Summary: Drop partitions treats partition columns as strings
 Key: HIVE-6961
 URL: https://issues.apache.org/jira/browse/HIVE-6961
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


Discovered just now while testing HIVE-6945



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6958) update union_remove_*, other tests for hadoop-2

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977619#comment-13977619
 ] 

Jason Dere commented on HIVE-6958:
--

Szehon, almost all of the tests (except stats_partialscan_autogether.q) were 
hadoop-2 only tests.  I'll take a look at that stats_partialscan_autogether.q 
and see what's up with that one.
Prasanth, you are correct that in these cases the difference in size is due to 
different folder file size on the different platforms.

> update union_remove_*, other tests for hadoop-2
> ---
>
> Key: HIVE-6958
> URL: https://issues.apache.org/jira/browse/HIVE-6958
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6958.1.patch
>
>
> Update q.out files to match totalSize for Linux platform.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977616#comment-13977616
 ] 

Xuefu Zhang commented on HIVE-6835:
---

Not sure if I understand your problem correctly, but I do understand that the 
scope of the proposed change has got bigger than your original approach. For 
any caller of serde initialization, we should be able to find whether serde 
instance extends AbstractSerde. If so, we cast the serde instance to 
AbstractSerde and call initialize(arg1, arg2, arg3). Otherwise, call 
serde.initialize(arg1, arg2). Does this solve the problem?

> Reading of partitioned Avro data fails if partition schema does not match 
> table schema
> --
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch, HIVE-6835.3.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde 
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties 
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type": 
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"} 
> } ] }')  STORED as INPUTFORMAT  
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'  OUTPUTFORMAT  
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde 
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with 
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
>  "record", "fields": [ {"name":"intfield","type":"int","default":0},{ 
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector 
> cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6960) Set Hive pom to use Hadoop-2.4

[Jason Dere (JIRA)]

Jason Dere created HIVE-6960:


 Summary: Set Hive pom to use Hadoop-2.4
 Key: HIVE-6960
 URL: https://issues.apache.org/jira/browse/HIVE-6960
 Project: Hive
  Issue Type: Bug
  Components: Build Infrastructure
Affects Versions: 0.14.0
Reporter: Jason Dere


A number of the hadoop-2 unit test failures are due to HADOOP-10425, fixed in 
Hadoop 2.4.  Perhaps we should move onto that version.

- org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
- org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
- org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
- 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
- org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
- 
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6927) Add support for MSSQL in schematool

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977607#comment-13977607
 ] 

Lefty Leverenz commented on HIVE-6927:
--

This doesn't seem to need any documentation in the wiki ... or does it?

* [Hive Metastore Administration 
|https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin]
* [Hive Schema Tool 
|https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool]

> Add support for MSSQL in schematool
> ---
>
> Key: HIVE-6927
> URL: https://issues.apache.org/jira/browse/HIVE-6927
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
> Fix For: 0.14.0
>
> Attachments: HIVE-6927.patch
>
>
> Schematool is the preferred way of initializing schema for Hive. Since 
> HIVE-6862 provided the script for MSSQL it would be nice to add the support 
> for it in schematool.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Plan: permanently move hive builds from bigtop

[Szehon Ho]

Patch upload auto-trigger is now enabled, thanks Ashutosh for helping me
fix the Jenkins url from Apache-Infra side.  Now there's no need to
manually submit Pre-commit builds, unless it got missed somehow.  And in
this Jenkins, you can easily see all the JIRA's in the queue:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/

Sorry, please use this URL and not the first one, as job was renamed due to
the auto-trigger expecting this format.

And to repeat, Test Reporting and the 'Logs located at:' link is still
broken and tracked in HIVE-6937.  For now, to see the logs of a failed
build, you have to manually navigate.
Example, for PreCommit-Hive-BUILD#6, logs are at:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-6/.
The HiveQA comment on the JIRA should mention the build number.

Thanks,
Szehon


On Sun, Apr 20, 2014 at 11:07 PM, Szehon Ho  wrote:

> It looks great, thanks Lefty!
>
>
> On Sun, Apr 20, 2014 at 2:22 PM, Lefty Leverenz 
> wrote:
>
>> Nice doc, Szehon.  I did some minor editing so you might want to make sure
>> I didn't introduce any errors.
>>
>>
>> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
>>
>> -- Lefty
>>
>>
>> On Sat, Apr 19, 2014 at 9:45 PM, Szehon Ho  wrote:
>>
>> > Migration is done, I updated the wiki to add all the details of the new
>> > setup:
>> >
>> https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure
>> >
>> > New Jenkins URL to submit pre-commit jobs:
>> >
>> >
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/precommit-hive/
>> >   Again, this has to be done manually for time being, by clicking on
>> 'build
>> > with parameters', and entering the issue number as a parameter.  I've
>> > submitted some already.  I'll reach out to some committers to get the
>> > auto-trigger working.
>> >
>> > As I mentioned, there is some work to fix the test-reporting, due to the
>> > framework using old url scheme.  I am tracking it at
>> > HIVE-6937.
>> >  For now I am hosting log directory separately, if you want to see test
>> > logs, you have to manually go the url corresponding to your build, like:
>> >
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/precommit-hive-11/for
>> > run#11.  Sorry about that.
>> >
>> > Let me know if you see other issues, thanks!
>> > Szehon
>> >
>> >
>> > On Fri, Apr 18, 2014 at 2:11 PM, Thejas Nair 
>> > wrote:
>> >
>> > > Sounds good.
>> > > Thanks Szehon!
>> > >
>> > >
>> > > On Fri, Apr 18, 2014 at 10:17 AM, Ashutosh Chauhan <
>> hashut...@apache.org
>> > >
>> > > wrote:
>> > > > +1 Thanks Szehon!
>> > > >
>> > > >
>> > > > On Fri, Apr 18, 2014 at 6:29 AM, Xuefu Zhang 
>> > > wrote:
>> > > >
>> > > >> +1. Thanks for taking care of this.
>> > > >>
>> > > >>
>> > > >> On Thu, Apr 17, 2014 at 11:00 PM, Szehon Ho 
>> > > wrote:
>> > > >>
>> > > >> > Hi,
>> > > >> >
>> > > >> > This week the machine running Hive builds at
>> > > >> > http://bigtop01.cloudera.org:8080/view/Hive/?  ran out of
>> space, so
>> > > new
>> > > >> > jobs like Precommit tests stopped.  Its still not resolved there,
>> > > there
>> > > >> was
>> > > >> > another email today on Bigtop list, but there's very few people
>> with
>> > > root
>> > > >> > access to that host, and they still haven't responded.
>> > > >> >
>> > > >> > I chatted with Brock, he has also seen various issues with Bigtop
>> > > jenkins
>> > > >> > in the past, so I am thinking to move the Jenkins jobs to the
>> PTest
>> > > >> master
>> > > >> > itself, where some PMC already have access and can admin if
>> needed.
>> > > >> >  Currently I am hosting the pre-commit Jenkins job on my own EC2
>> > > instance
>> > > >> > as stop-gap.
>> > > >> >
>> > > >> > Other advantages of hosting our own Jenkins:
>> > > >> > 1. No need to wait for other Bigtop jobs to run.
>> > > >> > 2. Bigtop is using a version of Jenkins that doesnt show
>> parameters
>> > > like
>> > > >> > JIRA number for queued jobs, so impossible to tell whether a
>> patch
>> > got
>> > > >> > picked up and where it is in queue.
>> > > >> > 3. Eliminate network hop from Bigtop box to our PTest master.
>> > > >> >
>> > > >> > The disadvantage is:
>> > > >> > 1. We don't have much experience doing Jenkins admin, but it
>> doesn't
>> > > look
>> > > >> > too bad.  Mostly, restart if there's issue and clean up if out of
>> > > space.
>> > > >> >
>> > > >> > I wonder what people think, and if there's any objections to
>> this?
>> >  If
>> > > >> not,
>> > > >> > I'll try setting up this weekend.  Then, there is some follow-up
>> > work,
>> > > >> like
>> > > >> > changing the Jenkins url's displayed in the test report.
>> > > >> >
>> > > >> > Thanks!
>> > > >> > Szehon
>> > > >> >
>> > > >>
>> > >
>> > > --
>> > > CONFIDENTIALITY NOTICE
>> > > NOTICE: This message is intended for the use of the 

[jira] [Updated] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-5092:


Status: Patch Available  (was: Open)

> Fix hiveserver2 mapreduce local job on Windows
> --
>
> Key: HIVE-5092
> URL: https://issues.apache.org/jira/browse/HIVE-5092
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Windows
>Reporter: Daniel Dai
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-5092-1.patch, HIVE-5092.2.patch
>
>
> Hiveserver2 fail on Mapreduce local job fail. For example:
> {code}
> select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v 
> on (s.name = v.name);
> {code}
> The root cause is "class not found" in the local hadoop job 
> (MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib. 
> Set HADOOP_CLASSPATH correctly will fix the issue.
> However, there is one complexity in Windows. We start Hiveserver2 using 
> Windows service console (services.msc), which takes hiveserver2.xml generated 
> by hive.cmd. There is no way to pass environment variable in hiveserver2.xml 
> (weird but reality). I attach a patch which pass it through command line 
> arguments and relay to HADOOP_CLASSPATH in Hive code. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-5092:


Attachment: HIVE-5092.2.patch

cc-ing [~daijy] for review. I have rebased the original patch without the 
change in hiveserver2.cmd changes. We will have the hiveserver2.cmd once we 
introduce windows scripts for Hive installation. The current changes provide a 
mechanism for accepting hive.hadoop.classpath as part of hive.

Thanks
Hari

> Fix hiveserver2 mapreduce local job on Windows
> --
>
> Key: HIVE-5092
> URL: https://issues.apache.org/jira/browse/HIVE-5092
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Windows
>Reporter: Daniel Dai
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-5092-1.patch, HIVE-5092.2.patch
>
>
> Hiveserver2 fail on Mapreduce local job fail. For example:
> {code}
> select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v 
> on (s.name = v.name);
> {code}
> The root cause is "class not found" in the local hadoop job 
> (MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib. 
> Set HADOOP_CLASSPATH correctly will fix the issue.
> However, there is one complexity in Windows. We start Hiveserver2 using 
> Windows service console (services.msc), which takes hiveserver2.xml generated 
> by hive.cmd. There is no way to pass environment variable in hiveserver2.xml 
> (weird but reality). I attach a patch which pass it through command line 
> arguments and relay to HADOOP_CLASSPATH in Hive code. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6950) Parsing Error in GROUPING SETS

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977595#comment-13977595
 ] 

Harish Butani commented on HIVE-6950:
-

My 2 cents would be: no. This is a bug. Should get fixed. 
Documenting every bug and then updating when we fix, would be asking too much...


> Parsing Error in GROUPING SETS
> --
>
> Key: HIVE-6950
> URL: https://issues.apache.org/jira/browse/HIVE-6950
> Project: Hive
>  Issue Type: Bug
>Reporter: Rohit Agarwal
>
> The following query:
> {code}
> SELECT tab1.a,
>tab1.b,
>SUM(tab1.c)
> FROM tab1
> GROUP BY tab1.a,
>  tab1.b
> GROUPING SETS ((tab1.a, tab1.b))
> {code}
> results in the following error:
> {code}
> ParseException line 7:22 missing ) at ',' near ''
> line 7:31 extraneous input ')' expecting EOF near ''
> {code}
> Changing the query to:
> {code}
> SELECT tab1.a,
>tab1.b,
>SUM(tab1.c)
> FROM tab1
> GROUP BY tab1.a,
>  tab1.b
> GROUPING SETS ((a, tab1.b))
> {code}
> makes it work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-5092) Fix hiveserver2 mapreduce local job on Windows

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan reassigned HIVE-5092:
---

Assignee: Hari Sankar Sivarama Subramaniyan  (was: Daniel Dai)

> Fix hiveserver2 mapreduce local job on Windows
> --
>
> Key: HIVE-5092
> URL: https://issues.apache.org/jira/browse/HIVE-5092
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Windows
>Reporter: Daniel Dai
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-5092-1.patch
>
>
> Hiveserver2 fail on Mapreduce local job fail. For example:
> {code}
> select /*+ MAPJOIN(v) */ registration from studenttab10k s join votertab10k v 
> on (s.name = v.name);
> {code}
> The root cause is "class not found" in the local hadoop job 
> (MapredLocalTask.execute). HADOOP_CLASSPATH does not include $HIVE_HOME/lib. 
> Set HADOOP_CLASSPATH correctly will fix the issue.
> However, there is one complexity in Windows. We start Hiveserver2 using 
> Windows service console (services.msc), which takes hiveserver2.xml generated 
> by hive.cmd. There is no way to pass environment variable in hiveserver2.xml 
> (weird but reality). I attach a patch which pass it through command line 
> arguments and relay to HADOOP_CLASSPATH in Hive code. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6958) update union_remove_*, other tests for hadoop-2

[Prasanth J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977570#comment-13977570
 ] 

Prasanth J commented on HIVE-6958:
--

I suspect the new diffs are showing folder size as opposed to the files inside 
them. I will take a look at it in my linux box and will report back.

> update union_remove_*, other tests for hadoop-2
> ---
>
> Key: HIVE-6958
> URL: https://issues.apache.org/jira/browse/HIVE-6958
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6958.1.patch
>
>
> Update q.out files to match totalSize for Linux platform.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive

[Hari Sankar Sivarama Subramaniyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977566#comment-13977566
 ] 

Hari Sankar Sivarama Subramaniyan commented on HIVE-5771:
-

Hi [~tedxu]  I have a  minor comment on the implementation.
Is there a reason for these rules sharing same val.
+opRules.put(new RuleRegExp("R5", "UNION%"), ConstantPropagateProcFactory
+.getUnionProc());
+opRules.put(new RuleRegExp("R5", "RS%"), ConstantPropagateProcFactory
+.getReduceSinkProc());

Thanks
Hari

> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, 
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, 
> HIVE-5771.8.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions 
> are evaluated at runtime. 
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase, 
> however, it is still a runtime evaluation and it doesn't propagate constants 
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6956) Duplicate partitioning column for union when dynamic partition sort optimization is enabled

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977564#comment-13977564
 ] 

Ashutosh Chauhan commented on HIVE-6956:


+1

> Duplicate partitioning column for union when dynamic partition sort 
> optimization is enabled
> ---
>
> Key: HIVE-6956
> URL: https://issues.apache.org/jira/browse/HIVE-6956
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: optimization
> Attachments: HIVE-6956.1.patch
>
>
> HIVE-6455 optimization uses PreOrder graph walker resulting in double 
> invocation of SortedDynPartitionOptimizer(). Because of this there are 
> duplicate partitioning columns in RSOp for few test cases (union_remove_17.q 
> and similar test cases..)..
> HIVE-6455 optimization does not actually need a PreOrder graph walker. The 
> fix is to revert it to DefaultGraphWalker.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive

[Hari Sankar Sivarama Subramaniyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977563#comment-13977563
 ] 

Hari Sankar Sivarama Subramaniyan commented on HIVE-5771:
-

[~ashutoshc] Created  HIVE-6959  to remove vectorization related constant 
expression folding code once HIVE-5771 is committed.

> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, 
> HIVE-5771.4.patch, HIVE-5771.5.patch, HIVE-5771.6.patch, HIVE-5771.7.patch, 
> HIVE-5771.8.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions 
> are evaluated at runtime. 
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase, 
> however, it is still a runtime evaluation and it doesn't propagate constants 
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6959) Remove vectorization related constant expression folding code once Constant propagation optimizer for Hive is committed

[Hari Sankar Sivarama Subramaniyan (JIRA)]

Hari Sankar Sivarama Subramaniyan created HIVE-6959:
---

 Summary: Remove vectorization related constant expression folding 
code once Constant propagation optimizer for Hive is committed
 Key: HIVE-6959
 URL: https://issues.apache.org/jira/browse/HIVE-6959
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan


HIVE-5771 covers Constant propagation optimizer for Hive. We should remove any 
vectorization related code which duplicates this feature once HIVE-5771 is 
committed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6934) PartitionPruner doesn't handle top level constant expression correctly

[Hari Sankar Sivarama Subramaniyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977554#comment-13977554
 ] 

Hari Sankar Sivarama Subramaniyan commented on HIVE-6934:
-

encountered another hive issue as part of this fix. 
explain select * from table where 1;
The above query passes where as ' select * from table where 1;' fails when 
trying to convert 1 to boolean. We should be able to catch this exception 
semantically.


> PartitionPruner doesn't handle top level constant expression correctly
> --
>
> Key: HIVE-6934
> URL: https://issues.apache.org/jira/browse/HIVE-6934
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-6934.1.patch
>
>
> You hit this error indirectly, because how we handle invalid constant 
> comparisons. Consider:
> {code}
> create table x(key int, value string) partitioned by (dt int, ts string);
> -- both these queries hit this issue
> select * from x where key = 'abc';
> select * from x where dt = 'abc';
> -- the issue is the comparison get converted to the constant false
> -- and the PartitionPruner doesn't handle top level constant exprs corrcetly
> {code}
> Thanks to [~hsubramaniyan] for uncovering this as part of adding tests for 
> HIVE-5376



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6958) update union_remove_*, other tests for hadoop-2

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977553#comment-13977553
 ] 

Szehon Ho commented on HIVE-6958:
-

I'm just curious, are these the same file size when run on hadoop-1, on linux?

> update union_remove_*, other tests for hadoop-2
> ---
>
> Key: HIVE-6958
> URL: https://issues.apache.org/jira/browse/HIVE-6958
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6958.1.patch
>
>
> Update q.out files to match totalSize for Linux platform.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6956) Duplicate partitioning column for union when dynamic partition sort optimization is enabled

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6956:
-

Attachment: HIVE-6956.1.patch

The test diffs are valid since SEL->RS->EX->SEL->FS is same as 
SEL->RS->EX->FS.. The diffs have the redundant SEL in reducer removed after 
changing the graph walker.

> Duplicate partitioning column for union when dynamic partition sort 
> optimization is enabled
> ---
>
> Key: HIVE-6956
> URL: https://issues.apache.org/jira/browse/HIVE-6956
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: optimization
> Attachments: HIVE-6956.1.patch
>
>
> HIVE-6455 optimization uses PreOrder graph walker resulting in double 
> invocation of SortedDynPartitionOptimizer(). Because of this there are 
> duplicate partitioning columns in RSOp for few test cases (union_remove_17.q 
> and similar test cases..)..
> HIVE-6455 optimization does not actually need a PreOrder graph walker. The 
> fix is to revert it to DefaultGraphWalker.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6956) Duplicate partitioning column for union when dynamic partition sort optimization is enabled

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6956:
-

Status: Patch Available  (was: Open)

> Duplicate partitioning column for union when dynamic partition sort 
> optimization is enabled
> ---
>
> Key: HIVE-6956
> URL: https://issues.apache.org/jira/browse/HIVE-6956
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0, 0.14.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: optimization
> Attachments: HIVE-6956.1.patch
>
>
> HIVE-6455 optimization uses PreOrder graph walker resulting in double 
> invocation of SortedDynPartitionOptimizer(). Because of this there are 
> duplicate partitioning columns in RSOp for few test cases (union_remove_17.q 
> and similar test cases..)..
> HIVE-6455 optimization does not actually need a PreOrder graph walker. The 
> fix is to revert it to DefaultGraphWalker.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6934) PartitionPruner doesn't handle top level constant expression correctly

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-6934:


Attachment: HIVE-6934.1.patch

> PartitionPruner doesn't handle top level constant expression correctly
> --
>
> Key: HIVE-6934
> URL: https://issues.apache.org/jira/browse/HIVE-6934
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-6934.1.patch
>
>
> You hit this error indirectly, because how we handle invalid constant 
> comparisons. Consider:
> {code}
> create table x(key int, value string) partitioned by (dt int, ts string);
> -- both these queries hit this issue
> select * from x where key = 'abc';
> select * from x where dt = 'abc';
> -- the issue is the comparison get converted to the constant false
> -- and the PartitionPruner doesn't handle top level constant exprs corrcetly
> {code}
> Thanks to [~hsubramaniyan] for uncovering this as part of adding tests for 
> HIVE-5376



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6934) PartitionPruner doesn't handle top level constant expression correctly

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-6934:


Status: Patch Available  (was: Open)

> PartitionPruner doesn't handle top level constant expression correctly
> --
>
> Key: HIVE-6934
> URL: https://issues.apache.org/jira/browse/HIVE-6934
> Project: Hive
>  Issue Type: Bug
>Reporter: Harish Butani
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-6934.1.patch
>
>
> You hit this error indirectly, because how we handle invalid constant 
> comparisons. Consider:
> {code}
> create table x(key int, value string) partitioned by (dt int, ts string);
> -- both these queries hit this issue
> select * from x where key = 'abc';
> select * from x where dt = 'abc';
> -- the issue is the comparison get converted to the constant false
> -- and the PartitionPruner doesn't handle top level constant exprs corrcetly
> {code}
> Thanks to [~hsubramaniyan] for uncovering this as part of adding tests for 
> HIVE-5376



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6950) Parsing Error in GROUPING SETS

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977545#comment-13977545
 ] 

Lefty Leverenz commented on HIVE-6950:
--

Should this be documented in the wiki?

* [GROUPING SETS clause 
|https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup#EnhancedAggregation,Cube,GroupingandRollup-GROUPINGSETSclause]

> Parsing Error in GROUPING SETS
> --
>
> Key: HIVE-6950
> URL: https://issues.apache.org/jira/browse/HIVE-6950
> Project: Hive
>  Issue Type: Bug
>Reporter: Rohit Agarwal
>
> The following query:
> {code}
> SELECT tab1.a,
>tab1.b,
>SUM(tab1.c)
> FROM tab1
> GROUP BY tab1.a,
>  tab1.b
> GROUPING SETS ((tab1.a, tab1.b))
> {code}
> results in the following error:
> {code}
> ParseException line 7:22 missing ) at ',' near ''
> line 7:31 extraneous input ')' expecting EOF near ''
> {code}
> Changing the query to:
> {code}
> SELECT tab1.a,
>tab1.b,
>SUM(tab1.c)
> FROM tab1
> GROUP BY tab1.a,
>  tab1.b
> GROUPING SETS ((a, tab1.b))
> {code}
> makes it work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6958) update union_remove_*, other tests for hadoop-2

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-6958:
-

Attachment: HIVE-6958.1.patch

> update union_remove_*, other tests for hadoop-2
> ---
>
> Key: HIVE-6958
> URL: https://issues.apache.org/jira/browse/HIVE-6958
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6958.1.patch
>
>
> Update q.out files to match totalSize for Linux platform.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6958) update union_remove_*, other tests for hadoop-2

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-6958:
-

Status: Patch Available  (was: Open)

> update union_remove_*, other tests for hadoop-2
> ---
>
> Key: HIVE-6958
> URL: https://issues.apache.org/jira/browse/HIVE-6958
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-6958.1.patch
>
>
> Update q.out files to match totalSize for Linux platform.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6958) update union_remove_*, other tests for hadoop-2

[Jason Dere (JIRA)]

Jason Dere created HIVE-6958:


 Summary: update union_remove_*, other tests for hadoop-2
 Key: HIVE-6958
 URL: https://issues.apache.org/jira/browse/HIVE-6958
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere


Update q.out files to match totalSize for Linux platform.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 20565: HIVE-6945 issues with dropping partitions on Oracle

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20565/
---

(Updated April 22, 2014, 9:58 p.m.)


Review request for hive, Ashutosh Chauhan and Thejas Nair.


Repository: hive-git


Description
---

see JIRA


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
a5c7daf 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 6a0eabe 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java fe15101 

Diff: https://reviews.apache.org/r/20565/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6945:
---

Attachment: (was: HIVE-6945.01.patch)

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6945.01.patch, HIVE-6945.patch
>
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6945:
---

Attachment: HIVE-6945.01.patch

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6945.01.patch, HIVE-6945.patch
>
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6936) Provide table properties to InputFormats

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-6936:


Attachment: HIVE-6936.patch

I've added a test case where I use a custom input format and run "select *" to 
ensure that the properties are correctly passed down to the input format.

> Provide table properties to InputFormats
> 
>
> Key: HIVE-6936
> URL: https://issues.apache.org/jira/browse/HIVE-6936
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 0.14.0
>
> Attachments: HIVE-6936.patch, HIVE-6936.patch
>
>
> Some advanced file formats need the table properties made available to them. 
> Additionally, it would be convenient to provide a unique id for fetch 
> operators and the complete list of directories.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6957:


Status: Patch Available  (was: Open)

> SQL authorization does not work with HS2 binary mode and Kerberos auth
> --
>
> Key: HIVE-6957
> URL: https://issues.apache.org/jira/browse/HIVE-6957
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6957.1.patch
>
>
> In HiveServer2, when Kerberos auth and binary transport modes are used, the 
> user name that gets passed on to authorization is the long kerberos username.
> The username that is used in grant/revoke statements tend to be the short 
> usernames.
> This also fails in authorizing statements that involve URI, as the 
> authorization mode checks the file system permissions for given user. It does 
> not recognize that the given long username actually owns the file or belongs 
> to the group that owns the file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6945:
---

Attachment: HIVE-6945.01.patch

some missed places on sql path

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6945.01.patch, HIVE-6945.patch
>
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6947) More fixes for tests on hadoop-2

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6947:
-

Attachment: HIVE-6947.1.patch

Remove union_remove_17.q as its not an expected output. The duplication of 
partition columns in RSOp will be fixed in HIVE-6956.

> More fixes for tests on hadoop-2 
> -
>
> Key: HIVE-6947
> URL: https://issues.apache.org/jira/browse/HIVE-6947
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6947.1.patch, HIVE-6947.patch
>
>
> Few more fixes for test cases on hadoop-2



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5072) [WebHCat]Enable directly invoke Sqoop job through Templeton

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977476#comment-13977476
 ] 

Thejas M Nair commented on HIVE-5072:
-

+1

> [WebHCat]Enable directly invoke Sqoop job through Templeton
> ---
>
> Key: HIVE-5072
> URL: https://issues.apache.org/jira/browse/HIVE-5072
> Project: Hive
>  Issue Type: Improvement
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-5072.1.patch, HIVE-5072.2.patch, HIVE-5072.3.patch, 
> HIVE-5072.4.patch, HIVE-5072.5.patch, Templeton-Sqoop-Action.pdf
>
>
> Now it is hard to invoke a Sqoop job through templeton. The only way is to 
> use the classpath jar generated by a sqoop job and use the jar delegator in 
> Templeton. We should implement Sqoop Delegator to enable directly invoke 
> Sqoop job through Templeton.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 20578: HIVE-6957 - SQL authorization does not work with HS2 binary mode and Kerberos auth

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20578/
---

Review request for hive, Ashutosh Chauhan and Vaibhav Gumashta.


Bugs: HIVE-6957
https://issues.apache.org/jira/browse/HIVE-6957


Repository: hive-git


Description
---

In HiveServer2, when Kerberos auth and binary transport modes are used, the 
user name that gets passed on to authorization is the long kerberos username.
The username that is used in grant/revoke statements tend to be the short 
usernames.
This also fails in authorizing statements that involve URI, as the 
authorization mode checks the file system permissions for given user. It does 
not recognize that the given long username actually owns the file or belongs to 
the group that owns the file.


Diffs
-

  itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/MiniHiveKdc.java 
f7ec93d 
  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
 62bfa1e 
  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdcSQLAuth.java
 PRE-CREATION 
  
shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java
 9e296de 

Diff: https://reviews.apache.org/r/20578/diff/


Testing
---

Unit test included.


Thanks,

Thejas Nair

[jira] [Commented] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977468#comment-13977468
 ] 

Thejas M Nair commented on HIVE-6957:
-

Error looks like this 
{code}
java.sql.SQLException: Error while compiling statement: FAILED: 
HiveAccessControlException Permission denied. Principal 
[name=us...@example.com, type=USER] does not have following privileges on 
Object [type=TABLE_OR_VIEW, name=default.test_jdbc_sql_auth2] : [SELECT]
{code}

> SQL authorization does not work with HS2 binary mode and Kerberos auth
> --
>
> Key: HIVE-6957
> URL: https://issues.apache.org/jira/browse/HIVE-6957
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6957.1.patch
>
>
> In HiveServer2, when Kerberos auth and binary transport modes are used, the 
> user name that gets passed on to authorization is the long kerberos username.
> The username that is used in grant/revoke statements tend to be the short 
> usernames.
> This also fails in authorizing statements that involve URI, as the 
> authorization mode checks the file system permissions for given user. It does 
> not recognize that the given long username actually owns the file or belongs 
> to the group that owns the file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977462#comment-13977462
 ] 

Thejas M Nair commented on HIVE-6957:
-

The long username is not of any significance within hive. We always use the 
short username for all purposes including the owner in metastore.
This patch changes the username that gets set for HS2 purposes, to the short 
username.


> SQL authorization does not work with HS2 binary mode and Kerberos auth
> --
>
> Key: HIVE-6957
> URL: https://issues.apache.org/jira/browse/HIVE-6957
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6957.1.patch
>
>
> In HiveServer2, when Kerberos auth and binary transport modes are used, the 
> user name that gets passed on to authorization is the long kerberos username.
> The username that is used in grant/revoke statements tend to be the short 
> usernames.
> This also fails in authorizing statements that involve URI, as the 
> authorization mode checks the file system permissions for given user. It does 
> not recognize that the given long username actually owns the file or belongs 
> to the group that owns the file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6957:


Attachment: HIVE-6957.1.patch

> SQL authorization does not work with HS2 binary mode and Kerberos auth
> --
>
> Key: HIVE-6957
> URL: https://issues.apache.org/jira/browse/HIVE-6957
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6957.1.patch
>
>
> In HiveServer2, when Kerberos auth and binary transport modes are used, the 
> user name that gets passed on to authorization is the long kerberos username.
> The username that is used in grant/revoke statements tend to be the short 
> usernames.
> This also fails in authorizing statements that involve URI, as the 
> authorization mode checks the file system permissions for given user. It does 
> not recognize that the given long username actually owns the file or belongs 
> to the group that owns the file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977461#comment-13977461
 ] 

Thejas M Nair commented on HIVE-6957:
-

A workaround is to use the http transport mode for HS2.

> SQL authorization does not work with HS2 binary mode and Kerberos auth
> --
>
> Key: HIVE-6957
> URL: https://issues.apache.org/jira/browse/HIVE-6957
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HiveServer2
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6957.1.patch
>
>
> In HiveServer2, when Kerberos auth and binary transport modes are used, the 
> user name that gets passed on to authorization is the long kerberos username.
> The username that is used in grant/revoke statements tend to be the short 
> usernames.
> This also fails in authorizing statements that involve URI, as the 
> authorization mode checks the file system permissions for given user. It does 
> not recognize that the given long username actually owns the file or belongs 
> to the group that owns the file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6957) SQL authorization does not work with HS2 binary mode and Kerberos auth

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-6957:
---

 Summary: SQL authorization does not work with HS2 binary mode and 
Kerberos auth
 Key: HIVE-6957
 URL: https://issues.apache.org/jira/browse/HIVE-6957
 Project: Hive
  Issue Type: Bug
  Components: Authorization, HiveServer2
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Thejas M Nair
 Attachments: HIVE-6957.1.patch

In HiveServer2, when Kerberos auth and binary transport modes are used, the 
user name that gets passed on to authorization is the long kerberos username.
The username that is used in grant/revoke statements tend to be the short 
usernames.
This also fails in authorizing statements that involve URI, as the 
authorization mode checks the file system permissions for given user. It does 
not recognize that the given long username actually owns the file or belongs to 
the group that owns the file.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

java.lang.NoSuchMethodError: org/apache/hadoop/conf/Configuration.unset(Ljava/lang/String;)V

[Dmitry Vasilenko]

Team:

I am getting the exception below on attempt to "show tables;" in Hive 
0.13.0. What is the minimum version of Hadoop that Hive 0.13.0 can be used 
with? Hive 0.12.0 worked with Hadoop 0.20.x, 0.23.x.y, 1.x.y, 2.x.y 

Exception in thread "main" java.lang.NoSuchMethodError: 
org/apache/hadoop/conf/Configuration.unset(Ljava/lang/String;)V
at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3416)
at 
org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3401)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214)
at org.apache.hadoop.hive.ql.Context.getLocalScratchDir(Context.java:241)
at org.apache.hadoop.hive.ql.Context.getLocalTmpPath(Context.java:333)
at 
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:296)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:391)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:291)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:944)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1009)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:880)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:870)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156) 

Regards
Dmitry Vasilenko

[jira] [Created] (HIVE-6956) Duplicate partitioning column for union when dynamic partition sort optimization is enabled

[Prasanth J (JIRA)]

Prasanth J created HIVE-6956:


 Summary: Duplicate partitioning column for union when dynamic 
partition sort optimization is enabled
 Key: HIVE-6956
 URL: https://issues.apache.org/jira/browse/HIVE-6956
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.13.0, 0.14.0
Reporter: Prasanth J
Assignee: Prasanth J


HIVE-6455 optimization uses PreOrder graph walker resulting in double 
invocation of SortedDynPartitionOptimizer(). Because of this there are 
duplicate partitioning columns in RSOp for few test cases (union_remove_17.q 
and similar test cases..)..

HIVE-6455 optimization does not actually need a PreOrder graph walker. The fix 
is to revert it to DefaultGraphWalker.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6946) Make it easier to run WebHCat e2e tests

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977442#comment-13977442
 ] 

Hive QA commented on HIVE-6946:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12641165/HIVE-6946.patch

{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 5417 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby2_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_infer_bucket_sort_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullgroup3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_dummy_source
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_partscan_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_symlink_text_input_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_truncate_column_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_current_database
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_17
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_dynamic_partitions_with_whitelist
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_stats_partialscan_autogether
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/6/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/6/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 43 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12641165

> Make it easier to run WebHCat e2e tests
> ---
>
> Key: HIVE-6946
> URL: https://issues.apache.org/jira/browse/HIVE-6946
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-6946.patch
>
>
> Right now hcatalog/src/test/e2e/templeton/README.txt explains the steps to 
> set up WebHCat e2e tests but it's cumbersome and error prone.  Need to make 
> some improvements here.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6945) issues with dropping partitions on Oracle

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6945:


Affects Version/s: 0.13.0

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6945.patch
>
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6924) MapJoinKeyBytes::hashCode() should use Murmur hash

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6924:
---

Attachment: HIVE-6924.01.patch

Updated patch to use Apache implementation. I am assuming +1 still stands as 
the patch is essentially the same. Running tests

> MapJoinKeyBytes::hashCode() should use Murmur hash
> --
>
> Key: HIVE-6924
> URL: https://issues.apache.org/jira/browse/HIVE-6924
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6924.01.patch, HIVE-6924.patch
>
>
> Existing hashCode is bad, causes HashMap to cluster



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6950) Parsing Error in GROUPING SETS

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977346#comment-13977346
 ] 

Harish Butani commented on HIVE-6950:
-

Yes there s an ambiguity in the grammar. 
The dfa for the groupingSetExpression chooses to interpret the '((tab1.a, 
tab1.b))' as 
'((tab1.a), (tab1.b))' and so it complains that the ')' is missing.

In the case of '((a, tab1.b))' the dfa chooses the right path.

The fix is to add Syntactic predicate for this rule.

> Parsing Error in GROUPING SETS
> --
>
> Key: HIVE-6950
> URL: https://issues.apache.org/jira/browse/HIVE-6950
> Project: Hive
>  Issue Type: Bug
>Reporter: Rohit Agarwal
>
> The following query:
> {code}
> SELECT tab1.a,
>tab1.b,
>SUM(tab1.c)
> FROM tab1
> GROUP BY tab1.a,
>  tab1.b
> GROUPING SETS ((tab1.a, tab1.b))
> {code}
> results in the following error:
> {code}
> ParseException line 7:22 missing ) at ',' near ''
> line 7:31 extraneous input ')' expecting EOF near ''
> {code}
> Changing the query to:
> {code}
> SELECT tab1.a,
>tab1.b,
>SUM(tab1.c)
> FROM tab1
> GROUP BY tab1.a,
>  tab1.b
> GROUPING SETS ((a, tab1.b))
> {code}
> makes it work.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6835) Reading of partitioned Avro data fails if partition schema does not match table schema

[Anthony Hsu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977337#comment-13977337
 ] 

Anthony Hsu commented on HIVE-6835:
---

I started looking into this alternative and encountered an issue.  Most calls 
to serde.initialize() are treating serde as a Deserializer (interface).  I 
would either have to change the interface (and change all the implementations) 
or cast the Deserializer as an AbstractSerDe (whenever I want to use the new 
initialize() method), neither of which seems like a great solution. So I am 
back to supporting my original "table." prefix approach. Any thoughts on this?

> Reading of partitioned Avro data fails if partition schema does not match 
> table schema
> --
>
> Key: HIVE-6835
> URL: https://issues.apache.org/jira/browse/HIVE-6835
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-6835.1.patch, HIVE-6835.2.patch, HIVE-6835.3.patch
>
>
> To reproduce:
> {code}
> create table testarray (a array);
> load data local inpath '/home/ahsu/test/array.txt' into table testarray;
> # create partitioned Avro table with one array column
> create table avroarray partitioned by (y string) row format serde 
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with serdeproperties 
> ('avro.schema.literal'='{"namespace":"test","name":"avroarray","type": 
> "record", "fields": [ { "name":"a", "type":{"type":"array","items":"string"} 
> } ] }')  STORED as INPUTFORMAT  
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'  OUTPUTFORMAT  
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat';
> insert into table avroarray partition(y=1) select * from testarray;
> # add an int column with a default value of 0
> alter table avroarray set serde 
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' with 
> serdeproperties('avro.schema.literal'='{"namespace":"test","name":"avroarray","type":
>  "record", "fields": [ {"name":"intfield","type":"int","default":0},{ 
> "name":"a", "type":{"type":"array","items":"string"} } ] }');
> # fails with ClassCastException
> select * from avroarray;
> {code}
> The select * fails with:
> {code}
> Failed with exception java.io.IOException:java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector 
> cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6932) hive README needs update

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977287#comment-13977287
 ] 

Lefty Leverenz commented on HIVE-6932:
--

Good, I'd wondered about that phrase but moved on to minutiae.

> hive README needs update
> 
>
> Key: HIVE-6932
> URL: https://issues.apache.org/jira/browse/HIVE-6932
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6932.1.patch, HIVE-6932.2.patch
>
>
> It needs to be updated to include Tez as a runtime. Also, it talks about 
> average latency being in minutes, which is very misleading.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6946) Make it easier to run WebHCat e2e tests

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977283#comment-13977283
 ] 

Eugene Koifman commented on HIVE-6946:
--

hcatalog/src/test/e2e/templeton/deployers/README.txt in the attached patch 
explains in detail what's added in the patch

> Make it easier to run WebHCat e2e tests
> ---
>
> Key: HIVE-6946
> URL: https://issues.apache.org/jira/browse/HIVE-6946
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-6946.patch
>
>
> Right now hcatalog/src/test/e2e/templeton/README.txt explains the steps to 
> set up WebHCat e2e tests but it's cumbersome and error prone.  Need to make 
> some improvements here.
> NO PRECOMMIT TESTS



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6893) out of sequence error in HiveMetastore server

[Naveen Gangam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977259#comment-13977259
 ] 

Naveen Gangam commented on HIVE-6893:
-

Review posted to the review board.
https://reviews.apache.org/r/20569/



> out of sequence error in HiveMetastore server
> -
>
> Key: HIVE-6893
> URL: https://issues.apache.org/jira/browse/HIVE-6893
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Romain Rigaux
>Assignee: Naveen Gangam
> Fix For: 0.13.0
>
> Attachments: HIVE-6893.1.patch
>
>
> Calls listing databases or tables fail. It seems to be a concurrency problem.
> {code}
> 014-03-06 05:34:00,785 ERROR hive.log: 
> org.apache.thrift.TApplicationException: get_databases failed: out of 
> sequence response
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
> at 
> org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
> at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
> at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
> at 
> org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
> at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
> at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5376) Hive does not honor type for partition columns when altering column type

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-5376:
---

Fix Version/s: 0.14.0

> Hive does not honor type for partition columns when altering column type
> 
>
> Key: HIVE-5376
> URL: https://issues.apache.org/jira/browse/HIVE-5376
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Reporter: Sergey Shelukhin
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 0.14.0
>
> Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch
>
>
> Followup for HIVE-5297. If partition column of type string is changed to int, 
> the data is not verified. The values for partition columns are all in 
> metastore db, so it's easy to check and fail the type change.
> alter_partition_coltype.q (or some other test?) checks this behavior right 
> now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5376) Hive does not honor type for partition columns when altering column type

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-5376:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

thanks Hari, Sergey

> Hive does not honor type for partition columns when altering column type
> 
>
> Key: HIVE-5376
> URL: https://issues.apache.org/jira/browse/HIVE-5376
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Reporter: Sergey Shelukhin
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch
>
>
> Followup for HIVE-5297. If partition column of type string is changed to int, 
> the data is not verified. The values for partition columns are all in 
> metastore db, so it's easy to check and fail the type change.
> alter_partition_coltype.q (or some other test?) checks this behavior right 
> now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6947) More fixes for tests on hadoop-2

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977218#comment-13977218
 ] 

Jason Dere commented on HIVE-6947:
--

Just a bit of explanation on auto_sortmerge_join_16: the query results had been 
changed by HIVE-6455, looks like a subsequent fix has restored the result set 
back to its original results.

+1

> More fixes for tests on hadoop-2 
> -
>
> Key: HIVE-6947
> URL: https://issues.apache.org/jira/browse/HIVE-6947
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6947.patch
>
>
> Few more fixes for test cases on hadoop-2



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6893) out of sequence error in HiveMetastore server

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977213#comment-13977213
 ] 

Vaibhav Gumashta commented on HIVE-6893:


[~ngangam] Can you also upload the patch to review board?

> out of sequence error in HiveMetastore server
> -
>
> Key: HIVE-6893
> URL: https://issues.apache.org/jira/browse/HIVE-6893
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Romain Rigaux
>Assignee: Naveen Gangam
> Fix For: 0.13.0
>
> Attachments: HIVE-6893.1.patch
>
>
> Calls listing databases or tables fail. It seems to be a concurrency problem.
> {code}
> 014-03-06 05:34:00,785 ERROR hive.log: 
> org.apache.thrift.TApplicationException: get_databases failed: out of 
> sequence response
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
> at 
> org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
> at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
> at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
> at 
> org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
> at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
> at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6893) out of sequence error in HiveMetastore server

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-6893:


Fix Version/s: 0.13.0
   Status: Patch Available  (was: Open)

> out of sequence error in HiveMetastore server
> -
>
> Key: HIVE-6893
> URL: https://issues.apache.org/jira/browse/HIVE-6893
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Romain Rigaux
>Assignee: Naveen Gangam
> Fix For: 0.13.0
>
> Attachments: HIVE-6893.1.patch
>
>
> Calls listing databases or tables fail. It seems to be a concurrency problem.
> {code}
> 014-03-06 05:34:00,785 ERROR hive.log: 
> org.apache.thrift.TApplicationException: get_databases failed: out of 
> sequence response
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
> at 
> org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
> at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
> at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
> at 
> org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
> at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
> at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6893) out of sequence error in HiveMetastore server

[Naveen Gangam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naveen Gangam updated HIVE-6893:


Attachment: HIVE-6893.1.patch

This proposed fix makes HS2 use a synchronized wrapper for the metastore client 
when the HMS is running as a standalone process. In embedded mode, we would not 
be reading sequenced responses over a socket.

This fix does NOT synchronize all the calls to the HMS. For example, 
Hive.getMSC() returns an metastore client instance that is not thread-safe. 
There is a bit of risk synchronizing these calls, as it might have a regressive 
performance impact. 

I will be measure performance impact of using a synchronized wrapper all over 
and see if the reward outweighs the risk. 

> out of sequence error in HiveMetastore server
> -
>
> Key: HIVE-6893
> URL: https://issues.apache.org/jira/browse/HIVE-6893
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
>Reporter: Romain Rigaux
>Assignee: Naveen Gangam
> Fix For: 0.13.0
>
> Attachments: HIVE-6893.1.patch
>
>
> Calls listing databases or tables fail. It seems to be a concurrency problem.
> {code}
> 014-03-06 05:34:00,785 ERROR hive.log: 
> org.apache.thrift.TApplicationException: get_databases failed: out of 
> sequence response
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:472)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:459)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:648)
> at 
> org.apache.hive.service.cli.operation.GetSchemasOperation.run(GetSchemasOperation.java:66)
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:278)
> at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:62)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:582)
> at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:57)
> at com.sun.proxy.$Proxy9.getSchemas(Unknown Source)
> at 
> org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:192)
> at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:263)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1433)
> at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1418)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.hive.service.cli.thrift.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:38)
> at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5376) Hive does not honor type for partition columns when altering column type

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-5376:


Status: Patch Available  (was: Open)

> Hive does not honor type for partition columns when altering column type
> 
>
> Key: HIVE-5376
> URL: https://issues.apache.org/jira/browse/HIVE-5376
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Reporter: Sergey Shelukhin
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch
>
>
> Followup for HIVE-5297. If partition column of type string is changed to int, 
> the data is not verified. The values for partition columns are all in 
> metastore db, so it's easy to check and fail the type change.
> alter_partition_coltype.q (or some other test?) checks this behavior right 
> now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6945) issues with dropping partitions on Oracle

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977200#comment-13977200
 ] 

Sergey Shelukhin commented on HIVE-6945:


https://reviews.apache.org/r/20565/

> issues with dropping partitions on Oracle
> -
>
> Key: HIVE-6945
> URL: https://issues.apache.org/jira/browse/HIVE-6945
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6945.patch
>
>
> 1) Direct SQL is broken on Oracle due to the usage of NUMBER type which is 
> translated by DN into decimal rather than long. This appears to be specific 
> to some cases because it seemed to have worked before (different version of 
> Oracle? JDBC? DN? Maybe depends on whether db was auto-created).
> 2) When partition dropping code falls back to JDO, it creates objects to 
> return, then drops partitions. It appears that dropping makes DN objects 
> invalid. We create metastore partition objects out of DN objects before drop, 
> however the list of partition column values is re-used, rather than copied, 
> into these. DN appears to clear this list during drop, so the returned object 
> becomes invalid and the exception is thrown.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5376) Hive does not honor type for partition columns when altering column type

[Hari Sankar Sivarama Subramaniyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-5376:


Status: Open  (was: Patch Available)

> Hive does not honor type for partition columns when altering column type
> 
>
> Key: HIVE-5376
> URL: https://issues.apache.org/jira/browse/HIVE-5376
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Reporter: Sergey Shelukhin
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-5376.1.patch, HIVE-5376.2.patch, HIVE-5376.3.patch
>
>
> Followup for HIVE-5297. If partition column of type string is changed to int, 
> the data is not verified. The values for partition columns are all in 
> metastore db, so it's easy to check and fail the type change.
> alter_partition_coltype.q (or some other test?) checks this behavior right 
> now.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 20565: HIVE-6945 issues with dropping partitions on Oracle

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/20565/
---

Review request for hive, Ashutosh Chauhan and Thejas Nair.


Repository: hive-git


Description
---

see JIRA


Diffs
-

  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
a5c7daf 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 6a0eabe 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java fe15101 

Diff: https://reviews.apache.org/r/20565/diff/


Testing
---


Thanks,

Sergey Shelukhin