[jira] [Commented] (HIVE-5849) Improve the stats of operators based on heuristics in the absence of any column statistics

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828537#comment-13828537
 ] 

Hive QA commented on HIVE-5849:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12615014/HIVE-5849.5.patch

{color:red}ERROR:{color} -1 due to 114 failed/errored test(s), 4680 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_def

[jira] [Commented] (HIVE-4956) Allow multiple tables in from clause if all them have the same schema, but can be partitioned differently

[Amareshwari Sriramadasu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828528#comment-13828528
 ] 

Amareshwari Sriramadasu commented on HIVE-4956:
---

I agree with the concerns above that this is deviating from SQL. But it gives 
lot of performance improvement in distributed systems. How about change the 
separator to '+' instead of ',', as part of Hive QL? 

The query will look like the following :
{noformat}
select t.x, t.y,  from T1+T2 t where t.p1='x' OR t.p1='y' ... 
[groupby-clause] [having-clause] [orderby-clause]
{noformat}

If the proposal is fine, I can upload the patch.

> Allow multiple tables in from clause if all them have the same schema, but 
> can be partitioned differently
> -
>
> Key: HIVE-4956
> URL: https://issues.apache.org/jira/browse/HIVE-4956
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Amareshwari Sriramadasu
>Assignee: Amareshwari Sriramadasu
>
> We have a usecase where the table storage partitioning changes over time.
> For ex:
>  we can have a table T1 which is partitioned by p1. But overtime, we want to 
> partition the table on p1 and p2 as well. The new table can be T2. So, if we 
> have to query table on partition p1, it will be a union query across two 
> table T1 and T2. Especially with aggregations like avg, it becomes costly 
> union query because we cannot make use of mapside aggregations and other 
> optimizations.
> The proposal is to support queries of the following format :
> select t.x, t.y,  from T1,T2 t where t.p1='x' OR t.p1='y' ... 
> [groupby-clause] [having-clause] [orderby-clause] and so on.
> Here we allow from clause as a comma separated list of tables with an alias 
> and alias will be used in the full query, and partition pruning will happen 
> on the actual tables to pick up the right paths. This will work because the 
> difference is only on picking up the input paths and whole operator tree does 
> not change. If this sounds a good usecase, I can put up the changes required 
> to support the same.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5614) Subquery support: allow subquery expressions in having clause

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828514#comment-13828514
 ] 

Hive QA commented on HIVE-5614:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614936/HIVE-5614.3.patch

{color:green}SUCCESS:{color} +1 4684 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/381/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/381/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614936

> Subquery support: allow subquery expressions in having clause
> -
>
> Key: HIVE-5614
> URL: https://issues.apache.org/jira/browse/HIVE-5614
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-5614.1.patch, HIVE-5614.2.patch, HIVE-5614.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-3815) hive table rename fails if filesystem cache is disabled

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3815:


Status: Patch Available  (was: Open)

> hive table rename fails if filesystem cache is disabled
> ---
>
> Key: HIVE-3815
> URL: https://issues.apache.org/jira/browse/HIVE-3815
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-3815.1.patch
>
>
> If fs..impl.disable.cache  (eg fs.hdfs.impl.disable.cache) is set 
> to true, then table rename fails.
> The exception that gets thrown (though not logged!) is 
> {quote}
> Caused by: InvalidOperationException(message:table new location 
> hdfs://host1:8020/apps/hive/warehouse/t2 is on a different file system than 
> the old location hdfs://host1:8020/apps/hive/warehouse/t1. This operation is 
> not supported)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_result$alter_table_resultStandardScheme.read(ThriftHiveMetastore.java:28825)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_result$alter_table_resultStandardScheme.read(ThriftHiveMetastore.java:28811)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_table_result.read(ThriftHiveMetastore.java:28753)
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_alter_table(ThriftHiveMetastore.java:977)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.alter_table(ThriftHiveMetastore.java:962)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:208)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
> at $Proxy7.alter_table(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:373)
> ... 18 more
> {quote}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-4977) HS2: support an alternate resultset serialization format between client and server

[Carl Steinbach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-4977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carl Steinbach updated HIVE-4977:
-

Summary: HS2: support an alternate resultset serialization format between 
client and server  (was: HS2: support an alternate serialization protocol 
between client and server)

> HS2: support an alternate resultset serialization format between client and 
> server
> --
>
> Key: HIVE-4977
> URL: https://issues.apache.org/jira/browse/HIVE-4977
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Chris Drome
>Assignee: Chris Drome
>
> Current serialization protocol between client and server as defined in 
> cli_service.thrift results in 2x (or more) throughput degradation compared to 
> HS1.
> Initial proposal is to introduce HS1 serialization protocol as a negotiable 
> alternative.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5692) Make VectorGroupByOperator parameters configurable

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-5692:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Remus!

> Make VectorGroupByOperator parameters configurable
> --
>
> Key: HIVE-5692
> URL: https://issues.apache.org/jira/browse/HIVE-5692
> Project: Hive
>  Issue Type: Bug
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
> Fix For: 0.13.0
>
> Attachments: HIVE-5692.1.patch, HIVE-5692.2.patch, HIVE-5692.3.patch, 
> HIVE-5692.4.patch, HIVE-5692.5.patch, HIVE-5692.6.patch
>
>
> The FLUSH_CHECK_THRESHOLD and PERCENT_ENTRIES_TO_FLUSH should be configurable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5618) Hive local task fails to run when run from oozie in a secure cluster

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828449#comment-13828449
 ] 

Thejas M Nair commented on HIVE-5618:
-

+1 
I think we can take the discussion about delegation token being needed without 
doAs being enabled in a different jira, if that is the case, it looks like a 
hadoop bug to me. Getting additional token in this case should not cause any 
noticeable problems.



> Hive local task fails to run when run from oozie in a secure cluster
> 
>
> Key: HIVE-5618
> URL: https://issues.apache.org/jira/browse/HIVE-5618
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
> Environment: Hadoop 2.2.0
>Reporter: Venkat Ranganathan
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5618.2.patch
>
>
> When a hive query like the one below
> ==
> INSERT OVERWRITE DIRECTORY '' SELECT table1.*, table2.* FROM table1 
> JOIN table2 ON (table1.col = table2.col);
> ==
> is run from a hive action in Oozie in a secure cluster, the hive action fails 
> with the following stack trace
> ===
> {noformat}
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token 
> can be issued only with kerberos or web authentication
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:5886)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:447)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:833)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59648)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:396)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1347)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1300)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at $Proxy10.getDelegationToken(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>   at $Proxy10.getDelegationToken(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:805)
>   at 
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:847)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1318)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.createDelegationTokenFile(HadoopShimsSecure.java:535)
>   at 
> org.apache.hadoop.hive.ql.exec.SecureCmdDoAs.(SecureCmdDoAs.java:38)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.execute(MapredLocalTask.java:238)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1437)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1215)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1043)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:446)
>  

[jira] [Commented] (HIVE-5858) Alter table fails with exception when fs.hdfs.impl.disable.cache is set false

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828445#comment-13828445
 ] 

Thejas M Nair commented on HIVE-5858:
-

Looks good to me.
I have same fix, with more comments in HIVE-3815 .
Shall we use that ?


> Alter table fails with exception when fs.hdfs.impl.disable.cache is set false
> -
>
> Key: HIVE-5858
> URL: https://issues.apache.org/jira/browse/HIVE-5858
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5858.1.patch.txt
>
>
> http://www.mail-archive.com/user@hive.apache.org/msg10098.html
> {noformat}
> 2013-11-20 10:07:05,759 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(141)) - 
> InvalidOperationException(message:table new location 
> hdfs://localhost:9000/user/hive/warehouse/xa is on a different file system 
> than the old location hdfs://localhost:9000/user/hive/warehouse/zz. This 
> operation is not supported)
> at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:146)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:2352)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
> at com.sun.proxy.$Proxy9.alter_table_with_environment_context(Unknown 
> Source)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:216)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
> at com.sun.proxy.$Proxy10.alter_table(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:385)
> at 
> org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3432)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:149)
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:87)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1256)
> at org.apache.hadoop.hive.ql.Driver.executePlan(Driver.java:1054)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:969)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:764)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Hive-trunk-hadoop2 - Build # 561 - Still Failing

[Apache Jenkins Server]

Changes for Build #529
[hashutosh] HIVE-5698 : A few test file missing apache license header (Xuefu 
Zhang via Ashutosh Chauhan)

[daijy] HIVE-5696: WebHCat e2e tests/jobsubmission.conf file is malformed and 
loosing tests ( Eugene Koifman via Daniel Dai )

[hashutosh] HIVE-5681 : Validation doesn't catch SMBMapJoin (Jitendra Nath 
Pandey via Ashutosh Chauhan)


Changes for Build #530
[hashutosh] HIVE-5670 : annoying ZK exceptions are annoying (Sergey Shelukhin 
via Ashutosh Chauhan)

[hashutosh] HIVE-5613 : Subquery support: disallow nesting of SubQueries 
(Harish Butani via Ashutosh Chauhan)


Changes for Build #531
[thejas] HIVE-5483 : use metastore statistics to optimize max/min/etc. queries 
(Ashutosh Chauhan via Thejas Nair)

[daijy] HIVE-5510: [WebHCat] GET job/queue return wrong job information

[brock] HIVE-5610 - Merge maven branch into trunk (delete ant)

[brock] HIVE-5610 - Merge maven branch into trunk (maven rollforward)

[brock] HIVE-5610 - Merge maven branch into trunk (patch)

[hashutosh] HIVE-5693 : Rewrite some tests to reduce test time (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5582 : Implement BETWEEN filter in vectorized mode (Eric 
Hanson via Ashutosh Chauhan)

[hashutosh] HIVE-5556 : Pushdown join conditions (Harish Butani via Ashutosh 
Chauhan)


Changes for Build #532
[brock] HIVE-5716 - Fix broken tests after maven merge (1) (Brock Noland 
reviewed by Thejas M Nair and Ashutosh Chauhan)


Changes for Build #533
[hashutosh] HIVE-3959 : Update Partition Statistics in Metastore Layer 
(Ashutosh Chauhan, Bhushan Mandhani, Gang Tim Liu via Thejas Nair)


Changes for Build #534
[hashutosh] HIVE-5503 : TopN optimization in VectorReduceSink (Sergey Shelukhin 
via Ashutosh Chauhan)

[brock] HIVE-5695 - PTest2 fix shutdown, duplicate runs, and add client retry

[brock] HIVE-5708 - PTest2 should trim long logs when posting to jira


Changes for Build #535
[thejas] HIVE-5547 : webhcat pig job submission should ship hive tar if 
-usehcatalog is specified (Eugene Koifman via Thejas Nair)

[thejas] HIVE-5715 : HS2 should not "start" a session for every command 
(Gunther Hagleitner via Thejas Nair)


Changes for Build #536

Changes for Build #537
[brock] HIVE-5740: Tar files should extract to the directory of the same name 
minus tar.gz (Brock Noland reviewed by Xuefu Zhang)

[brock] HIVE-5611: Add assembly (i.e.) tar creation to pom (Szehon Ho via Brock 
Noland)

[brock] HIVE-5707: Validate values for ConfVar (Navis via Brock Noland)

[brock] HIVE-5721: Incremental build is disabled by MCOMPILER-209 (Navis via 
Brock Noland)


Changes for Build #538
[brock] HIVE-5354 - Decimal precision/scale support in ORC file (Xuefu Zhang 
via Brock Noland)

[brock] HIVE-4523 - round() function with specified decimal places not 
consistent with mysql (Xuefu Zhang via Brock Noland)

[thejas] HIVE-5542 : Webhcat is failing to run ddl command on a secure cluster 
(Sushanth Sowmyan via Thejas Nair)


Changes for Build #539
[brock] HIVE-5730: Beeline throws non-terminal NPE upon starting, after 
mavenization (Szehon Ho reviewed by Navis)


Changes for Build #540
[omalley] HIVE-5425 Provide a configuration option to control the default stripe
size for ORC. (omalley reviewed by gunther)

[omalley] Revert HIVE-5583 since it broke the build.

[hashutosh] HIVE-5583 : Implement support for IN (list-of-constants) filter in 
vectorized mode (Eric Hanson via Ashutosh Chauhan)

[brock] HIVE-5355 - JDBC support for decimal precision/scale


Changes for Build #541
[hashutosh] HIVE-5583 : Implement support for IN (list-of-constants) filter in 
vectorized mode (Eric Hanson via Ashutosh Chauhan)

[brock] HIVE-5773 - Fix build due to conflict between HIVE-5711 and HIVE-5713

[brock] HIVE-5711 - Fix eclipse:eclipse maven goal (Carl Steinbach via Brock 
Noland)

[brock] HIVE-5752 - log4j properties appear to have been lost in maven upgrade 
(Sergey Shelukhin via Brock Noland)

[brock] HIVE-5713 - Verify versions of libraries post maven merge (Brock Noland 
reviewed by Gunther Hagleitner)

[brock] HIVE-5765 - Beeline throws NPE when -e option is used (Szehon Ho via 
Brock Noland)

[xuefu] HIVE-5726: The DecimalTypeInfo instance associated with a decimal 
constant is not in line with the precision/scale of the constant (reviewed by 
Brock)

[xuefu] HIVE-5655: Hive incorrecly handles divide-by-zero case (reviewed by 
Edward and Brock)

[xuefu] HIVE-5191: Add char data type (Jason via Xuefu)


Changes for Build #542
[brock] HIVE-5351 - Secure-Socket-Layer (SSL) support for HiveServer2 (Prasad 
Mujumdar via Brock Noland)


Changes for Build #543
[brock] HIVE-5780 - Add the missing declaration of HIVE_CLI_SERVICE_PROTOCOL_V4 
in TCLIService.thrift (Prasad Mujumdar via Brock Noland)


Changes for Build #544
[gunther] HIVE-5601: NPE in ORC's PPD when using select * from table with where 
predicate (Prasanth J via Owen O'Malley and Gunther Hagleitner)

[gunther] HIVE-5562: Provide stripe level column statistics in ORC (Patch by 
Prasa

[jira] [Commented] (HIVE-5692) Make VectorGroupByOperator parameters configurable

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828422#comment-13828422
 ] 

Hive QA commented on HIVE-5692:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614897/HIVE-5692.6.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4666 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.listener.TestNotificationListener.testAMQListener
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/379/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/379/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614897

> Make VectorGroupByOperator parameters configurable
> --
>
> Key: HIVE-5692
> URL: https://issues.apache.org/jira/browse/HIVE-5692
> Project: Hive
>  Issue Type: Bug
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-5692.1.patch, HIVE-5692.2.patch, HIVE-5692.3.patch, 
> HIVE-5692.4.patch, HIVE-5692.5.patch, HIVE-5692.6.patch
>
>
> The FLUSH_CHECK_THRESHOLD and PERCENT_ENTRIES_TO_FLUSH should be configurable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5845) CTAS failed on vectorized code path

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-5845:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Remus!

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Fix For: 0.13.0
>
> Attachments: HIVE-5845.1.patch
>
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-4256) JDBC2 HiveConnection does not use the specified database

[Anandha L Ranganathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828402#comment-13828402
 ] 

Anandha L Ranganathan commented on HIVE-4256:
-

[~cdrome]  If you are not working on this, can I take up this ticket.

> JDBC2 HiveConnection does not use the specified database
> 
>
> Key: HIVE-4256
> URL: https://issues.apache.org/jira/browse/HIVE-4256
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.11.0
>Reporter: Chris Drome
>Assignee: Chris Drome
>
> HiveConnection ignores the database specified in the connection string when 
> configuring the connection.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5817) column name to index mapping in VectorizationContext is broken

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828397#comment-13828397
 ] 

Eric Hanson commented on HIVE-5817:
---

I was able to create a smaller repro as follows.

First, create the table alltypesorc (this is a standard table that is in the 
Hive source code as test data).

set hive.vectorized.execution.enabled = false;

create table store(s_store_sk int, s_city string)
stored as orc;

insert overwrite table store
select cint, cstring1
from alltypesorc
where cint not in (
-3728, -563, 762, 6981, 253665376, 528534767, 626923679);

create table store_sales(ss_store_sk int, ss_hdemo_sk int, ss_net_profit double)
stored as orc;

insert overwrite table store_sales
select cint, cint, cdouble
from alltypesorc
where cint not in (
-3728, -563, 762, 6981, 253665376, 528534767, 626923679);

create table household_demographics(hd_demo_sk int)
stored as orc;

insert overwrite table household_demographics
select cint 
from alltypesorc
where cint not in (
-3728, -563, 762, 6981, 253665376, 528534767, 626923679);

-- the NOT IN condition makes sure all the cint values are unique

-- finally, run this:
set hive.vectorized.execution.enabled = true;

select store.s_city, ss_net_profit
from store_sales
 JOIN store ON store_sales.ss_store_sk = store.s_store_sk  
 JOIN household_demographics ON store_sales.ss_hdemo_sk = 
household_demographics.hd_demo_sk
limit 100;

Expected result: 100 rows of output are produced
Actual result:
...
2013-11-20 17:57:37,487 Stage-4 map = 0%,  reduce = 0%
2013-11-20 17:58:04,585 Stage-4 map = 100%,  reduce = 100%
Ended Job = job_201311191600_0022 with errors
Error during job, obtaining debugging information...
Job Tracking URL: 
http://localhost:50030/jobdetails.jsp?jobid=job_201311191600_0022
Examining task ID: task_201311191600_0022_m_02 (and more) from job 
job_201311191600_0022

Task with the most failures(4):
-
Task ID:
  task_201311191600_0022_m_00

URL:
  
http://localhost:50030/taskdetails.jsp?jobid=job_201311191600_0022&tipid=task_201311191600_0022_m_00
-
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:181)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error 
while processing row
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Long 
vector column and primitive category STRING
at 
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory.buildObjectAssign(VectorColumnAssignFactory.java:278)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory.buildAssigners(VectorColumnAssignFactory.java:365)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:235)
at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:675)
at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:224)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.processOp(VectorMapJoinOperator.java:293)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.flushOutput(VectorMapJoinOperator.java:249)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:244)
at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:675)
at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:224)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOpe

[jira] [Commented] (HIVE-5845) CTAS failed on vectorized code path

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828393#comment-13828393
 ] 

Hive QA commented on HIVE-5845:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614892/HIVE-5845.1.patch

{color:green}SUCCESS:{color} +1 4679 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/378/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/378/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614892

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Attachments: HIVE-5845.1.patch
>
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-4256) JDBC2 HiveConnection does not use the specified database

[Anandha L Ranganathan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828379#comment-13828379
 ] 

Anandha L Ranganathan commented on HIVE-4256:
-

I can take it up this ticket.

> JDBC2 HiveConnection does not use the specified database
> 
>
> Key: HIVE-4256
> URL: https://issues.apache.org/jira/browse/HIVE-4256
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 0.11.0
>Reporter: Chris Drome
>Assignee: Chris Drome
>
> HiveConnection ignores the database specified in the connection string when 
> configuring the connection.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5550) Import fails for tables created with default text and sequence file formats using HCatalog API

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5550:
---

Component/s: HCatalog

> Import fails for tables created with default text and sequence file formats 
> using HCatalog API
> --
>
> Key: HIVE-5550
> URL: https://issues.apache.org/jira/browse/HIVE-5550
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.11.0
>Reporter: Venkatesh Seetharam
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5550.2.patch, HIVE-5550.patch
>
>
> A table was created using HCatalog API with out specifying the file format, 
> it defaults to:
> {code}
> fileFormat=TextFile, inputformat=org.apache.hadoop.mapred.TextInputFormat, 
> outputformat=org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> {code}
> But, when hive fetches the table from the metastore, it strangely replaces 
> the output format with 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> and the comparison between source and target table fails.
> The code in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable 
> does a string comparison of classes and fails.
> {code}
>   // check IF/OF/Serde
>   String existingifc = table.getInputFormatClass().getName();
>   String importedifc = tableDesc.getInputFormat();
>   String existingofc = table.getOutputFormatClass().getName();
>   String importedofc = tableDesc.getOutputFormat();
>   if ((!existingifc.equals(importedifc))
>   || (!existingofc.equals(importedofc))) {
> throw new SemanticException(
> ErrorMsg.INCOMPATIBLE_SCHEMA
> .getMsg(" Table inputformat/outputformats do not match"));
>   }
> {code}
> This only affects tables with text and sequence file formats but not rc or 
> orc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5863) INSERT OVERWRITE TABLE fails in vectorized mode for ORC format target table

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828347#comment-13828347
 ] 

Ashutosh Chauhan commented on HIVE-5863:


Can you try with patch on HIVE-5845 I think that one should fix this issue.

> INSERT OVERWRITE TABLE fails in vectorized mode for ORC format target table
> ---
>
> Key: HIVE-5863
> URL: https://issues.apache.org/jira/browse/HIVE-5863
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Eric Hanson
>
> create table store(s_store_key int, s_city string)
> stored as orc;
> set hive.vectorized.execution.enabled = true;
> insert overwrite table store
> select cint, cstring1
> from alltypesorc;
> Alltypesorc is a test table that is checked in to the Hive source.
> Expected result: data is added to store table.
> Actual result:
> Total MapReduce jobs = 3
> Launching Job 1 out of 3
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201311191600_0007, Tracking URL = 
> http://localhost:50030/jobdetails.jsp?jobid=job_201311191600_0007
> Kill Command = c:\Hadoop\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd job  -kill 
> job_201311191600_0007
> Hadoop job information for Stage-1: number of mappers: 1; number of reducers: > 0
> 2013-11-20 16:39:53,271 Stage-1 map = 0%,  reduce = 0%
> 2013-11-20 16:40:20,375 Stage-1 map = 100%,  reduce = 100%
> Ended Job = job_201311191600_0007 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL: 
> http://localhost:50030/jobdetails.jsp?jobid=job_201311191600_0007
> Examining task ID: task_201311191600_0007_m_02 (and more) from job 
> job_201311191600_0007
> Task with the most failures(4):
> -
> Task ID:
>   task_201311191600_0007_m_00
> URL:
>   
> http://localhost:50030/taskdetails.jsp?jobid=job_201311191600_0007&tipid=task_201311191600_0007_m_00
> -
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:181)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
> at org.apache.hadoop.mapred.Child.main(Child.java:260)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ... 8 more
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.io.orc.OrcStruct cannot be cast to 
> [Ljava.lang.Object;
> at 
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldData(StandardStructObjectInspec
> tor.java:173)
> at 
> org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.write(WriterImpl.java:1349)
> at 
> org.apache.hadoop.hive.ql.io.orc.WriterImpl.addRow(WriterImpl.java:1962)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:78)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.processOp(VectorFileSinkOperator.java:159)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:129)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> ... 9 more



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HIVE-5863) INSERT OVERWRITE TABLE fails in vectorized mode for ORC format target table

[Eric Hanson (JIRA)]

Eric Hanson created HIVE-5863:
-

 Summary: INSERT OVERWRITE TABLE fails in vectorized mode for ORC 
format target table
 Key: HIVE-5863
 URL: https://issues.apache.org/jira/browse/HIVE-5863
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Eric Hanson


create table store(s_store_key int, s_city string)
stored as orc;

set hive.vectorized.execution.enabled = true;

insert overwrite table store
select cint, cstring1
from alltypesorc;

Alltypesorc is a test table that is checked in to the Hive source.

Expected result: data is added to store table.

Actual result:

Total MapReduce jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201311191600_0007, Tracking URL = 
http://localhost:50030/jobdetails.jsp?jobid=job_201311191600_0007
Kill Command = c:\Hadoop\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd job  -kill 
job_201311191600_0007
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2013-11-20 16:39:53,271 Stage-1 map = 0%,  reduce = 0%
2013-11-20 16:40:20,375 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201311191600_0007 with errors
Error during job, obtaining debugging information...
Job Tracking URL: 
http://localhost:50030/jobdetails.jsp?jobid=job_201311191600_0007
Examining task ID: task_201311191600_0007_m_02 (and more) from job 
job_201311191600_0007

Task with the most failures(4):
-
Task ID:
  task_201311191600_0007_m_00

URL:
  
http://localhost:50030/taskdetails.jsp?jobid=job_201311191600_0007&tipid=task_201311191600_0007_m_00
-
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:181)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error 
while processing row
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
... 8 more
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.io.orc.OrcStruct cannot be cast to [Ljava.lang.Object;
at 
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldData(StandardStructObjectInspec
tor.java:173)
at 
org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.write(WriterImpl.java:1349)
at 
org.apache.hadoop.hive.ql.io.orc.WriterImpl.addRow(WriterImpl.java:1962)
at 
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:78)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.processOp(VectorFileSinkOperator.java:159)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:129)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:489)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:827)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
... 9 more




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5799) session/operation timeout for hiveserver2

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828326#comment-13828326
 ] 

Thejas M Nair commented on HIVE-5799:
-

For a once a minute heartbeat, I think it would be better to create a new 
connection, and close it after the heartbeat is sent. It would be possible to 
set reasonable timeout values of say 5 mins or 10 mins (with this one minute 
heartbeat).



> session/operation timeout for hiveserver2
> -
>
> Key: HIVE-5799
> URL: https://issues.apache.org/jira/browse/HIVE-5799
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5799.1.patch.txt, HIVE-5799.2.patch.txt, 
> HIVE-5799.3.patch.txt
>
>
> Need some timeout facility for preventing resource leakages from instable  or 
> bad clients.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5837) SQL standard based secure authorization for hive

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828325#comment-13828325
 ] 

Navis commented on HIVE-5837:
-

Ah, sorry. It's issue number of internal patches applied to our product. All of 
them are based on hive-11 but might be rebased to trunk.

> SQL standard based secure authorization for hive
> 
>
> Key: HIVE-5837
> URL: https://issues.apache.org/jira/browse/HIVE-5837
> Project: Hive
>  Issue Type: New Feature
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: SQL standard authorization hive.pdf
>
>
> The current default authorization is incomplete and not secure. The 
> alternative of storage based authorization provides security but does not 
> provide fine grained authorization.
> The proposal is to support secure fine grained authorization in hive using 
> SQL standard based authorization model.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5799) session/operation timeout for hiveserver2

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828324#comment-13828324
 ] 

Navis commented on HIVE-5799:
-

It's the way I've solved this issue for oozie-hive integration. In that case, 
we know that the all the connection is from single entity(oozie) and one more 
connection was not big burden. But if connections are to be connected from 
various entities, the number of connection for hive server2 would be simply 
doubled. Would it be ok? Then I'll proceed the work.

> session/operation timeout for hiveserver2
> -
>
> Key: HIVE-5799
> URL: https://issues.apache.org/jira/browse/HIVE-5799
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5799.1.patch.txt, HIVE-5799.2.patch.txt, 
> HIVE-5799.3.patch.txt
>
>
> Need some timeout facility for preventing resource leakages from instable  or 
> bad clients.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5550) Import fails for tables created with default text and sequence file formats using HCatalog API

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5550:
---

Attachment: HIVE-5550.2.patch

Updated patch attached.

> Import fails for tables created with default text and sequence file formats 
> using HCatalog API
> --
>
> Key: HIVE-5550
> URL: https://issues.apache.org/jira/browse/HIVE-5550
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Venkatesh Seetharam
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5550.2.patch, HIVE-5550.patch
>
>
> A table was created using HCatalog API with out specifying the file format, 
> it defaults to:
> {code}
> fileFormat=TextFile, inputformat=org.apache.hadoop.mapred.TextInputFormat, 
> outputformat=org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> {code}
> But, when hive fetches the table from the metastore, it strangely replaces 
> the output format with 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> and the comparison between source and target table fails.
> The code in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable 
> does a string comparison of classes and fails.
> {code}
>   // check IF/OF/Serde
>   String existingifc = table.getInputFormatClass().getName();
>   String importedifc = tableDesc.getInputFormat();
>   String existingofc = table.getOutputFormatClass().getName();
>   String importedofc = tableDesc.getOutputFormat();
>   if ((!existingifc.equals(importedifc))
>   || (!existingofc.equals(importedofc))) {
> throw new SemanticException(
> ErrorMsg.INCOMPATIBLE_SCHEMA
> .getMsg(" Table inputformat/outputformats do not match"));
>   }
> {code}
> This only affects tables with text and sequence file formats but not rc or 
> orc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5550) Import fails for tables created with default text and sequence file formats using HCatalog API

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5550:
---

Status: Patch Available  (was: Open)

> Import fails for tables created with default text and sequence file formats 
> using HCatalog API
> --
>
> Key: HIVE-5550
> URL: https://issues.apache.org/jira/browse/HIVE-5550
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Venkatesh Seetharam
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5550.2.patch, HIVE-5550.patch
>
>
> A table was created using HCatalog API with out specifying the file format, 
> it defaults to:
> {code}
> fileFormat=TextFile, inputformat=org.apache.hadoop.mapred.TextInputFormat, 
> outputformat=org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> {code}
> But, when hive fetches the table from the metastore, it strangely replaces 
> the output format with 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> and the comparison between source and target table fails.
> The code in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable 
> does a string comparison of classes and fails.
> {code}
>   // check IF/OF/Serde
>   String existingifc = table.getInputFormatClass().getName();
>   String importedifc = tableDesc.getInputFormat();
>   String existingofc = table.getOutputFormatClass().getName();
>   String importedofc = tableDesc.getOutputFormat();
>   if ((!existingifc.equals(importedifc))
>   || (!existingofc.equals(importedofc))) {
> throw new SemanticException(
> ErrorMsg.INCOMPATIBLE_SCHEMA
> .getMsg(" Table inputformat/outputformats do not match"));
>   }
> {code}
> This only affects tables with text and sequence file formats but not rc or 
> orc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5550) Import fails for tables created with default text and sequence file formats using HCatalog API

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5550:
---

Status: Open  (was: Patch Available)

Cancelling patch, patch did not contain change to SequenceFile as well.

> Import fails for tables created with default text and sequence file formats 
> using HCatalog API
> --
>
> Key: HIVE-5550
> URL: https://issues.apache.org/jira/browse/HIVE-5550
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Venkatesh Seetharam
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5550.patch
>
>
> A table was created using HCatalog API with out specifying the file format, 
> it defaults to:
> {code}
> fileFormat=TextFile, inputformat=org.apache.hadoop.mapred.TextInputFormat, 
> outputformat=org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> {code}
> But, when hive fetches the table from the metastore, it strangely replaces 
> the output format with 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> and the comparison between source and target table fails.
> The code in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable 
> does a string comparison of classes and fails.
> {code}
>   // check IF/OF/Serde
>   String existingifc = table.getInputFormatClass().getName();
>   String importedifc = tableDesc.getInputFormat();
>   String existingofc = table.getOutputFormatClass().getName();
>   String importedofc = tableDesc.getOutputFormat();
>   if ((!existingifc.equals(importedifc))
>   || (!existingofc.equals(importedofc))) {
> throw new SemanticException(
> ErrorMsg.INCOMPATIBLE_SCHEMA
> .getMsg(" Table inputformat/outputformats do not match"));
>   }
> {code}
> This only affects tables with text and sequence file formats but not rc or 
> orc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5550) Import fails for tables created with default text and sequence file formats using HCatalog API

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5550:
---

Attachment: HIVE-5550.patch

Uploading patch updating to HiveIgnoreKeyOutputFormat.

Please note, however, that this api is a candidate for deprecation pretty soon.

> Import fails for tables created with default text and sequence file formats 
> using HCatalog API
> --
>
> Key: HIVE-5550
> URL: https://issues.apache.org/jira/browse/HIVE-5550
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Venkatesh Seetharam
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5550.patch
>
>
> A table was created using HCatalog API with out specifying the file format, 
> it defaults to:
> {code}
> fileFormat=TextFile, inputformat=org.apache.hadoop.mapred.TextInputFormat, 
> outputformat=org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> {code}
> But, when hive fetches the table from the metastore, it strangely replaces 
> the output format with 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> and the comparison between source and target table fails.
> The code in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable 
> does a string comparison of classes and fails.
> {code}
>   // check IF/OF/Serde
>   String existingifc = table.getInputFormatClass().getName();
>   String importedifc = tableDesc.getInputFormat();
>   String existingofc = table.getOutputFormatClass().getName();
>   String importedofc = tableDesc.getOutputFormat();
>   if ((!existingifc.equals(importedifc))
>   || (!existingofc.equals(importedofc))) {
> throw new SemanticException(
> ErrorMsg.INCOMPATIBLE_SCHEMA
> .getMsg(" Table inputformat/outputformats do not match"));
>   }
> {code}
> This only affects tables with text and sequence file formats but not rc or 
> orc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5550) Import fails for tables created with default text and sequence file formats using HCatalog API

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5550:
---

Status: Patch Available  (was: Open)

> Import fails for tables created with default text and sequence file formats 
> using HCatalog API
> --
>
> Key: HIVE-5550
> URL: https://issues.apache.org/jira/browse/HIVE-5550
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Venkatesh Seetharam
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5550.patch
>
>
> A table was created using HCatalog API with out specifying the file format, 
> it defaults to:
> {code}
> fileFormat=TextFile, inputformat=org.apache.hadoop.mapred.TextInputFormat, 
> outputformat=org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> {code}
> But, when hive fetches the table from the metastore, it strangely replaces 
> the output format with 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> and the comparison between source and target table fails.
> The code in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable 
> does a string comparison of classes and fails.
> {code}
>   // check IF/OF/Serde
>   String existingifc = table.getInputFormatClass().getName();
>   String importedifc = tableDesc.getInputFormat();
>   String existingofc = table.getOutputFormatClass().getName();
>   String importedofc = tableDesc.getOutputFormat();
>   if ((!existingifc.equals(importedifc))
>   || (!existingofc.equals(importedofc))) {
> throw new SemanticException(
> ErrorMsg.INCOMPATIBLE_SCHEMA
> .getMsg(" Table inputformat/outputformats do not match"));
>   }
> {code}
> This only affects tables with text and sequence file formats but not rc or 
> orc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Assigned] (HIVE-5550) Import fails for tables created with default text and sequence file formats using HCatalog API

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan reassigned HIVE-5550:
--

Assignee: Sushanth Sowmyan

> Import fails for tables created with default text and sequence file formats 
> using HCatalog API
> --
>
> Key: HIVE-5550
> URL: https://issues.apache.org/jira/browse/HIVE-5550
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Venkatesh Seetharam
>Assignee: Sushanth Sowmyan
>
> A table was created using HCatalog API with out specifying the file format, 
> it defaults to:
> {code}
> fileFormat=TextFile, inputformat=org.apache.hadoop.mapred.TextInputFormat, 
> outputformat=org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat
> {code}
> But, when hive fetches the table from the metastore, it strangely replaces 
> the output format with 
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> and the comparison between source and target table fails.
> The code in org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer#checkTable 
> does a string comparison of classes and fails.
> {code}
>   // check IF/OF/Serde
>   String existingifc = table.getInputFormatClass().getName();
>   String importedifc = tableDesc.getInputFormat();
>   String existingofc = table.getOutputFormatClass().getName();
>   String importedofc = tableDesc.getOutputFormat();
>   if ((!existingifc.equals(importedifc))
>   || (!existingofc.equals(importedofc))) {
> throw new SemanticException(
> ErrorMsg.INCOMPATIBLE_SCHEMA
> .getMsg(" Table inputformat/outputformats do not match"));
>   }
> {code}
> This only affects tables with text and sequence file formats but not rc or 
> orc.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5799) session/operation timeout for hiveserver2

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828269#comment-13828269
 ] 

Thejas M Nair commented on HIVE-5799:
-

[~navis] I think using a different thread spawned by jdbc driver and using a 
new connection from that thread every minute or so would be much simpler. As 
long as the heartbeats are sent only every minute or so, the load it puts on 
HS2 should be negligible. What do you think?



> session/operation timeout for hiveserver2
> -
>
> Key: HIVE-5799
> URL: https://issues.apache.org/jira/browse/HIVE-5799
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5799.1.patch.txt, HIVE-5799.2.patch.txt, 
> HIVE-5799.3.patch.txt
>
>
> Need some timeout facility for preventing resource leakages from instable  or 
> bad clients.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5635) WebHCatJTShim23 ignores security/user context

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5635:


   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Patch committed to trunk.
Thanks for the contribution Eugene!


> WebHCatJTShim23 ignores security/user context
> -
>
> Key: HIVE-5635
> URL: https://issues.apache.org/jira/browse/HIVE-5635
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.13.0
>
> Attachments: HIVE-5635.2.patch, HIVE-5635.3.patch, HIVE-5635.patch
>
>
> WebHCatJTShim23 takes UserGroupInformation object as argument (which 
> represents the user make the call to WebHCat or doAs user) but ignores.
> WebHCatJTShim20S uses the UserGroupInformation
> This is inconsistent and may be a security hole because in with Hadoop 2 the  
> methods on WebHCatJTShim are likely running with 'hcat' as the user context.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5817) column name to index mapping in VectorizationContext is broken

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828257#comment-13828257
 ] 

Eric Hanson commented on HIVE-5817:
---

Uploaded HIVE-5817.00-broken.patch here to make it easier to review the 
differences:

https://reviews.apache.org/r/15740/

> column name to index mapping in VectorizationContext is broken
> --
>
> Key: HIVE-5817
> URL: https://issues.apache.org/jira/browse/HIVE-5817
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-5817-uniquecols.broken.patch, 
> HIVE-5817.00-broken.patch
>
>
> Columns coming from different operators may have the same internal names 
> ("_colNN"). There exists a query in the form {{select b.cb, a.ca from a JOIN 
> b ON ... JOIN x ON ...;}}  (distilled from a more complex query), which runs 
> ok w/o vectorization. With vectorization, it will run ok for most ca, but for 
> some ca it will fail (or can probably return incorrect results). That is 
> because when building column-to-VRG-index map in VectorizationContext, 
> internal column name for ca that the first map join operator adds to the 
> mapping may be the same as internal name for cb that the 2nd one tries to 
> add. 2nd VMJ doesn't add it (see code in ctor), and when it's time for it to 
> output stuff, it retrieves wrong index from the map by name, and then wrong 
> vector from VRG.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5837) SQL standard based secure authorization for hive

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828242#comment-13828242
 ] 

Thejas M Nair commented on HIVE-5837:
-

[~navis] Thanks, yes, I would certainly appreciate help with this.

What/where are the NHIVE-* jiras(?) that you refer to ? 


> SQL standard based secure authorization for hive
> 
>
> Key: HIVE-5837
> URL: https://issues.apache.org/jira/browse/HIVE-5837
> Project: Hive
>  Issue Type: New Feature
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: SQL standard authorization hive.pdf
>
>
> The current default authorization is incomplete and not secure. The 
> alternative of storage based authorization provides security but does not 
> provide fine grained authorization.
> The proposal is to support secure fine grained authorization in hive using 
> SQL standard based authorization model.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (HIVE-5862) While running some queries on large data using tez, we OOM.

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner resolved HIVE-5862.
--

Resolution: Fixed

Committed to branch. Thanks Vikram!

> While running some queries on large data using tez, we OOM.
> ---
>
> Key: HIVE-5862
> URL: https://issues.apache.org/jira/browse/HIVE-5862
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-5862.1.patch, HIVE-5862.2.patch
>
>
> Running out of memory while running map joins in tez on large data sets.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5849) Improve the stats of operators based on heuristics in the absence of any column statistics

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828229#comment-13828229
 ] 

Harish Butani commented on HIVE-5849:
-

+1

> Improve the stats of operators based on heuristics in the absence of any 
> column statistics
> --
>
> Key: HIVE-5849
> URL: https://issues.apache.org/jira/browse/HIVE-5849
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor, Statistics
>Reporter: Prasanth J
>Assignee: Prasanth J
> Fix For: 0.13.0
>
> Attachments: HIVE-5849.1.patch.txt, HIVE-5849.2.patch.txt, 
> HIVE-5849.3.patch, HIVE-5849.3.patch.txt, HIVE-5849.4.javaonly.patch, 
> HIVE-5849.5.patch
>
>
> In the absence of any column statistics, operators will simply use the 
> statistics from its parents. It is useful to apply some heuristics to update 
> basic statistics (number of rows and data size) in the absence of any column 
> statistics. This will be worst case scenario.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5862) While running some queries on large data using tez, we OOM.

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-5862:
-

Attachment: HIVE-5862.2.patch

.2 has one more location

> While running some queries on large data using tez, we OOM.
> ---
>
> Key: HIVE-5862
> URL: https://issues.apache.org/jira/browse/HIVE-5862
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-5862.1.patch, HIVE-5862.2.patch
>
>
> Running out of memory while running map joins in tez on large data sets.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5849) Improve the stats of operators based on heuristics in the absence of any column statistics

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5849:
-

Attachment: HIVE-5849.5.patch

Addressed [~rhbutani]'s review comments. Regenerated golden files of affected 
test cases.

> Improve the stats of operators based on heuristics in the absence of any 
> column statistics
> --
>
> Key: HIVE-5849
> URL: https://issues.apache.org/jira/browse/HIVE-5849
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor, Statistics
>Reporter: Prasanth J
>Assignee: Prasanth J
> Fix For: 0.13.0
>
> Attachments: HIVE-5849.1.patch.txt, HIVE-5849.2.patch.txt, 
> HIVE-5849.3.patch, HIVE-5849.3.patch.txt, HIVE-5849.4.javaonly.patch, 
> HIVE-5849.5.patch
>
>
> In the absence of any column statistics, operators will simply use the 
> statistics from its parents. It is useful to apply some heuristics to update 
> basic statistics (number of rows and data size) in the absence of any column 
> statistics. This will be worst case scenario.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5862) While running some queries on large data using tez, we OOM.

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-5862:
-

Attachment: HIVE-5862.1.patch

Fixes couple of the leaks found.

> While running some queries on large data using tez, we OOM.
> ---
>
> Key: HIVE-5862
> URL: https://issues.apache.org/jira/browse/HIVE-5862
> Project: Hive
>  Issue Type: Bug
>  Components: Tez
>Affects Versions: tez-branch
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-5862.1.patch
>
>
> Running out of memory while running map joins in tez on large data sets.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HIVE-5862) While running some queries on large data using tez, we OOM.

[Vikram Dixit K (JIRA)]

Vikram Dixit K created HIVE-5862:


 Summary: While running some queries on large data using tez, we 
OOM.
 Key: HIVE-5862
 URL: https://issues.apache.org/jira/browse/HIVE-5862
 Project: Hive
  Issue Type: Bug
  Components: Tez
Affects Versions: tez-branch
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K


Running out of memory while running map joins in tez on large data sets.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5861) Fix exception in multi insert statement on Tez

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-5861:
-

Attachment: HIVE-5861.1.patch

> Fix exception in multi insert statement on Tez
> --
>
> Key: HIVE-5861
> URL: https://issues.apache.org/jira/browse/HIVE-5861
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Fix For: tez-branch
>
> Attachments: HIVE-5861.1.patch
>
>
> Multi insert statements that have multiple group by clauses aren't handled 
> properly in tez.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HIVE-5861) Fix exception in multi insert statement on Tez

[Gunther Hagleitner (JIRA)]

Gunther Hagleitner created HIVE-5861:


 Summary: Fix exception in multi insert statement on Tez
 Key: HIVE-5861
 URL: https://issues.apache.org/jira/browse/HIVE-5861
 Project: Hive
  Issue Type: Bug
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
 Fix For: tez-branch


Multi insert statements that have multiple group by clauses aren't handled 
properly in tez.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5859) Create view does not captures inputs

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828118#comment-13828118
 ] 

Hive QA commented on HIVE-5859:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614791/D14235.1.patch

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 4665 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_tbl_props
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_deletejar
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_columns
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/376/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/376/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614791

> Create view does not captures inputs
> 
>
> Key: HIVE-5859
> URL: https://issues.apache.org/jira/browse/HIVE-5859
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D14235.1.patch
>
>
> For example, 
> CREATE VIEW view_j5jbymsx8e_1 as SELECT * FROM tbl_j5jbymsx8e;
> should capture "default.tbl_j5jbymsx8e" as input entity for authorization 
> process but currently it's not.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5663) Refactor ORC RecordReader to operate on direct & wrapped ByteBuffers

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-5663:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Comitted to trunk. Thanks, Gopal!

> Refactor ORC RecordReader to operate on direct & wrapped ByteBuffers
> 
>
> Key: HIVE-5663
> URL: https://issues.apache.org/jira/browse/HIVE-5663
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 0.13.0
> Environment: Ubuntu LXC 
>Reporter: Gopal V
>Assignee: Gopal V
>  Labels: ORC
> Fix For: 0.13.0
>
> Attachments: HIVE-5663.01.patch, HIVE-5663.02.patch, 
> HIVE-5663.03.patch
>
>
> The current ORC RecordReader implementation assumes array structures backing 
> the ByteBuffers it passes around between RecordReaderImpl and 
> Compressed/Uncompressed InStream objects.
> This patch attempts to refactor those assumptions out of both classes, 
> allowing the future use of direct byte buffers within ORC (as might come from 
> HDFS zero-copy readers).



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5224) When creating table with AVRO serde, the "avro.schema.url" should be about to load serde schema from file system beside HDFS

[Jakob Homan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828051#comment-13828051
 ] 

Jakob Homan commented on HIVE-5224:
---

So the goal is to use o.a.h.FileSystem to read other derivative file systems? 
Sounds reasonable.  But doesn't this lead to a similar situation when trying to 
open a URI that's not http or file?  Not sure that's important though.

> When creating table with AVRO serde, the "avro.schema.url" should be about to 
> load serde schema from file system beside HDFS
> 
>
> Key: HIVE-5224
> URL: https://issues.apache.org/jira/browse/HIVE-5224
> Project: Hive
>  Issue Type: Bug
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-5224.1.patch, HIVE-5224.2.patch, Hive-5224.3.patch
>
>
> Now when loading schema for table with AVRO serde, the file system is hard 
> coded to hdfs in AvroSerdeUtils.java. This should enable loading schema from 
> file system beside hdfs.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5845) CTAS failed on vectorized code path

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828056#comment-13828056
 ] 

Ashutosh Chauhan commented on HIVE-5845:


+1

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Attachments: HIVE-5845.1.patch
>
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Hive-trunk-h0.21 - Build # 2461 - Still Failing

[Apache Jenkins Server]

Changes for Build #2429
[hashutosh] HIVE-5681 : Validation doesn't catch SMBMapJoin (Jitendra Nath 
Pandey via Ashutosh Chauhan)

[thejas] HIVE-5519 : Use paging mechanism for templeton get requests. (Hari 
Sankar Sivarama Subramaniyan via Thejas Nair)


Changes for Build #2430
[hashutosh] HIVE-5670 : annoying ZK exceptions are annoying (Sergey Shelukhin 
via Ashutosh Chauhan)

[hashutosh] HIVE-5613 : Subquery support: disallow nesting of SubQueries 
(Harish Butani via Ashutosh Chauhan)

[hashutosh] HIVE-5698 : A few test file missing apache license header (Xuefu 
Zhang via Ashutosh Chauhan)

[daijy] HIVE-5696: WebHCat e2e tests/jobsubmission.conf file is malformed and 
loosing tests ( Eugene Koifman via Daniel Dai )


Changes for Build #2431
[hashutosh] HIVE-5693 : Rewrite some tests to reduce test time (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-5582 : Implement BETWEEN filter in vectorized mode (Eric 
Hanson via Ashutosh Chauhan)

[hashutosh] HIVE-5556 : Pushdown join conditions (Harish Butani via Ashutosh 
Chauhan)


Changes for Build #2432
[brock] HIVE-5716 - Fix broken tests after maven merge (1) (Brock Noland 
reviewed by Thejas M Nair and Ashutosh Chauhan)

[thejas] HIVE-5483 : use metastore statistics to optimize max/min/etc. queries 
(Ashutosh Chauhan via Thejas Nair)

[daijy] HIVE-5510: [WebHCat] GET job/queue return wrong job information

[brock] HIVE-5610 - Merge maven branch into trunk (delete ant)

[brock] HIVE-5610 - Merge maven branch into trunk (maven rollforward)

[brock] HIVE-5610 - Merge maven branch into trunk (patch)


Changes for Build #2433

Changes for Build #2434
[hashutosh] HIVE-3959 : Update Partition Statistics in Metastore Layer 
(Ashutosh Chauhan, Bhushan Mandhani, Gang Tim Liu via Thejas Nair)


Changes for Build #2435
[hashutosh] HIVE-5503 : TopN optimization in VectorReduceSink (Sergey Shelukhin 
via Ashutosh Chauhan)

[brock] HIVE-5695 - PTest2 fix shutdown, duplicate runs, and add client retry

[brock] HIVE-5708 - PTest2 should trim long logs when posting to jira


Changes for Build #2436
[thejas] HIVE-5547 : webhcat pig job submission should ship hive tar if 
-usehcatalog is specified (Eugene Koifman via Thejas Nair)

[thejas] HIVE-5715 : HS2 should not "start" a session for every command 
(Gunther Hagleitner via Thejas Nair)


Changes for Build #2437

Changes for Build #2438
[brock] HIVE-5740: Tar files should extract to the directory of the same name 
minus tar.gz (Brock Noland reviewed by Xuefu Zhang)

[brock] HIVE-5611: Add assembly (i.e.) tar creation to pom (Szehon Ho via Brock 
Noland)

[brock] HIVE-5707: Validate values for ConfVar (Navis via Brock Noland)

[brock] HIVE-5721: Incremental build is disabled by MCOMPILER-209 (Navis via 
Brock Noland)


Changes for Build #2439
[brock] HIVE-5354 - Decimal precision/scale support in ORC file (Xuefu Zhang 
via Brock Noland)

[brock] HIVE-4523 - round() function with specified decimal places not 
consistent with mysql (Xuefu Zhang via Brock Noland)

[thejas] HIVE-5542 : Webhcat is failing to run ddl command on a secure cluster 
(Sushanth Sowmyan via Thejas Nair)


Changes for Build #2440
[brock] HIVE-5730: Beeline throws non-terminal NPE upon starting, after 
mavenization (Szehon Ho reviewed by Navis)


Changes for Build #2441
[omalley] HIVE-5425 Provide a configuration option to control the default stripe
size for ORC. (omalley reviewed by gunther)

[omalley] Revert HIVE-5583 since it broke the build.

[hashutosh] HIVE-5583 : Implement support for IN (list-of-constants) filter in 
vectorized mode (Eric Hanson via Ashutosh Chauhan)

[brock] HIVE-5355 - JDBC support for decimal precision/scale


Changes for Build #2443
[brock] HIVE-5351 - Secure-Socket-Layer (SSL) support for HiveServer2 (Prasad 
Mujumdar via Brock Noland)

[hashutosh] HIVE-5583 : Implement support for IN (list-of-constants) filter in 
vectorized mode (Eric Hanson via Ashutosh Chauhan)

[brock] HIVE-5773 - Fix build due to conflict between HIVE-5711 and HIVE-5713

[brock] HIVE-5711 - Fix eclipse:eclipse maven goal (Carl Steinbach via Brock 
Noland)

[brock] HIVE-5752 - log4j properties appear to have been lost in maven upgrade 
(Sergey Shelukhin via Brock Noland)

[brock] HIVE-5713 - Verify versions of libraries post maven merge (Brock Noland 
reviewed by Gunther Hagleitner)

[brock] HIVE-5765 - Beeline throws NPE when -e option is used (Szehon Ho via 
Brock Noland)

[xuefu] HIVE-5726: The DecimalTypeInfo instance associated with a decimal 
constant is not in line with the precision/scale of the constant (reviewed by 
Brock)

[xuefu] HIVE-5655: Hive incorrecly handles divide-by-zero case (reviewed by 
Edward and Brock)

[xuefu] HIVE-5191: Add char data type (Jason via Xuefu)


Changes for Build #2444
[brock] HIVE-5780 - Add the missing declaration of HIVE_CLI_SERVICE_PROTOCOL_V4 
in TCLIService.thrift (Prasad Mujumdar via Brock Noland)


Changes for Build #2445
[gunther] HIVE-5601: NPE in ORC's PPD when using select * from table with where 
predicat

[jira] [Commented] (HIVE-5850) Multiple table join error for avro

[Jakob Homan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828055#comment-13828055
 ] 

Jakob Homan commented on HIVE-5850:
---

This has been a recurring problem.  The code to figure out what schema goes 
where has been problematic and the information passed to the mapper has changed 
from Hive version to Hive version.  Using the parent may not always get the 
latest schema, yes?

> Multiple table join error for avro 
> ---
>
> Key: HIVE-5850
> URL: https://issues.apache.org/jira/browse/HIVE-5850
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Shengjun Xin
> Attachments: part.tar.gz, partsupp.tar.gz, schema.tar.gz
>
>
> Reproduce step:
> {code}
> -- Create table Part.
> CREATE EXTERNAL TABLE part
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS
> INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION 'hdfs:///user/hadoop/tpc-h/data/part'
> TBLPROPERTIES 
> ('avro.schema.url'='hdfs:///user/hadoop/tpc-h/schema/part.avsc');
> -- Create table Part Supplier.
> CREATE EXTERNAL TABLE partsupp
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS
> INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION 'hdfs:///user/hadoop/tpc-h/data/partsupp'
> TBLPROPERTIES 
> ('avro.schema.url'='hdfs:///user/hadoop/tpc-h/schema/partsupp.avsc');
> --- Query
> select * from partsupp ps join part p on ps.ps_partkey = p.p_partkey where 
> p.p_partkey=1;
> {code}
> {code}
> Error message is:
> Error: java.io.IOException: java.io.IOException: 
> org.apache.avro.AvroTypeException: Found {
>   "type" : "record",
>   "name" : "partsupp",
>   "namespace" : "com.gs.sdst.pl.avro.tpch",
>   "fields" : [ {
> "name" : "ps_partkey",
> "type" : "long"
>   }, {
> "name" : "ps_suppkey",
> "type" : "long"
>   }, {
> "name" : "ps_availqty",
> "type" : "long"
>   }, {
> "name" : "ps_supplycost",
> "type" : "double"
>   }, {
> "name" : "ps_comment",
> "type" : "string"
>   }, {
> "name" : "systimestamp",
> "type" : "long"
>   } ]
> }, expecting {
>   "type" : "record",
>   "name" : "part",
>   "namespace" : "com.gs.sdst.pl.avro.tpch",
>   "fields" : [ {
> "name" : "p_partkey",
> "type" : "long"
>   }, {
> "name" : "p_name",
> "type" : "string"
>   }, {
> "name" : "p_mfgr",
> "type" : "string"
>   }, {
> "name" : "p_brand",
> "type" : "string"
>   }, {
> "name" : "p_type",
> "type" : "string"
>   }, {
> "name" : "p_size",
> "type" : "int"
>   }, {
> "name" : "p_container",
> "type" : "string"
>   }, {
> "name" : "p_retailprice",
> "type" : "double"
>   }, {
> "name" : "p_comment",
> "type" : "string"
>   }, {
> "name" : "systimestamp",
> "type" : "long"
>   } ]
> }
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:302)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:218)
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:197)
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:183)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5565) Limit Hive decimal type maximum precision and scale to 38

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-5565:
--

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Patch committed to trunk. Thanks to Brock for the review.

> Limit Hive decimal type maximum precision and scale to 38
> -
>
> Key: HIVE-5565
> URL: https://issues.apache.org/jira/browse/HIVE-5565
> Project: Hive
>  Issue Type: Task
>  Components: Types
>Affects Versions: 0.13.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-5565.1.patch, HIVE-5565.2.patch, HIVE-5565.3.patch, 
> HIVE-5565.4.patch, HIVE-5565.5.patch, HIVE-5565.6.patch, HIVE-5565.patch
>
>
> With HIVE-3976, the maximum precision is set to 65, and maximum scale is to 
> 30. After discussing with several folks in the community, it's determined 
> that 38 as a maximum for both precision and scale are probably sufficient, in 
> addition to the potential performance boost that might become possible to 
> some implementation.
> This task is to make such a change. The change is expected to be trivial, but 
> it may impact many test cases. The reason for a separate JIRA is that patch 
> in HIVE-3976 is already in a good shape. Rather than destabilizing a bigger 
> patch, a dedicate patch will facilitates both reviews.
> The wiki document will be updated shortly.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5849) Improve the stats of operators based on heuristics in the absence of any column statistics

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5849:
-

Attachment: HIVE-5849.4.javaonly.patch

Uploading java only patch for review. Q file tests are running. I will reupload 
the patch once done.

> Improve the stats of operators based on heuristics in the absence of any 
> column statistics
> --
>
> Key: HIVE-5849
> URL: https://issues.apache.org/jira/browse/HIVE-5849
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor, Statistics
>Reporter: Prasanth J
>Assignee: Prasanth J
> Fix For: 0.13.0
>
> Attachments: HIVE-5849.1.patch.txt, HIVE-5849.2.patch.txt, 
> HIVE-5849.3.patch, HIVE-5849.3.patch.txt, HIVE-5849.4.javaonly.patch
>
>
> In the absence of any column statistics, operators will simply use the 
> statistics from its parents. It is useful to apply some heuristics to update 
> basic statistics (number of rows and data size) in the absence of any column 
> statistics. This will be worst case scenario.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5565) Limit Hive decimal type maximum precision and scale to 38

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828005#comment-13828005
 ] 

Hive QA commented on HIVE-5565:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614937/HIVE-5565.6.patch

{color:green}SUCCESS:{color} +1 4665 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/375/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/375/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614937

> Limit Hive decimal type maximum precision and scale to 38
> -
>
> Key: HIVE-5565
> URL: https://issues.apache.org/jira/browse/HIVE-5565
> Project: Hive
>  Issue Type: Task
>  Components: Types
>Affects Versions: 0.13.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5565.1.patch, HIVE-5565.2.patch, HIVE-5565.3.patch, 
> HIVE-5565.4.patch, HIVE-5565.5.patch, HIVE-5565.6.patch, HIVE-5565.patch
>
>
> With HIVE-3976, the maximum precision is set to 65, and maximum scale is to 
> 30. After discussing with several folks in the community, it's determined 
> that 38 as a maximum for both precision and scale are probably sufficient, in 
> addition to the potential performance boost that might become possible to 
> some implementation.
> This task is to make such a change. The change is expected to be trivial, but 
> it may impact many test cases. The reason for a separate JIRA is that patch 
> in HIVE-3976 is already in a good shape. Rather than destabilizing a bigger 
> patch, a dedicate patch will facilitates both reviews.
> The wiki document will be updated shortly.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15295: HIVE-5618: Hive local task fails to run when run from oozie in a secure cluster

[Prasad Mujumdar]

> On Nov. 19, 2013, 8:53 p.m., Thejas Nair wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java, line 233
> > 
> >
> > Is this a behavior change between 2.1.x and 2.2.x ? or between 2.1.x 
> > and 1.x ?
> >

I believe it's between 1.x and 2.1.x, however I can't pinpoint to a specific 
Hadoop ticket that changed the behavior.


- Prasad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15295/#review29133
---


On Nov. 7, 2013, 12:15 a.m., Prasad Mujumdar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15295/
> ---
> 
> (Updated Nov. 7, 2013, 12:15 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5618
> https://issues.apache.org/jira/browse/HIVE-5618
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch is to pass the delegation token to child JVM if the parent is using 
> a keytab based login.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 540a9a1 
>   shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 
> 623ebcd 
>   
> shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
>  fd274a6 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> 62ff878 
> 
> Diff: https://reviews.apache.org/r/15295/diff/
> 
> 
> Testing
> ---
> 
> Manually tested on secure cluster.
> 
> 
> Thanks,
> 
> Prasad Mujumdar
> 
>

[jira] [Updated] (HIVE-5833) Remove versions from child module dependencies

[Kousuke Saruta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kousuke Saruta updated HIVE-5833:
-

Attachment: HIVE-5833.2.patch

[~brocknoland] Thank you for your comment!
I've just modified that.

> Remove versions from child module dependencies
> --
>
> Key: HIVE-5833
> URL: https://issues.apache.org/jira/browse/HIVE-5833
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Brock Noland
> Attachments: HIVE-5833.2.patch, HIVE-5833.patch
>
>
> HIVE-5741 moved all dependencies to the plugin management section of the 
> parent pom therefore we can remove 
> {noformat}${dep.version}{noformat} from all dependencies 
> in child modules.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5857) Reduce tasks do not work in uber mode in YARN

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827950#comment-13827950
 ] 

Hive QA commented on HIVE-5857:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614761/HIVE-5857.1.patch.txt

{color:green}SUCCESS:{color} +1 4665 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/374/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/374/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614761

> Reduce tasks do not work in uber mode in YARN
> -
>
> Key: HIVE-5857
> URL: https://issues.apache.org/jira/browse/HIVE-5857
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Adam Kawa
>Priority: Critical
>  Labels: plan, uber-jar, yarn
> Attachments: HIVE-5857.1.patch.txt
>
>
> A Hive query fails when it tries to run a reduce task in uber mode in YARN.
> The NullPointerException is thrown in the ExecReducer.configure method, 
> because the plan file (reduce.xml) for a reduce task is not found.
> The Utilities.getBaseWork method is expected to return BaseWork object, but 
> it returns NULL due to FileNotFoundException. 
> {code}
> // org.apache.hadoop.hive.ql.exec.Utilities
> public static BaseWork getBaseWork(Configuration conf, String name) {
>   ...
> try {
> ...
>   if (gWork == null) {
> Path localPath;
> if (ShimLoader.getHadoopShims().isLocalMode(conf)) {
>   localPath = path;
> } else {
>   localPath = new Path(name);
> }
> InputStream in = new FileInputStream(localPath.toUri().getPath());
> BaseWork ret = deserializePlan(in);
> 
>   }
>   return gWork;
> } catch (FileNotFoundException fnf) {
>   // happens. e.g.: no reduce work.
>   LOG.debug("No plan file found: "+path);
>   return null;
> } ...
> }
> {code}
> It happens because, the ShimLoader.getHadoopShims().isLocalMode(conf)) method 
> returns true, because immediately before running a reduce task, 
> org.apache.hadoop.mapred.LocalContainerLauncher changes its configuration to 
> local mode ("mapreduce.framework.name" is changed from" "yarn" to "local"). 
> On the other hand map tasks run successfully, because its configuration is 
> not changed and still remains "yarn".
> {code}
> // org.apache.hadoop.mapred.LocalContainerLauncher
> private void runSubtask(..) {
>   ...
>   conf.set(MRConfig.FRAMEWORK_NAME, MRConfig.LOCAL_FRAMEWORK_NAME);
>   conf.set(MRConfig.MASTER_ADDRESS, "local");  // bypass shuffle
>   ReduceTask reduce = (ReduceTask)task;
>   reduce.setConf(conf);  
>   reduce.run(conf, umbilical);
> }
> {code}
> A super quick fix could just an additional if-branch, where we check if we 
> run a reduce task in uber mode, and then look for a plan file in a different 
> location.
> *Java stacktrace*
> {code}
> 2013-11-20 00:50:56,862 INFO [uber-SubtaskRunner] 
> org.apache.hadoop.hive.ql.exec.Utilities: No plan file found: 
> hdfs://namenode.c.lon.spotify.net:54310/var/tmp/kawaa/hive_2013-11-20_00-50-43_888_3938384086824086680-2/-mr-10003/e3caacf6-15d6-4987-b186-d2906791b5b0/reduce.xml
> 2013-11-20 00:50:56,862 WARN [uber-SubtaskRunner] 
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local 
> (uberized) 'child' : java.lang.RuntimeException: Error in configuring object
>   at 
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
>   at 
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
>   at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>   at 
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:427)
>   at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.runSubtask(LocalContainerLauncher.java:340)
>   at 
> org.apache.hadoop.mapred.LocalContainerLauncher$SubtaskRunner.run(LocalContainerLauncher.java:225)
>   at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.reflect.InvocationTargetException
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at jav

[jira] [Updated] (HIVE-5849) Improve the stats of operators based on heuristics in the absence of any column statistics

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5849:
-

Attachment: HIVE-5849.3.patch

Reuploading .3 patch for HIVE QA to run precommit tests.

> Improve the stats of operators based on heuristics in the absence of any 
> column statistics
> --
>
> Key: HIVE-5849
> URL: https://issues.apache.org/jira/browse/HIVE-5849
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor, Statistics
>Reporter: Prasanth J
>Assignee: Prasanth J
> Fix For: 0.13.0
>
> Attachments: HIVE-5849.1.patch.txt, HIVE-5849.2.patch.txt, 
> HIVE-5849.3.patch, HIVE-5849.3.patch.txt
>
>
> In the absence of any column statistics, operators will simply use the 
> statistics from its parents. It is useful to apply some heuristics to update 
> basic statistics (number of rows and data size) in the absence of any column 
> statistics. This will be worst case scenario.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5849) Improve the stats of operators based on heuristics in the absence of any column statistics

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5849:
-

Attachment: HIVE-5849.3.patch.txt

Fixed failing test case bucketmapjoin7.q in TestMinimrCliDriver.

> Improve the stats of operators based on heuristics in the absence of any 
> column statistics
> --
>
> Key: HIVE-5849
> URL: https://issues.apache.org/jira/browse/HIVE-5849
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor, Statistics
>Reporter: Prasanth J
>Assignee: Prasanth J
> Fix For: 0.13.0
>
> Attachments: HIVE-5849.1.patch.txt, HIVE-5849.2.patch.txt, 
> HIVE-5849.3.patch.txt
>
>
> In the absence of any column statistics, operators will simply use the 
> statistics from its parents. It is useful to apply some heuristics to update 
> basic statistics (number of rows and data size) in the absence of any column 
> statistics. This will be worst case scenario.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5614) Subquery support: allow subquery expressions in having clause

[Harish Butani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827915#comment-13827915
 ] 

Harish Butani commented on HIVE-5614:
-

review request at https://reviews.apache.org/r/15718/

> Subquery support: allow subquery expressions in having clause
> -
>
> Key: HIVE-5614
> URL: https://issues.apache.org/jira/browse/HIVE-5614
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-5614.1.patch, HIVE-5614.2.patch, HIVE-5614.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5565) Limit Hive decimal type maximum precision and scale to 38

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-5565:
--

Attachment: HIVE-5565.6.patch

Patch #6 fixed the test failures above.

> Limit Hive decimal type maximum precision and scale to 38
> -
>
> Key: HIVE-5565
> URL: https://issues.apache.org/jira/browse/HIVE-5565
> Project: Hive
>  Issue Type: Task
>  Components: Types
>Affects Versions: 0.13.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5565.1.patch, HIVE-5565.2.patch, HIVE-5565.3.patch, 
> HIVE-5565.4.patch, HIVE-5565.5.patch, HIVE-5565.6.patch, HIVE-5565.patch
>
>
> With HIVE-3976, the maximum precision is set to 65, and maximum scale is to 
> 30. After discussing with several folks in the community, it's determined 
> that 38 as a maximum for both precision and scale are probably sufficient, in 
> addition to the potential performance boost that might become possible to 
> some implementation.
> This task is to make such a change. The change is expected to be trivial, but 
> it may impact many test cases. The reason for a separate JIRA is that patch 
> in HIVE-3976 is already in a good shape. Rather than destabilizing a bigger 
> patch, a dedicate patch will facilitates both reviews.
> The wiki document will be updated shortly.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5614) Subquery support: allow subquery expressions in having clause

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-5614:


Status: Patch Available  (was: Open)

> Subquery support: allow subquery expressions in having clause
> -
>
> Key: HIVE-5614
> URL: https://issues.apache.org/jira/browse/HIVE-5614
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-5614.1.patch, HIVE-5614.2.patch, HIVE-5614.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5614) Subquery support: allow subquery expressions in having clause

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani updated HIVE-5614:


Attachment: HIVE-5614.3.patch

> Subquery support: allow subquery expressions in having clause
> -
>
> Key: HIVE-5614
> URL: https://issues.apache.org/jira/browse/HIVE-5614
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-5614.1.patch, HIVE-5614.2.patch, HIVE-5614.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Review Request 15718: HIVE-5614: Subquery support: allow subquery expressions in having clause

[Harish Butani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15718/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-5614
https://issues.apache.org/jira/browse/HIVE-5614


Repository: hive-git


Description
---

support for subquery predicates in having clause. SubTask of HIVE-784


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java fa111cc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java 3e8215d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 7979873 
  ql/src/test/queries/clientpositive/subquery_exists_having.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_in_having.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_notexists_having.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_notin_having.q PRE-CREATION 
  ql/src/test/results/clientpositive/subquery_exists_having.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/subquery_in_having.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/subquery_multiinsert.q.out 8dfb485 
  ql/src/test/results/clientpositive/subquery_notexists_having.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/subquery_notin_having.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/15718/diff/


Testing
---

added new tests: subquery_in_having.q, subquery_notin_having.q, 
subquery_exists_having.q, subquery_notexists_having.q


Thanks,

Harish Butani

[jira] [Commented] (HIVE-4887) hive should have an option to disable non sql commands that impose security risk

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827890#comment-13827890
 ] 

Prasad Mujumdar commented on HIVE-4887:
---

hmm .. I don't think the HiveServer2 client can submit a shell command to the 
server directly. The IDL doesn't include any RPC call for that. You execute 
commands like 'dfs -rm /user/hive/warehouse/tab1' via ExecuteStatement() 
interface. HiveServer2 (similar to CLI), supports non-sql commands like DFS, 
SET, RESET, ADD, DELET. Anything other than that will be directed to compiler.

Beeline treats '!' as escape sequence. Some of the commands are translated to 
JDBC metadata requests (eg !tables will make beeline call 
DatabaseMetaData.getTables()). However beeline won't redirect a shell command 
to HS2. If you are on the a cluster node with HDFS access, then you can execute 
shell operations. But those will be with your credentials and not as hive.

[~thejas] I apologize if I completely misunderstood your point and rambling 
something here ..

> hive should have an option to disable non sql commands that impose security 
> risk
> 
>
> Key: HIVE-4887
> URL: https://issues.apache.org/jira/browse/HIVE-4887
> Project: Hive
>  Issue Type: Task
>  Components: Authorization, Security
>Reporter: Thejas M Nair
>
> Hive's RDBMS style of authorization (using grant/revoke), relies on all data 
> access being done through hive select queries. But hive also supports running 
> dfs commands, shell commands (eg "!cat file"), and shell commands through 
> hive streaming.
> This creates problems in securing a hive server using this authorization 
> model. UDF is another way to write custom code that can compromise security, 
> but you can control that by restricting access to users to be only through 
> jdbc connection to hive server (2).
> (note that there are other major problems such as this one - HIVE-3271)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5842) Fix issues with new paths to jar in hcatalog

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5842:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Thank you Prasad!!

> Fix issues with new paths to jar in hcatalog
> 
>
> Key: HIVE-5842
> URL: https://issues.apache.org/jira/browse/HIVE-5842
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5842.patch
>
>
> HIVE-5741 included hcatalog in the binary tarball but some of the paths to 
> jars is slightly different requiring the scripts be updated.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5565) Limit Hive decimal type maximum precision and scale to 38

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827842#comment-13827842
 ] 

Hive QA commented on HIVE-5565:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614904/HIVE-5565.5.patch

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 4665 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/373/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/373/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614904

> Limit Hive decimal type maximum precision and scale to 38
> -
>
> Key: HIVE-5565
> URL: https://issues.apache.org/jira/browse/HIVE-5565
> Project: Hive
>  Issue Type: Task
>  Components: Types
>Affects Versions: 0.13.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5565.1.patch, HIVE-5565.2.patch, HIVE-5565.3.patch, 
> HIVE-5565.4.patch, HIVE-5565.5.patch, HIVE-5565.patch
>
>
> With HIVE-3976, the maximum precision is set to 65, and maximum scale is to 
> 30. After discussing with several folks in the community, it's determined 
> that 38 as a maximum for both precision and scale are probably sufficient, in 
> addition to the potential performance boost that might become possible to 
> some implementation.
> This task is to make such a change. The change is expected to be trivial, but 
> it may impact many test cases. The reason for a separate JIRA is that patch 
> in HIVE-3976 is already in a good shape. Rather than destabilizing a bigger 
> patch, a dedicate patch will facilitates both reviews.
> The wiki document will be updated shortly.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5663) Refactor ORC RecordReader to operate on direct & wrapped ByteBuffers

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827843#comment-13827843
 ] 

Gopal V commented on HIVE-5663:
---

bucket_num_reducers.q is

{code}
CREATE TABLE bucket_nr(key int, value string) CLUSTERED BY (key) INTO 50 
BUCKETS;

insert overwrite table bucket_nr
select * from src;
{code}

And src table is a text table from kv1.txt - trying to confirm if this test 
fails on trunk without the patch (which should only kick in for ORC tables & 
that too in the read path only).

> Refactor ORC RecordReader to operate on direct & wrapped ByteBuffers
> 
>
> Key: HIVE-5663
> URL: https://issues.apache.org/jira/browse/HIVE-5663
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 0.13.0
> Environment: Ubuntu LXC 
>Reporter: Gopal V
>Assignee: Gopal V
>  Labels: ORC
> Attachments: HIVE-5663.01.patch, HIVE-5663.02.patch, 
> HIVE-5663.03.patch
>
>
> The current ORC RecordReader implementation assumes array structures backing 
> the ByteBuffers it passes around between RecordReaderImpl and 
> Compressed/Uncompressed InStream objects.
> This patch attempts to refactor those assumptions out of both classes, 
> allowing the future use of direct byte buffers within ORC (as might come from 
> HDFS zero-copy readers).



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5860) Select * does not work when table has non-array columns.

[Anuj Ojha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anuj Ojha updated HIVE-5860:


Description: 
I was querying a table with non-array columns and I came across situation where 
if I use select * it would throw me index out of bound exception.

{code}
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: -1
{code}

{code}
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: -1
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1403)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:270)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at 
org.apache.hadoop.hive.serde2.lazy.LazyInteger.parseInt(LazyInteger.java:107)
at 
org.apache.hadoop.hive.serde2.lazy.LazyByte.parseByte(LazyByte.java:94)
at 
org.apache.hadoop.hive.serde2.lazy.LazyByte.parseByte(LazyByte.java:74)
at 
org.apache.hadoop.hive.serde2.lazy.LazyUnion.parse(LazyUnion.java:123)
at 
org.apache.hadoop.hive.serde2.lazy.LazyUnion.getTag(LazyUnion.java:178)
at 
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyUnionObjectInspector.getTag(LazyUnionObjectInspector.java:128)
at 
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:367)
at 
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:354)
at 
org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:307)
at 
org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:220)
at 
org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:59)
at 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:365)
at 
org.apache.hadoop.hive.ql.exec.ListSinkOperator.processOp(ListSinkOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at 
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:468)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
... 11 more
{code}

Table looked something like this:
{code}
Column 1 |  column 2  | Column 3 | 
Column4
{"a": "b", "c": "d", "e": ["f": "g"]}[ some stuff here]abcde aaa
{code}

When I used single fields in select it worked but when I do select * it would 
not.
{code}
select * from default.table where column1.a='b' and column3='abcde' and 
Column4='aaa'
The above query failed.
{code}
{code}
select count(column4) from default.table where column1.a='b' and 
column3='abcde' and Column4='aaa'
The above query works.
{code}

  was:
I was querying a table with non-array columns and I came across situation where 
if I use select * it would throw me index out of bound exception.

{code}
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: -1
{code}

{code}
13/11/20 10:01:28 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
BeeswaxException(message:java.io.IOException: 
java.lang.ArrayIndexOutOfBoundsException: -1, log_context:key1, 
handle:QueryHandle(id:abc, log_context:def), SQLState: )
at 
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.fetch(BeeswaxServiceImpl.java:545)
at 
com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:986)
at 
com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:981)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at 
com.cloudera.beeswax.BeeswaxServiceImpl.doWithState(BeeswaxServiceImpl.java:772)
at 
com.cloudera.beeswax.BeeswaxServiceImpl.fetch(BeeswaxServiceImpl.java:980)
at 
com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:987)
at 
com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:971)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
 

[jira] [Updated] (HIVE-5860) Select * does not work when table has non-array columns.

[Anuj Ojha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anuj Ojha updated HIVE-5860:


Description: 
I was querying a table with non-array columns and I came across situation where 
if I use select * it would throw me index out of bound exception.

{code}
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: -1
{code}

{code}
13/11/20 10:01:28 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
BeeswaxException(message:java.io.IOException: 
java.lang.ArrayIndexOutOfBoundsException: -1, log_context:key1, 
handle:QueryHandle(id:abc, log_context:def), SQLState: )
at 
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.fetch(BeeswaxServiceImpl.java:545)
at 
com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:986)
at 
com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:981)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at 
com.cloudera.beeswax.BeeswaxServiceImpl.doWithState(BeeswaxServiceImpl.java:772)
at 
com.cloudera.beeswax.BeeswaxServiceImpl.fetch(BeeswaxServiceImpl.java:980)
at 
com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:987)
at 
com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:971)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}

Table looked something like this:
{code}
Column 1 |  column 2  | Column 3 | 
Column4
{"a": "b", "c": "d", "e": ["f": "g"]}[ some stuff here]abcde aaa
{code}

When I used single fields in select it worked but when I do select * it would 
not.
{code}
select * from default.table where column1.a='b' and column3='abcde' and 
Column4='aaa'
The above query failed.
{code}
{code}
select count(column4) from default.table where column1.a='b' and 
column3='abcde' and Column4='aaa'
The above query works.
{code}

  was:
I was querying a table with non-array columns and I came across situation where 
if I use select * it would throw me index out of bound exception.

{code}
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: -1
{code}

{code}
13/11/20 10:01:28 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
BeeswaxException(message:java.io.IOException: 
java.lang.ArrayIndexOutOfBoundsException: -1, log_context:key1, 
handle:QueryHandle(id:abc, log_context:def), SQLState: )
at 
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.fetch(BeeswaxServiceImpl.java:545)
at 
com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:986)
at 
com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:981)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at 
com.cloudera.beeswax.BeeswaxServiceImpl.doWithState(BeeswaxServiceImpl.java:772)
at 
com.cloudera.beeswax.BeeswaxServiceImpl.fetch(BeeswaxServiceImpl.java:980)
at 
com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:987)
at 
com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:971)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}

Table looked something like this:

Column 1 |  column 2  | Column 3 | 
Column4
{"a": "b", "c": "d", "e": ["f": "g"]}[ some stuff here]abcde aaa

When I used single fields in select it worked but when I do select * it would 
not.

select * from default.table where column1.a='b' and column3='abcde' and 
Column4='aaa'
The above query failed.

select count(column4) from default.table where column1.a='b' an

[jira] [Created] (HIVE-5860) Select * does not work when table has non-array columns.

[Anuj Ojha (JIRA)]

Anuj Ojha created HIVE-5860:
---

 Summary: Select * does not work when table has non-array columns.
 Key: HIVE-5860
 URL: https://issues.apache.org/jira/browse/HIVE-5860
 Project: Hive
  Issue Type: Bug
Reporter: Anuj Ojha


I was querying a table with non-array columns and I came across situation where 
if I use select * it would throw me index out of bound exception.

{code}
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: -1
{code}

{code}
13/11/20 10:01:28 ERROR beeswax.BeeswaxServiceImpl: Caught BeeswaxException
BeeswaxException(message:java.io.IOException: 
java.lang.ArrayIndexOutOfBoundsException: -1, log_context:key1, 
handle:QueryHandle(id:abc, log_context:def), SQLState: )
at 
com.cloudera.beeswax.BeeswaxServiceImpl$RunningQueryState.fetch(BeeswaxServiceImpl.java:545)
at 
com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:986)
at 
com.cloudera.beeswax.BeeswaxServiceImpl$5.run(BeeswaxServiceImpl.java:981)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at 
com.cloudera.beeswax.BeeswaxServiceImpl.doWithState(BeeswaxServiceImpl.java:772)
at 
com.cloudera.beeswax.BeeswaxServiceImpl.fetch(BeeswaxServiceImpl.java:980)
at 
com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:987)
at 
com.cloudera.beeswax.api.BeeswaxService$Processor$fetch.getResult(BeeswaxService.java:971)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}

Table looked something like this:

Column 1 |  column 2  | Column 3 | 
Column4
{"a": "b", "c": "d", "e": ["f": "g"]}[ some stuff here]abcde aaa

When I used single fields in select it worked but when I do select * it would 
not.

select * from default.table where column1.a='b' and column3='abcde' and 
Column4='aaa'
The above query failed.

select count(column4) from default.table where column1.a='b' and 
column3='abcde' and Column4='aaa'
The above query works.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5858) Alter table fails with exception when fs.hdfs.impl.disable.cache is set false

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827790#comment-13827790
 ] 

Hive QA commented on HIVE-5858:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614758/HIVE-5858.1.patch.txt

{color:green}SUCCESS:{color} +1 4665 tests passed

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/372/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/372/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614758

> Alter table fails with exception when fs.hdfs.impl.disable.cache is set false
> -
>
> Key: HIVE-5858
> URL: https://issues.apache.org/jira/browse/HIVE-5858
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5858.1.patch.txt
>
>
> http://www.mail-archive.com/user@hive.apache.org/msg10098.html
> {noformat}
> 2013-11-20 10:07:05,759 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(141)) - 
> InvalidOperationException(message:table new location 
> hdfs://localhost:9000/user/hive/warehouse/xa is on a different file system 
> than the old location hdfs://localhost:9000/user/hive/warehouse/zz. This 
> operation is not supported)
> at 
> org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:146)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:2352)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
> at com.sun.proxy.$Proxy9.alter_table_with_environment_context(Unknown 
> Source)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:216)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_table(HiveMetaStoreClient.java:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
> at com.sun.proxy.$Proxy10.alter_table(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.alterTable(Hive.java:385)
> at 
> org.apache.hadoop.hive.ql.exec.DDLTask.alterTable(DDLTask.java:3432)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:149)
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:87)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1256)
> at org.apache.hadoop.hive.ql.Driver.executePlan(Driver.java:1054)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:969)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:764)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5565) Limit Hive decimal type maximum precision and scale to 38

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-5565:
--

Attachment: HIVE-5565.5.patch

Patch #5 rebased with latest trunk.

> Limit Hive decimal type maximum precision and scale to 38
> -
>
> Key: HIVE-5565
> URL: https://issues.apache.org/jira/browse/HIVE-5565
> Project: Hive
>  Issue Type: Task
>  Components: Types
>Affects Versions: 0.13.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5565.1.patch, HIVE-5565.2.patch, HIVE-5565.3.patch, 
> HIVE-5565.4.patch, HIVE-5565.5.patch, HIVE-5565.patch
>
>
> With HIVE-3976, the maximum precision is set to 65, and maximum scale is to 
> 30. After discussing with several folks in the community, it's determined 
> that 38 as a maximum for both precision and scale are probably sufficient, in 
> addition to the potential performance boost that might become possible to 
> some implementation.
> This task is to make such a change. The change is expected to be trivial, but 
> it may impact many test cases. The reason for a separate JIRA is that patch 
> in HIVE-3976 is already in a good shape. Rather than destabilizing a bigger 
> patch, a dedicate patch will facilitates both reviews.
> The wiki document will be updated shortly.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5692) Make VectorGroupByOperator parameters configurable

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827735#comment-13827735
 ] 

Remus Rusanu commented on HIVE-5692:


the .5 patch failed in QA build 
https://builds.apache.org/job/PreCommit-HIVE-Build/1551/console during the 
subversion sync:
Caused by: svn: E204899: Cannot create new file 
'/home/hudson/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/common/src/test/org/apache/hadoop/hive/common/metrics/.svn/lock':
 No such file or directory





> Make VectorGroupByOperator parameters configurable
> --
>
> Key: HIVE-5692
> URL: https://issues.apache.org/jira/browse/HIVE-5692
> Project: Hive
>  Issue Type: Bug
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-5692.1.patch, HIVE-5692.2.patch, HIVE-5692.3.patch, 
> HIVE-5692.4.patch, HIVE-5692.5.patch, HIVE-5692.6.patch
>
>
> The FLUSH_CHECK_THRESHOLD and PERCENT_ENTRIES_TO_FLUSH should be configurable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5692) Make VectorGroupByOperator parameters configurable

[Remus Rusanu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-5692:
---

Attachment: HIVE-5692.6.patch

> Make VectorGroupByOperator parameters configurable
> --
>
> Key: HIVE-5692
> URL: https://issues.apache.org/jira/browse/HIVE-5692
> Project: Hive
>  Issue Type: Bug
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-5692.1.patch, HIVE-5692.2.patch, HIVE-5692.3.patch, 
> HIVE-5692.4.patch, HIVE-5692.5.patch, HIVE-5692.6.patch
>
>
> The FLUSH_CHECK_THRESHOLD and PERCENT_ENTRIES_TO_FLUSH should be configurable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5692) Make VectorGroupByOperator parameters configurable

[Remus Rusanu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-5692:
---

Status: Patch Available  (was: Open)

6th time is a charm

> Make VectorGroupByOperator parameters configurable
> --
>
> Key: HIVE-5692
> URL: https://issues.apache.org/jira/browse/HIVE-5692
> Project: Hive
>  Issue Type: Bug
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-5692.1.patch, HIVE-5692.2.patch, HIVE-5692.3.patch, 
> HIVE-5692.4.patch, HIVE-5692.5.patch, HIVE-5692.6.patch
>
>
> The FLUSH_CHECK_THRESHOLD and PERCENT_ENTRIES_TO_FLUSH should be configurable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5692) Make VectorGroupByOperator parameters configurable

[Remus Rusanu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-5692:
---

Status: Open  (was: Patch Available)

> Make VectorGroupByOperator parameters configurable
> --
>
> Key: HIVE-5692
> URL: https://issues.apache.org/jira/browse/HIVE-5692
> Project: Hive
>  Issue Type: Bug
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-5692.1.patch, HIVE-5692.2.patch, HIVE-5692.3.patch, 
> HIVE-5692.4.patch, HIVE-5692.5.patch
>
>
> The FLUSH_CHECK_THRESHOLD and PERCENT_ENTRIES_TO_FLUSH should be configurable.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (HIVE-5845) CTAS failed on vectorized code path

[Remus Rusanu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-5845:
---

Status: Patch Available  (was: Open)

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Attachments: HIVE-5845.1.patch
>
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5845) CTAS failed on vectorized code path

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827703#comment-13827703
 ] 

Remus Rusanu commented on HIVE-5845:


The uploaded fix addresses all the issues I found:

 - changes the OrcStruct serialization to use the correct object inspector (the 
one that writes into the OrcStruct fields)
 - the VectorExpressionWriterFactory was refactored to handle all 
assignments/writes and new object creation via the object inspectors, rather 
than assumming Writable types
 - a new API is available on the VectorExpressionWritter: setValue, which 
writes the value into the passed in object rather that returning a mutated the 
per-writer singleton. This was necessary for the OrcStruct vector 
serialization, which would end up reusing the same instance on all rows in the 
vector otherwise
 - changed the VectorExpressionWriter to use TypeInfo Category and 
PrimitiveCategory rather than type name string comparison
 - Have the VectorExpressionWriter generate writers from ObjectInspector and 
derive the OI from ExprNodeDesc rather than the other way around
 - extended the TestVectorExpressionWriter unit tests to cover the setValue API 
and struct fields assignment

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Attachments: HIVE-5845.1.patch
>
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5845) CTAS failed on vectorized code path

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827694#comment-13827694
 ] 

Remus Rusanu commented on HIVE-5845:


https://reviews.apache.org/r/15716/

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Attachments: HIVE-5845.1.patch
>
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Review Request 15716: HIVE-5845 CTAS failed on vectorized code path

[Remus Rusanu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15716/
---

Review request for hive, Ashutosh Chauhan and Eric Hanson.


Bugs: HIVE-5845
https://issues.apache.org/jira/browse/HIVE-5845


Repository: hive-git


Description
---

See HIVE-5845 for problem and fix discussion


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpressionWriter.java
 890cf4c 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorExpressionWriterFactory.java
 c5b026a 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java d765353 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcSerde.java 3c28efc 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorExpressionWriters.java
 1899c3b 

Diff: https://reviews.apache.org/r/15716/diff/


Testing
---

Repro, new unit tests


Thanks,

Remus Rusanu

[jira] [Updated] (HIVE-5845) CTAS failed on vectorized code path

[Remus Rusanu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Remus Rusanu updated HIVE-5845:
---

Attachment: HIVE-5845.1.patch

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
> Attachments: HIVE-5845.1.patch
>
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5771) Constant propagation optimizer for Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827671#comment-13827671
 ] 

Hive QA commented on HIVE-5771:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614749/HIVE-5771.4.patch

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/371/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/371/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash 
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and 
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-371/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/assembly/target shims/0.20S/target shims/0.23/target shims/common/target 
shims/common-secure/target packaging/target hbase-handler/target 
testutils/target jdbc/target metastore/target itests/target 
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/storage-handlers/hbase/target 
hcatalog/server-extensions/target hcatalog/core/target 
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target 
hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen 
service/target contrib/target serde/target beeline/target odbc/target 
cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1543817.

At revision 1543814.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614749

> Constant propagation optimizer for Hive
> ---
>
> Key: HIVE-5771
> URL: https://issues.apache.org/jira/browse/HIVE-5771
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Ted Xu
>Assignee: Ted Xu
> Attachments: HIVE-5771.1.patch, HIVE-5771.2.patch, HIVE-5771.3.patch, 
> HIVE-5771.4.patch, HIVE-5771.patch
>
>
> Currently there is no constant folding/propagation optimizer, all expressions 
> are evaluated at runtime. 
> HIVE-2470 did a great job on evaluating constants on UDF initializing phase, 
> however, it is still a runtime evaluation and it doesn't propagate constants 
> from a subquery to outside.
> It may reduce I/O and accelerate process if we introduce such an optimizer.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5663) Refactor ORC RecordReader to operate on direct & wrapped ByteBuffers

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827657#comment-13827657
 ] 

Hive QA commented on HIVE-5663:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614718/HIVE-5663.03.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4665 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/370/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/370/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12614718

> Refactor ORC RecordReader to operate on direct & wrapped ByteBuffers
> 
>
> Key: HIVE-5663
> URL: https://issues.apache.org/jira/browse/HIVE-5663
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Affects Versions: 0.13.0
> Environment: Ubuntu LXC 
>Reporter: Gopal V
>Assignee: Gopal V
>  Labels: ORC
> Attachments: HIVE-5663.01.patch, HIVE-5663.02.patch, 
> HIVE-5663.03.patch
>
>
> The current ORC RecordReader implementation assumes array structures backing 
> the ByteBuffers it passes around between RecordReaderImpl and 
> Compressed/Uncompressed InStream objects.
> This patch attempts to refactor those assumptions out of both classes, 
> allowing the future use of direct byte buffers within ORC (as might come from 
> HDFS zero-copy readers).



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5224) When creating table with AVRO serde, the "avro.schema.url" should be about to load serde schema from file system beside HDFS

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827525#comment-13827525
 ] 

Hive QA commented on HIVE-5224:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12614692/Hive-5224.3.patch

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/369/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/369/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash 
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and 
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-369/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/assembly/target shims/0.20S/target shims/0.23/target shims/common/target 
shims/common-secure/target packaging/target hbase-handler/target 
testutils/target jdbc/target metastore/target itests/target 
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/storage-handlers/hbase/target 
hcatalog/server-extensions/target hcatalog/core/target 
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target 
hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen 
service/target contrib/target serde/target beeline/target odbc/target 
cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Ucommon/src/test/org/apache/hadoop/hive/common/type/TestHiveDecimal.java
U
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java
U
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/AbstractPrimitiveObjectInspector.java
U
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantLongObjectInspector.java
U
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantShortObjectInspector.java
U
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantByteObjectInspector.java
U
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveDecimalObjectInspector.java
U
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantIntObjectInspector.java
Userde/src/java/org/apache/hadoop/hive/serde2/typeinfo/HiveDecimalUtils.java
Uql/src/test/results/clientnegative/invalid_arithmetic_type.q.out
Uql/src/test/results/clientnegative/udf_assert_true2.q.out
Uql/src/test/results/clientpositive/rcfile_createas1.q.out
Uql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out
Uql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out
Uql/src/test/results/clientpositive/udf_pmod.q.out
Uql/src/test/results/clientpositive/windowing_expressions.q.out
Uql/src/test/results/clientpositive/decimal_6.q.out
Uql/src/test/results/clientpositive/rcfile_merge1.q.out
Uql/src/test/results/clientpositive/input8.q.out
Uql/src/test/results/clientpositive/vectorization_5.q.out
Uql/src/test/results/clientpositive/udf_when.q.out
Uql/src/test/results/clientpositive/vectorized_math_funcs.q.out
Uql/src/test/results/clientpositive/ppd_constant_expr.q.out
Uql/src/test/results/clientpositive/num_op_type_conv.q.out
Uql/src/test/results/clientpositive/vectorization_15.q.out
Uql/src/test/results/clientpositive/auto_join13.q.out
Uql/src/test/results/clientpositive/vectorization_short_regress.q.out
Uql/src/test/results/clientpositive/orc_createas1.q.out
Uql/src/test/results/clientpositive/skewjoin.q.out
Uql/src/test/results/clientpositive/decimal_udf.q.out
Uql/src/test/results/clientpositive/rcfile_merge2.q.out
Uql/src/test/results/clientpositive/udf_case.q.out
Uql/src/test/results/clientpositive/auto_join2.q.out
Uql/src/test/results/compiler/plan/input8.q.xml
Uql/src/test/results/compiler/plan/i

[jira] [Updated] (HIVE-5581) Implement vectorized year/month/day... etc. for string arguments

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-5581:
-

Attachment: HIVE-5581.5.patch

I thought that the error is due to a system problem. But it shows 404 page now 
and I don't remember details. I will upload it again and will see its result. 
Thanks.

> Implement vectorized year/month/day... etc. for string arguments
> 
>
> Key: HIVE-5581
> URL: https://issues.apache.org/jira/browse/HIVE-5581
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Eric Hanson
>Assignee: Teddy Choi
> Attachments: HIVE-5581.1.patch.txt, HIVE-5581.2.patch, 
> HIVE-5581.3.patch, HIVE-5581.4.patch, HIVE-5581.5.patch, HIVE-5581.5.patch
>
>
> Functions year(), month(), day(), weekofyear(), hour(), minute(), second() 
> need to be implemented for string arguments in vectorized mode. 
> They already work for timestamp arguments.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5850) Multiple table join error for avro

[Shengjun Xin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827445#comment-13827445
 ] 

Shengjun Xin commented on HIVE-5850:


This issue is because of using wrong schema when process the split.

In getSchema function of AvroGenericRecordReader.java, if a partition is the 
prefix of a split, it will use schema of this partition to parse the split, but 
this is not always correct.

For example, partition '/user/hadoop/tpc-h/data/part' is the prefix of 
'/user/hadoop/tpc-h/data/partsupp/good_2013-01_partsupp_tbl_0002.avro', but we 
can not use the schema of '/user/hadoop/tpc-h/data/part' to parse 
'/user/hadoop/tpc-h/data/partsupp/good_2013-01_partsupp_tbl_0002.avro'

In my opinion, if a partition is the path parent of a split, we can use this 
partition's schema to parse the split. 

> Multiple table join error for avro 
> ---
>
> Key: HIVE-5850
> URL: https://issues.apache.org/jira/browse/HIVE-5850
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.11.0
>Reporter: Shengjun Xin
> Attachments: part.tar.gz, partsupp.tar.gz, schema.tar.gz
>
>
> Reproduce step:
> {code}
> -- Create table Part.
> CREATE EXTERNAL TABLE part
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS
> INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION 'hdfs:///user/hadoop/tpc-h/data/part'
> TBLPROPERTIES 
> ('avro.schema.url'='hdfs:///user/hadoop/tpc-h/schema/part.avsc');
> -- Create table Part Supplier.
> CREATE EXTERNAL TABLE partsupp
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS
> INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION 'hdfs:///user/hadoop/tpc-h/data/partsupp'
> TBLPROPERTIES 
> ('avro.schema.url'='hdfs:///user/hadoop/tpc-h/schema/partsupp.avsc');
> --- Query
> select * from partsupp ps join part p on ps.ps_partkey = p.p_partkey where 
> p.p_partkey=1;
> {code}
> {code}
> Error message is:
> Error: java.io.IOException: java.io.IOException: 
> org.apache.avro.AvroTypeException: Found {
>   "type" : "record",
>   "name" : "partsupp",
>   "namespace" : "com.gs.sdst.pl.avro.tpch",
>   "fields" : [ {
> "name" : "ps_partkey",
> "type" : "long"
>   }, {
> "name" : "ps_suppkey",
> "type" : "long"
>   }, {
> "name" : "ps_availqty",
> "type" : "long"
>   }, {
> "name" : "ps_supplycost",
> "type" : "double"
>   }, {
> "name" : "ps_comment",
> "type" : "string"
>   }, {
> "name" : "systimestamp",
> "type" : "long"
>   } ]
> }, expecting {
>   "type" : "record",
>   "name" : "part",
>   "namespace" : "com.gs.sdst.pl.avro.tpch",
>   "fields" : [ {
> "name" : "p_partkey",
> "type" : "long"
>   }, {
> "name" : "p_name",
> "type" : "string"
>   }, {
> "name" : "p_mfgr",
> "type" : "string"
>   }, {
> "name" : "p_brand",
> "type" : "string"
>   }, {
> "name" : "p_type",
> "type" : "string"
>   }, {
> "name" : "p_size",
> "type" : "int"
>   }, {
> "name" : "p_container",
> "type" : "string"
>   }, {
> "name" : "p_retailprice",
> "type" : "double"
>   }, {
> "name" : "p_comment",
> "type" : "string"
>   }, {
> "name" : "systimestamp",
> "type" : "long"
>   } ]
> }
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:302)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:218)
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:197)
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:183)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
> {code}



--
This message was sent by Atlassi

[jira] [Commented] (HIVE-4518) Counter Strike: Operation Operator

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827439#comment-13827439
 ] 

Jason Dere commented on HIVE-4518:
--

Do you mean we can just use fixed names rather than allowing configurable 
values for those counter names? You're probably right, no need for those names 
to be configurable, as long as we can configure the counter group for these 
counters.  Let me know if that is what you mean, I will make the change if so.

> Counter Strike: Operation Operator
> --
>
> Key: HIVE-4518
> URL: https://issues.apache.org/jira/browse/HIVE-4518
> Project: Hive
>  Issue Type: Improvement
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
> Attachments: HIVE-4518.1.patch, HIVE-4518.10.patch, 
> HIVE-4518.2.patch, HIVE-4518.3.patch, HIVE-4518.4.patch, HIVE-4518.5.patch, 
> HIVE-4518.6.patch.txt, HIVE-4518.7.patch, HIVE-4518.8.patch, HIVE-4518.9.patch
>
>
> Queries of the form:
> from foo
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> Generate a huge amount of counters. The reason is that task.progress is 
> turned on for dynamic partitioning queries.
> The counters not only make queries slower than necessary (up to 50%) you will 
> also eventually run out. That's because we're wrapping them in enum values to 
> comply with hadoop 0.17.
> The real reason we turn task.progress on is that we need CREATED_FILES and 
> FATAL counters to ensure dynamic partitioning queries don't go haywire.
> The counters have counter-intuitive names like C1 through C1000 and don't 
> seem really useful by themselves.
> With hadoop 20+ you don't need to wrap the counters anymore, each operator 
> can simply create and increment counters. That should simplify the code a lot.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5845) CTAS failed on vectorized code path

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827437#comment-13827437
 ] 

Remus Rusanu commented on HIVE-5845:


The root cause is not in OrcStruct/OrcSerde, but instead is in the 
VectorExpressionWriterFactory which does not honor properly the object 
inspectors passed in, assumes always to be a WritableXXObjectInspector and 
creates an WritableXXX object value. I am fixing this.

Why this was exposed is that OrcStruct.VectorExpressionWriterFactory creates 
writable object inspectors for most primitives, but for TIMESTAMP uses a native 
Java object inspector (also for DATE). I don't know why Orc does that, but none 
the less the VectorExpressionWriterFactory should handle this in a robust 
manner.

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (HIVE-5845) CTAS failed on vectorized code path

[Remus Rusanu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827438#comment-13827438
 ] 

Remus Rusanu commented on HIVE-5845:


 OrcStruct.createObjectInspector(TypeInfo info) that is (Eclipse copy/paste ...)

> CTAS failed on vectorized code path
> ---
>
> Key: HIVE-5845
> URL: https://issues.apache.org/jira/browse/HIVE-5845
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Remus Rusanu
>
> Following query fails:
>  create table store_sales_2 stored as orc as select * from alltypesorc;



--
This message was sent by Atlassian JIRA
(v6.1#6144)