[jira] [Commented] (HIVE-4975) Reading orc file throws exception after adding new column

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917817#comment-13917817
 ] 

Kevin Wilfong commented on HIVE-4975:
-

The goal of this is just feature parity with other file formats, e.g. RC file.  
AFAIK, no formats in Hive handle reordering of columns, or swapping the names 
of columns (I'm assuming that's what you're worried about with regards to 
changing the name of a column).

> Reading orc file throws exception after adding new column
> -
>
> Key: HIVE-4975
> URL: https://issues.apache.org/jira/browse/HIVE-4975
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.11.0
> Environment: hive 0.11.0 hadoop 1.0.0
>Reporter: cyril liao
>Assignee: Kevin Wilfong
>Priority: Critical
>  Labels: orcfile
> Fix For: 0.13.0
>
> Attachments: HIVE-4975.1.patch.txt
>
>
> ORC file read failure after add table column.
> create a table which have three column .(a string,b string,c string).
> add a new column after c by executing "ALTER TABLE table ADD COLUMNS (d 
> string)".
> execute hiveql "select d from table",the following exception goes:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row [Error getting row data with 
> exception java.lang.ArrayIndexOutOfBoundsException: 4
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcStructInspector.getStructFieldData(OrcStruct.java:206)
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:128)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:665)
>   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>   at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:396)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>   at org.apache.hadoop.mapred.Child.main(Child.java:249)
>  ]
>   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>   at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:396)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>   at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row [Error getting row data with exception 
> java.lang.ArrayIndexOutOfBoundsException: 4
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcStructInspector.getStructFieldData(OrcStruct.java:206)
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:128)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:665)
>   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>   at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:396)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>   at org.apache.hadoop.mapred.Child.main(Child.java:249)
>  ]
>   at 

[jira] [Comment Edited] (HIVE-4975) Reading orc file throws exception after adding new column

[Kevin Wilfong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917817#comment-13917817
 ] 

Kevin Wilfong edited comment on HIVE-4975 at 3/3/14 8:03 AM:
-

The goal of this patch is just feature parity with other file formats, e.g. RC 
file.  AFAIK, no formats in Hive handle reordering of columns, or swapping the 
names of columns (I'm assuming that's what you're worried about with regards to 
changing the name of a column).


was (Author: kevinwilfong):
The goal of this is just feature parity with other file formats, e.g. RC file.  
AFAIK, no formats in Hive handle reordering of columns, or swapping the names 
of columns (I'm assuming that's what you're worried about with regards to 
changing the name of a column).

> Reading orc file throws exception after adding new column
> -
>
> Key: HIVE-4975
> URL: https://issues.apache.org/jira/browse/HIVE-4975
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.11.0
> Environment: hive 0.11.0 hadoop 1.0.0
>Reporter: cyril liao
>Assignee: Kevin Wilfong
>Priority: Critical
>  Labels: orcfile
> Fix For: 0.13.0
>
> Attachments: HIVE-4975.1.patch.txt
>
>
> ORC file read failure after add table column.
> create a table which have three column .(a string,b string,c string).
> add a new column after c by executing "ALTER TABLE table ADD COLUMNS (d 
> string)".
> execute hiveql "select d from table",the following exception goes:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row [Error getting row data with 
> exception java.lang.ArrayIndexOutOfBoundsException: 4
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcStructInspector.getStructFieldData(OrcStruct.java:206)
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:128)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:665)
>   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>   at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:396)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>   at org.apache.hadoop.mapred.Child.main(Child.java:249)
>  ]
>   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>   at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:396)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>   at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row [Error getting row data with exception 
> java.lang.ArrayIndexOutOfBoundsException: 4
>   at 
> org.apache.hadoop.hive.ql.io.orc.OrcStruct$OrcStructInspector.getStructFieldData(OrcStruct.java:206)
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldData(UnionStructObjectInspector.java:128)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236)
>   at 
> org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222)
>   at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:665)
>   at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>   at org.apache.ha

[jira] [Commented] (HIVE-3938) Hive MetaStore should send a single AddPartitionEvent for atomically added partition-set.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917841#comment-13917841
 ] 

Hive QA commented on HIVE-3938:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632169/HIVE-3938.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5208 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1598/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1598/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12632169

> Hive MetaStore should send a single AddPartitionEvent for atomically added 
> partition-set.
> -
>
> Key: HIVE-3938
> URL: https://issues.apache.org/jira/browse/HIVE-3938
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-3938.patch, 
> Hive-3938-Support_for_Multi-table-insert.patch
>
>
> HiveMetaStore::add_partitions() currently adds all partitions specified in 
> one call using a single meta-store transaction. This acts correctly. However, 
> there's one AddPartitionEvent created per partition specified.
> Ideally, the set of partitions added atomically can be communicated using a 
> single AddPartitionEvent, such that they are consumed together.
> I'll post a patch that does this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5926) Load Data OverWrite Into Table Throw org.apache.hadoop.hive.ql.metadata.HiveException

[Yi Tian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Tian updated HIVE-5926:
--

   Resolution: Duplicate
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

> Load Data OverWrite Into Table Throw 
> org.apache.hadoop.hive.ql.metadata.HiveException
> -
>
> Key: HIVE-5926
> URL: https://issues.apache.org/jira/browse/HIVE-5926
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 0.12.0
> Environment: OS: Red Hat Enterprise Linux Server release 6.2
> HDFS: CDH-4.2.1
> MAPRED: CDH-4.2.1-mr1
>Reporter: Yi Tian
>Assignee: Yi Tian
> Fix For: 0.13.0
>
> Attachments: HIVE-5926.patch
>
>
> step1: create table 
> step2: load data 
> load data inpath '/tianyi/usys_etl_map_total.del' overwrite into table 
> tianyi_test3
> step3: copy file back
> hadoop fs -cp /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del /tianyi
> step4: load data again
> load data inpath '/tianyi/usys_etl_map_total.del' overwrite into table 
> tianyi_test3
> here we can see the error in console:
> Failed with exception Error moving: 
> hdfs://ocdccluster/tianyi/usys_etl_map_total.del into: 
> /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.MoveTask
> we can find error detail in hive.log:
> 2013-12-03 17:26:41,717 ERROR exec.Task (SessionState.java:printError(419)) - 
> Failed with exception Error moving: 
> hdfs://ocdccluster/tianyi/usys_etl_map_total.del into: 
> /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del
> org.apache.hadoop.hive.ql.metadata.HiveException: Error moving: 
> hdfs://ocdccluster/tianyi/usys_etl_map_total.del into: 
> /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del
>   at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2323)
>   at org.apache.hadoop.hive.ql.metadata.Table.replaceFiles(Table.java:639)
>   at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1441)
>   at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:283)
>   at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
>   at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
>   at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
>   at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
>   at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>   at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
>   at 
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
>   at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
>   at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.io.IOException: Error moving: 
> hdfs://ocdccluster/tianyi/usys_etl_map_total.del into: 
> /user/hive/warehouse/tianyi_test3/usys_etl_map_total.del
>   at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2317)
>   ... 20 more
> 2013-12-03 17:26:41,718 ERROR ql.Driver (SessionState.java:printError(419)) - 
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.MoveTask



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-6455:
-

Attachment: HIVE-6455.10.patch

To kick-off precommit tests earlier I reuploaded .9 version instead of .10 
version. Reuploading .10 version again to kick off another precommit test run.

> Scalable dynamic partitioning and bucketing optimization
> 
>
> Key: HIVE-6455
> URL: https://issues.apache.org/jira/browse/HIVE-6455
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: optimization
> Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch, 
> HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.2.patch, HIVE-6455.3.patch, 
> HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch, 
> HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch
>
>
> The current implementation of dynamic partition works by keeping at least one 
> record writer open per dynamic partition directory. In case of bucketing 
> there can be multispray file writers which further adds up to the number of 
> open record writers. The record writers of column oriented file format (like 
> ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or 
> compression buffers) open all the time to buffer up the rows and compress 
> them before flushing it to disk. Since these buffers are maintained per 
> column basis the amount of constant memory that will required at runtime 
> increases as the number of partitions and number of columns per partition 
> increases. This often leads to OutOfMemory (OOM) exception in mappers or 
> reducers depending on the number of open record writers. Users often tune the 
> JVM heapsize (runtime memory) to get over such OOM issues. 
> With this optimization, the dynamic partition columns and bucketing columns 
> (in case of bucketed tables) are sorted before being fed to the reducers. 
> Since the partitioning and bucketing columns are sorted, each reducers can 
> keep only one record writer open at any time thereby reducing the memory 
> pressure on the reducers. This optimization is highly scalable as the number 
> of partition and number of columns per partition increases at the cost of 
> sorting the columns.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6530) JDK 7 trunk build fails after HIVE-6418 patch

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917912#comment-13917912
 ] 

Hive QA commented on HIVE-6530:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632172/HIVE-6530.2.patch.txt

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5208 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1599/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1599/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12632172

> JDK 7 trunk build fails after HIVE-6418 patch
> -
>
> Key: HIVE-6530
> URL: https://issues.apache.org/jira/browse/HIVE-6530
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Navis
>Priority: Blocker
> Attachments: HIVE-6530.1.patch.txt, HIVE-6530.2.patch.txt
>
>
> JDK7 build fails with following error 
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) 
> on project hive-exec: Compilation failure
> [ERROR] 
> /home/prasadm/repos/apache/hive-trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/LazyFlatRowContainer.java:[118,15]
>  name clash: add(java.util.List) in 
> org.apache.hadoop.hive.ql.exec.persistence.LazyFlatRowContainer overrides a 
> method whose erasure is the same as another method, yet neither overrides the 
> other
> [ERROR] first method:  add(E) in java.util.AbstractCollection
> [ERROR] second method: add(ROW) in 
> org.apache.hadoop.hive.ql.exec.persistence.AbstractRowContainer
> [ERROR] -> [Help 1]
> [ERROR] 
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR] 
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR] 
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR]   mvn  -rf :hive-exec
> {noformat}
> This LazyFlatRowContainer.java is  a new file added as part of  HIVE-6418 
> patch. It's extending AbstractCollection and implements AbstractRowContainer. 
> Looks like the both these have a add() method that's conflicting.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6535) jdbc async wait should happen during fetch for results

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-6535:
---

 Summary: jdbc async wait should happen during fetch for results
 Key: HIVE-6535
 URL: https://issues.apache.org/jira/browse/HIVE-6535
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair


The hive jdbc client waits query completion during execute() call. It would be 
better to block in the jdbc for completion when the results are being fetched.
This way the application using hive jdbc driver can do other tasks while 
asynchronous query execution is happening, until it needs to fetch the result 
set.

 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6535) jdbc async wait should happen during fetch for results

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917939#comment-13917939
 ] 

Thejas M Nair commented on HIVE-6535:
-

See discussion in review of https://reviews.apache.org/r/14950/


> jdbc async wait should happen during fetch for results
> --
>
> Key: HIVE-6535
> URL: https://issues.apache.org/jira/browse/HIVE-6535
> Project: Hive
>  Issue Type: Bug
>Reporter: Thejas M Nair
>
> The hive jdbc client waits query completion during execute() call. It would 
> be better to block in the jdbc for completion when the results are being 
> fetched.
> This way the application using hive jdbc driver can do other tasks while 
> asynchronous query execution is happening, until it needs to fetch the result 
> set.
>  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5232) Make JDBC use the new HiveServer2 async execution API by default

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5232:


Resolution: Fixed
Status: Resolved  (was: Patch Available)

Created jira for moving the blocking from jdbc from execute to fetch-results - 
HIVE-6535
Patch committed to trunk.
Thanks for the contribution Vaibhav!


> Make JDBC use the new HiveServer2 async execution API by default
> 
>
> Key: HIVE-5232
> URL: https://issues.apache.org/jira/browse/HIVE-5232
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2
>Affects Versions: 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.13.0
>
> Attachments: HIVE-5232.1.patch, HIVE-5232.2.patch, HIVE-5232.3.patch
>
>
> HIVE-4617 provides support for async execution in HS2. There are some 
> proposed improvements in followup JIRAs:
> HIVE-5217
> HIVE-5229
> HIVE-5230
> HIVE-5441
> There is also [HIVE-5060] which assumes that execute to be asynchronous by 
> default.
>  
> Once they are in, we can think of using the async API as the default for 
> JDBC. This can enable the server to report back error sooner to the client. 
> It can also be useful in cases where a statement.cancel is done in a 
> different thread - the original thread will now be able to detect the cancel, 
> as opposed to the use of the blocking execute calls, in which 
> statement.cancel will be a no-op. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6440) sql std auth - add command to change owner of database

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917988#comment-13917988
 ] 

Hive QA commented on HIVE-6440:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12631983/HIVE-6440.3.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5190 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_input16_cc
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1601/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1601/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12631983

> sql std auth - add command to change owner of database
> --
>
> Key: HIVE-6440
> URL: https://issues.apache.org/jira/browse/HIVE-6440
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-6440.1.patch, HIVE-6440.2.patch, HIVE-6440.3.patch
>
>
> It should be possible to change the owner of a database once it is created.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-4598) Incorrect results when using subquery in multi table insert

[pandeeswaran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13917990#comment-13917990
 ] 

pandeeswaran commented on HIVE-4598:


Is anyone working on this issue?
I would like to fix this ,if no-one started already?

> Incorrect results when using subquery in multi table insert
> ---
>
> Key: HIVE-4598
> URL: https://issues.apache.org/jira/browse/HIVE-4598
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0, 0.11.0
>Reporter: Sebastian
>
> I'm using a multi table insert like this:
> FROM 
> INSERT INTO TABLE t PARTITION (type='x')
> SELECT * WHERE type='x'
> INSERT INTO TABLE t PARTITION (type='y')
> SELECT * WHERE type='y';
> Now when  is the name of a table, everything works as expected.
> However if I use a subquery as , the query runs but it inserts all results 
> from the subquery into each partition, as if there were no "WHERE" clauses in 
> the selects.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6440) sql std auth - add command to change owner of database

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-6440:


   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

The 2 test failures are unrelated. I verified that the tests pass locally.
Patch committed to trunk.
Thanks for the review Ashutosh!


> sql std auth - add command to change owner of database
> --
>
> Key: HIVE-6440
> URL: https://issues.apache.org/jira/browse/HIVE-6440
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Fix For: 0.13.0
>
> Attachments: HIVE-6440.1.patch, HIVE-6440.2.patch, HIVE-6440.3.patch
>
>
> It should be possible to change the owner of a database once it is created.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6455) Scalable dynamic partitioning and bucketing optimization

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918047#comment-13918047
 ] 

Hive QA commented on HIVE-6455:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632210/HIVE-6455.10.patch

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5209 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_union
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1603/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1603/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12632210

> Scalable dynamic partitioning and bucketing optimization
> 
>
> Key: HIVE-6455
> URL: https://issues.apache.org/jira/browse/HIVE-6455
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: optimization
> Attachments: HIVE-6455.1.patch, HIVE-6455.1.patch, 
> HIVE-6455.10.patch, HIVE-6455.10.patch, HIVE-6455.2.patch, HIVE-6455.3.patch, 
> HIVE-6455.4.patch, HIVE-6455.4.patch, HIVE-6455.5.patch, HIVE-6455.6.patch, 
> HIVE-6455.7.patch, HIVE-6455.8.patch, HIVE-6455.9.patch, HIVE-6455.9.patch
>
>
> The current implementation of dynamic partition works by keeping at least one 
> record writer open per dynamic partition directory. In case of bucketing 
> there can be multispray file writers which further adds up to the number of 
> open record writers. The record writers of column oriented file format (like 
> ORC, RCFile etc.) keeps some sort of in-memory buffers (value buffer or 
> compression buffers) open all the time to buffer up the rows and compress 
> them before flushing it to disk. Since these buffers are maintained per 
> column basis the amount of constant memory that will required at runtime 
> increases as the number of partitions and number of columns per partition 
> increases. This often leads to OutOfMemory (OOM) exception in mappers or 
> reducers depending on the number of open record writers. Users often tune the 
> JVM heapsize (runtime memory) to get over such OOM issues. 
> With this optimization, the dynamic partition columns and bucketing columns 
> (in case of bucketed tables) are sorted before being fed to the reducers. 
> Since the partitioning and bucketing columns are sorted, each reducers can 
> keep only one record writer open at any time thereby reducing the memory 
> pressure on the reducers. This optimization is highly scalable as the number 
> of partition and number of columns per partition increases at the cost of 
> sorting the columns.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions

[Andrey Stepachev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev updated HIVE-6427:
---

Attachment: 0001-HDPDEV-53-HiveServer-should-reconnect-automatically-.patch

Patch was updated.

> Hive Server2 should reopen Metastore client in case of any Thrift exceptions
> 
>
> Key: HIVE-6427
> URL: https://issues.apache.org/jira/browse/HIVE-6427
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
> Environment: cloudera cdh5 beta2
>Reporter: Andrey Stepachev
>Priority: Critical
> Attachments: 
> 0001-HDPDEV-53-HiveServer-should-reconnect-automatically-.patch, 
> 0001-HIVE-6427-Hive-Server2-should-reopen-Metastore-clien.patch
>
>
> In case of metastore restart hive server doesn't reopen connection to 
> metastore. Any command gives broken pipe or similar exceptions.
> http://paste.ubuntu.com/6926215/
> Any subsequent command doesn't reestablish connection and tries to use stale 
> (closed) connection.
> Looks like we shouldn't blindly convert any MetaException to 
> HiveSQLException, but should distinguish between fatal exceptions and logical 
> exceptions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions

[Andrey Stepachev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev updated HIVE-6427:
---

Attachment: (was: 
0001-HDPDEV-53-HiveServer-should-reconnect-automatically-.patch)

> Hive Server2 should reopen Metastore client in case of any Thrift exceptions
> 
>
> Key: HIVE-6427
> URL: https://issues.apache.org/jira/browse/HIVE-6427
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
> Environment: cloudera cdh5 beta2
>Reporter: Andrey Stepachev
>Priority: Critical
>
> In case of metastore restart hive server doesn't reopen connection to 
> metastore. Any command gives broken pipe or similar exceptions.
> http://paste.ubuntu.com/6926215/
> Any subsequent command doesn't reestablish connection and tries to use stale 
> (closed) connection.
> Looks like we shouldn't blindly convert any MetaException to 
> HiveSQLException, but should distinguish between fatal exceptions and logical 
> exceptions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions

[Andrey Stepachev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev updated HIVE-6427:
---

Attachment: (was: 
0001-HIVE-6427-Hive-Server2-should-reopen-Metastore-clien.patch)

> Hive Server2 should reopen Metastore client in case of any Thrift exceptions
> 
>
> Key: HIVE-6427
> URL: https://issues.apache.org/jira/browse/HIVE-6427
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
> Environment: cloudera cdh5 beta2
>Reporter: Andrey Stepachev
>Priority: Critical
>
> In case of metastore restart hive server doesn't reopen connection to 
> metastore. Any command gives broken pipe or similar exceptions.
> http://paste.ubuntu.com/6926215/
> Any subsequent command doesn't reestablish connection and tries to use stale 
> (closed) connection.
> Looks like we shouldn't blindly convert any MetaException to 
> HiveSQLException, but should distinguish between fatal exceptions and logical 
> exceptions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6427) Hive Server2 should reopen Metastore client in case of any Thrift exceptions

[Andrey Stepachev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev updated HIVE-6427:
---

Attachment: HIVE-6427-2.patch

> Hive Server2 should reopen Metastore client in case of any Thrift exceptions
> 
>
> Key: HIVE-6427
> URL: https://issues.apache.org/jira/browse/HIVE-6427
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.12.0
> Environment: cloudera cdh5 beta2
>Reporter: Andrey Stepachev
>Priority: Critical
> Attachments: HIVE-6427-2.patch
>
>
> In case of metastore restart hive server doesn't reopen connection to 
> metastore. Any command gives broken pipe or similar exceptions.
> http://paste.ubuntu.com/6926215/
> Any subsequent command doesn't reestablish connection and tries to use stale 
> (closed) connection.
> Looks like we shouldn't blindly convert any MetaException to 
> HiveSQLException, but should distinguish between fatal exceptions and logical 
> exceptions.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-3938) Hive MetaStore should send a single AddPartitionEvent for atomically added partition-set.

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918140#comment-13918140
 ] 

Mithun Radhakrishnan commented on HIVE-3938:


With the updated patch, TestNegativeMinimrCliDriver has a failure.

This looks unrelated, but would someone please confirm?

> Hive MetaStore should send a single AddPartitionEvent for atomically added 
> partition-set.
> -
>
> Key: HIVE-3938
> URL: https://issues.apache.org/jira/browse/HIVE-3938
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-3938.patch, 
> Hive-3938-Support_for_Multi-table-insert.patch
>
>
> HiveMetaStore::add_partitions() currently adds all partitions specified in 
> one call using a single meta-store transaction. This acts correctly. However, 
> there's one AddPartitionEvent created per partition specified.
> Ideally, the set of partitions added atomically can be communicated using a 
> single AddPartitionEvent, such that they are consumed together.
> I'll post a patch that does this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-6389:
---

Status: Open  (was: Patch Available)

Pulling patch. Will capitalize and resubmit.

> LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in 
> null-maps.
> 
>
> Key: HIVE-6389
> URL: https://issues.apache.org/jira/browse/HIVE-6389
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.12.0, 0.11.0, 0.10.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>
> RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle 
> look-ups into map-columns when the value of the column is null.
> When an RCFile table is created with LazyBinaryColumnarSerDe (as is default 
> in 0.12), and queried as follows:
> {code}
> select mymap['1024'] from mytable;
> {code}
> and if the mymap column has nulls, then one is treated to the following 
> guttural utterance:
> {code}
> 2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"id":null,"mymap":null,"isnull":null}
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
>   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
> org.apache.hadoop.io.Text
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
>   ... 10 more
> {code}
> A patch is on the way, but the short of it is that the LazyBinaryMapOI needs 
> to return nulls if either the map or the lookup-key is null.
> This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918146#comment-13918146
 ] 

Hive QA commented on HIVE-5504:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12631095/HIVE-5504.2.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5218 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1604/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1604/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12631095

> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.2.patch, HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-6389:
---

Attachment: (was: Hive-6389.patch)

> LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in 
> null-maps.
> 
>
> Key: HIVE-6389
> URL: https://issues.apache.org/jira/browse/HIVE-6389
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
>
> RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle 
> look-ups into map-columns when the value of the column is null.
> When an RCFile table is created with LazyBinaryColumnarSerDe (as is default 
> in 0.12), and queried as follows:
> {code}
> select mymap['1024'] from mytable;
> {code}
> and if the mymap column has nulls, then one is treated to the following 
> guttural utterance:
> {code}
> 2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"id":null,"mymap":null,"isnull":null}
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
>   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
> org.apache.hadoop.io.Text
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
>   ... 10 more
> {code}
> A patch is on the way, but the short of it is that the LazyBinaryMapOI needs 
> to return nulls if either the map or the lookup-key is null.
> This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-6389:
---

Attachment: HIVE-6389.patch

Renamed.

> LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in 
> null-maps.
> 
>
> Key: HIVE-6389
> URL: https://issues.apache.org/jira/browse/HIVE-6389
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6389.patch
>
>
> RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle 
> look-ups into map-columns when the value of the column is null.
> When an RCFile table is created with LazyBinaryColumnarSerDe (as is default 
> in 0.12), and queried as follows:
> {code}
> select mymap['1024'] from mytable;
> {code}
> and if the mymap column has nulls, then one is treated to the following 
> guttural utterance:
> {code}
> 2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"id":null,"mymap":null,"isnull":null}
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
>   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
> org.apache.hadoop.io.Text
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
>   ... 10 more
> {code}
> A patch is on the way, but the short of it is that the LazyBinaryMapOI needs 
> to return nulls if either the map or the lookup-key is null.
> This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-6389:
---

Status: Patch Available  (was: Open)

> LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in 
> null-maps.
> 
>
> Key: HIVE-6389
> URL: https://issues.apache.org/jira/browse/HIVE-6389
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.12.0, 0.11.0, 0.10.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6389.patch
>
>
> RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle 
> look-ups into map-columns when the value of the column is null.
> When an RCFile table is created with LazyBinaryColumnarSerDe (as is default 
> in 0.12), and queried as follows:
> {code}
> select mymap['1024'] from mytable;
> {code}
> and if the mymap column has nulls, then one is treated to the following 
> guttural utterance:
> {code}
> 2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"id":null,"mymap":null,"isnull":null}
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
>   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
> org.apache.hadoop.io.Text
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
>   ... 10 more
> {code}
> A patch is on the way, but the short of it is that the LazyBinaryMapOI needs 
> to return nulls if either the map or the lookup-key is null.
> This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-4598) Incorrect results when using subquery in multi table insert

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918168#comment-13918168
 ] 

Ashutosh Chauhan commented on HIVE-4598:


Go ahead [~pandeesh] Looks like no one is working on it.

> Incorrect results when using subquery in multi table insert
> ---
>
> Key: HIVE-4598
> URL: https://issues.apache.org/jira/browse/HIVE-4598
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.10.0, 0.11.0
>Reporter: Sebastian
>
> I'm using a multi table insert like this:
> FROM 
> INSERT INTO TABLE t PARTITION (type='x')
> SELECT * WHERE type='x'
> INSERT INTO TABLE t PARTITION (type='y')
> SELECT * WHERE type='y';
> Now when  is the name of a table, everything works as expected.
> However if I use a subquery as , the query runs but it inserts all results 
> from the subquery into each partition, as if there were no "WHERE" clauses in 
> the selects.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-3938) Hive MetaStore should send a single AddPartitionEvent for atomically added partition-set.

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-3938:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Ya, that test case is flaky. Committed to trunk. Thanks, Mithun!

> Hive MetaStore should send a single AddPartitionEvent for atomically added 
> partition-set.
> -
>
> Key: HIVE-3938
> URL: https://issues.apache.org/jira/browse/HIVE-3938
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 0.13.0
>
> Attachments: HIVE-3938.patch, 
> Hive-3938-Support_for_Multi-table-insert.patch
>
>
> HiveMetaStore::add_partitions() currently adds all partitions specified in 
> one call using a single meta-store transaction. This acts correctly. However, 
> there's one AddPartitionEvent created per partition specified.
> Ideally, the set of partitions added atomically can be communicated using a 
> single AddPartitionEvent, such that they are consumed together.
> I'll post a patch that does this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6505) Make stats optimizer more robust in presence of distinct clause

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6505:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Reran the tests. They passed locally. Committed to trunk.

> Make stats optimizer more robust in presence of distinct clause
> ---
>
> Key: HIVE-6505
> URL: https://issues.apache.org/jira/browse/HIVE-6505
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 0.13.0
>
> Attachments: HIVE-6505.1.patch, HIVE-6505.patch
>
>
> Currently it throws exceptions in few cases.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6536) Reduce dependencies of org.apache.hive:hive-jdbc maven module

[Kevin Minder (JIRA)]

Kevin Minder created HIVE-6536:
--

 Summary: Reduce dependencies of org.apache.hive:hive-jdbc maven 
module
 Key: HIVE-6536
 URL: https://issues.apache.org/jira/browse/HIVE-6536
 Project: Hive
  Issue Type: Improvement
  Components: JDBC
Affects Versions: 0.12.0
 Environment: org.apache.hive:hive-jdbc:jar:0.12.0
Reporter: Kevin Minder


The Hive JDBC driver maven module requires a significant number of dependencies 
that are likely unnecessary and will result in bloating of consumers.  Most of 
this is a result of the dependency on org.apache.hive:hive-cli.  I have 
attached a portion of the output from mvn depedency:tree output for a client 
that depends on the org.apache.hive:hive-jdbc module.  Note the extra 
2.0.6.1-102 in the output is the result of our local build and publish to a 
local nexus repo.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6536) Reduce dependencies of org.apache.hive:hive-jdbc maven module

[Kevin Minder (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Minder updated HIVE-6536:
---

Attachment: hive-jdbc-maven-dependencies.log

> Reduce dependencies of org.apache.hive:hive-jdbc maven module
> -
>
> Key: HIVE-6536
> URL: https://issues.apache.org/jira/browse/HIVE-6536
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Affects Versions: 0.12.0
> Environment: org.apache.hive:hive-jdbc:jar:0.12.0
>Reporter: Kevin Minder
> Attachments: hive-jdbc-maven-dependencies.log
>
>
> The Hive JDBC driver maven module requires a significant number of 
> dependencies that are likely unnecessary and will result in bloating of 
> consumers.  Most of this is a result of the dependency on 
> org.apache.hive:hive-cli.  I have attached a portion of the output from mvn 
> depedency:tree output for a client that depends on the 
> org.apache.hive:hive-jdbc module.  Note the extra 2.0.6.1-102 in the output 
> is the result of our local build and publish to a local nexus repo.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918277#comment-13918277
 ] 

Hive QA commented on HIVE-6389:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632273/HIVE-6389.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5218 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketizedhiveinputformat
org.apache.hive.hcatalog.mapreduce.TestHCatMutablePartitioned.testHCatPartitionedTable
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1605/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1605/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12632273

> LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in 
> null-maps.
> 
>
> Key: HIVE-6389
> URL: https://issues.apache.org/jira/browse/HIVE-6389
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6389.patch
>
>
> RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle 
> look-ups into map-columns when the value of the column is null.
> When an RCFile table is created with LazyBinaryColumnarSerDe (as is default 
> in 0.12), and queried as follows:
> {code}
> select mymap['1024'] from mytable;
> {code}
> and if the mymap column has nulls, then one is treated to the following 
> guttural utterance:
> {code}
> 2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"id":null,"mymap":null,"isnull":null}
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
>   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
> org.apache.hadoop.io.Text
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
>   ... 10 more
> {code}
> A patch is on the way, but the short of it is that the LazyBinaryMapOI needs 
> to return nulls if either the map or the lookup-key is null.
> This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6511) casting from decimal to tinyint,smallint, int and bigint generates different result when vectorization is on

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918290#comment-13918290
 ] 

Eric Hanson commented on HIVE-6511:
---

Can you put this up on ReviewBoard?

> casting from decimal to tinyint,smallint, int and bigint generates different 
> result when vectorization is on
> 
>
> Key: HIVE-6511
> URL: https://issues.apache.org/jira/browse/HIVE-6511
> Project: Hive
>  Issue Type: Bug
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-6511.1.patch
>
>
> select dc,cast(dc as int), cast(dc as smallint),cast(dc as tinyint) from 
> vectortab10korc limit 20 generates following result when vectorization is 
> enabled:
> {code}
> 4619756289662.078125  -1628520834 -16770  126
> 1553532646710.316406  -1245514442 -2762   54
> 3367942487288.360352  688127224   -776-8
> 4386447830839.337891  1286221623  12087   55
> -3234165331139.458008 -54957251   27453   61
> -488378613475.326172  1247658269  -16099  29
> -493942492598.691406  -21253559   -19895  73
> 3101852523586.039062  886135874   23618   66
> 2544105595941.381836  1484956709  -23515  37
> -3997512403067.0625   1102149509  30597   -123
> -1183754978977.589355 1655994718  31070   94
> 1408783849655.676758  34576568-26440  -72
> -2993175106993.426758 417098319   27215   79
> 3004723551798.100586  -1753555402 -8650   54
> 1103792083527.786133  -14511544   -28088  72
> 469767055288.485352   1615620024  26552   -72
> -1263700791098.294434 -980406074  12486   -58
> -4244889766496.484375 -1462078048 30112   -96
> -3962729491139.782715 1525323068  -27332  60
> NULL  NULLNULLNULL
> {code}
> When vectorization is disabled, result looks like this:
> {code}
> 4619756289662.078125  -1628520834 -16770  126
> 1553532646710.316406  -1245514442 -2762   54
> 3367942487288.360352  688127224   -776-8
> 4386447830839.337891  1286221623  12087   55
> -3234165331139.458008 -54957251   27453   61
> -488378613475.326172  1247658269  -16099  29
> -493942492598.691406  -21253558   -19894  74
> 3101852523586.039062  886135874   23618   66
> 2544105595941.381836  1484956709  -23515  37
> -3997512403067.0625   1102149509  30597   -123
> -1183754978977.589355 1655994719  31071   95
> 1408783849655.676758  34576567-26441  -73
> -2993175106993.426758 417098319   27215   79
> 3004723551798.100586  -1753555402 -8650   54
> 1103792083527.786133  -14511545   -28089  71
> 469767055288.485352   1615620024  26552   -72
> -1263700791098.294434 -980406074  12486   -58
> -4244889766496.484375 -1462078048 30112   -96
> -3962729491139.782715 1525323069  -27331  61
> NULL  NULLNULLNULL
> {code}
> This issue is visible only for certain decimal values. In above example, row 
> 7,11,12, and 15 generates different results.
> vectortab10korc table schema:
> {code}
> t tinyint from deserializer   
> sismallintfrom deserializer   
> i int from deserializer   
> b bigint  from deserializer   
> f float   from deserializer   
> d double  from deserializer   
> dcdecimal(38,18)  from deserializer   
> boboolean from deserializer   
> s string  from deserializer   
> s2string  from deserializer   
> tstimestamp   from deserializer   
>
> # Detailed Table Information   
> Database: default  
> Owner:xyz  
> CreateTime:   Tue Feb 25 21:54:28 UTC 2014 
> LastAccessTime:   UNKNOWN  
> Protect Mode: None 
> Retention:0
> Location: 
> hdfs://host1.domain.com:8020/apps/hive/warehouse/vectortab10korc 
> Table Type:   MANAGED_TABLE
> Table Parameters:  
>   COLUMN_STATS_ACCURATE   true
>   numFiles1   
>   numRows 1   
>   rawDataSize 0   
>   totalSize   344748  
>   transient_lastDdlTime   1393365281  
>
> # Storage Information  
> SerDe Library:org.apache.hadoop.hive.ql.io.orc.OrcSerde
> In

[jira] [Commented] (HIVE-5950) ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918320#comment-13918320
 ] 

Gunther Hagleitner commented on HIVE-5950:
--

comments on rb

> ORC SARG creation fails with NPE for predicate conditions with 
> decimal/date/char/varchar datatypes
> --
>
> Key: HIVE-5950
> URL: https://issues.apache.org/jira/browse/HIVE-5950
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5950.1.patch, HIVE-5950.2.patch, HIVE-5950.3.patch, 
> HIVE-5950.4.patch
>
>
> When decimal or date column is used, the type field in PredicateLeafImpl will 
> be set to null. This will result in NPE during predicate leaf generation 
> because of null dereferencing in hashcode computation. SARG creation should 
> be extended to support/handle decimal and date data types.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 13845: HIVE-5155: Support secure proxy user access to HiveServer2

[Prasad Mujumdar]

> On Jan. 8, 2014, 9:43 p.m., Thejas Nair wrote:
> > beeline/src/java/org/apache/hive/beeline/BeeLine.java, line 547
> > 
> >
> > we should document what this option means, in the usage output, and 
> > that it is a hive specific option.
> >

Agreed. will updated the docs according.


> On Jan. 8, 2014, 9:43 p.m., Thejas Nair wrote:
> > conf/hive-default.xml.template, line 2111
> > 
> >
> > should "altername" be "alternate" ?
> > requestion => request
> >

Done.


> On Jan. 8, 2014, 9:43 p.m., Thejas Nair wrote:
> > shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java,
> >  line 528
> > 
> >
> > This would mean that to make a user a proxy user, you would need to 
> > make the user a proxy user for all of hadoop. In general for security, it 
> > is useful to be able to give users only what they need.
> > 
> > Webhcat and oozie follow this model AFAIK. Granting a user proxy user 
> > privilege for these services does not require you to make the user a proxy 
> > user for hadoop (HDFS, MR).
> >

I do agree with the point that we shouldn't be requiring to grant permissions 
beyond the minimum required. Here's the rationale for the proposed approach -
- For impersonation cases, the middleware user needs to impersonate the end 
user at Hadoop level (eg Oozie). If we use a different configuration format, 
then you need to keep those two setting in sycn. That's an administration 
nightmare.
- If you do want this to be a hive specific setting (eg. for middleware tools 
that don't need impersonation), then you can always add it to hive-site.xml. 
This way you don't need a different configuration format or file, and yet keep 
the privilege specific to hive service.


- Prasad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13845/#review31384
---


On Dec. 5, 2013, 8:08 p.m., Prasad Mujumdar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13845/
> ---
> 
> (Updated Dec. 5, 2013, 8:08 p.m.)
> 
> 
> Review request for hive, Brock Noland, Carl Steinbach, and Thejas Nair.
> 
> 
> Bugs: HIVE-5155
> https://issues.apache.org/jira/browse/HIVE-5155
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Delegation token support -
> Enable delegation token connection for HiveServer2
> Enhance the TCLIService interface to support delegation token requests
> Support passing the delegation token connection type via JDBC URL and Beeline 
> option
> 
> Direct proxy access -
> Define new proxy user property
> Shim interfaces to validate proxy access for a given user
> 
> Note that the diff doesn't include thrift generated code.
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java c5e36a5 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java c3abba3 
>   beeline/src/java/org/apache/hive/beeline/Commands.java d2d7fd3 
>   beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java 1de5829 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36503fa 
>   conf/hive-default.xml.template c61a0bb 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> 7b1c9da 
>   jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java d08e05b 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java ef39573 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 4d75d98 
>   service/if/TCLIService.thrift 62a9730 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java d80649f 
>   service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java 
> 519556c 
>   service/src/java/org/apache/hive/service/auth/PlainSaslHelper.java 15b1675 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 8c85386 
>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
> 9dca874 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 00058cc 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> cfda752 
>   
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
>  708f4e4 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> e262b72 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 9df110e 
>   
> service/src/java/org/apache/hive/service

[jira] [Updated] (HIVE-5933) SQL std auth - add support to metastore api to list all privileges for a user

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5933:


Attachment: HIVE-5933.thriftapi.patch

HIVE-5933.thriftapi.patch - thrift api that I plan to introduce.

I also think it will be cleaner to use an api that returns RoleGrant for the 
show-role-grant instead of adding the RoleGrant information to Role (which 
becomes confusing in the context of create-role). I am thinking of making that 
change in a separate follow up patch for 0.13 . I am hoping we can include that 
during the stabilization phase of 0.13 (ie, post branching).

cc [~navis]


> SQL std auth - add support to metastore api to list all privileges for a user
> -
>
> Key: HIVE-5933
> URL: https://issues.apache.org/jira/browse/HIVE-5933
> Project: Hive
>  Issue Type: Sub-task
>  Components: Authorization
>Reporter: Thejas M Nair
> Attachments: HIVE-5933.thriftapi.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> This is for supporting SHOW GRANTS statements -
> SHOW GRANTS;
> SHOW GRANTS FOR user;
> SHOW GRANTS FOR role;



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6389) LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in null-maps.

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6389:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Mithun!

> LazyBinaryColumnarSerDe-based RCFile tables break when looking up elements in 
> null-maps.
> 
>
> Key: HIVE-6389
> URL: https://issues.apache.org/jira/browse/HIVE-6389
> Project: Hive
>  Issue Type: Bug
>  Components: Serializers/Deserializers
>Affects Versions: 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 0.13.0
>
> Attachments: HIVE-6389.patch
>
>
> RCFile tables that use the LazyBinaryColumnarSerDe don't seem to handle 
> look-ups into map-columns when the value of the column is null.
> When an RCFile table is created with LazyBinaryColumnarSerDe (as is default 
> in 0.12), and queried as follows:
> {code}
> select mymap['1024'] from mytable;
> {code}
> and if the mymap column has nulls, then one is treated to the following 
> guttural utterance:
> {code}
> 2014-02-05 21:50:25,050 FATAL mr.ExecMapper (ExecMapper.java:map(194)) - 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row {"id":null,"mymap":null,"isnull":null}
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
>   at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>   at 
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
>   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
> org.apache.hadoop.io.Text
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveWritableObject(WritableStringObjectInspector.java:41)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:226)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:486)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:439)
>   at 
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:423)
>   at 
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:560)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:87)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:790)
>   at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
>   ... 10 more
> {code}
> A patch is on the way, but the short of it is that the LazyBinaryMapOI needs 
> to return nulls if either the map or the lookup-key is null.
> This is handled correctly for Text data, and for RCFiles using ColumnarSerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 13845: HIVE-5155: Support secure proxy user access to HiveServer2

[Prasad Mujumdar]

> On Feb. 23, 2014, 9:47 a.m., Lefty Leverenz wrote:
> > conf/hive-default.xml.template, line 2111
> > 
> >
> > Agreed on both points.  But as typos go, "requestion" is such a lovely 
> > word that it ought to be added to the English language.  ;)
> 
> Lefty Leverenz wrote:
> Oops, thought this would appear with Thejas's comment (the points are 
> "altername" -> "alternate" and "requestion" -> "request").

:) My linguistic creativity is inversely proportional to my blood caffeine 
level ...
sorry about the typos.


- Prasad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13845/#review35241
---


On Dec. 5, 2013, 8:08 p.m., Prasad Mujumdar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13845/
> ---
> 
> (Updated Dec. 5, 2013, 8:08 p.m.)
> 
> 
> Review request for hive, Brock Noland, Carl Steinbach, and Thejas Nair.
> 
> 
> Bugs: HIVE-5155
> https://issues.apache.org/jira/browse/HIVE-5155
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Delegation token support -
> Enable delegation token connection for HiveServer2
> Enhance the TCLIService interface to support delegation token requests
> Support passing the delegation token connection type via JDBC URL and Beeline 
> option
> 
> Direct proxy access -
> Define new proxy user property
> Shim interfaces to validate proxy access for a given user
> 
> Note that the diff doesn't include thrift generated code.
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java c5e36a5 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java c3abba3 
>   beeline/src/java/org/apache/hive/beeline/Commands.java d2d7fd3 
>   beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java 1de5829 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36503fa 
>   conf/hive-default.xml.template c61a0bb 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> 7b1c9da 
>   jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java d08e05b 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java ef39573 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 4d75d98 
>   service/if/TCLIService.thrift 62a9730 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java d80649f 
>   service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java 
> 519556c 
>   service/src/java/org/apache/hive/service/auth/PlainSaslHelper.java 15b1675 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 8c85386 
>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
> 9dca874 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 00058cc 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> cfda752 
>   
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
>  708f4e4 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> e262b72 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 9df110e 
>   
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
>  9bb2a0f 
>   service/src/test/org/apache/hive/service/auth/TestPlainSaslHelper.java 
> 8fa4afd 
>   service/src/test/org/apache/hive/service/cli/session/TestSessionHooks.java 
> 2fac800 
>   shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 
> 6ff1a84 
>   
> shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
>  84f3ddc 
>   
> shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java
>  dc89de1 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> 0d5615c 
>   
> shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java
>  03f4e51 
> 
> Diff: https://reviews.apache.org/r/13845/diff/
> 
> 
> Testing
> ---
> 
> Since this requires kerberos setup, its tested by a standalone test program 
> that runs various existing and new secure connection scenarios. The test code 
> is attached to the ticket at 
> https://issues.apache.org/jira/secure/attachment/12600119/ProxyAuth.java
> 
> 
> Thanks,
> 
> Prasad Mujumdar
> 
>

[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918341#comment-13918341
 ] 

Prasad Mujumdar commented on HIVE-5155:
---

[~thejas] I responded to your comments on the review board. Please let me your 
feedback.

[~vaibhavgumashta] Thanks for the offer. I am working on rebase and testing the 
updated patch.

> Support secure proxy user access to HiveServer2
> ---
>
> Key: HIVE-5155
> URL: https://issues.apache.org/jira/browse/HIVE-5155
> Project: Hive
>  Issue Type: Improvement
>  Components: Authentication, HiveServer2, JDBC
>Affects Versions: 0.12.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, 
> HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, 
> HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, 
> HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java
>
>
> The HiveServer2 can authenticate a client using via Kerberos and impersonate 
> the connecting user with underlying secure hadoop. This becomes a gateway for 
> a remote client to access secure hadoop cluster. Now this works fine for when 
> the client obtains Kerberos ticket and directly connects to HiveServer2. 
> There's another big use case for middleware tools where the end user wants to 
> access Hive via another server. For example Oozie action or Hue submitting 
> queries or a BI tool server accessing to HiveServer2. In these cases, the 
> third party server doesn't have end user's Kerberos credentials and hence it 
> can't submit queries to HiveServer2 on behalf of the end user.
> This ticket is for enabling proxy access to HiveServer2 for third party tools 
> on behalf of end users. There are two parts of the solution proposed in this 
> ticket:
> 1) Delegation token based connection for Oozie (OOZIE-1457)
> This is the common mechanism for Hadoop ecosystem components. Hive Remote 
> Metastore and HCatalog already support this. This is suitable for tool like 
> Oozie that submits the MR jobs as actions on behalf of its client. Oozie 
> already uses similar mechanism for Metastore/HCatalog access.
> 2) Direct proxy access for privileged hadoop users
> The delegation token implementation can be a challenge for non-hadoop 
> (especially non-java) components. This second part enables a privileged user 
> to directly specify an alternate session user during the connection. If the 
> connecting user has hadoop level privilege to impersonate the requested 
> userid, then HiveServer2 will run the session as that requested user. For 
> example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy 
> user configuration). Then user Hue can connect to HiveServer2 and specify Bob 
> as session user via a session property. HiveServer2 will verify Hue's proxy 
> user privilege and then impersonate user Bob instead of Hue. This will enable 
> any third party tool to impersonate alternate userid without having to 
> implement delegation token connection.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-6499:
---

Status: Patch Available  (was: Open)

> Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
> --
>
> Key: HIVE-6499
> URL: https://issues.apache.org/jira/browse/HIVE-6499
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-6499.patch
>
>
> In cases where a user needs to use a custom IF/OF/SerDe that is not 
> accessible from the metastore, calls like msc.createTable and msc.dropTable 
> should still work without being able to load the class. This is possible as 
> long as one does not enable MetaStore-side authorization, at which point this 
> becomes impossible, erroring out with a ClassNotFoundException.
> The reason this happens is that since the AuthorizationProvider interface is 
> defined against a ql.metadata.Table, we wind up needing to instantiate a 
> ql.metadata.Table object, which, in its constructor tries to instantiate 
> IF/OF/SerDe elements in an attempt to pre-load those fields. And if we do not 
> have access to those classes in the metastore, this is when that fails. The 
> constructor/initialize methods of Table and Partition do not really need to 
> pre-initialize these fields, since the fields are accessed only through the 
> accessor, and will be instantiated on first-use.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6499) Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-6499:
---

Status: Open  (was: Patch Available)

> Using Metastore-side Auth errors on non-resolvable IF/OF/SerDe
> --
>
> Key: HIVE-6499
> URL: https://issues.apache.org/jira/browse/HIVE-6499
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-6499.patch
>
>
> In cases where a user needs to use a custom IF/OF/SerDe that is not 
> accessible from the metastore, calls like msc.createTable and msc.dropTable 
> should still work without being able to load the class. This is possible as 
> long as one does not enable MetaStore-side authorization, at which point this 
> becomes impossible, erroring out with a ClassNotFoundException.
> The reason this happens is that since the AuthorizationProvider interface is 
> defined against a ql.metadata.Table, we wind up needing to instantiate a 
> ql.metadata.Table object, which, in its constructor tries to instantiate 
> IF/OF/SerDe elements in an attempt to pre-load those fields. And if we do not 
> have access to those classes in the metastore, this is when that fails. The 
> constructor/initialize methods of Table and Partition do not really need to 
> pre-initialize these fields, since the fields are accessed only through the 
> accessor, and will be instantiated on first-use.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5504:
---

Status: Patch Available  (was: Open)

> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.11.0, 0.13.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.2.patch, HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5504:
---

Status: Open  (was: Patch Available)

> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.11.0, 0.13.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.2.patch, HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6530) JDK 7 trunk build fails after HIVE-6418 patch

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918389#comment-13918389
 ] 

Prasad Mujumdar commented on HIVE-6530:
---

+1

[~navis] Thanks for taking care of it! I guess test failures are unrelated ..

> JDK 7 trunk build fails after HIVE-6418 patch
> -
>
> Key: HIVE-6530
> URL: https://issues.apache.org/jira/browse/HIVE-6530
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Navis
>Priority: Blocker
> Attachments: HIVE-6530.1.patch.txt, HIVE-6530.2.patch.txt
>
>
> JDK7 build fails with following error 
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) 
> on project hive-exec: Compilation failure
> [ERROR] 
> /home/prasadm/repos/apache/hive-trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/LazyFlatRowContainer.java:[118,15]
>  name clash: add(java.util.List) in 
> org.apache.hadoop.hive.ql.exec.persistence.LazyFlatRowContainer overrides a 
> method whose erasure is the same as another method, yet neither overrides the 
> other
> [ERROR] first method:  add(E) in java.util.AbstractCollection
> [ERROR] second method: add(ROW) in 
> org.apache.hadoop.hive.ql.exec.persistence.AbstractRowContainer
> [ERROR] -> [Help 1]
> [ERROR] 
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR] 
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR] 
> [ERROR] After correcting the problems, you can resume the build with the 
> command
> [ERROR]   mvn  -rf :hive-exec
> {noformat}
> This LazyFlatRowContainer.java is  a new file added as part of  HIVE-6418 
> patch. It's extending AbstractCollection and implements AbstractRowContainer. 
> Looks like the both these have a add() method that's conflicting.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6486) Support secure Subject.doAs() in HiveServer2 JDBC client.

[Shivaraju Gowda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shivaraju Gowda updated HIVE-6486:
--

Component/s: HiveServer2
 Authentication

> Support secure Subject.doAs() in HiveServer2 JDBC client.
> -
>
> Key: HIVE-6486
> URL: https://issues.apache.org/jira/browse/HIVE-6486
> Project: Hive
>  Issue Type: Improvement
>  Components: Authentication, HiveServer2, JDBC
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Shivaraju Gowda
> Fix For: 0.13.0
>
> Attachments: HIVE-6486.1.patch, Hive_011_Support-Subject_doAS.patch, 
> TestHive_SujectDoAs.java
>
>
> HIVE-5155 addresses the problem of kerberos authentication in multi-user 
> middleware server using proxy user.  In this mode the principal used by the 
> middle ware server has privileges to impersonate selected users in 
> Hive/Hadoop. 
> This enhancement is to support Subject.doAs() authentication in  Hive JDBC 
> layer so that the end users Kerberos Subject is passed through in the middle 
> ware server. With this improvement there won't be any additional setup in the 
> server to grant proxy privileges to some users and there won't be need to 
> specify a proxy user in the JDBC client. This version should also be more 
> secure since it won't require principals with the privileges to impersonate 
> other users in Hive/Hadoop setup.
>  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918392#comment-13918392
 ] 

Sushanth Sowmyan commented on HIVE-5504:


Looks like the tests finally ran! :)

And the error reported by the pre-commit tests do not seem related to this 
patch - the table in question that's failing on that test is not even created 
as an orc table.

> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-5504.2.patch, HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 13845: HIVE-5155: Support secure proxy user access to HiveServer2

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13845/#review36000
---



shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java


Good point about adding this setting only in hive-site.xml, that way this 
privilege will be specific only to hive.

On other hand, all hdfs/mr level proxy users will end up getting privileges 
in hive as well. But I think that is OK, as they already would be privileged 
users at webhdfs level etc.



- Thejas Nair


On Dec. 5, 2013, 8:08 p.m., Prasad Mujumdar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13845/
> ---
> 
> (Updated Dec. 5, 2013, 8:08 p.m.)
> 
> 
> Review request for hive, Brock Noland, Carl Steinbach, and Thejas Nair.
> 
> 
> Bugs: HIVE-5155
> https://issues.apache.org/jira/browse/HIVE-5155
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Delegation token support -
> Enable delegation token connection for HiveServer2
> Enhance the TCLIService interface to support delegation token requests
> Support passing the delegation token connection type via JDBC URL and Beeline 
> option
> 
> Direct proxy access -
> Define new proxy user property
> Shim interfaces to validate proxy access for a given user
> 
> Note that the diff doesn't include thrift generated code.
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java c5e36a5 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java c3abba3 
>   beeline/src/java/org/apache/hive/beeline/Commands.java d2d7fd3 
>   beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java 1de5829 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 36503fa 
>   conf/hive-default.xml.template c61a0bb 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> 7b1c9da 
>   jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java d08e05b 
>   jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java ef39573 
>   jdbc/src/java/org/apache/hive/jdbc/Utils.java 4d75d98 
>   service/if/TCLIService.thrift 62a9730 
>   service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java d80649f 
>   service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java 
> 519556c 
>   service/src/java/org/apache/hive/service/auth/PlainSaslHelper.java 15b1675 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 8c85386 
>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 14ef54f 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
> 9dca874 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java f647ce6 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 00058cc 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> cfda752 
>   
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java
>  708f4e4 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> e262b72 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 9df110e 
>   
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
>  9bb2a0f 
>   service/src/test/org/apache/hive/service/auth/TestPlainSaslHelper.java 
> 8fa4afd 
>   service/src/test/org/apache/hive/service/cli/session/TestSessionHooks.java 
> 2fac800 
>   shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 
> 6ff1a84 
>   
> shims/common-secure/src/main/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
>  84f3ddc 
>   
> shims/common-secure/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java
>  dc89de1 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> 0d5615c 
>   
> shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java
>  03f4e51 
> 
> Diff: https://reviews.apache.org/r/13845/diff/
> 
> 
> Testing
> ---
> 
> Since this requires kerberos setup, its tested by a standalone test program 
> that runs various existing and new secure connection scenarios. The test code 
> is attached to the ticket at 
> https://issues.apache.org/jira/secure/attachment/12600119/ProxyAuth.java
> 
> 
> Thanks,
> 
> Prasad Mujumdar
> 
>

[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918410#comment-13918410
 ] 

Thejas M Nair commented on HIVE-5155:
-

[~prasadm] Yes, I think putting this in hive-site.xml is a reasonable way to 
restrict proxy user privileges to hs2.

[~owen.omalley] Please refer to the conversation in reviewboard about the 
config settings. 


> Support secure proxy user access to HiveServer2
> ---
>
> Key: HIVE-5155
> URL: https://issues.apache.org/jira/browse/HIVE-5155
> Project: Hive
>  Issue Type: Improvement
>  Components: Authentication, HiveServer2, JDBC
>Affects Versions: 0.12.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, 
> HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, 
> HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, 
> HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java
>
>
> The HiveServer2 can authenticate a client using via Kerberos and impersonate 
> the connecting user with underlying secure hadoop. This becomes a gateway for 
> a remote client to access secure hadoop cluster. Now this works fine for when 
> the client obtains Kerberos ticket and directly connects to HiveServer2. 
> There's another big use case for middleware tools where the end user wants to 
> access Hive via another server. For example Oozie action or Hue submitting 
> queries or a BI tool server accessing to HiveServer2. In these cases, the 
> third party server doesn't have end user's Kerberos credentials and hence it 
> can't submit queries to HiveServer2 on behalf of the end user.
> This ticket is for enabling proxy access to HiveServer2 for third party tools 
> on behalf of end users. There are two parts of the solution proposed in this 
> ticket:
> 1) Delegation token based connection for Oozie (OOZIE-1457)
> This is the common mechanism for Hadoop ecosystem components. Hive Remote 
> Metastore and HCatalog already support this. This is suitable for tool like 
> Oozie that submits the MR jobs as actions on behalf of its client. Oozie 
> already uses similar mechanism for Metastore/HCatalog access.
> 2) Direct proxy access for privileged hadoop users
> The delegation token implementation can be a challenge for non-hadoop 
> (especially non-java) components. This second part enables a privileged user 
> to directly specify an alternate session user during the connection. If the 
> connecting user has hadoop level privilege to impersonate the requested 
> userid, then HiveServer2 will run the session as that requested user. For 
> example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy 
> user configuration). Then user Hue can connect to HiveServer2 and specify Bob 
> as session user via a session property. HiveServer2 will verify Hue's proxy 
> user privilege and then impersonate user Bob instead of Hue. This will enable 
> any third party tool to impersonate alternate userid without having to 
> implement delegation token connection.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918416#comment-13918416
 ] 

Sushanth Sowmyan commented on HIVE-5193:


Thanks for the update, viraj. I'm still reviewing this patch, and will be done 
by EOD today. Since the precommit tests picked it up, and the errors being 
reported do not seem related, if I have no further feedback request for this 
patch, I will commit it once I'm done.

> Columnar Pushdown for RC/ORC File not happening in HCatLoader 
> --
>
> Key: HIVE-5193
> URL: https://issues.apache.org/jira/browse/HIVE-5193
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Viraj Bhat
>Assignee: Viraj Bhat
>  Labels: hcatalog
> Fix For: 0.13.0
>
> Attachments: HIVE-5193.2.patch, HIVE-5193.patch
>
>
> Currently the HCatLoader is not taking advantage of the 
> ColumnProjectionUtils. where it could skip columns during read. The 
> information is available in Pig it just needs to get to the Readers.
> Viraj



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5504) OrcOutputFormat honors compression properties only from within hive

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-5504:


   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

Patch committed to trunk.
Thanks for the contribution [~sushanth]

> OrcOutputFormat honors  compression  properties only from within hive
> -
>
> Key: HIVE-5504
> URL: https://issues.apache.org/jira/browse/HIVE-5504
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.11.0, 0.12.0, 0.13.0
>Reporter: Venkat Ranganathan
>Assignee: Sushanth Sowmyan
> Fix For: 0.13.0
>
> Attachments: HIVE-5504.2.patch, HIVE-5504.patch
>
>
> When we import data into a HCatalog table created with the following storage  
> description
> .. stored as orc tblproperties ("orc.compress"="SNAPPY") 
> the resultant orc file still uses the default zlib compression
> It looks like HCatOutputFormat is ignoring the tblproperties specified.   
> show tblproperties shows that the table indeed has the properties properly 
> saved.
> An insert/select into the table has the resulting orc file honor the tbl 
> property.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-6537:
--

 Summary: NullPointerException when loading hashtable for MapJoin 
directly
 Key: HIVE-6537
 URL: https://issues.apache.org/jira/browse/HIVE-6537
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


We see the following error:
{noformat}
2014-02-20 23:33:15,743 FATAL [main] 
org.apache.hadoop.hive.ql.exec.mr.ExecMapper: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
at 
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
at 
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at 
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.NullPointerException
at java.util.Arrays.fill(Arrays.java:2685)
at 
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
at 
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
... 15 more
{noformat}

It appears that the tables in Arrays.fill call is nulls. I don't really have 
full understanding of this path, but what I gleaned so far is this...
>From what I see, tables would be set unconditionally in initializeOp of the 
>sink, and in no other place, so I assume for this code to ever  work that 
>startForward calls it at least some time.
Here, it doesn't call it, so it's null. 
Previous loop also uses tables, and should have NPE-d before fill was ever 
called; it didn't, so I'd assume it never executed. 
There's a little bit of inconsistency in the above code where directWorks are 
added to parents unconditionally but sink is only added as child conditionally. 
I think it may be that some of the direct works are not table scans; in fact 
given that loop never executes they may be null (which is rather strange). 
Regardless, it seems that the logic should be fixed, it may be the root cause



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6409) FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918424#comment-13918424
 ] 

Sushanth Sowmyan commented on HIVE-6409:


Committed (with some minor indenting fixes).

Thanks, Mithun!

> FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
> --
>
> Key: HIVE-6409
> URL: https://issues.apache.org/jira/browse/HIVE-6409
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6409.patch
>
>
> When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls 
> the underlying OutputCommitter and then attempts to register partitions in 
> HCatalog.
> If the commit fails (for example, because of HIVE-4996), commitJob() cancels 
> delegation tokens retrieved from HCatalog before the exception is rethrown.
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
> ... 5 more
> Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
> partition to metastore. Cause :
> MetaException(message:java.lang.RuntimeException: commitTransaction was called
> but openTransactionCalls = 0. This probably indicates that there are 
> unbalanced
> calls to openTransaction/commitTransaction)
> at
> org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
> {code}
> The problem is that this happens before abortJob() has had a chance to run, 
> thus yielding the following error:
> {code}
> MetaException(message:Could not connect to meta store using any of the URIs 
> provided. Most recent failure: 
> org.apache.thrift.transport.TTransportException: Peer indicated failure: 
> DIGEST-MD5: IO error acquiring password
>   at 
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
>   at 
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>   at 
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>   at 
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>   at java.security.AccessController.doPrivileged(Native Method)
> ...
> {code}
> I'll have a patch out that only cancels delegation tokens if the commitJob() 
> has succeeded.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6409) FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918414#comment-13918414
 ] 

Sushanth Sowmyan commented on HIVE-6409:


I'm +1 on this new patch, I'll go ahead and commit it.

As to your other concerns, this is per HIVE-5274, and per discussions that I 
brought up in the next hive-dev meetup(after that patch) as well.

As to visibly breaking code, we will do that, as of 0.14, which will be trunk 
in a couple more days as soon as we branch out for 0.13 - we have HIVE-6432 
open for just that. The idea is that org.apache.hcatalog.* works predictably in 
the way it has for all of those that have already been using it and find it 
usable in the state it is in, whereas to all new users who may be evaluating, 
they must be moving to org.apache.hive.hcatalog.


> FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
> --
>
> Key: HIVE-6409
> URL: https://issues.apache.org/jira/browse/HIVE-6409
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6409.patch
>
>
> When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls 
> the underlying OutputCommitter and then attempts to register partitions in 
> HCatalog.
> If the commit fails (for example, because of HIVE-4996), commitJob() cancels 
> delegation tokens retrieved from HCatalog before the exception is rethrown.
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
> ... 5 more
> Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
> partition to metastore. Cause :
> MetaException(message:java.lang.RuntimeException: commitTransaction was called
> but openTransactionCalls = 0. This probably indicates that there are 
> unbalanced
> calls to openTransaction/commitTransaction)
> at
> org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
> {code}
> The problem is that this happens before abortJob() has had a chance to run, 
> thus yielding the following error:
> {code}
> MetaException(message:Could not connect to meta store using any of the URIs 
> provided. Most recent failure: 
> org.apache.thrift.transport.TTransportException: Peer indicated failure: 
> DIGEST-MD5: IO error acquiring password
>   at 
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
>   at 
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>   at 
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>   at 
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>   at java.security.AccessController.doPrivileged(Native Method)
> ...
> {code}
> I'll have a patch out that only cancels delegation tokens if the commitJob() 
> has succeeded.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6537:
---

Attachment: HIVE-6537.patch

> NullPointerException when loading hashtable for MapJoin directly
> 
>
> Key: HIVE-6537
> URL: https://issues.apache.org/jira/browse/HIVE-6537
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> We see the following error:
> {noformat}
> 2014-02-20 23:33:15,743 FATAL [main] 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.NullPointerException
> at java.util.Arrays.fill(Arrays.java:2685)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
> ... 15 more
> {noformat}
> It appears that the tables in Arrays.fill call is nulls. I don't really have 
> full understanding of this path, but what I gleaned so far is this...
> From what I see, tables would be set unconditionally in initializeOp of the 
> sink, and in no other place, so I assume for this code to ever  work that 
> startForward calls it at least some time.
> Here, it doesn't call it, so it's null. 
> Previous loop also uses tables, and should have NPE-d before fill was ever 
> called; it didn't, so I'd assume it never executed. 
> There's a little bit of inconsistency in the above code where directWorks are 
> added to parents unconditionally but sink is only added as child 
> conditionally. I think it may be that some of the direct works are not table 
> scans; in fact given that loop never executes they may be null (which is 
> rather strange). 
> Regardless, it seems that the logic should be fixed, it may be the root cause



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6537:
---

Attachment: (was: HIVE-6537.patch)

> NullPointerException when loading hashtable for MapJoin directly
> 
>
> Key: HIVE-6537
> URL: https://issues.apache.org/jira/browse/HIVE-6537
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> We see the following error:
> {noformat}
> 2014-02-20 23:33:15,743 FATAL [main] 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.NullPointerException
> at java.util.Arrays.fill(Arrays.java:2685)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
> ... 15 more
> {noformat}
> It appears that the tables in Arrays.fill call is nulls. I don't really have 
> full understanding of this path, but what I gleaned so far is this...
> From what I see, tables would be set unconditionally in initializeOp of the 
> sink, and in no other place, so I assume for this code to ever  work that 
> startForward calls it at least some time.
> Here, it doesn't call it, so it's null. 
> Previous loop also uses tables, and should have NPE-d before fill was ever 
> called; it didn't, so I'd assume it never executed. 
> There's a little bit of inconsistency in the above code where directWorks are 
> added to parents unconditionally but sink is only added as child 
> conditionally. I think it may be that some of the direct works are not table 
> scans; in fact given that loop never executes they may be null (which is 
> rather strange). 
> Regardless, it seems that the logic should be fixed, it may be the root cause



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5155) Support secure proxy user access to HiveServer2

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918428#comment-13918428
 ] 

Vaibhav Gumashta commented on HIVE-5155:


[~prasadm] Thanks a lot for the heads up Prasad!

> Support secure proxy user access to HiveServer2
> ---
>
> Key: HIVE-5155
> URL: https://issues.apache.org/jira/browse/HIVE-5155
> Project: Hive
>  Issue Type: Improvement
>  Components: Authentication, HiveServer2, JDBC
>Affects Versions: 0.12.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Attachments: HIVE-5155-1-nothrift.patch, HIVE-5155-noThrift.2.patch, 
> HIVE-5155-noThrift.4.patch, HIVE-5155-noThrift.5.patch, 
> HIVE-5155-noThrift.6.patch, HIVE-5155.1.patch, HIVE-5155.2.patch, 
> HIVE-5155.3.patch, ProxyAuth.java, ProxyAuth.out, TestKERBEROS_Hive_JDBC.java
>
>
> The HiveServer2 can authenticate a client using via Kerberos and impersonate 
> the connecting user with underlying secure hadoop. This becomes a gateway for 
> a remote client to access secure hadoop cluster. Now this works fine for when 
> the client obtains Kerberos ticket and directly connects to HiveServer2. 
> There's another big use case for middleware tools where the end user wants to 
> access Hive via another server. For example Oozie action or Hue submitting 
> queries or a BI tool server accessing to HiveServer2. In these cases, the 
> third party server doesn't have end user's Kerberos credentials and hence it 
> can't submit queries to HiveServer2 on behalf of the end user.
> This ticket is for enabling proxy access to HiveServer2 for third party tools 
> on behalf of end users. There are two parts of the solution proposed in this 
> ticket:
> 1) Delegation token based connection for Oozie (OOZIE-1457)
> This is the common mechanism for Hadoop ecosystem components. Hive Remote 
> Metastore and HCatalog already support this. This is suitable for tool like 
> Oozie that submits the MR jobs as actions on behalf of its client. Oozie 
> already uses similar mechanism for Metastore/HCatalog access.
> 2) Direct proxy access for privileged hadoop users
> The delegation token implementation can be a challenge for non-hadoop 
> (especially non-java) components. This second part enables a privileged user 
> to directly specify an alternate session user during the connection. If the 
> connecting user has hadoop level privilege to impersonate the requested 
> userid, then HiveServer2 will run the session as that requested user. For 
> example, user Hue is allowed to impersonate user Bob (via core-site.xml proxy 
> user configuration). Then user Hue can connect to HiveServer2 and specify Bob 
> as session user via a session property. HiveServer2 will verify Hue's proxy 
> user privilege and then impersonate user Bob instead of Hue. This will enable 
> any third party tool to impersonate alternate userid without having to 
> implement delegation token connection.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6537:
---

Status: Patch Available  (was: Open)

> NullPointerException when loading hashtable for MapJoin directly
> 
>
> Key: HIVE-6537
> URL: https://issues.apache.org/jira/browse/HIVE-6537
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6537.patch
>
>
> We see the following error:
> {noformat}
> 2014-02-20 23:33:15,743 FATAL [main] 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.NullPointerException
> at java.util.Arrays.fill(Arrays.java:2685)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
> ... 15 more
> {noformat}
> It appears that the tables in Arrays.fill call is nulls. I don't really have 
> full understanding of this path, but what I gleaned so far is this...
> From what I see, tables would be set unconditionally in initializeOp of the 
> sink, and in no other place, so I assume for this code to ever  work that 
> startForward calls it at least some time.
> Here, it doesn't call it, so it's null. 
> Previous loop also uses tables, and should have NPE-d before fill was ever 
> called; it didn't, so I'd assume it never executed. 
> There's a little bit of inconsistency in the above code where directWorks are 
> added to parents unconditionally but sink is only added as child 
> conditionally. I think it may be that some of the direct works are not table 
> scans; in fact given that loop never executes they may be null (which is 
> rather strange). 
> Regardless, it seems that the logic should be fixed, it may be the root cause



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-6537:
---

Attachment: HIVE-6537.patch

> NullPointerException when loading hashtable for MapJoin directly
> 
>
> Key: HIVE-6537
> URL: https://issues.apache.org/jira/browse/HIVE-6537
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6537.patch
>
>
> We see the following error:
> {noformat}
> 2014-02-20 23:33:15,743 FATAL [main] 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.NullPointerException
> at java.util.Arrays.fill(Arrays.java:2685)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
> ... 15 more
> {noformat}
> It appears that the tables in Arrays.fill call is nulls. I don't really have 
> full understanding of this path, but what I gleaned so far is this...
> From what I see, tables would be set unconditionally in initializeOp of the 
> sink, and in no other place, so I assume for this code to ever  work that 
> startForward calls it at least some time.
> Here, it doesn't call it, so it's null. 
> Previous loop also uses tables, and should have NPE-d before fill was ever 
> called; it didn't, so I'd assume it never executed. 
> There's a little bit of inconsistency in the above code where directWorks are 
> added to parents unconditionally but sink is only added as child 
> conditionally. I think it may be that some of the direct works are not table 
> scans; in fact given that loop never executes they may be null (which is 
> rather strange). 
> Regardless, it seems that the logic should be fixed, it may be the root cause



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader

[Viraj Bhat (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918460#comment-13918460
 ] 

Viraj Bhat commented on HIVE-5193:
--

Sushanth, Thanks for your update. Really appreciate
Viraj

> Columnar Pushdown for RC/ORC File not happening in HCatLoader 
> --
>
> Key: HIVE-5193
> URL: https://issues.apache.org/jira/browse/HIVE-5193
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Viraj Bhat
>Assignee: Viraj Bhat
>  Labels: hcatalog
> Fix For: 0.13.0
>
> Attachments: HIVE-5193.2.patch, HIVE-5193.patch
>
>
> Currently the HCatLoader is not taking advantage of the 
> ColumnProjectionUtils. where it could skip columns during read. The 
> information is available in Pig it just needs to get to the Readers.
> Viraj



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6379) webhcat is not operational due to Slf4j conflicts

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918456#comment-13918456
 ] 

Eugene Koifman commented on HIVE-6379:
--

to include .properties fiels in webhcat.jar

+++ b/hcatalog/webhcat/svr/pom.xml
@@ -141,6 +141,12 @@
   
 
   
+  
+  
+  .
+  src/main/config
+  
+  


> webhcat is not operational due to Slf4j conflicts
> -
>
> Key: HIVE-6379
> URL: https://issues.apache.org/jira/browse/HIVE-6379
> Project: Hive
>  Issue Type: Bug
>  Components: WebHCat
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> 1. build current trunk: mvn clean install -Pdist,hadoop-1 -DskipTests
> 2. 
> /Users/ekoifman/dev/hive/packaging/target/apache-hive-0.13.0-SNAPSHOT-bin/apache-hive-0.13.0-SNAPSHOT-bin/hcatalog
> 3. ./sbin/webhcat_server.sh start
> 4. http://localhost:50111/templeton/v1/ddl/database?user.name=ekoifman
> webhcat-console-error.log has
> log4j:WARN No appenders could be found for logger 
> (org.apache.hive.hcatalog.templeton.AppConfig).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.log(JettyAwareLogger.java:613)
> at 
> org.eclipse.jetty.util.log.JettyAwareLogger.debug(JettyAwareLogger.java:200)
> at org.eclipse.jetty.util.log.Slf4jLog.debug(Slf4jLog.java:94)
> at org.eclipse.jetty.util.component.Container.add(Container.java:201)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:164)
> at 
> org.eclipse.jetty.util.component.Container.update(Container.java:106)
> at org.eclipse.jetty.server.Server.setConnectors(Server.java:184)
> at org.eclipse.jetty.server.Server.(Server.java:97)
> at org.apache.hive.hcatalog.templeton.Main.runServer(Main.java:162)
> at org.apache.hive.hcatalog.templeton.Main.run(Main.java:114)
> at org.apache.hive.hcatalog.templeton.Main.main(Main.java:259)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> webhcat-console-error.log (END) 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 18588: sql std auth - add command to change owner of database

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18588/#review36013
---



ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java


nit: spacing here and above



ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java


why is this change in this patch?


- Sergey Shelukhin


On March 2, 2014, 12:52 a.m., Thejas Nair wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18588/
> ---
> 
> (Updated March 2, 2014, 12:52 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-6440
> https://issues.apache.org/jira/browse/HIVE-6440
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> It should be possible to change the owner of a database once it is created.
> Also includes a small change to authorize drop database, as it is better 
> tested with this alter-db-owner functionality.
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/negative/cascade_dbdrop_hadoop20.q.out 
> c5bfd6a 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
>  7fbb67e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> d60cb1b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 4310ab8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 2484576 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 8356155 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 129c6c6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 6bf2a17 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
> b1d3371 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/AuthorizationParseUtils.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/authorization/HiveAuthorizationTaskFactoryImpl.java
>  4dbe78c 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AlterDatabaseDesc.java 16bb95b 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java e661f00 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveOperationType.java
>  f1671ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/Operation2Privilege.java
>  c43bcea 
>   ql/src/test/queries/clientnegative/authorization_alter_db_owner.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/authorization_alter_db_owner_default.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/authorization_drop_db_cascade.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/authorization_drop_db_empty.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/alter_db_owner.q PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_alter_db_owner.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientnegative/authorization_alter_db_owner_default.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_drop_db_cascade.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_drop_db_empty.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/alter_db_owner.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/database.q.out cdaa405 
>   ql/src/test/results/clientpositive/database_drop.q.out 3128561 
>   ql/src/test/results/clientpositive/database_properties.q.out e3e2816 
>   
> ql/src/test/results/clientpositive/drop_database_removes_partition_dirs.q.out 
> 475d3d0 
>   ql/src/test/results/clientpositive/input46.q.out a891362 
>   ql/src/test/results/clientpositive/udf_using.q.out 69e5f3b 
> 
> Diff: https://reviews.apache.org/r/18588/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Thejas Nair
> 
>

[jira] [Commented] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918472#comment-13918472
 ] 

Sergey Shelukhin commented on HIVE-6537:


[~navis] [~vikram.dixit] fyi

> NullPointerException when loading hashtable for MapJoin directly
> 
>
> Key: HIVE-6537
> URL: https://issues.apache.org/jira/browse/HIVE-6537
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-6537.patch
>
>
> We see the following error:
> {noformat}
> 2014-02-20 23:33:15,743 FATAL [main] 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:103)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:149)
> at 
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:164)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1026)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1030)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
> at 
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.NullPointerException
> at java.util.Arrays.fill(Arrays.java:2685)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.loadDirectly(HashTableLoader.java:155)
> at 
> org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:81)
> ... 15 more
> {noformat}
> It appears that the tables in Arrays.fill call is nulls. I don't really have 
> full understanding of this path, but what I gleaned so far is this...
> From what I see, tables would be set unconditionally in initializeOp of the 
> sink, and in no other place, so I assume for this code to ever  work that 
> startForward calls it at least some time.
> Here, it doesn't call it, so it's null. 
> Previous loop also uses tables, and should have NPE-d before fill was ever 
> called; it didn't, so I'd assume it never executed. 
> There's a little bit of inconsistency in the above code where directWorks are 
> added to parents unconditionally but sink is only added as child 
> conditionally. I think it may be that some of the direct works are not table 
> scans; in fact given that loop never executes they may be null (which is 
> rather strange). 
> Regardless, it seems that the logic should be fixed, it may be the root cause



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5761) Implement vectorized support for the DATE data type

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918476#comment-13918476
 ] 

Jitendra Nath Pandey commented on HIVE-5761:


Committed to trunk. Thanks to Teddy!

> Implement vectorized support for the DATE data type
> ---
>
> Key: HIVE-5761
> URL: https://issues.apache.org/jira/browse/HIVE-5761
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Teddy Choi
> Attachments: HIVE-5761.1.patch, HIVE-5761.2.patch, HIVE-5761.3.patch, 
> HIVE-5761.4.patch, HIVE-5761.5.patch, HIVE-5761.6.patch, HIVE-5761.6.patch
>
>
> Add support to allow queries referencing DATE columns and expression results 
> to run efficiently in vectorized mode. This should re-use the code for the 
> the integer/timestamp types to the extent possible and beneficial. Include 
> unit tests and end-to-end tests. Consider re-using or extending existing 
> end-to-end tests for vectorized integer and/or timestamp operations.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5761) Implement vectorized support for the DATE data type

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5761:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Implement vectorized support for the DATE data type
> ---
>
> Key: HIVE-5761
> URL: https://issues.apache.org/jira/browse/HIVE-5761
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Teddy Choi
> Attachments: HIVE-5761.1.patch, HIVE-5761.2.patch, HIVE-5761.3.patch, 
> HIVE-5761.4.patch, HIVE-5761.5.patch, HIVE-5761.6.patch, HIVE-5761.6.patch
>
>
> Add support to allow queries referencing DATE columns and expression results 
> to run efficiently in vectorized mode. This should re-use the code for the 
> the integer/timestamp types to the extent possible and beneficial. Include 
> unit tests and end-to-end tests. Consider re-using or extending existing 
> end-to-end tests for vectorized integer and/or timestamp operations.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6538) yet another annoying exception in test logs

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-6538:
--

 Summary: yet another annoying exception in test logs
 Key: HIVE-6538
 URL: https://issues.apache.org/jira/browse/HIVE-6538
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Priority: Trivial


Whenever you look at failed q tests you have to go thru this useless exception.

{noformat}
2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invoke(143)) - 
MetaException(message:NoSuchObjectException(message:Function 
default.qtest_get_java_boolean does not exist))
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at $Proxy8.get_function(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at $Proxy9.getFunction(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
at 
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
at 
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
at 
org.apache.hadoop.hive.cli.TestCliDriver.(TestCliDriver.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:23)
at 
org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
at 
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at 
org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at 
org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at 
org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(Fork

[jira] [Created] (HIVE-6539) Couple of issues in fs based stats collection

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-6539:
--

 Summary: Couple of issues in fs based stats collection
 Key: HIVE-6539
 URL: https://issues.apache.org/jira/browse/HIVE-6539
 Project: Hive
  Issue Type: Bug
Reporter: Ashutosh Chauhan


While testing on cluster found couple of bugs:
* NPE in certain case.
* map object reuse causing problem



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6539) Couple of issues in fs based stats collection

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6539:
---

Attachment: HIVE-6539.patch

* Fixes NPE
* Do new Hashmap to avoid object reuse issue.
* Increased test coverage by putting fs based stats collection in tests.

> Couple of issues in fs based stats collection
> -
>
> Key: HIVE-6539
> URL: https://issues.apache.org/jira/browse/HIVE-6539
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
> Attachments: HIVE-6539.patch
>
>
> While testing on cluster found couple of bugs:
> * NPE in certain case.
> * map object reuse causing problem



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6539) Couple of issues in fs based stats collection

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6539:
---

Assignee: Ashutosh Chauhan
  Status: Patch Available  (was: Open)

> Couple of issues in fs based stats collection
> -
>
> Key: HIVE-6539
> URL: https://issues.apache.org/jira/browse/HIVE-6539
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6539.patch
>
>
> While testing on cluster found couple of bugs:
> * NPE in certain case.
> * map object reuse causing problem



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-5787) Move non-type specific column statistics to ColumnStatistics object in ORC

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J reassigned HIVE-5787:


Assignee: Prasanth J

> Move non-type specific column statistics to ColumnStatistics object in ORC
> --
>
> Key: HIVE-5787
> URL: https://issues.apache.org/jira/browse/HIVE-5787
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>Priority: Minor
>  Labels: orcfile
>
> Non-type statistics like min, max, numNulls etc. can be moved to 
> ColumnStatistics class. This avoids ugly checks like 
> {code} if (index instanceof IntegerColumnStatistics) {
> return ((IntegerColumnStatistics) index).getMinimum();
>   } else if (index instanceof DoubleColumnStatistics) {
> return ((DoubleColumnStatistics) index).getMinimum();
>   } else if (index instanceof StringColumnStatistics) {
> return ((StringColumnStatistics) index).getMinimum();
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6492) limit partition number involved in a table scan

[Selina Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Selina Zhang updated HIVE-6492:
---

Attachment: HIVE-6492.3.patch.txt

The test failure seems not related to this patch. Re-submit the patch. 

> limit partition number involved in a table scan
> ---
>
> Key: HIVE-6492
> URL: https://issues.apache.org/jira/browse/HIVE-6492
> Project: Hive
>  Issue Type: New Feature
>  Components: Query Processor
>Affects Versions: 0.12.0
>Reporter: Selina Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-6492.1.patch.txt, HIVE-6492.2.patch.txt, 
> HIVE-6492.3.patch.txt
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> To protect the cluster, a new configure variable 
> "hive.limit.query.max.table.partition" is added to hive configuration to
> limit the table partitions involved in a table scan. 
> The default value will be set to -1 which means there is no limit by default. 
> This variable will not affect "metadata only" query.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6486) Support secure Subject.doAs() in HiveServer2 JDBC client.

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-6486:
---

Assignee: Shivaraju Gowda

> Support secure Subject.doAs() in HiveServer2 JDBC client.
> -
>
> Key: HIVE-6486
> URL: https://issues.apache.org/jira/browse/HIVE-6486
> Project: Hive
>  Issue Type: Improvement
>  Components: Authentication, HiveServer2, JDBC
>Affects Versions: 0.11.0, 0.12.0
>Reporter: Shivaraju Gowda
>Assignee: Shivaraju Gowda
> Fix For: 0.13.0
>
> Attachments: HIVE-6486.1.patch, Hive_011_Support-Subject_doAS.patch, 
> TestHive_SujectDoAs.java
>
>
> HIVE-5155 addresses the problem of kerberos authentication in multi-user 
> middleware server using proxy user.  In this mode the principal used by the 
> middle ware server has privileges to impersonate selected users in 
> Hive/Hadoop. 
> This enhancement is to support Subject.doAs() authentication in  Hive JDBC 
> layer so that the end users Kerberos Subject is passed through in the middle 
> ware server. With this improvement there won't be any additional setup in the 
> server to grant proxy privileges to some users and there won't be need to 
> specify a proxy user in the JDBC client. This version should also be more 
> secure since it won't require principals with the privileges to impersonate 
> other users in Hive/Hadoop setup.
>  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5728) Make ORC InputFormat/OutputFormat usable outside Hive

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918590#comment-13918590
 ] 

Rohini Palaniswamy commented on HIVE-5728:
--

[~owen.omalley],
Shouldn't this jira be marked Resolved or is there some reason to keep it 
open?

> Make ORC InputFormat/OutputFormat usable outside Hive
> -
>
> Key: HIVE-5728
> URL: https://issues.apache.org/jira/browse/HIVE-5728
> Project: Hive
>  Issue Type: Improvement
>  Components: File Formats
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.13.0
>
> Attachments: HIVE-5728-1.patch, HIVE-5728-10.patch, 
> HIVE-5728-2.patch, HIVE-5728-3.patch, HIVE-5728-4.patch, HIVE-5728-5.patch, 
> HIVE-5728-6.patch, HIVE-5728-7.patch, HIVE-5728-8.patch, HIVE-5728-9.patch, 
> HIVE-5728.10.patch, HIVE-5728.11.patch, HIVE-5728.12.patch, HIVE-5728.13.patch
>
>
> ORC InputFormat/OutputFormat is currently not usable outside Hive. There are 
> several issues need to solve:
> 1. Several class is not public, eg: OrcStruct
> 2. There is no InputFormat/OutputFormat for new api (Some tools such as Pig 
> need new api)
> 3. Has no way to push WriteOption to OutputFormat outside Hive



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6538) yet another annoying exception in test logs

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918593#comment-13918593
 ] 

Szehon Ho commented on HIVE-6538:
-

In RetryingHMSHandler, there are checks for get_table and get_partitions not to 
log to error, I guess similar thing needs to apply for new method 
"get_function".

> yet another annoying exception in test logs
> ---
>
> Key: HIVE-6538
> URL: https://issues.apache.org/jira/browse/HIVE-6538
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Trivial
>
> Whenever you look at failed q tests you have to go thru this useless 
> exception.
> {noformat}
> 2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(143)) - 
> MetaException(message:NoSuchObjectException(message:Function 
> default.qtest_get_java_boolean does not exist))
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>   at $Proxy8.get_function(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>   at $Proxy9.getFunction(Unknown Source)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
>   at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
>   at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.(TestCliDriver.java:46)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
>   at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:23)
>   at 
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
>   at 
> org.apache.maven.surefire.

Review Request 18709: Bug fix for fs based stats collection.

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18709/
---

Review request for hive.


Bugs: HIVE-6539
https://issues.apache.org/jira/browse/HIVE-6539


Repository: hive-git


Description
---

Bug fix for fs based stats collection.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java 3255031 
  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java d898a26 
  ql/src/test/queries/clientpositive/metadata_only_queries.q 9cec4da 
  ql/src/test/queries/clientpositive/metadata_only_queries_with_filters.q 
998f6a1 
  ql/src/test/queries/clientpositive/stats_only_null.q 38524b4 

Diff: https://reviews.apache.org/r/18709/diff/


Testing
---

Made few stats based test to use fs based stats collection.


Thanks,

Ashutosh Chauhan

[jira] [Created] (HIVE-6540) Support Multi Column Stats

[Laljo John Pullokkaran (JIRA)]

Laljo John Pullokkaran created HIVE-6540:


 Summary: Support Multi Column Stats
 Key: HIVE-6540
 URL: https://issues.apache.org/jira/browse/HIVE-6540
 Project: Hive
  Issue Type: Improvement
Reporter: Laljo John Pullokkaran
Assignee: Laljo John Pullokkaran


For Joins involving compound predicates, multi column stats can be used to 
accurately compute the NDV.

Objective is to compute NDV of more than one columns.

Compute NDV of (x,y,z).

R1 IJ R2 on R1.x=R2.x and R1.y=R2.y and R1.z=R2.z can use max(NDV(R1.x, R1.y, 
R1.z), NDV(R2.x, R2.y, R2.z)) for Join NDV (& hence selectivity).

http://www.oracle-base.com/articles/11g/statistics-collection-enhancements-11gr1.php#multi_column_statistics
http://blogs.msdn.com/b/ianjo/archive/2005/11/10/491548.aspx
http://developer.teradata.com/database/articles/removing-multi-column-statistics-a-process-for-identification-of-redundant-statist



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6507) OrcFile table property names are specified as strings

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-6507:
---

Status: Open  (was: Patch Available)

> OrcFile table property names are specified as strings
> -
>
> Key: HIVE-6507
> URL: https://issues.apache.org/jira/browse/HIVE-6507
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-6507.patch
>
>
> In HIVE-5504, we had to do some special casing in HCatalog to add a 
> particular set of orc table properties from table properties to job 
> properties.
> In doing so, it's obvious that that is a bit cumbersome, and ideally, the 
> list of all orc file table properties should really be an enum, rather than 
> individual loosely tied constant strings. If we were to clean this up, we can 
> clean up other code that references this to reference the entire enum, and 
> avoid future errors when new table properties are introduced, but other 
> referencing code is not updated.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6409) FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-6409:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

> FileOutputCommitterContainer::commitJob() cancels delegation tokens too early.
> --
>
> Key: HIVE-6409
> URL: https://issues.apache.org/jira/browse/HIVE-6409
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Fix For: 0.13.0
>
> Attachments: HIVE-6409.patch
>
>
> When HCatalog's FileOutputCommitterContainer::commitJob() is run, it calls 
> the underlying OutputCommitter and then attempts to register partitions in 
> HCatalog.
> If the commit fails (for example, because of HIVE-4996), commitJob() cancels 
> delegation tokens retrieved from HCatalog before the exception is rethrown.
> {code}
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:185)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:249)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:212)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter.commitJob(PigOutputCommitter.java:183)
> ... 5 more
> Caused by: org.apache.hcatalog.common.HCatException : 2006 : Error adding
> partition to metastore. Cause :
> MetaException(message:java.lang.RuntimeException: commitTransaction was called
> but openTransactionCalls = 0. This probably indicates that there are 
> unbalanced
> calls to openTransaction/commitTransaction)
> at
> org.apache.hcatalog.mapreduce.FileOutputCommitterContainer.registerPartitions(FileOutputCommitterContainer.java:712)
> {code}
> The problem is that this happens before abortJob() has had a chance to run, 
> thus yielding the following error:
> {code}
> MetaException(message:Could not connect to meta store using any of the URIs 
> provided. Most recent failure: 
> org.apache.thrift.transport.TTransportException: Peer indicated failure: 
> DIGEST-MD5: IO error acquiring password
>   at 
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
>   at 
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
>   at 
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
>   at 
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
>   at java.security.AccessController.doPrivileged(Native Method)
> ...
> {code}
> I'll have a patch out that only cancels delegation tokens if the commitJob() 
> has succeeded.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6507) OrcFile table property names are specified as strings

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-6507:
---

Status: Patch Available  (was: Open)

> OrcFile table property names are specified as strings
> -
>
> Key: HIVE-6507
> URL: https://issues.apache.org/jira/browse/HIVE-6507
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog, Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-6507.patch
>
>
> In HIVE-5504, we had to do some special casing in HCatalog to add a 
> particular set of orc table properties from table properties to job 
> properties.
> In doing so, it's obvious that that is a bit cumbersome, and ideally, the 
> list of all orc file table properties should really be an enum, rather than 
> individual loosely tied constant strings. If we were to clean this up, we can 
> clean up other code that references this to reference the entire enum, and 
> avoid future errors when new table properties are introduced, but other 
> referencing code is not updated.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6537) NullPointerException when loading hashtable for MapJoin directly

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918618#comment-13918618
 ] 

Hive QA commented on HIVE-6537:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632331/HIVE-6537.patch

{color:red}ERROR:{color} -1 due to 61 failed/errored test(s), 5236 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cp_mj_rc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32_lessSize
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join33
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join34
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join35
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_distinct
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_subquery2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_test_outer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats11
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1606/testReport
Conso

[jira] [Updated] (HIVE-5950) ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5950:
-

Attachment: HIVE-5950.5.patch

Addressed [~hagleitn]'s code review comments.

> ORC SARG creation fails with NPE for predicate conditions with 
> decimal/date/char/varchar datatypes
> --
>
> Key: HIVE-5950
> URL: https://issues.apache.org/jira/browse/HIVE-5950
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5950.1.patch, HIVE-5950.2.patch, HIVE-5950.3.patch, 
> HIVE-5950.4.patch, HIVE-5950.5.patch
>
>
> When decimal or date column is used, the type field in PredicateLeafImpl will 
> be set to null. This will result in NPE during predicate leaf generation 
> because of null dereferencing in hashcode computation. SARG creation should 
> be extended to support/handle decimal and date data types.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Gunther Hagleitner]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/#review36039
---



ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java


can you remove trailing ws in the files. This shouldn't require a new run 
of the tests...



service/src/java/org/apache/hive/service/server/HiveServer2.java


i believe this should be hidden behind conf check


- Gunther Hagleitner


On Feb. 28, 2014, 2:25 a.m., Vikram Dixit Kumaraswamy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/17471/
> ---
> 
> (Updated Feb. 28, 2014, 2:25 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-6325
> https://issues.apache.org/jira/browse/HIVE-6325
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Enable using multiple concurrent sessions in tez.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6802b4d 
>   conf/hive-default.xml.template 0b86b9c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java b8552a3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java bef5ba3 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 16e9166 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
> PRE-CREATION 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 
> 
> Diff: https://reviews.apache.org/r/17471/diff/
> 
> 
> Testing
> ---
> 
> Added multi-threaded junit tests.
> 
> 
> Thanks,
> 
> Vikram Dixit Kumaraswamy
> 
>

[jira] [Commented] (HIVE-6325) Enable using multiple concurrent sessions in tez

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918645#comment-13918645
 ] 

Gunther Hagleitner commented on HIVE-6325:
--

minor comments on rb. Otherwise +1 LGTM.

> Enable using multiple concurrent sessions in tez
> 
>
> Key: HIVE-6325
> URL: https://issues.apache.org/jira/browse/HIVE-6325
> Project: Hive
>  Issue Type: Improvement
>  Components: Tez
>Affects Versions: 0.13.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-6325.1.patch, HIVE-6325.2.patch, HIVE-6325.3.patch, 
> HIVE-6325.4.patch, HIVE-6325.5.patch, HIVE-6325.6.patch, HIVE-6325.7.patch, 
> HIVE-6325.8.patch, HIVE-6325.9.patch
>
>
> We would like to enable multiple concurrent sessions in tez via hive server 
> 2. This will enable users to make efficient use of the cluster when it has 
> been partitioned using yarn queues.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918660#comment-13918660
 ] 

Sushanth Sowmyan commented on HIVE-5193:


Looks good to me, +1.

> Columnar Pushdown for RC/ORC File not happening in HCatLoader 
> --
>
> Key: HIVE-5193
> URL: https://issues.apache.org/jira/browse/HIVE-5193
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Viraj Bhat
>Assignee: Viraj Bhat
>  Labels: hcatalog
> Fix For: 0.13.0
>
> Attachments: HIVE-5193.2.patch, HIVE-5193.patch
>
>
> Currently the HCatLoader is not taking advantage of the 
> ColumnProjectionUtils. where it could skip columns during read. The 
> information is available in Pig it just needs to get to the Readers.
> Viraj



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6392) Hive (and HCatalog) don't allow super-users to add partitions to tables.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-6392:
---

Status: Patch Available  (was: Open)

> Hive (and HCatalog) don't allow super-users to add partitions to tables.
> 
>
> Key: HIVE-6392
> URL: https://issues.apache.org/jira/browse/HIVE-6392
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6392.branch-0.12.patch, HIVE-6392.patch
>
>
> HDFS allows for users to be added to a "supergroup" (identified by the 
> "dfs.permissions.superusergroup" key in hdfs-site.xml). Users in this group 
> are allowed to modify HDFS contents regardless of the path's ogw permissions.
> However, Hive's StorageBasedAuthProvider disallows such a superuser from 
> adding partitions to any table that doesn't explicitly grant write 
> permissions to said superuser. This causes the odd scenario where the 
> superuser writes data to a partition-directory (under the table's path), but 
> can't register the appropriate partition.
> I have a patch that brings the Metastore's behaviour in line with what the 
> HDFS allows.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6392) Hive (and HCatalog) don't allow super-users to add partitions to tables.

[Mithun Radhakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mithun Radhakrishnan updated HIVE-6392:
---

Status: Open  (was: Patch Available)

> Hive (and HCatalog) don't allow super-users to add partitions to tables.
> 
>
> Key: HIVE-6392
> URL: https://issues.apache.org/jira/browse/HIVE-6392
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6392.branch-0.12.patch, HIVE-6392.patch
>
>
> HDFS allows for users to be added to a "supergroup" (identified by the 
> "dfs.permissions.superusergroup" key in hdfs-site.xml). Users in this group 
> are allowed to modify HDFS contents regardless of the path's ogw permissions.
> However, Hive's StorageBasedAuthProvider disallows such a superuser from 
> adding partitions to any table that doesn't explicitly grant write 
> permissions to said superuser. This causes the odd scenario where the 
> superuser writes data to a partition-directory (under the table's path), but 
> can't register the appropriate partition.
> I have a patch that brings the Metastore's behaviour in line with what the 
> HDFS allows.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 17471: HIVE-6325: Enable using multiple concurrent sessions in tez

[Vikram Dixit Kumaraswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17471/
---

(Updated March 3, 2014, 10:39 p.m.)


Review request for hive.


Changes
---

Addressed comments.


Bugs: HIVE-6325
https://issues.apache.org/jira/browse/HIVE-6325


Repository: hive-git


Description
---

Enable using multiple concurrent sessions in tez.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 388a604 
  conf/hive-default.xml.template 3f01e0b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java aef6e68 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java d30ec8e 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 16e9166 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionPool.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezSessionState.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 8700c5b 

Diff: https://reviews.apache.org/r/17471/diff/


Testing
---

Added multi-threaded junit tests.


Thanks,

Vikram Dixit Kumaraswamy

[jira] [Updated] (HIVE-6538) yet another annoying exception in test logs

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-6538:


Attachment: HIVE-6538.patch

This seems to work and get rid of the stack trace in the log.  

However, FunctionRegistry.getFunctionInfoFromMetastore() still seems to print 
the top-level error, not sure if its by design. + [~jdere]

> yet another annoying exception in test logs
> ---
>
> Key: HIVE-6538
> URL: https://issues.apache.org/jira/browse/HIVE-6538
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Trivial
> Attachments: HIVE-6538.patch
>
>
> Whenever you look at failed q tests you have to go thru this useless 
> exception.
> {noformat}
> 2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(143)) - 
> MetaException(message:NoSuchObjectException(message:Function 
> default.qtest_get_java_boolean does not exist))
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>   at $Proxy8.get_function(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>   at $Proxy9.getFunction(Unknown Source)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
>   at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
>   at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.(TestCliDriver.java:46)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
>   at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:23)
>   at 
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Pro

[jira] [Updated] (HIVE-6538) yet another annoying exception in test logs

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-6538:


Status: Patch Available  (was: Open)

> yet another annoying exception in test logs
> ---
>
> Key: HIVE-6538
> URL: https://issues.apache.org/jira/browse/HIVE-6538
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Szehon Ho
>Priority: Trivial
> Attachments: HIVE-6538.patch
>
>
> Whenever you look at failed q tests you have to go thru this useless 
> exception.
> {noformat}
> 2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(143)) - 
> MetaException(message:NoSuchObjectException(message:Function 
> default.qtest_get_java_boolean does not exist))
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>   at $Proxy8.get_function(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>   at $Proxy9.getFunction(Unknown Source)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
>   at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
>   at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.(TestCliDriver.java:46)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
>   at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:23)
>   at 
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.juni

[jira] [Assigned] (HIVE-6538) yet another annoying exception in test logs

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho reassigned HIVE-6538:
---

Assignee: Szehon Ho

> yet another annoying exception in test logs
> ---
>
> Key: HIVE-6538
> URL: https://issues.apache.org/jira/browse/HIVE-6538
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Szehon Ho
>Priority: Trivial
> Attachments: HIVE-6538.patch
>
>
> Whenever you look at failed q tests you have to go thru this useless 
> exception.
> {noformat}
> 2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(143)) - 
> MetaException(message:NoSuchObjectException(message:Function 
> default.qtest_get_java_boolean does not exist))
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>   at $Proxy8.get_function(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>   at $Proxy9.getFunction(Unknown Source)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
>   at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
>   at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.(TestCliDriver.java:46)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
>   at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:23)
>   at 
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Pr

[jira] [Commented] (HIVE-6532) ORC NPE on null Map

[Prasanth J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918675#comment-13918675
 ] 

Prasanth J commented on HIVE-6532:
--

ORC creates SARG (searchable argument) when there are where predicates in the 
query. SARG creation fails for non-primitive columns. This is already tracked 
in HIVE-5567. The fix for it should go in next release.

> ORC NPE on null Map
> ---
>
> Key: HIVE-6532
> URL: https://issues.apache.org/jira/browse/HIVE-6532
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.12.0
> Environment: ubuntu 12.04 
>Reporter: nicolas maillard
>Priority: Trivial
>
> On a select query map tasks will fail with npe for a where clause on a null 
> map. on an ORC table 
> example
>  select * from my table where mymap['entry'] = 'something'
> If my map where to be null in a line we will get an npe exception.
> Maybe the line should just be ignored.
> The same query on a text format table will resolve correctly



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6325) Enable using multiple concurrent sessions in tez

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-6325:
-

Status: Open  (was: Patch Available)

> Enable using multiple concurrent sessions in tez
> 
>
> Key: HIVE-6325
> URL: https://issues.apache.org/jira/browse/HIVE-6325
> Project: Hive
>  Issue Type: Improvement
>  Components: Tez
>Affects Versions: 0.13.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-6325.1.patch, HIVE-6325.10.patch, 
> HIVE-6325.2.patch, HIVE-6325.3.patch, HIVE-6325.4.patch, HIVE-6325.5.patch, 
> HIVE-6325.6.patch, HIVE-6325.7.patch, HIVE-6325.8.patch, HIVE-6325.9.patch
>
>
> We would like to enable multiple concurrent sessions in tez via hive server 
> 2. This will enable users to make efficient use of the cluster when it has 
> been partitioned using yarn queues.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6325) Enable using multiple concurrent sessions in tez

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-6325:
-

Attachment: HIVE-6325.10.patch

Address comments.

> Enable using multiple concurrent sessions in tez
> 
>
> Key: HIVE-6325
> URL: https://issues.apache.org/jira/browse/HIVE-6325
> Project: Hive
>  Issue Type: Improvement
>  Components: Tez
>Affects Versions: 0.13.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-6325.1.patch, HIVE-6325.10.patch, 
> HIVE-6325.2.patch, HIVE-6325.3.patch, HIVE-6325.4.patch, HIVE-6325.5.patch, 
> HIVE-6325.6.patch, HIVE-6325.7.patch, HIVE-6325.8.patch, HIVE-6325.9.patch
>
>
> We would like to enable multiple concurrent sessions in tez via hive server 
> 2. This will enable users to make efficient use of the cluster when it has 
> been partitioned using yarn queues.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918679#comment-13918679
 ] 

Sushanth Sowmyan commented on HIVE-5193:


Committed. Thanks, Viraj!

> Columnar Pushdown for RC/ORC File not happening in HCatLoader 
> --
>
> Key: HIVE-5193
> URL: https://issues.apache.org/jira/browse/HIVE-5193
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Viraj Bhat
>Assignee: Viraj Bhat
>  Labels: hcatalog
> Fix For: 0.13.0
>
> Attachments: HIVE-5193.2.patch, HIVE-5193.patch
>
>
> Currently the HCatLoader is not taking advantage of the 
> ColumnProjectionUtils. where it could skip columns during read. The 
> information is available in Pig it just needs to get to the Readers.
> Viraj



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5193:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Columnar Pushdown for RC/ORC File not happening in HCatLoader 
> --
>
> Key: HIVE-5193
> URL: https://issues.apache.org/jira/browse/HIVE-5193
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Viraj Bhat
>Assignee: Viraj Bhat
>  Labels: hcatalog
> Fix For: 0.13.0
>
> Attachments: HIVE-5193.2.patch, HIVE-5193.patch
>
>
> Currently the HCatLoader is not taking advantage of the 
> ColumnProjectionUtils. where it could skip columns during read. The 
> information is available in Pig it just needs to get to the Readers.
> Viraj



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-5193) Columnar Pushdown for RC/ORC File not happening in HCatLoader

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5193:
---

Release Note: (Columnar pushdown will now work with HCatLoader - basically, 
what this means is that when using RC/ORC tables, not all rows are necessarily 
read and processed if pig would discard (by filtering) it anyway. This allows 
reads to be a bit faster.)

> Columnar Pushdown for RC/ORC File not happening in HCatLoader 
> --
>
> Key: HIVE-5193
> URL: https://issues.apache.org/jira/browse/HIVE-5193
> Project: Hive
>  Issue Type: Improvement
>  Components: HCatalog
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Viraj Bhat
>Assignee: Viraj Bhat
>  Labels: hcatalog
> Fix For: 0.13.0
>
> Attachments: HIVE-5193.2.patch, HIVE-5193.patch
>
>
> Currently the HCatLoader is not taking advantage of the 
> ColumnProjectionUtils. where it could skip columns during read. The 
> information is available in Pig it just needs to get to the Readers.
> Viraj



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6538) yet another annoying exception in test logs

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918684#comment-13918684
 ] 

Sergey Shelukhin commented on HIVE-6538:


+1 assuming tests pass, esp. negative cli 

> yet another annoying exception in test logs
> ---
>
> Key: HIVE-6538
> URL: https://issues.apache.org/jira/browse/HIVE-6538
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Szehon Ho
>Priority: Trivial
> Attachments: HIVE-6538.patch
>
>
> Whenever you look at failed q tests you have to go thru this useless 
> exception.
> {noformat}
> 2014-03-03 11:22:54,872 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(143)) - 
> MetaException(message:NoSuchObjectException(message:Function 
> default.qtest_get_java_boolean does not exist))
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:4575)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_function(HiveMetaStore.java:4702)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>   at $Proxy8.get_function(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunction(HiveMetaStoreClient.java:1526)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
>   at $Proxy9.getFunction(Unknown Source)
>   at org.apache.hadoop.hive.ql.metadata.Hive.getFunction(Hive.java:2603)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfoFromMetastore(FunctionRegistry.java:546)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getQualifiedFunctionInfo(FunctionRegistry.java:578)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:599)
>   at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:606)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:94)
>   at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:60)
>   at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:445)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:345)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1078)
>   at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1121)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1014)
>   at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
>   at org.apache.hadoop.hive.ql.QTestUtil.runCmd(QTestUtil.java:655)
>   at org.apache.hadoop.hive.ql.QTestUtil.createSources(QTestUtil.java:772)
>   at 
> org.apache.hadoop.hive.cli.TestCliDriver.(TestCliDriver.java:46)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   at java.lang.reflect.Method.invoke(Method.java:597)
>   at 
> org.junit.internal.runners.SuiteMethod.testFromSuiteMethod(SuiteMethod.java:34)
>   at org.junit.internal.runners.SuiteMethod.(SuiteMethod.java:23)
>   at 
> org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:14)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
>   at 
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
>   at 
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:262)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTest

[jira] [Commented] (HIVE-5950) ORC SARG creation fails with NPE for predicate conditions with decimal/date/char/varchar datatypes

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918652#comment-13918652
 ] 

Gunther Hagleitner commented on HIVE-5950:
--

+1

> ORC SARG creation fails with NPE for predicate conditions with 
> decimal/date/char/varchar datatypes
> --
>
> Key: HIVE-5950
> URL: https://issues.apache.org/jira/browse/HIVE-5950
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5950.1.patch, HIVE-5950.2.patch, HIVE-5950.3.patch, 
> HIVE-5950.4.patch, HIVE-5950.5.patch
>
>
> When decimal or date column is used, the type field in PredicateLeafImpl will 
> be set to null. This will result in NPE during predicate leaf generation 
> because of null dereferencing in hashcode computation. SARG creation should 
> be extended to support/handle decimal and date data types.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-6541) Need to write documentation for ACID work

[Alan Gates (JIRA)]

Alan Gates created HIVE-6541:


 Summary: Need to write documentation for ACID work
 Key: HIVE-6541
 URL: https://issues.apache.org/jira/browse/HIVE-6541
 Project: Hive
  Issue Type: Sub-task
  Components: Documentation
Affects Versions: 0.13.0
Reporter: Alan Gates
Assignee: Alan Gates
 Fix For: 0.13.0


ACID introduces a number of new config file options, tables in the metastore, 
keywords in the grammar, and a new interface for use of tools like storm and 
flume.  These need to be documented.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6518) Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-6518:
--

Status: Patch Available  (was: Open)

> Add a GC canary to the VectorGroupByOperator to flush whenever a GC is 
> triggered
> 
>
> Key: HIVE-6518
> URL: https://issues.apache.org/jira/browse/HIVE-6518
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-6518.1-tez.patch, HIVE-6518.2-tez.patch, 
> HIVE-6518.2.patch
>
>
> The current VectorGroupByOperator implementation flushes the in-memory hashes 
> when the maximum entries or fraction of memory is hit.
> This works for most cases, but there are some corner cases where we hit GC 
> ovehead limits or heap size limits before either of those conditions are 
> reached due to the rest of the pipeline.
> This patch adds a SoftReference as a GC canary. If the soft reference is 
> dead, then a full GC pass happened sometime in the near past & the 
> aggregation hashtables should be flushed immediately before another full GC 
> is triggered.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6518) Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-6518:
--

Status: Open  (was: Patch Available)

> Add a GC canary to the VectorGroupByOperator to flush whenever a GC is 
> triggered
> 
>
> Key: HIVE-6518
> URL: https://issues.apache.org/jira/browse/HIVE-6518
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
> Attachments: HIVE-6518.1-tez.patch, HIVE-6518.2-tez.patch, 
> HIVE-6518.2.patch
>
>
> The current VectorGroupByOperator implementation flushes the in-memory hashes 
> when the maximum entries or fraction of memory is hit.
> This works for most cases, but there are some corner cases where we hit GC 
> ovehead limits or heap size limits before either of those conditions are 
> reached due to the rest of the pipeline.
> This patch adds a SoftReference as a GC canary. If the soft reference is 
> dead, then a full GC pass happened sometime in the near past & the 
> aggregation hashtables should be flushed immediately before another full GC 
> is triggered.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6392) Hive (and HCatalog) don't allow super-users to add partitions to tables.

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918697#comment-13918697
 ] 

Mithun Radhakrishnan commented on HIVE-6392:


Hey, Thejas. This failure doesn't look related to the code change. I've 
resubmitted the patch. I'm hoping for a cleaner run.

> Hive (and HCatalog) don't allow super-users to add partitions to tables.
> 
>
> Key: HIVE-6392
> URL: https://issues.apache.org/jira/browse/HIVE-6392
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-6392.branch-0.12.patch, HIVE-6392.patch
>
>
> HDFS allows for users to be added to a "supergroup" (identified by the 
> "dfs.permissions.superusergroup" key in hdfs-site.xml). Users in this group 
> are allowed to modify HDFS contents regardless of the path's ogw permissions.
> However, Hive's StorageBasedAuthProvider disallows such a superuser from 
> adding partitions to any table that doesn't explicitly grant write 
> permissions to said superuser. This causes the odd scenario where the 
> superuser writes data to a partition-directory (under the table's path), but 
> can't register the appropriate partition.
> I have a patch that brings the Metastore's behaviour in line with what the 
> HDFS allows.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6290) Add support for hbase filters for composite keys

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918718#comment-13918718
 ] 

Xuefu Zhang commented on HIVE-6290:
---

[~swarnim] The review board (https://reviews.apache.org/r/17483/) doesn't seem 
having your latest code change. Do you mind updating that? You can reuse the 
same review request, but uploading diff with your latest patch.

> Add support for hbase filters for composite keys
> 
>
> Key: HIVE-6290
> URL: https://issues.apache.org/jira/browse/HIVE-6290
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler
>Affects Versions: 0.12.0
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-6290.1.patch.txt, HIVE-6290.2.patch.txt, 
> HIVE-6290.3.patch.txt
>
>
> Add support for filters to be provided via the composite key class



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6539) Couple of issues in fs based stats collection

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918732#comment-13918732
 ] 

Hive QA commented on HIVE-6539:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12632345/HIVE-6539.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5236 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hive.hcatalog.mapreduce.TestHCatMutablePartitioned.testHCatPartitionedTable
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1607/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1607/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12632345

> Couple of issues in fs based stats collection
> -
>
> Key: HIVE-6539
> URL: https://issues.apache.org/jira/browse/HIVE-6539
> Project: Hive
>  Issue Type: Bug
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-6539.patch
>
>
> While testing on cluster found couple of bugs:
> * NPE in certain case.
> * map object reuse causing problem



--
This message was sent by Atlassian JIRA
(v6.2#6252)