[jira] [Commented] (HIVE-5979) Failure in cast to timestamps.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844051#comment-13844051
 ] 

Hive QA commented on HIVE-5979:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617968/HIVE-5979.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4762 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/592/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/592/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12617968

> Failure in cast to timestamps.
> --
>
> Key: HIVE-5979
> URL: https://issues.apache.org/jira/browse/HIVE-5979
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5979.1.patch
>
>
> Query ran:
> {code}
> select cast(t as timestamp), cast(si as timestamp),
>cast(i as timestamp), cast(b as timestamp),
>cast(f as string), cast(d as timestamp),
>cast(bo as timestamp), cast(b * 0 as timestamp),
>cast(ts as timestamp), cast(s as timestamp),
>cast(substr(s, 1, 1) as timestamp)
> from Table1;
> {code}
> Running this query with hive.vectorized.execution.enabled=true fails with the 
> following exception:
> {noformat}
> 13/12/05 07:56:36 ERROR tez.TezJobMonitor: Status: Failed
> Vertex failed, vertexName=Map 1, vertexId=vertex_1386227234886_0482_1_00, 
> diagnostics=[Task failed, taskId=task_1386227234886_0482_1_00_00, 
> diagnostics=[AttemptID:attempt_1386227234886_0482_1_00_00_0 Info:Error: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:205)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:171)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:112)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:201)
> at 
> org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:484)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at 
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:474)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:193)
> ... 8 more
> Caused by: java.lang.IllegalArgumentException: nanos > 9 or < 0
> at java.sql.Timestamp.setNanos(Timestamp.java:383)
> at 
> org.apache.hadoop.hive.ql.exec.vector.TimestampUtils.assignTimeInNanoSec(TimestampUtils.java:27)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$1.writeValue(VectorExpressionWriterFactory.java:412)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:162)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:152)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.processOp(VectorFileSinkOperator.java:85)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:129)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:93)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> ... 9 more
> {noformat}
> Full log is attached.
> Schema for the table i

[jira] [Resolved] (HIVE-5995) decimal_precision.q test case fails in trunk

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis resolved HIVE-5995.
-

Resolution: Fixed

Committed to trunk for quick fix. Thanks, Prasanth!

> decimal_precision.q test case fails in trunk
> 
>
> Key: HIVE-5995
> URL: https://issues.apache.org/jira/browse/HIVE-5995
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: test
> Attachments: HIVE-5995.1.patch
>
>
> decimal_precision test case fails reporting diff of one additional blank line.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5994) ORC RLEv2 decodes wrongly for large negative BIGINTs (64 bits )

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844012#comment-13844012
 ] 

Hive QA commented on HIVE-5994:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617962/HIVE-5994.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4762 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/591/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/591/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12617962

> ORC RLEv2 decodes wrongly for large negative BIGINTs  (64 bits )
> 
>
> Key: HIVE-5994
> URL: https://issues.apache.org/jira/browse/HIVE-5994
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5994.1.patch
>
>
> For large negative BIGINTs, zigzag encoding will yield large value (64bit 
> value) with MSB set to 1. This value is interpreted as negative value in 
> SerializationUtils.findClosestNumBits(long value) function. This resulted in 
> wrong computation of total number of bits required which results in wrong 
> encoding/decoding of values.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5945) ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those tables which are not used in the child of this conditional task.

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5945:


Priority: Critical  (was: Major)

> ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask also sums those 
> tables which are not used in the child of this conditional task.
> -
>
> Key: HIVE-5945
> URL: https://issues.apache.org/jira/browse/HIVE-5945
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0
>Reporter: Yin Huai
>Priority: Critical
>
> Here is an example
> {code}
> select
>i_item_id,
>s_state,
>avg(ss_quantity) agg1,
>avg(ss_list_price) agg2,
>avg(ss_coupon_amt) agg3,
>avg(ss_sales_price) agg4
> FROM store_sales
> JOIN date_dim on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)
> JOIN item on (store_sales.ss_item_sk = item.i_item_sk)
> JOIN customer_demographics on (store_sales.ss_cdemo_sk = 
> customer_demographics.cd_demo_sk)
> JOIN store on (store_sales.ss_store_sk = store.s_store_sk)
> where
>cd_gender = 'F' and
>cd_marital_status = 'U' and
>cd_education_status = 'Primary' and
>d_year = 2002 and
>s_state in ('GA','PA', 'LA', 'SC', 'MI', 'AL')
> group by
>i_item_id,
>s_state
> order by
>i_item_id,
>s_state
> limit 100;
> {\code}
> I turned off noconditionaltask. So, I expected that there will be 4 Map-only 
> jobs for this query. However, I got 1 Map-only job (joining strore_sales and 
> date_dim) and 3 MR job (for reduce joins.)
> So, I checked the conditional task determining the plan of the join involving 
> item. In ql.plan.ConditionalResolverCommonJoin.resolveMapJoinTask, 
> aliasToFileSizeMap contains all input tables used in this query and the 
> intermediate table generated by joining store_sales and date_dim. So, when we 
> sum the size of all small tables, the size of store_sales (which is around 
> 45GB in my test) will be also counted.  



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-5997) Replace SemanticException with HiveException for method signatues

[Navis (JIRA)]

Navis created HIVE-5997:
---

 Summary: Replace SemanticException with HiveException for method 
signatues
 Key: HIVE-5997
 URL: https://issues.apache.org/jira/browse/HIVE-5997
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Trivial


There is so many codes just wrapping HiveException to SemanticException in 
planing stage which seemed totally meaningless. How about replacing it all?



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission

[Phabricator (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phabricator updated HIVE-2093:
--

Attachment: D12807.4.patch

navis updated the revision "HIVE-2093 [jira] create/drop database should 
populate inputs/outputs and check concurrency and user permission".

  addressed comments

Reviewers: JIRA

REVISION DETAIL
  https://reviews.facebook.net/D12807

CHANGE SINCE LAST DIFF
  https://reviews.facebook.net/D12807?vs=41577&id=45177#toc

AFFECTED FILES
  hbase-handler/src/test/results/negative/cascade_dbdrop_hadoop20.q.out
  
hcatalog/core/src/main/java/org/apache/hcatalog/security/HdfsAuthorizationProvider.java
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java
  ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/LockDatabaseDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/ShowLocksDesc.java
  ql/src/java/org/apache/hadoop/hive/ql/plan/UnlockDatabaseDesc.java
  ql/src/test/queries/clientnegative/authorization_fail_create_db.q
  ql/src/test/queries/clientnegative/authorization_fail_drop_db.q
  ql/src/test/queries/clientnegative/lockneg_query_tbl_in_locked_db.q
  ql/src/test/queries/clientnegative/lockneg_try_db_lock_conflict.q
  ql/src/test/queries/clientnegative/lockneg_try_drop_locked_db.q
  ql/src/test/queries/clientnegative/lockneg_try_lock_db_in_use.q
  ql/src/test/results/clientnegative/authorization_fail_2.q.out
  ql/src/test/results/clientnegative/authorization_fail_3.q.out
  ql/src/test/results/clientnegative/authorization_fail_4.q.out
  ql/src/test/results/clientnegative/authorization_fail_5.q.out
  ql/src/test/results/clientnegative/authorization_fail_6.q.out
  ql/src/test/results/clientnegative/authorization_fail_7.q.out
  ql/src/test/results/clientnegative/authorization_fail_create_db.q.out
  ql/src/test/results/clientnegative/authorization_fail_drop_db.q.out
  ql/src/test/results/clientnegative/authorization_part.q.out
  ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out
  ql/src/test/results/clientnegative/database_drop_not_empty.q.out
  ql/src/test/results/clientnegative/database_drop_not_empty_restrict.q.out
  ql/src/test/results/clientnegative/exim_22_export_authfail.q.out
  ql/src/test/results/clientnegative/exim_23_import_exist_authfail.q.out
  ql/src/test/results/clientnegative/exim_24_import_part_authfail.q.out
  ql/src/test/results/clientnegative/exim_25_import_nonexist_authfail.q.out
  ql/src/test/results/clientnegative/join_nonexistent_part.q.out
  ql/src/test/results/clientnegative/load_exist_part_authfail.q.out
  ql/src/test/results/clientnegative/load_nonpart_authfail.q.out
  ql/src/test/results/clientnegative/load_part_authfail.q.out
  ql/src/test/results/clientnegative/lockneg_query_tbl_in_locked_db.q.out
  ql/src/test/results/clientnegative/lockneg_try_db_lock_conflict.q.out
  ql/src/test/results/clientnegative/lockneg_try_drop_locked_db.q.out
  ql/src/test/results/clientnegative/lockneg_try_lock_db_in_use.q.out
  ql/src/test/results/clientpositive/alter1.q.out
  ql/src/test/results/clientpositive/alter2.q.out
  ql/src/test/results/clientpositive/alter4.q.out
  ql/src/test/results/clientpositive/authorization_5.q.out
  ql/src/test/results/clientpositive/database.q.out
  ql/src/test/results/clientpositive/database_drop.q.out
  ql/src/test/results/clientpositive/describe_database_json.q.out
  ql/src/test/results/clientpositive/drop_database_removes_partition_dirs.q.out
  ql/src/test/results/clientpositive/exim_00_nonpart_empty.q.out
  ql/src/test/results/clientpositive/exim_01_nonpart.q.out
  ql/src/test/results/clientpositive/exim_02_00_part_empty.q.out
  ql/src/test/results/clientpositive/exim_02_part.q.out
  ql/src/test/results/clientpositive/exim_03_nonpart_over_compat.q.out
  ql/src/test/results/clientpositive/exim_04_all_part.q.out
  ql/src/test/results/clientpositive/exim_04_evolved_parts.q.out
  ql/src/test/results/clientpositive/exim_05_some_part.q.out
  ql/src/test/results/clientpositive/exim_06_one_part.q.out
  ql/src/test/results/clientpositive/exim_07_all_part_over_nonoverlap.q.out
  ql/src/test/results/clientpositive/exim_

[jira] [Commented] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843991#comment-13843991
 ] 

Ashutosh Chauhan commented on HIVE-5936:


+1

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt, HIVE-5936.8.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5936:


Status: Patch Available  (was: Open)

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt, HIVE-5936.8.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5936:


Attachment: HIVE-5936.8.patch.txt

Addressed comments

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt, HIVE-5936.8.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 15993: analyze command failing to collect stats with counter mechanism

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
---

(Updated Dec. 10, 2013, 6:03 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936


Repository: hive-git


Description
---

With counter mechanism, MR job is successful, but StatsTask on client fails 
with NPE


Diffs (updated)
-

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java 
a9c3136 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
d0e94bf 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f 
  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 
7e701f4 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java af729e6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7443ea4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d0a0ec7 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03 
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 
fa430eb 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 
fb5f50e 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 
8c23b87 
  ql/src/test/queries/clientpositive/stats_counter.q 20769e4 
  ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b 
  ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5 
  ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94 
  ql/src/test/results/clientpositive/truncate_column.q.out a247c4a 

Diff: https://reviews.apache.org/r/15993/diff/


Testing
---


Thanks,

Navis Ryu

Re: Review Request 15993: analyze command failing to collect stats with counter mechanism

[Navis Ryu]

> On Dec. 10, 2013, 2:56 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java, line 960
> > 
> >
> > I don't think we need to call init() from task nodes. Semantics is : 
> > init() is called once from client before job is launched. Is there some 
> > reason to call init() here now?

It's mistakenly included while testing. But for "jdbc:derby" type with 
non-local mode execution, it throws exception that "there is no such table", 
etc. Maybe we might need some handling for this case (ignore?)


> On Dec. 10, 2013, 2:56 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java, line 970
> > 
> >
> > This can simply be written as
> >String spSpec = conf.getStaticSpec();

Right. It's simple, yeah.


> On Dec. 10, 2013, 2:56 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 2299
> > 
> >
> > Not necessary to fix todo in this jira. But will be good an example 
> > where prefix could be longer than maxPrefixLength.

ok.


> On Dec. 10, 2013, 2:56 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java, line 2318
> > 
> >
> > Looks like this method is no longer used anywhere. if so, can be 
> > removed.

Soley used in getHashedStatsPrefix(). I've inlined that.


- Navis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/#review30071
---


On Dec. 10, 2013, 12:42 a.m., Navis Ryu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15993/
> ---
> 
> (Updated Dec. 10, 2013, 12:42 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-5936
> https://issues.apache.org/jira/browse/HIVE-5936
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE
> 
> 
> Diffs
> -
> 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java 
> a9c3136 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> d0e94bf 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 
> 7e701f4 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> af729e6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 7443ea4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d0a0ec7 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 
> fa430eb 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 
> fb5f50e 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 
> 8c23b87 
>   ql/src/test/queries/clientpositive/stats_counter.q 20769e4 
>   ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b 
>   ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5 
>   ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607 
>   ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94 
>   ql/src/test/results/clientpositive/truncate_column.q.out a247c4a 
> 
> Diff: https://reviews.apache.org/r/15993/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

[jira] [Updated] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5936:


Status: Open  (was: Patch Available)

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5276) Skip redundant string encoding/decoding for hiveserver2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843986#comment-13843986
 ] 

Hive QA commented on HIVE-5276:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617958/HIVE-5276.6.patch.txt

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4761 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/590/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/590/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12617958

> Skip redundant string encoding/decoding for hiveserver2
> ---
>
> Key: HIVE-5276
> URL: https://issues.apache.org/jira/browse/HIVE-5276
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5276.3.patch.txt, HIVE-5276.4.patch.txt, 
> HIVE-5276.5.patch.txt, HIVE-5276.6.patch.txt
>
>
> Current hiveserver2 acquires rows in string format which is used for cli 
> output. Then convert them into row again and convert to final format lastly. 
> This is inefficient and memory consuming. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5996) Query for sum of a long column of a table with only two rows produces wrong result

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843972#comment-13843972
 ] 

Xuefu Zhang commented on HIVE-5996:
---

For information, the follow is the text from SQL-92[1] standard w.r.t SUM 
function:
{quote}
b) If SUM is specified and DT is exact numeric with scale
  S, then the data type of the result is exact numeric with
  implementation-defined precision and scale S.
{quote}
For DT as long, currently Hive returns long, which doesn't violate the 
standard. However, such implementation is problematic as demonstrated in this 
JIRA. Plus, for decimal, Hive sum function accommodates at least 10 billion 
rows of data. Letting sum(long) return long is not able to uphold that 
assurance. Thus, we need to change the return type to make the function more 
useful.

[1] http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt


> Query for sum of a long column of a table with only two rows produces wrong 
> result
> --
>
> Key: HIVE-5996
> URL: https://issues.apache.org/jira/browse/HIVE-5996
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>
> {code}
> hive> desc test2;
> OK
> l bigint  None
> hive> select * from test2; 
> OK
> 666
> 555
> hive> select sum(l) from test2;
> OK
> -6224521851487329395
> {code}
> It's believed that a wrap-around error occurred. It's surprising that it 
> happens only with two rows. Same query in MySQL returns:
> {code}
> mysql> select sum(l) from test;
> +--+
> | sum(l)   |
> +--+
> | 1221 |
> +--+
> 1 row in set (0.00 sec)
> {code}
> Hive should accommodate large number of rows. Overflowing with only two rows 
> is very unusable.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5995) decimal_precision.q test case fails in trunk

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843966#comment-13843966
 ] 

Navis commented on HIVE-5995:
-

HIVE-5598 removed last dummy line. Seemed caused by that.

> decimal_precision.q test case fails in trunk
> 
>
> Key: HIVE-5995
> URL: https://issues.apache.org/jira/browse/HIVE-5995
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: test
> Attachments: HIVE-5995.1.patch
>
>
> decimal_precision test case fails reporting diff of one additional blank line.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5973) SMB joins produce incorrect results with multiple partitions and buckets

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843946#comment-13843946
 ] 

Vikram Dixit K commented on HIVE-5973:
--

It is quite easy to reproduce this on a cluster but I haven't had success
with our unit tests. I will come up with one and post it here.

Thanks
Vikram.






-- 
Nothing better than when appreciated for hard work.
-Mark


> SMB joins produce incorrect results with multiple partitions and buckets
> 
>
> Key: HIVE-5973
> URL: https://issues.apache.org/jira/browse/HIVE-5973
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Fix For: 0.13.0
>
>
> It looks like there is an issue with re-using the output object array in the 
> select operator. When we read rows of the non-big tables, we hold on to the 
> output object in the priority queue. This causes hive to produce incorrect 
> results because all the elements in the priority queue refer to the same 
> object and the join happens on only one of the buckets.
> {noformat}
> output[i] = eval[i].evaluate(row);
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5961) Add explain authorize for checking privileges

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843921#comment-13843921
 ] 

Hive QA commented on HIVE-5961:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617947/HIVE-5961.3.patch.txt

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4762 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/589/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/589/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12617947

> Add explain authorize for checking privileges
> -
>
> Key: HIVE-5961
> URL: https://issues.apache.org/jira/browse/HIVE-5961
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt, 
> HIVE-5961.3.patch.txt
>
>
> For easy checking of need privileges for a query, 
> {noformat}
> explain authorize select * from src join srcpart
> INPUTS: 
>   default@srcpart
>   default@srcpart@ds=2008-04-08/hr=11
>   default@srcpart@ds=2008-04-08/hr=12
>   default@srcpart@ds=2008-04-09/hr=11
>   default@srcpart@ds=2008-04-09/hr=12
>   default@src
> OUTPUTS: 
>   
> file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
> CURRENT_USER: 
>   hive_test_user
> OPERATION: 
>   QUERY
> AUTHORIZATION_FAILURES: 
>   No privilege 'Select' found for inputs { database:default, table:srcpart, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
> {noformat}
> Hopefully good for debugging of authorization, which is in progress on 
> HIVE-5837.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5783) Native Parquet Support in Hive

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843918#comment-13843918
 ] 

Carl Steinbach commented on HIVE-5783:
--

bq. on the parquet-hive side, we're good to submit a new patch with direct 
serde integration

:)

bq. I humbly submit that the two are not linked and one should not impede the 
other.

I agree. It wasn't my intention to imply that these issues were linked. Sorry 
if that wasn't clear.

In addition to the SerDe can please also include some test cases? I think it 
would be good to aim for coverage on par with what was provided with OrcFile. 
Also, the data/files directory contains two files (alltypes.txt and 
alltypesorc) which will make testing type support a lot easier.

> Native Parquet Support in Hive
> --
>
> Key: HIVE-5783
> URL: https://issues.apache.org/jira/browse/HIVE-5783
> Project: Hive
>  Issue Type: New Feature
>Reporter: Justin Coffey
>Assignee: Justin Coffey
>Priority: Minor
> Fix For: 0.11.0
>
> Attachments: HIVE-5783.patch, hive-0.11-parquet.patch
>
>
> Problem Statement:
> Hive would be easier to use if it had native Parquet support. Our 
> organization, Criteo, uses Hive extensively. Therefore we built the Parquet 
> Hive integration and would like to now contribute that integration to Hive.
> About Parquet:
> Parquet is a columnar storage format for Hadoop and integrates with many 
> Hadoop ecosystem tools such as Thrift, Avro, Hadoop MapReduce, Cascading, 
> Pig, Drill, Crunch, and Hive. Pig, Crunch, and Drill all contain native 
> Parquet integration.
> Changes Details:
> Parquet was built with dependency management in mind and therefore only a 
> single Parquet jar will be added as a dependency.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5992) Hive inconsistently converts timestamp in AVG and SUM UDAF's

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-5992:
--

Component/s: UDF

> Hive inconsistently converts timestamp in AVG and SUM UDAF's
> 
>
> Key: HIVE-5992
> URL: https://issues.apache.org/jira/browse/HIVE-5992
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>
> {code}
> hive> select t, sum(t), count(*), sum(t)/count(*), avg(t) from ts group by t;
> ...
> OK
> 1977-03-15 12:34:22.345678 227306062  1  227306062
> 2.27306062345678E8
> {code}
> As it can be seen, timestamp value (1977-03-15 12:34:22.345678) is converted 
> with fractional part ignored in sum, while preserved in avg. As a further 
> result, sum()/count() is not equivalent to avg.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-5996) Query for sum of a long column of a table with only two rows produces wrong result

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-5996:
-

 Summary: Query for sum of a long column of a table with only two 
rows produces wrong result
 Key: HIVE-5996
 URL: https://issues.apache.org/jira/browse/HIVE-5996
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


{quote}
hive> desc test2;
OK
l   bigint  None
hive> select * from test2; 
OK
666
555
hive> select sum(l) from test2;
OK
-6224521851487329395
{quote}
It's believed that a wrap-around error occurred. It's surprising that it 
happens only with two rows. Same query in MySQL returns:
{quote}
mysql> select sum(l) from test;
+--+
| sum(l)   |
+--+
| 1221 |
+--+
1 row in set (0.00 sec)
{quote}
Hive should accommodate large number of rows. Overflowing with only two rows is 
very unusable.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5996) Query for sum of a long column of a table with only two rows produces wrong result

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-5996:
--

Description: 
{code}
hive> desc test2;
OK
l   bigint  None
hive> select * from test2; 
OK
666
555
hive> select sum(l) from test2;
OK
-6224521851487329395
{code}
It's believed that a wrap-around error occurred. It's surprising that it 
happens only with two rows. Same query in MySQL returns:
{code}
mysql> select sum(l) from test;
+--+
| sum(l)   |
+--+
| 1221 |
+--+
1 row in set (0.00 sec)
{code}
Hive should accommodate large number of rows. Overflowing with only two rows is 
very unusable.

  was:
{quote}
hive> desc test2;
OK
l   bigint  None
hive> select * from test2; 
OK
666
555
hive> select sum(l) from test2;
OK
-6224521851487329395
{quote}
It's believed that a wrap-around error occurred. It's surprising that it 
happens only with two rows. Same query in MySQL returns:
{quote}
mysql> select sum(l) from test;
+--+
| sum(l)   |
+--+
| 1221 |
+--+
1 row in set (0.00 sec)
{quote}
Hive should accommodate large number of rows. Overflowing with only two rows is 
very unusable.


> Query for sum of a long column of a table with only two rows produces wrong 
> result
> --
>
> Key: HIVE-5996
> URL: https://issues.apache.org/jira/browse/HIVE-5996
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>
> {code}
> hive> desc test2;
> OK
> l bigint  None
> hive> select * from test2; 
> OK
> 666
> 555
> hive> select sum(l) from test2;
> OK
> -6224521851487329395
> {code}
> It's believed that a wrap-around error occurred. It's surprising that it 
> happens only with two rows. Same query in MySQL returns:
> {code}
> mysql> select sum(l) from test;
> +--+
> | sum(l)   |
> +--+
> | 1221 |
> +--+
> 1 row in set (0.00 sec)
> {code}
> Hive should accommodate large number of rows. Overflowing with only two rows 
> is very unusable.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5524) Unwanted delay in getting Hive metastore connection with METASTORE_CLIENT_CONNECT_RETRY_DELAY/

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843905#comment-13843905
 ] 

Brock Noland commented on HIVE-5524:


+1 if tests pass

> Unwanted delay in getting Hive metastore connection with 
> METASTORE_CLIENT_CONNECT_RETRY_DELAY/
> --
>
> Key: HIVE-5524
> URL: https://issues.apache.org/jira/browse/HIVE-5524
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Rajesh Balamohan
> Attachments: HIVE-5524.patch
>
>
> Reference:  
> http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> 
>  for (URI store : metastoreUris) {
> 
> ...
>  if (isConnected) {
>   break;
> }
>   }
> // Wait before launching the next round of connection retries.
>   if (retryDelaySeconds > 0) {
> try {
>   LOG.info("Waiting " + retryDelaySeconds + " seconds before next 
> connection attempt.");
>   Thread.sleep(retryDelaySeconds * 1000);
> } catch (InterruptedException ignore) {}
>   }
> 
> By default "hive.metastore.client.connect.retry.delay" is set to 1 second.  
> If it is set to 10 seconds, this code will wait for 10 seconds even if a 
> successful connection is made in first attempt itself.
> This can be avoided by changing to 
> 
>  if (!isConnected && retryDelaySeconds > 0) {
> 
> 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5524) Unwanted delay in getting Hive metastore connection with METASTORE_CLIENT_CONNECT_RETRY_DELAY/

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated HIVE-5524:
--

Affects Version/s: (was: 0.11.0)
   Status: Patch Available  (was: Open)

Marking as Patch Available for review.

> Unwanted delay in getting Hive metastore connection with 
> METASTORE_CLIENT_CONNECT_RETRY_DELAY/
> --
>
> Key: HIVE-5524
> URL: https://issues.apache.org/jira/browse/HIVE-5524
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Rajesh Balamohan
> Attachments: HIVE-5524.patch
>
>
> Reference:  
> http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
> 
>  for (URI store : metastoreUris) {
> 
> ...
>  if (isConnected) {
>   break;
> }
>   }
> // Wait before launching the next round of connection retries.
>   if (retryDelaySeconds > 0) {
> try {
>   LOG.info("Waiting " + retryDelaySeconds + " seconds before next 
> connection attempt.");
>   Thread.sleep(retryDelaySeconds * 1000);
> } catch (InterruptedException ignore) {}
>   }
> 
> By default "hive.metastore.client.connect.retry.delay" is set to 1 second.  
> If it is set to 10 seconds, this code will wait for 10 seconds even if a 
> successful connection is made in first attempt itself.
> This can be avoided by changing to 
> 
>  if (!isConnected && retryDelaySeconds > 0) {
> 
> 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5983) Fix name of ColumnProjectionUtils.appendReadColumnIDs

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-5983:
---

   Resolution: Fixed
Fix Version/s: 0.13.0
   Status: Resolved  (was: Patch Available)

> Fix name of ColumnProjectionUtils.appendReadColumnIDs
> -
>
> Key: HIVE-5983
> URL: https://issues.apache.org/jira/browse/HIVE-5983
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5983.patch
>
>
> In HIVE-5855 I tried to remove a backwards incompatibility, but I named the 
> method appendReadColumnID where it should be appendReadColumnIDs:
> https://github.com/apache/hive/blob/branch-0.12/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java#L51



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission

[Phabricator (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843900#comment-13843900
 ] 

Phabricator commented on HIVE-2093:
---

navis has commented on the revision "HIVE-2093 [jira] create/drop database 
should populate inputs/outputs and check concurrency and user permission".

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java:510 I've copied it from 
somewhere, forget where it was. Could you suggest better state code?
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java:737 Ah, sure.
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:257 This method 
seemed not used in anywhere in current codebase, so it's hard to say the intent 
of initial contributor(Siying Dong). But this method seemed useful and I'll 
change it to return URI for database, too.
  ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java:83 Some complex 
operations like "import" need different privilege checks with that of 
initiative operation. I think this part is from HIVE-2818 and mistakenly mixed 
with this issue. I'll remove it.
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:2290 ok.
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:1233 ok.
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java:104 
ok.

REVISION DETAIL
  https://reviews.facebook.net/D12807

To: JIRA, navis
Cc: thejas


> create/drop database should populate inputs/outputs and check concurrency and 
> user permission
> -
>
> Key: HIVE-2093
> URL: https://issues.apache.org/jira/browse/HIVE-2093
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, Locking, Metastore, Security
>Reporter: Namit Jain
>Assignee: Navis
> Attachments: D12807.3.patch, HIVE-2093.6.patch, 
> HIVE-2093.7.patch.txt, HIVE-2093.8.patch.txt, HIVE-2093.D12807.1.patch, 
> HIVE-2093.D12807.2.patch, HIVE.2093.1.patch, HIVE.2093.2.patch, 
> HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.5.patch
>
>
> concurrency and authorization are needed for create/drop table. Also to make 
> concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS 
> DATABASE



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5983) Fix name of ColumnProjectionUtils.appendReadColumnIDs

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843895#comment-13843895
 ] 

Brock Noland commented on HIVE-5983:


bq.  Sorry for missed that.

No worries it was my fault :)

> Fix name of ColumnProjectionUtils.appendReadColumnIDs
> -
>
> Key: HIVE-5983
> URL: https://issues.apache.org/jira/browse/HIVE-5983
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5983.patch
>
>
> In HIVE-5855 I tried to remove a backwards incompatibility, but I named the 
> method appendReadColumnID where it should be appendReadColumnIDs:
> https://github.com/apache/hive/blob/branch-0.12/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java#L51



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5993) JDBC Driver should not hard-code the database name

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843891#comment-13843891
 ] 

Hive QA commented on HIVE-5993:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617944/HIVE-5993.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4761 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/588/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/588/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12617944

> JDBC Driver should not hard-code the database name
> --
>
> Key: HIVE-5993
> URL: https://issues.apache.org/jira/browse/HIVE-5993
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-5993.patch
>
>
> Method HiveDatabaseMetadata.getDatabaseProductName() returns a hard-coded 
> string "hive".
> This should instead call the existing Hive-server2 api to return the db name.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5995) decimal_precision.q test case fails in trunk

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843890#comment-13843890
 ] 

Xuefu Zhang commented on HIVE-5995:
---

[~prasanth_j] Thanks for bringing this to my attention. I believe this was 
caused by a merge. My bad. Your patch looked good. +1.

I will commit it right way after tests run.

> decimal_precision.q test case fails in trunk
> 
>
> Key: HIVE-5995
> URL: https://issues.apache.org/jira/browse/HIVE-5995
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: test
> Attachments: HIVE-5995.1.patch
>
>
> decimal_precision test case fails reporting diff of one additional blank line.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5897) Fix hadoop2 execution environment Milestone 2

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843888#comment-13843888
 ] 

Brock Noland commented on HIVE-5897:


+1, thanks Vikram!

> Fix hadoop2 execution environment Milestone 2
> -
>
> Key: HIVE-5897
> URL: https://issues.apache.org/jira/browse/HIVE-5897
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Brock Noland
>Assignee: Vikram Dixit K
> Attachments: HIVE-5897.4.patch, HIVE-5897.5.patch, HIVE-5897.patch, 
> HIVE-5897.patch, HIVE-5897.patch
>
>
> Follow on to HIVE-5755.
> List of known issues:
> hcatalog-pig-adapter and ql need
> {noformat}
>  
>org.apache.hadoop
>hadoop-mapreduce-client-common
>${hadoop-23.version}
>test
> 
> {noformat}
> hcatalog core and hbase storage handler needs 
> {noformat}
>  
>org.apache.hadoop
>   hadoop-common
>   ${hadoop-23.version}
>   tests
>   test
> 
> 
>   org.apache.hadoop
>   hadoop-mapreduce-client-hs
>   ${hadoop-23.version}
>   test
> 
>
>  org.apache.hadoop
>  hadoop-yarn-server-tests
>  ${hadoop-23.version}
>  tests
>  test
>
> {noformat}
> hcatalog core needs:
> {noformat}
>
>   org.apache.hadoop
>   hadoop-mapreduce-client-jobclient
>   ${hadoop-23.version}
>   test
> 
> {noformat}
> beeline needs 
> {noformat}
> 
>   org.apache.hadoop
>   hadoop-mapreduce-client-core
>   ${hadoop-23.version}
>   test
> 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5878) Hive standard avg UDAF returns double as the return type for some exact input types

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843882#comment-13843882
 ] 

Xuefu Zhang commented on HIVE-5878:
---

For information, the follow is the text from SQL-92[1] standard w.r.t AVG 
function:

{quote}
c) If AVG is specified and DT is exact numeric, then the data
  type of the result is exact numeric with implementation-
  defined precision not less than the precision of DT and
  implementation-defined scale not less than the scale of DT.
{quote}

Clearly, currently Hive deviates from this. Both MySQL and SQL server are in 
line with this.

[1] http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt

> Hive standard avg UDAF returns double as the return type for some exact input 
> types
> ---
>
> Key: HIVE-5878
> URL: https://issues.apache.org/jira/browse/HIVE-5878
> Project: Hive
>  Issue Type: Bug
>  Components: Types, UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-5878.1.patch, HIVE-5878.patch
>
>
> For standard, no-partial avg result, hive currently returns double as the 
> result type.
> {code}
> hive> desc test;
> OK
> d int None
> Time taken: 0.051 seconds, Fetched: 1 row(s)
> hive> explain select avg(`d`) from test;  
> ...
>   Reduce Operator Tree:
> Group By Operator
>   aggregations:
> expr: avg(VALUE._col0)
>   bucketGroup: false
>   mode: mergepartial
>   outputColumnNames: _col0
>   Select Operator
> expressions:
>   expr: _col0
>   type: double
> {code}
> However, exact types including integers and decimal should yield exact type. 
> Here is what MySQL does:
> {code}
> mysql> desc test;
> +---+--+--+-+-+---+
> | Field | Type | Null | Key | Default | Extra |
> +---+--+--+-+-+---+
> | i | int(11)  | YES  | | NULL|   |
> | b | tinyint(1)   | YES  | | NULL|   |
> | d | double   | YES  | | NULL|   |
> | s | varchar(5)   | YES  | | NULL|   |
> | dd| decimal(5,2) | YES  | | NULL|   |
> +---+--+--+-+-+---+
> mysql> create table test62 as select avg(i) from test;
> mysql> desc test62;
> +---+---+--+-+-+---+
> | Field | Type  | Null | Key | Default | Extra |
> +---+---+--+-+-+---+
> | avg(i) | decimal(14,4) | YES  | | NULL|   |
> +---+---+--+-+-+---+
> 1 row in set (0.00 sec)
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5979) Failure in cast to timestamps.

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5979:
---

Attachment: HIVE-5979.1.patch

The attached patch fixes the above issue. A test case is also added that 
reproduces the issue without this patch. 

> Failure in cast to timestamps.
> --
>
> Key: HIVE-5979
> URL: https://issues.apache.org/jira/browse/HIVE-5979
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5979.1.patch
>
>
> Query ran:
> {code}
> select cast(t as timestamp), cast(si as timestamp),
>cast(i as timestamp), cast(b as timestamp),
>cast(f as string), cast(d as timestamp),
>cast(bo as timestamp), cast(b * 0 as timestamp),
>cast(ts as timestamp), cast(s as timestamp),
>cast(substr(s, 1, 1) as timestamp)
> from Table1;
> {code}
> Running this query with hive.vectorized.execution.enabled=true fails with the 
> following exception:
> {noformat}
> 13/12/05 07:56:36 ERROR tez.TezJobMonitor: Status: Failed
> Vertex failed, vertexName=Map 1, vertexId=vertex_1386227234886_0482_1_00, 
> diagnostics=[Task failed, taskId=task_1386227234886_0482_1_00_00, 
> diagnostics=[AttemptID:attempt_1386227234886_0482_1_00_00_0 Info:Error: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:205)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:171)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:112)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:201)
> at 
> org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:484)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at 
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:474)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:193)
> ... 8 more
> Caused by: java.lang.IllegalArgumentException: nanos > 9 or < 0
> at java.sql.Timestamp.setNanos(Timestamp.java:383)
> at 
> org.apache.hadoop.hive.ql.exec.vector.TimestampUtils.assignTimeInNanoSec(TimestampUtils.java:27)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$1.writeValue(VectorExpressionWriterFactory.java:412)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:162)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:152)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.processOp(VectorFileSinkOperator.java:85)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:129)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:93)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> ... 9 more
> {noformat}
> Full log is attached.
> Schema for the table is as follows:
> {code}
> hive> desc Table1;
> OK
> t tinyint from deserializer
> sismallintfrom deserializer
> i int from deserializer
> b bigint  from deserializer
> f float   from deserializer
> d double  from deserializer
> boboolean from deserializer
> s string  from deserializer
> s2string  from deserializer
> tstimestamp   from deserializer
> Time taken: 0.521 seconds, Fetched: 10 row(s)
> {cod

[jira] [Updated] (HIVE-5979) Failure in cast to timestamps.

[Jitendra Nath Pandey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jitendra Nath Pandey updated HIVE-5979:
---

Status: Patch Available  (was: Open)

> Failure in cast to timestamps.
> --
>
> Key: HIVE-5979
> URL: https://issues.apache.org/jira/browse/HIVE-5979
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
> Attachments: HIVE-5979.1.patch
>
>
> Query ran:
> {code}
> select cast(t as timestamp), cast(si as timestamp),
>cast(i as timestamp), cast(b as timestamp),
>cast(f as string), cast(d as timestamp),
>cast(bo as timestamp), cast(b * 0 as timestamp),
>cast(ts as timestamp), cast(s as timestamp),
>cast(substr(s, 1, 1) as timestamp)
> from Table1;
> {code}
> Running this query with hive.vectorized.execution.enabled=true fails with the 
> following exception:
> {noformat}
> 13/12/05 07:56:36 ERROR tez.TezJobMonitor: Status: Failed
> Vertex failed, vertexName=Map 1, vertexId=vertex_1386227234886_0482_1_00, 
> diagnostics=[Task failed, taskId=task_1386227234886_0482_1_00_00, 
> diagnostics=[AttemptID:attempt_1386227234886_0482_1_00_00_0 Info:Error: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:205)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:171)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:112)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:201)
> at 
> org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:484)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at 
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:474)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:193)
> ... 8 more
> Caused by: java.lang.IllegalArgumentException: nanos > 9 or < 0
> at java.sql.Timestamp.setNanos(Timestamp.java:383)
> at 
> org.apache.hadoop.hive.ql.exec.vector.TimestampUtils.assignTimeInNanoSec(TimestampUtils.java:27)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$1.writeValue(VectorExpressionWriterFactory.java:412)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:162)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:152)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.processOp(VectorFileSinkOperator.java:85)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:129)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:93)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> ... 9 more
> {noformat}
> Full log is attached.
> Schema for the table is as follows:
> {code}
> hive> desc Table1;
> OK
> t tinyint from deserializer
> sismallintfrom deserializer
> i int from deserializer
> b bigint  from deserializer
> f float   from deserializer
> d double  from deserializer
> boboolean from deserializer
> s string  from deserializer
> s2string  from deserializer
> tstimestamp   from deserializer
> Time taken: 0.521 seconds, Fetched: 10 row(s)
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5979) Failure in cast to timestamps.

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843878#comment-13843878
 ] 

Jitendra Nath Pandey commented on HIVE-5979:


The reason for error is an edge case with negative times in nanosecond. In the 
following code:
{code}
   public static void assignTimeInNanoSec(long timeInNanoSec, Timestamp t) {
 t.setTime((timeInNanoSec)/100);
 t.setNanos((int)((t.getNanos()) + (timeInNanoSec % 100)));
   }
{code}

The "t.setTime()" call sets the nanos part as well from 
(timeInNanoSec)/100. The remaining nano seconds (timeInNanoSec % 100) 
are added in "t.setNanos()". For negative values of timeInNanoSec it is 
expected that t.getNanos() will be positive and much larger than (timeInNanoSec 
% 100) so that "t.setNanos()" call works fine. But, when 
(timeInNanoSec)/100 yields zero nanoseconds the above logic fails. An 
example number that fails above code is "-179475023828416L".


> Failure in cast to timestamps.
> --
>
> Key: HIVE-5979
> URL: https://issues.apache.org/jira/browse/HIVE-5979
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jitendra Nath Pandey
>Assignee: Jitendra Nath Pandey
>
> Query ran:
> {code}
> select cast(t as timestamp), cast(si as timestamp),
>cast(i as timestamp), cast(b as timestamp),
>cast(f as string), cast(d as timestamp),
>cast(bo as timestamp), cast(b * 0 as timestamp),
>cast(ts as timestamp), cast(s as timestamp),
>cast(substr(s, 1, 1) as timestamp)
> from Table1;
> {code}
> Running this query with hive.vectorized.execution.enabled=true fails with the 
> following exception:
> {noformat}
> 13/12/05 07:56:36 ERROR tez.TezJobMonitor: Status: Failed
> Vertex failed, vertexName=Map 1, vertexId=vertex_1386227234886_0482_1_00, 
> diagnostics=[Task failed, taskId=task_1386227234886_0482_1_00_00, 
> diagnostics=[AttemptID:attempt_1386227234886_0482_1_00_00_0 Info:Error: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:205)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:171)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:112)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:201)
> at 
> org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:484)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at 
> org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:474)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.processRow(MapRecordProcessor.java:193)
> ... 8 more
> Caused by: java.lang.IllegalArgumentException: nanos > 9 or < 0
> at java.sql.Timestamp.setNanos(Timestamp.java:383)
> at 
> org.apache.hadoop.hive.ql.exec.vector.TimestampUtils.assignTimeInNanoSec(TimestampUtils.java:27)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$1.writeValue(VectorExpressionWriterFactory.java:412)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$VectorExpressionWriterLong.writeValue(VectorExpressionWriterFactory.java:162)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch.toString(VectorizedRowBatch.java:152)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.processOp(VectorFileSinkOperator.java:85)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.processOp(VectorSelectOperator.java:129)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:93)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:786)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:43)
> ... 9 more
> {noformat}
> Full log is attached.
> Schema for the table is as follows:
> {code}
> hive> desc Table1;
> OK
> t tinyint from des

[jira] [Commented] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843876#comment-13843876
 ] 

Ashutosh Chauhan commented on HIVE-5936:


Patch looks good. Left some comments on RB.

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 15993: analyze command failing to collect stats with counter mechanism

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/#review30071
---



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java


I don't think we need to call init() from task nodes. Semantics is : init() 
is called once from client before job is launched. Is there some reason to call 
init() here now?



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java


This can simply be written as
   String spSpec = conf.getStaticSpec();



ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java



Will be good to add a comment in javadoc about which strings are contained 
in different positions of array.



ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java


Nice work to batch together these metastore updates.



ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java


invalid semantics (todo) we should probably fix in seperate jira. There I 
think we need to make a distinction between insert overwrite and insert into 
cases, for earlier we can replace invalid stats  with current value but for 
insert into they should remain invalid.



ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java


Same comment about needing init() here.



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java


Not necessary to fix todo in this jira. But will be good an example where 
prefix could be longer than maxPrefixLength.



ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java


Looks like this method is no longer used anywhere. if so, can be removed.


- Ashutosh Chauhan


On Dec. 10, 2013, 12:42 a.m., Navis Ryu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15993/
> ---
> 
> (Updated Dec. 10, 2013, 12:42 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-5936
> https://issues.apache.org/jira/browse/HIVE-5936
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE
> 
> 
> Diffs
> -
> 
>   
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java 
> a9c3136 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> d0e94bf 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b 
>   ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 
> 7e701f4 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> af729e6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> 7443ea4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d0a0ec7 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 
> fa430eb 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 
> fb5f50e 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 
> 8c23b87 
>   ql/src/test/queries/clientpositive/stats_counter.q 20769e4 
>   ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b 
>   ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5 
>   ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607 
>   ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94 
>   ql/src/test/results/clientpositive/truncate_column.q.out a247c4a 
> 
> Diff: https://reviews.apache.org/r/15993/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

[jira] [Updated] (HIVE-5995) decimal_precision.q test case fails in trunk

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5995:
-

Attachment: HIVE-5995.1.patch

[~xuefuz] looking at this commit history 
https://github.com/apache/hive/commit/771fceec84bc4c7f13465e0cfd2a7f580e7945a1#diff-65a6d9e7aa2b97ae5b4e6f2f7e81c63f
 your change inserted a blank line at line no:585. This causes failure of 
decimal_precision.q in precommit tests. I just regenerated the test output 
file. Any idea why that blank like was inserted in your commit?

> decimal_precision.q test case fails in trunk
> 
>
> Key: HIVE-5995
> URL: https://issues.apache.org/jira/browse/HIVE-5995
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: test
> Attachments: HIVE-5995.1.patch
>
>
> decimal_precision test case fails reporting diff of one additional blank line.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-5995) decimal_precision.q test case fails in trunk

[Prasanth J (JIRA)]

Prasanth J created HIVE-5995:


 Summary: decimal_precision.q test case fails in trunk
 Key: HIVE-5995
 URL: https://issues.apache.org/jira/browse/HIVE-5995
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J


decimal_precision test case fails reporting diff of one additional blank line.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843857#comment-13843857
 ] 

Hive QA commented on HIVE-5936:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617941/HIVE-5936.7.patch.txt

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/587/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/587/console

Messages:
{noformat}
 This message was trimmed, see log for full details 
[INFO] Hive Integration - Test Serde
[INFO] Hive Integration - QFile Tests
[INFO] 
[INFO] 
[INFO] Building Hive Integration - Parent 0.13.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it ---
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/itests 
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-it ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it ---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/tmp
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/warehouse
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/tmp/conf
 [copy] Copying 4 files to 
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it ---
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/pom.xml to 
/data/hive-ptest/working/maven/org/apache/hive/hive-it/0.13.0-SNAPSHOT/hive-it-0.13.0-SNAPSHOT.pom
[INFO] 
[INFO] 
[INFO] Building Hive Integration - Custom Serde 0.13.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-custom-serde 
---
[INFO] Deleting 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde (includes 
= [datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
hive-it-custom-serde ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/main/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ 
hive-it-custom-serde ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ 
hive-it-custom-serde ---
[INFO] Compiling 8 source files to 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
hive-it-custom-serde ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/test/resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-custom-serde 
---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/warehouse
[mkdir] Created dir: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp/conf
 [copy] Copying 4 files to 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
hive-it-custom-serde ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ 
hive-it-custom-serde ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-custom-serde ---
[INFO] Building jar: 
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-0.13.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ 
hive-it-custom-serde ---
[INFO] Installing 
/data/hive-ptest/working/apache-svn-trunk-source/itests/

[jira] [Commented] (HIVE-5991) ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843845#comment-13843845
 ] 

Hive QA commented on HIVE-5991:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617931/HIVE-5991.1.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4762 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/586/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/586/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12617931

> ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB 
> encoding 
> ---
>
> Key: HIVE-5991
> URL: https://issues.apache.org/jira/browse/HIVE-5991
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5991.1.patch
>
>
> PATCHED_BLOB encoding creates mask with number of bits required for 95th 
> percentile value. If the 95th percentile value requires 32 bits then the mask 
> creation will result in integer overflow.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5994) ORC RLEv2 decodes wrongly for large negative BIGINTs (64 bits )

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5994:
-

Status: Patch Available  (was: Open)

Making it as patch available for precommit tests.

> ORC RLEv2 decodes wrongly for large negative BIGINTs  (64 bits )
> 
>
> Key: HIVE-5994
> URL: https://issues.apache.org/jira/browse/HIVE-5994
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5994.1.patch
>
>
> For large negative BIGINTs, zigzag encoding will yield large value (64bit 
> value) with MSB set to 1. This value is interpreted as negative value in 
> SerializationUtils.findClosestNumBits(long value) function. This resulted in 
> wrong computation of total number of bits required which results in wrong 
> encoding/decoding of values.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5994) ORC RLEv2 decodes wrongly for large negative BIGINTs (64 bits )

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5994:
-

Attachment: HIVE-5994.1.patch

> ORC RLEv2 decodes wrongly for large negative BIGINTs  (64 bits )
> 
>
> Key: HIVE-5994
> URL: https://issues.apache.org/jira/browse/HIVE-5994
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5994.1.patch
>
>
> For large negative BIGINTs, zigzag encoding will yield large value (64bit 
> value) with MSB set to 1. This value is interpreted as negative value in 
> SerializationUtils.findClosestNumBits(long value) function. This resulted in 
> wrong computation of total number of bits required which results in wrong 
> encoding/decoding of values.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-5994) ORC RLEv2 decodes wrongly for large negative BIGINTs (64 bits )

[Prasanth J (JIRA)]

Prasanth J created HIVE-5994:


 Summary: ORC RLEv2 decodes wrongly for large negative BIGINTs  (64 
bits )
 Key: HIVE-5994
 URL: https://issues.apache.org/jira/browse/HIVE-5994
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J


For large negative BIGINTs, zigzag encoding will yield large value (64bit 
value) with MSB set to 1. This value is interpreted as negative value in 
SerializationUtils.findClosestNumBits(long value) function. This resulted in 
wrong computation of total number of bits required which results in wrong 
encoding/decoding of values.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5276) Skip redundant string encoding/decoding for hiveserver2

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5276:


Attachment: HIVE-5276.6.patch.txt

> Skip redundant string encoding/decoding for hiveserver2
> ---
>
> Key: HIVE-5276
> URL: https://issues.apache.org/jira/browse/HIVE-5276
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5276.3.patch.txt, HIVE-5276.4.patch.txt, 
> HIVE-5276.5.patch.txt, HIVE-5276.6.patch.txt
>
>
> Current hiveserver2 acquires rows in string format which is used for cli 
> output. Then convert them into row again and convert to final format lastly. 
> This is inefficient and memory consuming. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5276) Skip redundant string encoding/decoding for hiveserver2

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5276:


Status: Patch Available  (was: Open)

> Skip redundant string encoding/decoding for hiveserver2
> ---
>
> Key: HIVE-5276
> URL: https://issues.apache.org/jira/browse/HIVE-5276
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5276.3.patch.txt, HIVE-5276.4.patch.txt, 
> HIVE-5276.5.patch.txt, HIVE-5276.6.patch.txt
>
>
> Current hiveserver2 acquires rows in string format which is used for cli 
> output. Then convert them into row again and convert to final format lastly. 
> This is inefficient and memory consuming. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5276) Skip redundant string encoding/decoding for hiveserver2

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5276:


Status: Open  (was: Patch Available)

> Skip redundant string encoding/decoding for hiveserver2
> ---
>
> Key: HIVE-5276
> URL: https://issues.apache.org/jira/browse/HIVE-5276
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-5276.3.patch.txt, HIVE-5276.4.patch.txt, 
> HIVE-5276.5.patch.txt
>
>
> Current hiveserver2 acquires rows in string format which is used for cli 
> output. Then convert them into row again and convert to final format lastly. 
> This is inefficient and memory consuming. 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5356) Move arithmatic UDFs to generic UDF implementations

[Eric Hanson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843795#comment-13843795
 ] 

Eric Hanson commented on HIVE-5356:
---

"Vectorized execution: true" in the explain plain was essentially meant to be 
an assertion that vectorization is working for the query in question. So going 
forward, again, it's important not to regress to non-vectorized execution. If 
it needs to be regressed for a reason, then I'd like to see discussion and 
general agreement or consensus on it. I wish I'd been tuned in to this one -- 
it slipped by me at the time.

> Move arithmatic UDFs to generic UDF implementations
> ---
>
> Key: HIVE-5356
> URL: https://issues.apache.org/jira/browse/HIVE-5356
> Project: Hive
>  Issue Type: Task
>  Components: UDF
>Affects Versions: 0.11.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-5356.1.patch, HIVE-5356.10.patch, 
> HIVE-5356.11.patch, HIVE-5356.12.patch, HIVE-5356.2.patch, HIVE-5356.3.patch, 
> HIVE-5356.4.patch, HIVE-5356.5.patch, HIVE-5356.6.patch, HIVE-5356.7.patch, 
> HIVE-5356.8.patch, HIVE-5356.9.patch
>
>
> Currently, all of the arithmetic operators, such as add/sub/mult/div, are 
> implemented as old-style UDFs and java reflection is used to determine the 
> return type TypeInfos/ObjectInspectors, based on the return type of the 
> evaluate() method chosen for the expression. This works fine for types that 
> don't have type params.
> Hive decimal type participates in these operations just like int or double. 
> Different from double or int, however, decimal has precision and scale, which 
> cannot be determined by just looking at the return type (decimal) of the UDF 
> evaluate() method, even though the operands have certain precision/scale. 
> With the default of "decimal" without precision/scale, then (10, 0) will be 
> the type params. This is certainly not desirable.
> To solve this problem, all of the arithmetic operators would need to be 
> implemented as GenericUDFs, which allow returning ObjectInspector during the 
> initialize() method. The object inspectors returned can carry type params, 
> from which the "exact" return type can be determined.
> It's worth mentioning that, for user UDF implemented in non-generic way, if 
> the return type of the chosen evaluate() method is decimal, the return type 
> actually has (10,0) as precision/scale, which might not be desirable. This 
> needs to be documented.
> This JIRA will cover minus, plus, divide, multiply, mod, and pmod, to limit 
> the scope of review. The remaining ones will be covered under HIVE-5706.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5961) Add explain authorize for checking privileges

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5961:


Status: Patch Available  (was: Open)

> Add explain authorize for checking privileges
> -
>
> Key: HIVE-5961
> URL: https://issues.apache.org/jira/browse/HIVE-5961
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt, 
> HIVE-5961.3.patch.txt
>
>
> For easy checking of need privileges for a query, 
> {noformat}
> explain authorize select * from src join srcpart
> INPUTS: 
>   default@srcpart
>   default@srcpart@ds=2008-04-08/hr=11
>   default@srcpart@ds=2008-04-08/hr=12
>   default@srcpart@ds=2008-04-09/hr=11
>   default@srcpart@ds=2008-04-09/hr=12
>   default@src
> OUTPUTS: 
>   
> file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
> CURRENT_USER: 
>   hive_test_user
> OPERATION: 
>   QUERY
> AUTHORIZATION_FAILURES: 
>   No privilege 'Select' found for inputs { database:default, table:srcpart, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
> {noformat}
> Hopefully good for debugging of authorization, which is in progress on 
> HIVE-5837.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5961) Add explain authorize for checking privileges

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5961:


Attachment: HIVE-5961.3.patch.txt

> Add explain authorize for checking privileges
> -
>
> Key: HIVE-5961
> URL: https://issues.apache.org/jira/browse/HIVE-5961
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt, 
> HIVE-5961.3.patch.txt
>
>
> For easy checking of need privileges for a query, 
> {noformat}
> explain authorize select * from src join srcpart
> INPUTS: 
>   default@srcpart
>   default@srcpart@ds=2008-04-08/hr=11
>   default@srcpart@ds=2008-04-08/hr=12
>   default@srcpart@ds=2008-04-09/hr=11
>   default@srcpart@ds=2008-04-09/hr=12
>   default@src
> OUTPUTS: 
>   
> file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
> CURRENT_USER: 
>   hive_test_user
> OPERATION: 
>   QUERY
> AUTHORIZATION_FAILURES: 
>   No privilege 'Select' found for inputs { database:default, table:srcpart, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
> {noformat}
> Hopefully good for debugging of authorization, which is in progress on 
> HIVE-5837.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5961) Add explain authorize for checking privileges

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5961:


Attachment: (was: HIVE-5961.3.patch.txt)

> Add explain authorize for checking privileges
> -
>
> Key: HIVE-5961
> URL: https://issues.apache.org/jira/browse/HIVE-5961
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt, 
> HIVE-5961.3.patch.txt
>
>
> For easy checking of need privileges for a query, 
> {noformat}
> explain authorize select * from src join srcpart
> INPUTS: 
>   default@srcpart
>   default@srcpart@ds=2008-04-08/hr=11
>   default@srcpart@ds=2008-04-08/hr=12
>   default@srcpart@ds=2008-04-09/hr=11
>   default@srcpart@ds=2008-04-09/hr=12
>   default@src
> OUTPUTS: 
>   
> file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
> CURRENT_USER: 
>   hive_test_user
> OPERATION: 
>   QUERY
> AUTHORIZATION_FAILURES: 
>   No privilege 'Select' found for inputs { database:default, table:srcpart, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
> {noformat}
> Hopefully good for debugging of authorization, which is in progress on 
> HIVE-5837.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5961) Add explain authorize for checking privileges

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5961:


Status: Open  (was: Patch Available)

> Add explain authorize for checking privileges
> -
>
> Key: HIVE-5961
> URL: https://issues.apache.org/jira/browse/HIVE-5961
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt, 
> HIVE-5961.3.patch.txt
>
>
> For easy checking of need privileges for a query, 
> {noformat}
> explain authorize select * from src join srcpart
> INPUTS: 
>   default@srcpart
>   default@srcpart@ds=2008-04-08/hr=11
>   default@srcpart@ds=2008-04-08/hr=12
>   default@srcpart@ds=2008-04-09/hr=11
>   default@srcpart@ds=2008-04-09/hr=12
>   default@src
> OUTPUTS: 
>   
> file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
> CURRENT_USER: 
>   hive_test_user
> OPERATION: 
>   QUERY
> AUTHORIZATION_FAILURES: 
>   No privilege 'Select' found for inputs { database:default, table:srcpart, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
> {noformat}
> Hopefully good for debugging of authorization, which is in progress on 
> HIVE-5837.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5961) Add explain authorize for checking privileges

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5961:


Attachment: HIVE-5961.3.patch.txt

> Add explain authorize for checking privileges
> -
>
> Key: HIVE-5961
> URL: https://issues.apache.org/jira/browse/HIVE-5961
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt, 
> HIVE-5961.3.patch.txt
>
>
> For easy checking of need privileges for a query, 
> {noformat}
> explain authorize select * from src join srcpart
> INPUTS: 
>   default@srcpart
>   default@srcpart@ds=2008-04-08/hr=11
>   default@srcpart@ds=2008-04-08/hr=12
>   default@srcpart@ds=2008-04-09/hr=11
>   default@srcpart@ds=2008-04-09/hr=12
>   default@src
> OUTPUTS: 
>   
> file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
> CURRENT_USER: 
>   hive_test_user
> OPERATION: 
>   QUERY
> AUTHORIZATION_FAILURES: 
>   No privilege 'Select' found for inputs { database:default, table:srcpart, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
> {noformat}
> Hopefully good for debugging of authorization, which is in progress on 
> HIVE-5837.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16034: Add explain authorize for checking privileges

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16034/
---

(Updated Dec. 10, 2013, 12:57 a.m.)


Review request for hive.


Bugs: HIVE-5961
https://issues.apache.org/jira/browse/HIVE-5961


Repository: hive-git


Description
---

For easy checking of need privileges for a query, 
{noformat}
explain authorize select * from src join srcpart
INPUTS: 
  default@srcpart
  default@srcpart@ds=2008-04-08/hr=11
  default@srcpart@ds=2008-04-08/hr=12
  default@srcpart@ds=2008-04-09/hr=11
  default@srcpart@ds=2008-04-09/hr=12
  default@src
OUTPUTS: 
  
file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
CURRENT_USER: 
  hive_test_user
OPERATION: 
  QUERY
AUTHORIZATION_FAILURES: 
  No privilege 'Select' found for inputs { database:default, table:srcpart, 
columnName:key}
  No privilege 'Select' found for inputs { database:default, table:src, 
columnName:key}
  No privilege 'Select' found for inputs { database:default, table:src, 
columnName:key}
{noformat}

Hopefully good for debugging of authorization, which is in progress on 
HIVE-5837.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java d18243c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java f0da57d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 4b7fc73 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
9b1c36e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 366b714 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 5e5b8cf 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 
97454e4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java 0cb6a9b 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationFactory.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/DelegatableAuthorizationProvider.java
 PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_explain.q PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_explain.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/16034/diff/


Testing
---


Thanks,

Navis Ryu

[jira] [Updated] (HIVE-5961) Add explain authorize for checking privileges

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5961:


Status: Patch Available  (was: Open)

> Add explain authorize for checking privileges
> -
>
> Key: HIVE-5961
> URL: https://issues.apache.org/jira/browse/HIVE-5961
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt, 
> HIVE-5961.3.patch.txt
>
>
> For easy checking of need privileges for a query, 
> {noformat}
> explain authorize select * from src join srcpart
> INPUTS: 
>   default@srcpart
>   default@srcpart@ds=2008-04-08/hr=11
>   default@srcpart@ds=2008-04-08/hr=12
>   default@srcpart@ds=2008-04-09/hr=11
>   default@srcpart@ds=2008-04-09/hr=12
>   default@src
> OUTPUTS: 
>   
> file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
> CURRENT_USER: 
>   hive_test_user
> OPERATION: 
>   QUERY
> AUTHORIZATION_FAILURES: 
>   No privilege 'Select' found for inputs { database:default, table:srcpart, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
> {noformat}
> Hopefully good for debugging of authorization, which is in progress on 
> HIVE-5837.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5961) Add explain authorize for checking privileges

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5961:


Status: Open  (was: Patch Available)

> Add explain authorize for checking privileges
> -
>
> Key: HIVE-5961
> URL: https://issues.apache.org/jira/browse/HIVE-5961
> Project: Hive
>  Issue Type: Improvement
>  Components: Authorization
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-5961.1.patch.txt, HIVE-5961.2.patch.txt
>
>
> For easy checking of need privileges for a query, 
> {noformat}
> explain authorize select * from src join srcpart
> INPUTS: 
>   default@srcpart
>   default@srcpart@ds=2008-04-08/hr=11
>   default@srcpart@ds=2008-04-08/hr=12
>   default@srcpart@ds=2008-04-09/hr=11
>   default@srcpart@ds=2008-04-09/hr=12
>   default@src
> OUTPUTS: 
>   
> file:/home/navis/apache/oss-hive/itests/qtest/target/tmp/localscratchdir/hive_2013-12-04_21-57-53_748_5323811717799107868-1/-mr-1
> CURRENT_USER: 
>   hive_test_user
> OPERATION: 
>   QUERY
> AUTHORIZATION_FAILURES: 
>   No privilege 'Select' found for inputs { database:default, table:srcpart, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
>   No privilege 'Select' found for inputs { database:default, table:src, 
> columnName:key}
> {noformat}
> Hopefully good for debugging of authorization, which is in progress on 
> HIVE-5837.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5993) JDBC Driver should not hard-code the database name

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-5993:


Status: Patch Available  (was: Open)

> JDBC Driver should not hard-code the database name
> --
>
> Key: HIVE-5993
> URL: https://issues.apache.org/jira/browse/HIVE-5993
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-5993.patch
>
>
> Method HiveDatabaseMetadata.getDatabaseProductName() returns a hard-coded 
> string "hive".
> This should instead call the existing Hive-server2 api to return the db name.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5993) JDBC Driver should not hard-code the database name

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-5993:


Attachment: HIVE-5993.patch

> JDBC Driver should not hard-code the database name
> --
>
> Key: HIVE-5993
> URL: https://issues.apache.org/jira/browse/HIVE-5993
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-5993.patch
>
>
> Method HiveDatabaseMetadata.getDatabaseProductName() returns a hard-coded 
> string "hive".
> This should instead call the existing Hive-server2 api to return the db name.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Review Request 16146: HIVE-5993: JDBC Driver should not hard-code the database name

[Szehon Ho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16146/
---

Review request for hive and Prasad Mujumdar.


Bugs: HIVE-5993
https://issues.apache.org/jira/browse/HIVE-5993


Repository: hive-git


Description
---

Method HiveDatabaseMetadata.getDatabaseProductName() returns a hard-coded 
string "Hive".

This should instead call the existing Hive-server2 api to return the db name.  
Incidentally, the server returns "Apache Hive".


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
1ba8ad3 
  jdbc/src/java/org/apache/hive/jdbc/HiveDatabaseMetaData.java 5087ded 

Diff: https://reviews.apache.org/r/16146/diff/


Testing
---

Ran TestJdbcDriver2.


Thanks,

Szehon Ho

[jira] [Created] (HIVE-5993) JDBC Driver should not hard-code the database name

[Szehon Ho (JIRA)]

Szehon Ho created HIVE-5993:
---

 Summary: JDBC Driver should not hard-code the database name
 Key: HIVE-5993
 URL: https://issues.apache.org/jira/browse/HIVE-5993
 Project: Hive
  Issue Type: Improvement
  Components: JDBC
Reporter: Szehon Ho
Assignee: Szehon Ho


Method HiveDatabaseMetadata.getDatabaseProductName() returns a hard-coded 
string "hive".

This should instead call the existing Hive-server2 api to return the db name.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5756) Implement vectorization support for IF conditional expression for long, double, timestamp, boolean and string inputs

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843775#comment-13843775
 ] 

Hive QA commented on HIVE-5756:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617925/HIVE-5756.8.patch

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4775 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/585/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/585/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12617925

> Implement vectorization support for IF conditional expression for long, 
> double, timestamp, boolean and string inputs
> 
>
> Key: HIVE-5756
> URL: https://issues.apache.org/jira/browse/HIVE-5756
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-5756.1.patch, HIVE-5756.2.patch, HIVE-5756.3.patch, 
> HIVE-5756.4.patch, HIVE-5756.5.patch, HIVE-5756.6.patch.txt, 
> HIVE-5756.7.patch, HIVE-5756.8.patch
>
>
> Implement full, end-to-end support for IF in vectorized mode, including new 
> VectorExpression class(es), VectorizationContext translation to a 
> VectorExpression, and unit tests for these, as well as end-to-end ad hoc 
> testing. An end-to-end .q test is recommended but optional.
> This is high priority because IF is the most popular conditional expression.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5936:


Status: Patch Available  (was: Open)

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 15993: analyze command failing to collect stats with counter mechanism

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15993/
---

(Updated Dec. 10, 2013, 12:42 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-5936
https://issues.apache.org/jira/browse/HIVE-5936


Repository: hive-git


Description
---

With counter mechanism, MR job is successful, but StatsTask on client fails 
with NPE


Diffs (updated)
-

  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStatsAggregator.java 
a9c3136 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
d0e94bf 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java c43145b 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java cbc3cd2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java bd2c4a8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java a2ecc80 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 46d88ce 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 197a20f 
  ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanMapper.java 
7e701f4 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java cca8481 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java af729e6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 7443ea4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d0a0ec7 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 0dd0b03 
  ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 
fa430eb 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 661d648 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 8ae32f0 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 
fb5f50e 
  ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsPublisher.java 
8c23b87 
  ql/src/test/queries/clientpositive/stats_counter.q 20769e4 
  ql/src/test/queries/clientpositive/stats_noscan_2.q a19d01b 
  ql/src/test/results/clientpositive/stats_counter.q.out f15d8c5 
  ql/src/test/results/clientpositive/stats_noscan_1.q.out 5aa6607 
  ql/src/test/results/clientpositive/stats_noscan_2.q.out e55fa94 
  ql/src/test/results/clientpositive/truncate_column.q.out a247c4a 

Diff: https://reviews.apache.org/r/15993/diff/


Testing
---


Thanks,

Navis Ryu

[jira] [Updated] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5936:


Status: Open  (was: Patch Available)

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5936:


Status: Patch Available  (was: Open)

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5936:


Status: Open  (was: Patch Available)

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5936) analyze command failing to collect stats with counter mechanism

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-5936:


Attachment: HIVE-5936.7.patch.txt

> analyze command failing to collect stats with counter mechanism
> ---
>
> Key: HIVE-5936
> URL: https://issues.apache.org/jira/browse/HIVE-5936
> Project: Hive
>  Issue Type: Bug
>  Components: Statistics
>Affects Versions: 0.13.0
>Reporter: Ashutosh Chauhan
>Assignee: Navis
> Attachments: HIVE-5936.1.patch.txt, HIVE-5936.2.patch.txt, 
> HIVE-5936.3.patch.txt, HIVE-5936.4.patch.txt, HIVE-5936.5.patch.txt, 
> HIVE-5936.6.patch.txt, HIVE-5936.7.patch.txt
>
>
> With counter mechanism, MR job is successful, but StatsTask on client fails 
> with NPE.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5991) ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5991:
-

Status: Patch Available  (was: Open)

Making it as patch available for precommit tests.


> ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB 
> encoding 
> ---
>
> Key: HIVE-5991
> URL: https://issues.apache.org/jira/browse/HIVE-5991
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5991.1.patch
>
>
> PATCHED_BLOB encoding creates mask with number of bits required for 95th 
> percentile value. If the 95th percentile value requires 32 bits then the mask 
> creation will result in integer overflow.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5970) ArrayIndexOutOfBoundsException in RunLengthIntegerReaderV2.java

[Prasanth J (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843739#comment-13843739
 ] 

Prasanth J commented on HIVE-5970:
--

[~ericchu30] and [~Aleksei] thanks for looking into this issue and your 
findings. I tried to reproduce the issue with the test data that is being 
provided. I think the attached test data is a corrupted orc file and so I was 
not able to reproduce the issue. But I recently faced a similar issue, 
ArrayIndexOutOfBoundsException while writing data using the new 
RunLengthIntegerReaderV2. I suspect that this issue might be related to writing 
of data RLEv2. I posted a patch for it here 
https://issues.apache.org/jira/browse/HIVE-5991. Will it be possible to apply 
this patch and see if this issue happens again?

> ArrayIndexOutOfBoundsException in RunLengthIntegerReaderV2.java
> ---
>
> Key: HIVE-5970
> URL: https://issues.apache.org/jira/browse/HIVE-5970
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.12.0
>Reporter: Eric Chu
>Priority: Critical
>  Labels: orcfile
> Attachments: test_data
>
>
> A workload involving ORC tables starts getting the following 
> ArrayIndexOutOfBoundsException AFTER the upgrade to Hive 0.12. The file is 
> added as part of HIVE-4123. 
> 2013-12-04 14:42:08,537 ERROR 
> cause:java.io.IOException: java.io.IOException: 
> java.lang.ArrayIndexOutOfBoundsException: 0
> 2013-12-04 14:42:08,537 WARN org.apache.hadoop.mapred.Child: Error running 
> child
> java.io.IOException: java.io.IOException: 
> java.lang.ArrayIndexOutOfBoundsException: 0
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:304)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:220)
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:215)
> at 
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:200)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at org.apache.hadoop.mapred.Child.main(Child.java:262)
> Caused by: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 0
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at 
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:276)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
> at 
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108)
> at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:302)
> ... 11 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at 
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readPatchedBaseValues(RunLengthIntegerReaderV2.java:171)
> at 
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:54)
> at 
> org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:287)
> at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$LongTreeReader.next(RecordReaderImpl.java:473)
> at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1157)
> at 
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2196)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:129)
> at 
> o

[jira] [Commented] (HIVE-5230) Better error reporting by async threads in HiveServer2

[Carl Steinbach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843728#comment-13843728
 ] 

Carl Steinbach commented on HIVE-5230:
--

I'm looking at it now.

[~thejas] If you don't hear back from me before 11:56am tomorrow you should 
feel free to go ahead and commit the patch.

> Better error reporting by async threads in HiveServer2
> --
>
> Key: HIVE-5230
> URL: https://issues.apache.org/jira/browse/HIVE-5230
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 0.13.0
>
> Attachments: HIVE-5230.1.patch, HIVE-5230.1.patch, HIVE-5230.2.patch, 
> HIVE-5230.3.patch, HIVE-5230.4.patch, HIVE-5230.6.patch, HIVE-5230.7.patch, 
> HIVE-5230.8.patch, HIVE-5230.9.patch
>
>
> [HIVE-4617|https://issues.apache.org/jira/browse/HIVE-4617] provides support 
> for async execution in HS2. When a background thread gets an error, currently 
> the client can only poll for the operation state and also the error with its 
> stacktrace is logged. However, it will be useful to provide a richer error 
> response like thrift API does with TStatus (which is constructed while 
> building a Thrift response object). 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5991) ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding

[Prasanth J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth J updated HIVE-5991:
-

Attachment: HIVE-5991.1.patch

> ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB 
> encoding 
> ---
>
> Key: HIVE-5991
> URL: https://issues.apache.org/jira/browse/HIVE-5991
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Prasanth J
>Assignee: Prasanth J
>  Labels: orcfile
> Attachments: HIVE-5991.1.patch
>
>
> PATCHED_BLOB encoding creates mask with number of bits required for 95th 
> percentile value. If the 95th percentile value requires 32 bits then the mask 
> creation will result in integer overflow.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-5992) Hive inconsistently converts timestamp in AVG and SUM UDAF's

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-5992:
-

 Summary: Hive inconsistently converts timestamp in AVG and SUM 
UDAF's
 Key: HIVE-5992
 URL: https://issues.apache.org/jira/browse/HIVE-5992
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


{code}
hive> select t, sum(t), count(*), sum(t)/count(*), avg(t) from ts group by t;
...
OK
1977-03-15 12:34:22.345678   227306062  1  227306062
2.27306062345678E8
{code}

As it can be seen, timestamp value (1977-03-15 12:34:22.345678) is converted 
with fractional part ignored in sum, while preserved in avg. As a further 
result, sum()/count() is not equivalent to avg.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5872) Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843725#comment-13843725
 ] 

Prasad Mujumdar commented on HIVE-5872:
---

Looks fine to me.
+1

In general, it looks like we need more exception logic for handling decimals in 
UDAF (HIVE-5872, HIVE-5866). It might be useful to add a note in the dev guide 
for future  work ..

> Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal 
> types
> ---
>
> Key: HIVE-5872
> URL: https://issues.apache.org/jira/browse/HIVE-5872
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-5872.1.patch, HIVE-5872.2.patch, HIVE-5872.3.patch, 
> HIVE-5872.4.patch, HIVE-5872.patch
>
>
> Currently UDAFs are still reporting system default precision/scale (38, 18) 
> for decimal results. Not only this is coarse, but also this can cause 
> problems in subsequent operators such as division, where the result is 
> dependent on the precision/scale of the input, which can go out of bound 
> (38,38). Thus, these UDAFs should correctly report the precision/scale of the 
> result.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843723#comment-13843723
 ] 

Hive QA commented on HIVE-5989:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12617917/HIVE-5989.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 4761 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_precision
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
{noformat}

Test results: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/584/testReport
Console output: 
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/584/console

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12617917

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0, 0.12.0, 0.12.1
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: HIVE-5989.patch, SleepyAP.patch
>
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.
> The effect of this bug is that if there are two users that are concurrently 
> running jobs on the metastore, we might :
> a) Allow a user to do something they didn't have permission to, because the 
> other person did. (Security hole)
> b) Disallow a user from doing something they should have permission to (More 
> common - annoying and can cause job failures)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-5991) ORC RLEv2 fails with ArrayIndexOutOfBounds exception for PATCHED_BLOB encoding

[Prasanth J (JIRA)]

Prasanth J created HIVE-5991:


 Summary: ORC RLEv2 fails with ArrayIndexOutOfBounds exception for 
PATCHED_BLOB encoding 
 Key: HIVE-5991
 URL: https://issues.apache.org/jira/browse/HIVE-5991
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Prasanth J
Assignee: Prasanth J


PATCHED_BLOB encoding creates mask with number of bits required for 95th 
percentile value. If the 95th percentile value requires 32 bits then the mask 
creation will result in integer overflow.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Maven unit test question

[Brock Noland]

Can you share the change with me so I can debug?
On Dec 9, 2013 5:15 PM, "Alan Gates"  wrote:

> I was attempting to write unit tests for changes I'm making to
> HiveMetaStoreClient as part of the ACID transaction work (see
> https://issues.apache.org/jira/browse/HIVE-5843).  When I added the tests
> and attempted to run them using
> mvn tests -Dtest=TestHiveMetaStoreClient -Phadoop-1
>
> it failed with:
>
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/thrift/TUGIContainingTransport$Factory
>
> This class is contained in the hive-shims jar.  The error surprised me
> because according to metastore/pom.xml, hive-shims is a dependency of
> hive-metastore.  When I ran maven with -X to get debug information, I found
> that in the classpath it was including
> /Users/gates/git/apache/hive/shims/assembly/target/classes.  I'm guessing
> that rather than use the shims jar (which has been built by this time) it's
> trying to use the compiled classes, but failing in this case because the
> shims jar is actually constructed not by directly conglomerating a set of
> class files but by picking and choosing from several shim jar versions and
> then constructing a single jar.  But I could not figure out how to
> communicate to maven that is should use the already built shims jar rather
> than the classes.  To test my theory I took the shims jar and unpacked in
> the path maven was looking in, and sure enough my tests ran once I did that.
>
> The existing unit test TestMetastoreExpr in ql seems to have the same
> issue.  I tried to use it as a model, but when I ran it it failed with the
> same error, and unpacking the jar resolved it in the same way.
>
> Am I doing something wrong, or is there a change needed in the pom.xml to
> get it to look in the jar instead of the .class files for shims
> dependencies?
>
> Alan.
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

[jira] [Updated] (HIVE-5756) Implement vectorization support for IF conditional expression for long, double, timestamp, boolean and string inputs

[Eric Hanson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Hanson updated HIVE-5756:
--

Attachment: HIVE-5756.8.patch

Fixed error where undefined and possibly null string reference was examined for 
NULL data value. Now there is a conditional check so the string object 
reference is not examined if data value is NULL.

> Implement vectorization support for IF conditional expression for long, 
> double, timestamp, boolean and string inputs
> 
>
> Key: HIVE-5756
> URL: https://issues.apache.org/jira/browse/HIVE-5756
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Eric Hanson
>Assignee: Eric Hanson
> Attachments: HIVE-5756.1.patch, HIVE-5756.2.patch, HIVE-5756.3.patch, 
> HIVE-5756.4.patch, HIVE-5756.5.patch, HIVE-5756.6.patch.txt, 
> HIVE-5756.7.patch, HIVE-5756.8.patch
>
>
> Implement full, end-to-end support for IF in vectorized mode, including new 
> VectorExpression class(es), VectorizationContext translation to a 
> VectorExpression, and unit tests for these, as well as end-to-end ad hoc 
> testing. An end-to-end .q test is recommended but optional.
> This is high priority because IF is the most popular conditional expression.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 15495: Implement vectorization support for IF conditional expression for long and double inputs

[Eric Hanson]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15495/
---

(Updated Dec. 9, 2013, 11:50 p.m.)


Review request for hive, Ashutosh Chauhan, Jitendra Pandey, and Teddy Choi.


Changes
---

Fixed error where undefined string was examined for NULL data value.


Bugs: HIVE-5756
https://issues.apache.org/jira/browse/HIVE-5756


Repository: hive-git


Description
---

Implement vectorization support for IF conditional expression for long and 
double inputs


Diffs (updated)
-

  ant/src/org/apache/hadoop/hive/ant/GenVectorCode.java 1d3c5c4 
  ql/src/gen/vectorization/ExpressionTemplates/IfExprColumnColumn.txt 
PRE-CREATION 
  ql/src/gen/vectorization/ExpressionTemplates/IfExprColumnScalar.txt 
PRE-CREATION 
  ql/src/gen/vectorization/ExpressionTemplates/IfExprScalarColumn.txt 
PRE-CREATION 
  ql/src/gen/vectorization/ExpressionTemplates/IfExprScalarScalar.txt 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java 
e1d4543 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ColumnVector.java 48b87ea 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/DoubleColumnVector.java 
d3bb28e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/LongColumnVector.java 
f65e8fa 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprStringColumnStringColumn.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprStringColumnStringScalar.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprStringScalarStringColumn.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/IfExprStringScalarStringScalar.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
7859e56 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java 0c7e61c 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizationContext.java 
720ca54 
  ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorizedRowBatch.java 
a250c9d 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorConditionalExpressions.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/15495/diff/


Testing
---


Thanks,

Eric Hanson

[jira] [Commented] (HIVE-5990) round() function with large scale chokes

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843709#comment-13843709
 ] 

Xuefu Zhang commented on HIVE-5990:
---

Trunk has a new implementation. However, OOM is observed with the same query, 
seemingly due to the same issue as above.
{code}
hive> select ROUND(4.42330604244772E-305, 20) from ts; 
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.math.BigDecimal.bigTenToThe(BigDecimal.java:3376)
at java.math.BigDecimal.bigMultiplyPowerTen(BigDecimal.java:3508)
at java.math.BigDecimal.setScale(BigDecimal.java:2394)
at java.math.BigDecimal.setScale(BigDecimal.java:2335)
at 
org.apache.hadoop.hive.ql.udf.generic.RoundUtils.round(RoundUtils.java:49)
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFRound.round(GenericUDFRound.java:248)
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFRound.evaluate(GenericUDFRound.java:229)
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:145)
at 
org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:232)
{code}

[~mihirk] Do you have a fix for this, or do you plan to work on this?

> round() function with large scale chokes
> 
>
> Key: HIVE-5990
> URL: https://issues.apache.org/jira/browse/HIVE-5990
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor, UDF
>Affects Versions: 0.9.0
> Environment: CDH 4.2 running Hive 0.9
>Reporter: Mihir Kulkarni
>
> ROUND(4.42330604244772E-305, 20) chokes with following error:
> FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments 
> '20': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to 
> execute method public org.apache.hadoop.hive.serde2.io.DoubleWritable 
> org.apache.hadoop.hive.ql.udf.UDFRound.evaluate(org.apache.hadoop.hive.serde2.io.DoubleWritable,org.apache.hadoop.io.IntWritable)
>   on object org.apache.hadoop.hive.ql.udf.UDFRound@51809ab7 of class 
> org.apache.hadoop.hive.ql.udf.UDFRound with arguments 
> {1.001:org.apache.hadoop.hive.serde2.io.DoubleWritable, 
> 20:org.apache.hadoop.io.IntWritable} of size 2
> I think the problem might be in how Hive evaluates 2 arg ROUND(). The code 
> uses
> BigDecimal.valueOf(d).setScale(i, RoundingMode.HALF_UP)
> which chokes by itself. I saw some changes in implementation of ROUND in hive 
> trunk but this is still being used in RoundUtils.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5981) Add hive-unit back to itests pom

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843696#comment-13843696
 ] 

Brock Noland commented on HIVE-5981:


No worries :)

> Add hive-unit back to itests pom
> 
>
> Key: HIVE-5981
> URL: https://issues.apache.org/jira/browse/HIVE-5981
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5981.patch, HIVE-5981.patch
>
>
> HIVE-5755 removed hive-unit from the itests pom and it should not have been 
> removed.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5981) Add hive-unit back to itests pom

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843694#comment-13843694
 ] 

Vikram Dixit K commented on HIVE-5981:
--

Sorry about that. The change was unintended. 

> Add hive-unit back to itests pom
> 
>
> Key: HIVE-5981
> URL: https://issues.apache.org/jira/browse/HIVE-5981
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.13.0
>
> Attachments: HIVE-5981.patch, HIVE-5981.patch
>
>
> HIVE-5755 removed hive-unit from the itests pom and it should not have been 
> removed.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HIVE-5990) round() function with large scale chokes

[Mihir Kulkarni (JIRA)]

Mihir Kulkarni created HIVE-5990:


 Summary: round() function with large scale chokes
 Key: HIVE-5990
 URL: https://issues.apache.org/jira/browse/HIVE-5990
 Project: Hive
  Issue Type: Bug
  Components: Query Processor, UDF
Affects Versions: 0.9.0
 Environment: CDH 4.2 running Hive 0.9
Reporter: Mihir Kulkarni


ROUND(4.42330604244772E-305, 20) chokes with following error:

FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '20': 
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method 
public org.apache.hadoop.hive.serde2.io.DoubleWritable 
org.apache.hadoop.hive.ql.udf.UDFRound.evaluate(org.apache.hadoop.hive.serde2.io.DoubleWritable,org.apache.hadoop.io.IntWritable)
  on object org.apache.hadoop.hive.ql.udf.UDFRound@51809ab7 of class 
org.apache.hadoop.hive.ql.udf.UDFRound with arguments 
{1.001:org.apache.hadoop.hive.serde2.io.DoubleWritable, 
20:org.apache.hadoop.io.IntWritable} of size 2

I think the problem might be in how Hive evaluates 2 arg ROUND(). The code uses
BigDecimal.valueOf(d).setScale(i, RoundingMode.HALF_UP)
which chokes by itself. I saw some changes in implementation of ROUND in hive 
trunk but this is still being used in RoundUtils.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5872) Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843680#comment-13843680
 ] 

Thejas M Nair commented on HIVE-5872:
-

[~xuefuz] Sounds good.


> Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal 
> types
> ---
>
> Key: HIVE-5872
> URL: https://issues.apache.org/jira/browse/HIVE-5872
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-5872.1.patch, HIVE-5872.2.patch, HIVE-5872.3.patch, 
> HIVE-5872.4.patch, HIVE-5872.patch
>
>
> Currently UDAFs are still reporting system default precision/scale (38, 18) 
> for decimal results. Not only this is coarse, but also this can cause 
> problems in subsequent operators such as division, where the result is 
> dependent on the precision/scale of the input, which can go out of bound 
> (38,38). Thus, these UDAFs should correctly report the precision/scale of the 
> result.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5872) Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types

[Prasad Mujumdar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843675#comment-13843675
 ] 

Prasad Mujumdar commented on HIVE-5872:
---

[~xuefuz] I will take a look shortly.

> Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal 
> types
> ---
>
> Key: HIVE-5872
> URL: https://issues.apache.org/jira/browse/HIVE-5872
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-5872.1.patch, HIVE-5872.2.patch, HIVE-5872.3.patch, 
> HIVE-5872.4.patch, HIVE-5872.patch
>
>
> Currently UDAFs are still reporting system default precision/scale (38, 18) 
> for decimal results. Not only this is coarse, but also this can cause 
> problems in subsequent operators such as division, where the result is 
> dependent on the precision/scale of the input, which can go out of bound 
> (38,38). Thus, these UDAFs should correctly report the precision/scale of the 
> result.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Maven unit test question

[Alan Gates]

I was attempting to write unit tests for changes I'm making to 
HiveMetaStoreClient as part of the ACID transaction work (see 
https://issues.apache.org/jira/browse/HIVE-5843).  When I added the tests and 
attempted to run them using 
mvn tests -Dtest=TestHiveMetaStoreClient -Phadoop-1 

it failed with:

java.lang.NoClassDefFoundError: 
org/apache/hadoop/hive/thrift/TUGIContainingTransport$Factory

This class is contained in the hive-shims jar.  The error surprised me because 
according to metastore/pom.xml, hive-shims is a dependency of hive-metastore.  
When I ran maven with -X to get debug information, I found that in the 
classpath it was including 
/Users/gates/git/apache/hive/shims/assembly/target/classes.  I'm guessing that 
rather than use the shims jar (which has been built by this time) it's trying 
to use the compiled classes, but failing in this case because the shims jar is 
actually constructed not by directly conglomerating a set of class files but by 
picking and choosing from several shim jar versions and then constructing a 
single jar.  But I could not figure out how to communicate to maven that is 
should use the already built shims jar rather than the classes.  To test my 
theory I took the shims jar and unpacked in the path maven was looking in, and 
sure enough my tests ran once I did that.

The existing unit test TestMetastoreExpr in ql seems to have the same issue.  I 
tried to use it as a model, but when I ran it it failed with the same error, 
and unpacking the jar resolved it in the same way.

Am I doing something wrong, or is there a change needed in the pom.xml to get 
it to look in the jar instead of the .class files for shims dependencies?

Alan.
-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Hive-trunk-h0.21 - Build # 2493 - Still Failing

[Apache Jenkins Server]

Changes for Build #2458
[rhbutani] HIVE-5369 Annotate hive operator tree with statistics from metastore 
(Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5809 : incorrect stats in some cases with 
hive.stats.autogather=true (Ashutosh Chauhan via Navis)

[brock] HIVE-5741: Fix binary packaging build eg include hcatalog, resolve pom 
issues (Brock Noland reviewed by Xuefu Zhang)


Changes for Build #2459
[hashutosh] HIVE-5844 : dynamic_partition_skip_default.q test fails on trunk 
(Prasanth J via Ashutosh Chauhan)


Changes for Build #2460
[hashutosh] HIVE-5846 : Analyze command fails with vectorization on (Remus 
Rusanu via Ashutosh Chauhan)

[hashutosh] HIVE-2055 : Hive should add HBase classpath dependencies when 
available (Nick Dimiduk via Ashutosh Chauhan)

[hashutosh] HIVE-4632 : Use hadoop counter as a stat publisher (Navis via 
Ashutosh Chauhan)

[hashutosh] HIVE-3107 : Improve semantic analyzer to better handle column name 
references in group by/sort by clauses (Harish Butani via Ashutosh Chauhan)


Changes for Build #2461
[xuefu] HIVE-5565: Limit Hive decimal type maximum precision and scale to 38 
(reviewed by Brock)

[brock] HIVE-5842 - Fix issues with new paths to jar in hcatalog (Brock Noland 
reviewed by Prasad Mujumdar)

[xuefu] HIVE-5356: Move arithmatic UDFs to generic UDF implementations 
(reviewed by Brock)


Changes for Build #2462
[hashutosh] HIVE-5692 : Make VectorGroupByOperator parameters configurable 
(Remus Rusanu via Ashutosh Chauhan)

[hashutosh] HIVE-5845 : CTAS failed on vectorized code path (Remus Rusanu via 
Ashutosh Chauhan)

[thejas] HIVE-5635 : WebHCatJTShim23 ignores security/user context (Eugene 
Koifman via Thejas Nair)

[hashutosh] HIVE-5663 : Refactor ORC RecordReader to operate on direct & 
wrapped ByteBuffers (Gopal V via Owen Omalley)


Changes for Build #2463

Changes for Build #2464
[thejas] HIVE-5618 : Hive local task fails to run when run from oozie in a 
secure cluster (Prasad Mujumdar via Thejas Nair)


Changes for Build #2465

Changes for Build #2466
[thejas] HIVE-3815 : hive table rename fails if filesystem cache is disabled 
(Thejas Nair reviewed by Navis)


Changes for Build #2467

Changes for Build #2468
[hashutosh] HIVE-5614 : Subquery support: allow subquery expressions in having 
clause (Harish Butani via Ashutosh Chauhan)


Changes for Build #2469
[xuefu] HIVE-5763: ExprNodeGenericFuncDesc.toString() generating unbalanced 
parenthesises (reviewed by Ashutosh)


Changes for Build #2470

Changes for Build #2471
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #2472
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #2473
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #2474
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #2475
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #2476
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #2477

Changes for Build #2478

Changes for Build #2479

Changes for Build #2480
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #2481

Changes for Build #2482
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #2483
[rhbutani] 

[jira] [Commented] (HIVE-5897) Fix hadoop2 execution environment Milestone 2

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843670#comment-13843670
 ] 

Vikram Dixit K commented on HIVE-5897:
--

Ping [~brocknoland]

> Fix hadoop2 execution environment Milestone 2
> -
>
> Key: HIVE-5897
> URL: https://issues.apache.org/jira/browse/HIVE-5897
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Brock Noland
>Assignee: Vikram Dixit K
> Attachments: HIVE-5897.4.patch, HIVE-5897.5.patch, HIVE-5897.patch, 
> HIVE-5897.patch, HIVE-5897.patch
>
>
> Follow on to HIVE-5755.
> List of known issues:
> hcatalog-pig-adapter and ql need
> {noformat}
>  
>org.apache.hadoop
>hadoop-mapreduce-client-common
>${hadoop-23.version}
>test
> 
> {noformat}
> hcatalog core and hbase storage handler needs 
> {noformat}
>  
>org.apache.hadoop
>   hadoop-common
>   ${hadoop-23.version}
>   tests
>   test
> 
> 
>   org.apache.hadoop
>   hadoop-mapreduce-client-hs
>   ${hadoop-23.version}
>   test
> 
>
>  org.apache.hadoop
>  hadoop-yarn-server-tests
>  ${hadoop-23.version}
>  tests
>  test
>
> {noformat}
> hcatalog core needs:
> {noformat}
>
>   org.apache.hadoop
>   hadoop-mapreduce-client-jobclient
>   ${hadoop-23.version}
>   test
> 
> {noformat}
> beeline needs 
> {noformat}
> 
>   org.apache.hadoop
>   hadoop-mapreduce-client-core
>   ${hadoop-23.version}
>   test
> 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5872) Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843668#comment-13843668
 ] 

Xuefu Zhang commented on HIVE-5872:
---

Thanks for pointing this out, [~thejas]. That's exactly my impression. :) I'll 
ask a committer to review the patch. If changes are due, I will create followup 
jira to accommodate. 

> Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal 
> types
> ---
>
> Key: HIVE-5872
> URL: https://issues.apache.org/jira/browse/HIVE-5872
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-5872.1.patch, HIVE-5872.2.patch, HIVE-5872.3.patch, 
> HIVE-5872.4.patch, HIVE-5872.patch
>
>
> Currently UDAFs are still reporting system default precision/scale (38, 18) 
> for decimal results. Not only this is coarse, but also this can cause 
> problems in subsequent operators such as division, where the result is 
> dependent on the precision/scale of the input, which can go out of bound 
> (38,38). Thus, these UDAFs should correctly report the precision/scale of the 
> result.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5872) Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843663#comment-13843663
 ] 

Thejas M Nair commented on HIVE-5872:
-

I guess Sergey might be doing a good job at "If you want to become a committer, 
behave like a committer." (from 
https://cwiki.apache.org/confluence/display/Hive/BecomingACommitter)

> Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal 
> types
> ---
>
> Key: HIVE-5872
> URL: https://issues.apache.org/jira/browse/HIVE-5872
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-5872.1.patch, HIVE-5872.2.patch, HIVE-5872.3.patch, 
> HIVE-5872.4.patch, HIVE-5872.patch
>
>
> Currently UDAFs are still reporting system default precision/scale (38, 18) 
> for decimal results. Not only this is coarse, but also this can cause 
> problems in subsequent operators such as division, where the result is 
> dependent on the precision/scale of the input, which can go out of bound 
> (38,38). Thus, these UDAFs should correctly report the precision/scale of the 
> result.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5872) Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal types

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843654#comment-13843654
 ] 

Thejas M Nair commented on HIVE-5872:
-

[~xuefuz] Sergey is not a committer. The policy is to get a +1 from a committer 
who is not the author of the patch before committing it. See 
https://cwiki.apache.org/confluence/display/Hive/Bylaws#Bylaws-



> Make UDAFs such as GenericUDAFSum report accurate precision/scale for decimal 
> types
> ---
>
> Key: HIVE-5872
> URL: https://issues.apache.org/jira/browse/HIVE-5872
> Project: Hive
>  Issue Type: Improvement
>  Components: Types, UDF
>Affects Versions: 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Fix For: 0.13.0
>
> Attachments: HIVE-5872.1.patch, HIVE-5872.2.patch, HIVE-5872.3.patch, 
> HIVE-5872.4.patch, HIVE-5872.patch
>
>
> Currently UDAFs are still reporting system default precision/scale (38, 18) 
> for decimal results. Not only this is coarse, but also this can cause 
> problems in subsequent operators such as division, where the result is 
> dependent on the precision/scale of the input, which can go out of bound 
> (38,38). Thus, these UDAFs should correctly report the precision/scale of the 
> result.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Hive-trunk-hadoop2 - Build # 591 - Still Failing

[Apache Jenkins Server]

Changes for Build #558
[rhbutani] HIVE-5369 Annotate hive operator tree with statistics from metastore 
(Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5809 : incorrect stats in some cases with 
hive.stats.autogather=true (Ashutosh Chauhan via Navis)

[brock] HIVE-5741: Fix binary packaging build eg include hcatalog, resolve pom 
issues (Brock Noland reviewed by Xuefu Zhang)


Changes for Build #559
[hashutosh] HIVE-3107 : Improve semantic analyzer to better handle column name 
references in group by/sort by clauses (Harish Butani via Ashutosh Chauhan)

[hashutosh] HIVE-5844 : dynamic_partition_skip_default.q test fails on trunk 
(Prasanth J via Ashutosh Chauhan)


Changes for Build #560
[xuefu] HIVE-5356: Move arithmatic UDFs to generic UDF implementations 
(reviewed by Brock)

[hashutosh] HIVE-5846 : Analyze command fails with vectorization on (Remus 
Rusanu via Ashutosh Chauhan)

[hashutosh] HIVE-2055 : Hive should add HBase classpath dependencies when 
available (Nick Dimiduk via Ashutosh Chauhan)

[hashutosh] HIVE-4632 : Use hadoop counter as a stat publisher (Navis via 
Ashutosh Chauhan)


Changes for Build #561
[hashutosh] HIVE-5845 : CTAS failed on vectorized code path (Remus Rusanu via 
Ashutosh Chauhan)

[thejas] HIVE-5635 : WebHCatJTShim23 ignores security/user context (Eugene 
Koifman via Thejas Nair)

[hashutosh] HIVE-5663 : Refactor ORC RecordReader to operate on direct & 
wrapped ByteBuffers (Gopal V via Owen Omalley)

[xuefu] HIVE-5565: Limit Hive decimal type maximum precision and scale to 38 
(reviewed by Brock)

[brock] HIVE-5842 - Fix issues with new paths to jar in hcatalog (Brock Noland 
reviewed by Prasad Mujumdar)


Changes for Build #562
[hashutosh] HIVE-5692 : Make VectorGroupByOperator parameters configurable 
(Remus Rusanu via Ashutosh Chauhan)


Changes for Build #563
[thejas] HIVE-5618 : Hive local task fails to run when run from oozie in a 
secure cluster (Prasad Mujumdar via Thejas Nair)


Changes for Build #564

Changes for Build #565
[thejas] HIVE-3815 : hive table rename fails if filesystem cache is disabled 
(Thejas Nair reviewed by Navis)


Changes for Build #566

Changes for Build #567
[hashutosh] HIVE-5614 : Subquery support: allow subquery expressions in having 
clause (Harish Butani via Ashutosh Chauhan)


Changes for Build #568
[xuefu] HIVE-5763: ExprNodeGenericFuncDesc.toString() generating unbalanced 
parenthesises (reviewed by Ashutosh)


Changes for Build #569

Changes for Build #570
[rhbutani] HIVE-5849 Improve the stats of operators based on heuristics in the 
absence of any column statistics (Prasanth Jayachandran via Harish Butani)

[hashutosh] HIVE-5793 : Update hive-default.xml.template for HIVE4002 (Navis 
via Ashutosh Chauhan)


Changes for Build #571
[navis] HIVE-4518 : Should be removed files (OptrStatsGroupByHook, etc.)

[navis] HIVE-5839 : BytesRefArrayWritable compareTo violates contract (Xuefu 
Zhang via Navis)

[navis] HIVE-4518 : Missing file (HiveFatalException)

[navis] HIVE-4518 : Counter Strike: Operation Operator (Gunther Hagleitner and 
Jason Dere via Navis)


Changes for Build #572
[brock] HIVE-4741 - Add Hive config API to modify the restrict list (Prasad 
Mujumdar, Navis via Brock Noland)


Changes for Build #573
[navis] HIVE-5827 : Incorrect location of logs for failed tests (Vikram Dixit K 
and Szehon Ho via Navis)

[thejas] HIVE-4485 : beeline prints null as empty strings (Thejas Nair reviewed 
by Ashutosh Chauhan)

[brock] HIVE-5704 - A couple of generic UDFs are not in the right 
folder/package (Xuefu Zhang via Brock Noland)

[brock] HIVE-5706 - Move a few numeric UDFs to generic implementations (Xuefu 
Zhang via Brock Noland)

[hashutosh] HIVE-5817 : column name to index mapping in VectorizationContext is 
broken (Remus Rusanu, Sergey Shelukhin via Ashutosh Chauhan)

[hashutosh] HIVE-5876 : Split elimination in ORC breaks for partitioned tables 
(Prasanth J via Ashutosh Chauhan)

[hashutosh] HIVE-5886 : [Refactor] Remove unused class JobCloseFeedback 
(Ashutosh Chauhan via Thejas Nair)

[brock] HIVE-5894 - Fix minor PTest2 issues (Brock Noland)


Changes for Build #574
[brock] HIVE-5755 - Fix hadoop2 execution environment Milestone 1 (Vikram Dixit 
K via Brock Noland)


Changes for Build #575
[xuefu] HIVE-5893: hive-schema-0.13.0.mysql.sql contains reference to 
nonexistent column (Carl via Xuefu)

[xuefu] HIVE-5684: Serde support for char (Jason via Xuefu)


Changes for Build #576

Changes for Build #577

Changes for Build #578

Changes for Build #579
[brock] HIVE-5441 - Async query execution doesn't return resultset status 
(Prasad Mujumdar via Thejas M Nair)

[brock] HIVE-5880 - Rename HCatalog HBase Storage Handler artifact id (Brock 
Noland reviewed by Prasad Mujumdar)


Changes for Build #580
[ehans] HIVE-5581: Implement vectorized year/month/day... etc. for string 
arguments (Teddy Choi via Eric Hanson)


Changes for Build #581
[rhbutani] HIVE-5898 Make fetching of column statistics configurable (Prasanth 
Jayac

[jira] [Updated] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5989:
---

Status: Patch Available  (was: Open)

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: HIVE-5989.patch, SleepyAP.patch
>
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.
> The effect of this bug is that if there are two users that are concurrently 
> running jobs on the metastore, we might :
> a) Allow a user to do something they didn't have permission to, because the 
> other person did. (Security hole)
> b) Disallow a user from doing something they should have permission to (More 
> common - annoying and can cause job failures)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843648#comment-13843648
 ] 

Sushanth Sowmyan commented on HIVE-5989:


Setting to patch-available so Hive-QA can pick it up to run existing tests with 
this patch.

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0, 0.12.0, 0.12.1
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: HIVE-5989.patch, SleepyAP.patch
>
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.
> The effect of this bug is that if there are two users that are concurrently 
> running jobs on the metastore, we might :
> a) Allow a user to do something they didn't have permission to, because the 
> other person did. (Security hole)
> b) Disallow a user from doing something they should have permission to (More 
> common - annoying and can cause job failures)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5989:
---

Affects Version/s: 0.12.1
   0.12.0

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0, 0.12.0, 0.12.1
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: HIVE-5989.patch, SleepyAP.patch
>
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.
> The effect of this bug is that if there are two users that are concurrently 
> running jobs on the metastore, we might :
> a) Allow a user to do something they didn't have permission to, because the 
> other person did. (Security hole)
> b) Disallow a user from doing something they should have permission to (More 
> common - annoying and can cause job failures)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5989:
---

Description: 
Metastore-side authorization has a couple of pretty important threadsafety bugs 
in it:

a) The HiveMetastoreAuthenticated instantiated by the 
AuthorizationPreEventListener is static. This is a premature optimization and 
incorrect, as it will result in Authenticator implementations that store state 
potentially giving an incorrect result, and this bug very much exists with the 
DefaultMetastoreAuthenticator.

b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, which 
it is not. HMSHandler.getConf() is the appropriate thread-safe equivalent.

The effect of this bug is that if there are two users that are concurrently 
running jobs on the metastore, we might :

a) Allow a user to do something they didn't have permission to, because the 
other person did. (Security hole)
b) Disallow a user from doing something they should have permission to (More 
common - annoying and can cause job failures)


  was:
Metastore-side authorization has a couple of pretty important threadsafety bugs 
in it:

a) The HiveMetastoreAuthenticated instantiated by the 
AuthorizationPreEventListener is static. This is a premature optimization and 
incorrect, as it will result in Authenticator implementations that store state 
potentially giving an incorrect result, and this bug very much exists with the 
DefaultMetastoreAuthenticator.

b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, which 
it is not. HMSHandler.getConf() is the appropriate thread-safe equivalent.



> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: HIVE-5989.patch, SleepyAP.patch
>
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.
> The effect of this bug is that if there are two users that are concurrently 
> running jobs on the metastore, we might :
> a) Allow a user to do something they didn't have permission to, because the 
> other person did. (Security hole)
> b) Disallow a user from doing something they should have permission to (More 
> common - annoying and can cause job failures)



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5989:
---

Attachment: HIVE-5989.patch

Attached patch.

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: HIVE-5989.patch, SleepyAP.patch
>
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5982) Remove redundant filesystem operations and methods in FileSink

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843642#comment-13843642
 ] 

Thejas M Nair commented on HIVE-5982:
-

+1

> Remove redundant filesystem operations and methods in FileSink
> --
>
> Key: HIVE-5982
> URL: https://issues.apache.org/jira/browse/HIVE-5982
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-5982.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: Review Request 16105: Refactoring patch to remove unnecessary methods in FileSink and other cleanup

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16105/#review30054
---

Ship it!


Ship It!

- Thejas Nair


On Dec. 7, 2013, 10:28 p.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16105/
> ---
> 
> (Updated Dec. 7, 2013, 10:28 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-5982
> https://issues.apache.org/jira/browse/HIVE-5982
> 
> 
> Repository: hive
> 
> 
> Description
> ---
> 
> Refactoring patch to remove unnecessary methods in FileSink and other cleanup
> 
> 
> Diffs
> -
> 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
> 1548974 
>   trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1548974 
> 
> Diff: https://reviews.apache.org/r/16105/diff/
> 
> 
> Testing
> ---
> 
> No new functionality. Existing test case suffice.
> 
> 
> Thanks,
> 
> Ashutosh Chauhan
> 
>

[jira] [Updated] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5989:
---

Attachment: SleepyAP.patch

SleepyAP.patch attached - Note, before using this for testing, you'll have to 
set hive.metastore.client.socket.timeout to an appropriately high amount (I 
used 6 for 1 minute) to make sure that we don't hit a timeout on the client.

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
> Attachments: SleepyAP.patch
>
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5989:
---

Priority: Critical  (was: Major)

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>Priority: Critical
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843634#comment-13843634
 ] 

Sushanth Sowmyan commented on HIVE-5989:


I'll attach the fix patch for now as this is a pretty severe bug. As for 
testing, it is easy to test from manual tests or with e2e tests, but difficult 
to test with a unit test. I'll also attach a "SleepyAuthorizationProvider" that 
I used to test this manually, so as to elongate the critical section to 
demonstrate and reproduce this error easily.

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Updated] (HIVE-5989) Hive metastore authorization check is not threadsafe

[Sushanth Sowmyan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-5989:
---

Component/s: Security
 Metastore

> Hive metastore authorization check is not threadsafe
> 
>
> Key: HIVE-5989
> URL: https://issues.apache.org/jira/browse/HIVE-5989
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Security
>Affects Versions: 0.11.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
>
> Metastore-side authorization has a couple of pretty important threadsafety 
> bugs in it:
> a) The HiveMetastoreAuthenticated instantiated by the 
> AuthorizationPreEventListener is static. This is a premature optimization and 
> incorrect, as it will result in Authenticator implementations that store 
> state potentially giving an incorrect result, and this bug very much exists 
> with the DefaultMetastoreAuthenticator.
> b) It assumes HMSHandler.getHiveConf() is itself going to be thread-safe, 
> which it is not. HMSHandler.getConf() is the appropriate thread-safe 
> equivalent.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)