[jira] [Commented] (HIVE-13659) An empty where condition leads to vectorization exceptions instead of throwing a compile time error

2016-05-07 Thread Matt McCline (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275139#comment-15275139
 ] 

Matt McCline commented on HIVE-13659:
-

[~ashutoshc] thank you for your review!

There were 2 unrelated failures on an internal PTest run on HIVE-13659.03.patch:

{code}
 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2   
20 sec  1
 
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
  5 sec   1
{code}

> An empty where condition leads to vectorization exceptions instead of 
> throwing a compile time error
> ---
>
> Key: HIVE-13659
> URL: https://issues.apache.org/jira/browse/HIVE-13659
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HIVE-13659.01.patch, HIVE-13659.02.patch, 
> HIVE-13659.03.patch
>
>
> A partial query
> select count (distinct field) from table where field;
> Note the missing 'field=value'
> resulted in the following error in task logs, instead of failing early during 
> compile
> {code}
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> ... 17 more
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.SelectColumnIsTrue.evaluate(SelectColumnIsTrue.java:46)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:106)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> ... 18 more
> {code}
> Comment by Gunther:
> I think this works by implicitly converting the expr to boolean (if a cast is 
> necessary). This query looks legal to me and probably needs to be handle in 
> Vectorization
> Comment by Ashutosh:
> Oracle, postgres & sql server throws error for this if type of field is not 
> boolean. However, MySQL & Hive (with vectorization off) executes the query by 
> implicitly adding a cast to boolean. Hive shall be consistent in its b

[jira] [Updated] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

2016-05-07 Thread Matt McCline (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13713:

Description: 
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}

> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

2016-05-07 Thread Matt McCline (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275153#comment-15275153
 ] 

Matt McCline commented on HIVE-13713:
-

With the fix, vectorization_limit.q does vectorize Reducer 2:

{code}
Reducer 2 
Execution mode: vectorized
Reduce Operator Tree:
  Group By Operator
keys: KEY._col0 (type: tinyint), KEY._col1 (type: double)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 6144 Data size: 1320982 Basic stats: 
COMPLETE Column stats: NONE
Group By Operator
  aggregations: count(_col1)
  keys: _col0 (type: tinyint)
  mode: complete
  outputColumnNames: _col0, _col1
  Statistics: Num rows: 3072 Data size: 660491 Basic stats: 
COMPLETE Column stats: NONE
  Limit
Number of rows: 20
Statistics: Num rows: 20 Data size: 4300 Basic stats: 
COMPLETE Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 20 Data size: 4300 Basic stats: 
COMPLETE Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
{code}

> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13659) An empty where condition leads to vectorization exceptions instead of throwing a compile time error

2016-05-07 Thread Matt McCline (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275161#comment-15275161
 ] 

Matt McCline commented on HIVE-13659:
-

Committed to master.

> An empty where condition leads to vectorization exceptions instead of 
> throwing a compile time error
> ---
>
> Key: HIVE-13659
> URL: https://issues.apache.org/jira/browse/HIVE-13659
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HIVE-13659.01.patch, HIVE-13659.02.patch, 
> HIVE-13659.03.patch
>
>
> A partial query
> select count (distinct field) from table where field;
> Note the missing 'field=value'
> resulted in the following error in task logs, instead of failing early during 
> compile
> {code}
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> ... 17 more
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.SelectColumnIsTrue.evaluate(SelectColumnIsTrue.java:46)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:106)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> ... 18 more
> {code}
> Comment by Gunther:
> I think this works by implicitly converting the expr to boolean (if a cast is 
> necessary). This query looks legal to me and probably needs to be handle in 
> Vectorization
> Comment by Ashutosh:
> Oracle, postgres & sql server throws error for this if type of field is not 
> boolean. However, MySQL & Hive (with vectorization off) executes the query by 
> implicitly adding a cast to boolean. Hive shall be consistent in its behavior 
> regardless whether vectorization is on or off.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13659) An empty where condition leads to vectorization exceptions instead of throwing a compile time error

2016-05-07 Thread Matt McCline (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13659:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> An empty where condition leads to vectorization exceptions instead of 
> throwing a compile time error
> ---
>
> Key: HIVE-13659
> URL: https://issues.apache.org/jira/browse/HIVE-13659
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HIVE-13659.01.patch, HIVE-13659.02.patch, 
> HIVE-13659.03.patch
>
>
> A partial query
> select count (distinct field) from table where field;
> Note the missing 'field=value'
> resulted in the following error in task logs, instead of failing early during 
> compile
> {code}
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at 
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
> at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> ... 17 more
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to 
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.SelectColumnIsTrue.evaluate(SelectColumnIsTrue.java:46)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:106)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> ... 18 more
> {code}
> Comment by Gunther:
> I think this works by implicitly converting the expr to boolean (if a cast is 
> necessary). This query looks legal to me and probably needs to be handle in 
> Vectorization
> Comment by Ashutosh:
> Oracle, postgres & sql server throws error for this if type of field is not 
> boolean. However, MySQL & Hive (with vectorization off) executes the query by 
> implicitly adding a cast to boolean. Hive shall be consistent in its behavior 
> regardless whether vectorization is on or off.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12827) Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit isNull[offset] modification

2016-05-07 Thread Matt McCline (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275167#comment-15275167
 ] 

Matt McCline commented on HIVE-12827:
-

Added to errata.txt for branch-1.

> Vectorization: VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign 
> needs explicit isNull[offset] modification
> ---
>
> Key: HIVE-12827
> URL: https://issues.apache.org/jira/browse/HIVE-12827
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 1.3.0, 2.0.0, 2.1.0
>
> Attachments: HIVE-12827.2.patch
>
>
> Some scenarios do set Double.NaN instead of isNull=true, but all types aren't 
> consistent.
> Examples of un-set isNull for the valid values are 
> {code}
>   private class FloatReader extends AbstractDoubleReader {
> FloatReader(int columnIndex) {
>   super(columnIndex);
> }
> @Override
> void apply(VectorizedRowBatch batch, int batchIndex) throws IOException {
>   DoubleColumnVector colVector = (DoubleColumnVector) 
> batch.cols[columnIndex];
>   if (deserializeRead.readCheckNull()) {
> VectorizedBatchUtil.setNullColIsNullValue(colVector, batchIndex);
>   } else {
> float value = deserializeRead.readFloat();
> colVector.vector[batchIndex] = (double) value;
>   }
> }
>   }
> {code}
> {code}
>   private class DoubleCopyRow extends CopyRow {
> DoubleCopyRow(int inColumnIndex, int outColumnIndex) {
>   super(inColumnIndex, outColumnIndex);
> }
> @Override
> void copy(VectorizedRowBatch inBatch, int inBatchIndex, 
> VectorizedRowBatch outBatch, int outBatchIndex) {
>   DoubleColumnVector inColVector = (DoubleColumnVector) 
> inBatch.cols[inColumnIndex];
>   DoubleColumnVector outColVector = (DoubleColumnVector) 
> outBatch.cols[outColumnIndex];
>   if (inColVector.isRepeating) {
> if (inColVector.noNulls || !inColVector.isNull[0]) {
>   outColVector.vector[outBatchIndex] = inColVector.vector[0];
> } else {
>   VectorizedBatchUtil.setNullColIsNullValue(outColVector, 
> outBatchIndex);
> }
>   } else {
> if (inColVector.noNulls || !inColVector.isNull[inBatchIndex]) {
>   outColVector.vector[outBatchIndex] = 
> inColVector.vector[inBatchIndex];
> } else {
>   VectorizedBatchUtil.setNullColIsNullValue(outColVector, 
> outBatchIndex);
> }
>   }
> }
>   }
> {code}
> {code}
>  private static abstract class VectorDoubleColumnAssign
> extends VectorColumnAssignVectorBase {
> protected void assignDouble(double value, int destIndex) {
>   outCol.vector[destIndex] = value;
> }
>   }
> {code}
> The pattern to imitate would be the earlier code from VectorBatchUtil
> {code}
> case DOUBLE: {
>   DoubleColumnVector dcv = (DoubleColumnVector) batch.cols[offset + 
> colIndex];
>   if (writableCol != null) {
> dcv.vector[rowIndex] = ((DoubleWritable) writableCol).get();
> dcv.isNull[rowIndex] = false;
>   } else {
> dcv.vector[rowIndex] = Double.NaN;
> setNullColIsNullValue(dcv, rowIndex);
>   }
> }
>   break;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13352) Seems unnecessary for HBase tests to call QTestUtil.tearDown to close zookeeper and others.

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275180#comment-15275180
 ] 

Hive QA commented on HIVE-13352:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802387/HIVE-13352.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9937 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more 
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/201/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/201/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-201/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12802387 - PreCommit-HIVE-MASTER-Build

> Seems unnecessary for HBase tests to call QTestUtil.tearDown to close 
> zookeeper and others.
> ---
>
> Key: HIVE-13352
> URL: https://issues.apache.org/jira/browse/HIVE-13352
> Project: Hive
>  Issue Type: Improvement
>  Components: Test
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Balint Molnar
> Attachments: HIVE-13352.patch
>
>
> HBase tests TestHBaseCliDriver.java right now call QTestUtil.tearDown to turn 
> off Zookeeper and others after each test. Seems we can reuse them for all the 
> tests while we just need to clear all the test data similar to TestCliDriver.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13714) How to config hive metastore mysql connection pool size?

2016-05-07 Thread Tao Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275217#comment-15275217
 ] 

Tao Li commented on HIVE-13714:
---

The following config works me.


  datanucleus.connectionPool.maxPoolSize
  50


  datanucleus.connectionPool.minPoolSize
  20


> How to config hive metastore mysql connection pool size?
> 
>
> Key: HIVE-13714
> URL: https://issues.apache.org/jira/browse/HIVE-13714
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.1
>Reporter: Tao Li
>
> How to config hive metastore mysql connection pool size?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13696) Validate jobs submitted to fair-scheduler

2016-05-07 Thread Reuben Kuhnert (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reuben Kuhnert updated HIVE-13696:
--
Description: 
Ensure that jobs are placed into the correct queue according to 
{{fair-scheduler.xml}}. Jobs should be placed into the correct queue, and users 
should not be able to submit jobs to queues they do not have access to.

This patch builds on the existing functionality in {{FairSchedulerShim}} to 
route jobs to user-specific queue based on {{fair-scheduler.xml}} configuration 
(leveraging the Yarn {{QueuePlacementPolicy}} class). In addition to 
configuring job routing at session connect (current behavior), the routing is 
validated per submission to yarn (when impersonation is off). A 
{{FileSystemWatcher}} class is included to monitor changes in the 
{{fair-scheduler.xml}} file (so updates are automatically reloaded when the 
file pointed to by {{yarn.scheduler.fair.allocation.file}} is changed).

  was:Ensure that jobs are placed into the correct queue according to 
{{fair-scheduler.xml}}. Jobs should be placed into the correct queue, and users 
should not be able to submit jobs to queues they do not have access to.


> Validate jobs submitted to fair-scheduler
> -
>
> Key: HIVE-13696
> URL: https://issues.apache.org/jira/browse/HIVE-13696
> Project: Hive
>  Issue Type: Improvement
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
> Attachments: HIVE-13696.01.patch
>
>
> Ensure that jobs are placed into the correct queue according to 
> {{fair-scheduler.xml}}. Jobs should be placed into the correct queue, and 
> users should not be able to submit jobs to queues they do not have access to.
> This patch builds on the existing functionality in {{FairSchedulerShim}} to 
> route jobs to user-specific queue based on {{fair-scheduler.xml}} 
> configuration (leveraging the Yarn {{QueuePlacementPolicy}} class). In 
> addition to configuring job routing at session connect (current behavior), 
> the routing is validated per submission to yarn (when impersonation is off). 
> A {{FileSystemWatcher}} class is included to monitor changes in the 
> {{fair-scheduler.xml}} file (so updates are automatically reloaded when the 
> file pointed to by {{yarn.scheduler.fair.allocation.file}} is changed).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13693) Multi-insert query drops Filter before file output when there is a.val <> b.val

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275235#comment-15275235
 ] 

Hive QA commented on HIVE-13693:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802464/HIVE-13693.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 77 failed/errored test(s), 9939 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more 
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_multi_single_reducer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join38
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_map_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_masking_disablecbo_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_masking_disablecbo_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_masking_disablecbo_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_group_by
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query12
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query17
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query18
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query19
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query25
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query29

[jira] [Commented] (HIVE-13602) TPCH q16 return wrong result when CBO is on

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275236#comment-15275236
 ] 

Hive QA commented on HIVE-13602:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802463/HIVE-13602.03.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/203/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/203/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-203/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-203/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 3f3aa2a HIVE-12827: Vectorization: 
VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit 
isNull[offset] modification (errata.txt)
+ git clean -f -d
Removing ql/src/test/queries/clientpositive/multi_insert_with_join2.q
Removing ql/src/test/results/clientpositive/multi_insert_with_join2.q.out
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 3f3aa2a HIVE-12827: Vectorization: 
VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit 
isNull[offset] modification (errata.txt)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12802463 - PreCommit-HIVE-MASTER-Build

> TPCH q16 return wrong result when CBO is on
> ---
>
> Key: HIVE-13602
> URL: https://issues.apache.org/jira/browse/HIVE-13602
> Project: Hive
>  Issue Type: Bug
>  Components: CBO, Logical Optimizer
>Affects Versions: 1.3.0, 2.0.0, 1.2.2
>Reporter: Nemon Lou
>Assignee: Pengcheng Xiong
> Attachments: HIVE-13602.01.patch, HIVE-13602.03.patch, 
> calcite_cbo_bad.out, calcite_cbo_good.out, explain_cbo_bad_part1.out, 
> explain_cbo_bad_part2.out, explain_cbo_bad_part3.out, 
> explain_cbo_good(rewrite)_part1.out, explain_cbo_good(rewrite)_part2.out, 
> explain_cbo_good(rewrite)_part3.out
>
>
> Running tpch with factor 2, 
> q16 returns 1,160 rows when CBO is on,
> while returns 24,581 rows when CBO is off.
> See attachment for detail .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275237#comment-15275237
 ] 

Hive QA commented on HIVE-13542:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802469/HIVE-13542.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/204/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/204/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-204/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/lib64/qt-3.3/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-204/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 3f3aa2a HIVE-12827: Vectorization: 
VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit 
isNull[offset] modification (errata.txt)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 3f3aa2a HIVE-12827: Vectorization: 
VectorCopyRow/VectorAssignRow/VectorDeserializeRow assign needs explicit 
isNull[offset] modification (errata.txt)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12802469 - PreCommit-HIVE-MASTER-Build

> Missing stats for tables in TPCDS performance regression suite
> --
>
> Key: HIVE-13542
> URL: https://issues.apache.org/jira/browse/HIVE-13542
> Project: Hive
>  Issue Type: Bug
>  Components: Testing Infrastructure
>Affects Versions: 2.0.0
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Fix For: 2.1.0
>
> Attachments: HIVE-13542.1.patch, HIVE-13542.2.patch
>
>
> These are the tables whose stats are missing in 
> data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt:
> * catalog_returns
> * catalog_sales
> * inventory
> * store_returns
> * store_sales
> * web_returns
> * web_sales
> Thanks to [~jcamachorodriguez] for discovering this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13714) How to config hive metastore mysql connection pool size?

2016-05-07 Thread Tao Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275247#comment-15275247
 ] 

Tao Li commented on HIVE-13714:
---

The following config will change the mysql connection num. But the mysql 
connection num will be 75 in my case, which is greater than the maxPoolSize 50. 
I think if there is some bugs of metastore's mysql connection pool, which can't 
limit the max pool size?


datanucleus.connectionPool.maxPoolSize
50


datanucleus.connectionPool.minPoolSize
20


> How to config hive metastore mysql connection pool size?
> 
>
> Key: HIVE-13714
> URL: https://issues.apache.org/jira/browse/HIVE-13714
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.1
>Reporter: Tao Li
>
> How to config hive metastore mysql connection pool size?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command

2016-05-07 Thread Shinichi Yamashita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shinichi Yamashita updated HIVE-13425:
--
Attachment: HIVE-13425.3.patch

I attach a patch file with test query.

> Fix partition addition in MSCK REPAIR TABLE command
> ---
>
> Key: HIVE-13425
> URL: https://issues.apache.org/jira/browse/HIVE-13425
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch, 
> HIVE-13425.3.patch
>
>
> I create a table in following HiveQL.
> {code}
> hive> create table example (name string) partitioned by (id int);
> {code}
> And, I make some directories about this table in HDFS.
> {code}
> [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example
> drwxr-xr-x   - root hadoop  0 2016-04-05 22:21 
> /user/hive/warehouse/example/id=1
> drwxr-xr-x   - root hadoop  0 2016-04-05 22:22 
> /user/hive/warehouse/example/id=1/id=2
> -rw-r--r--   1 root hadoop  8 2016-04-05 22:22 
> /user/hive/warehouse/example/id=1/id=2/example.txt
> {code}
> Next I executed MSCK REPAIR TABLE command and added a partition. And this 
> result became as follows.
> {code}
> [root@hadoop ~]# hive -e 'msck repair table example'
> OK
> Partitions not in metastore:  example:id=1/id=2
> Repair: Added partition to metastore example:id=1/id=2
> Time taken: 1.243 seconds, Fetched: 2 row(s)
> [root@hadoop ~]# hive -e 'show partitions example'
> OK
> id=2
> {code}
> "id=1" should be a partition, but "id=2" was added. I will fix this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13525) HoS hangs when job is empty

2016-05-07 Thread Xuefu Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275251#comment-15275251
 ] 

Xuefu Zhang commented on HIVE-13525:


Hi [~lirui], thanks for working on this. The patch looks good to me. One thing 
I'm not very clear: what's the relationship between the deserialization error 
and the need for antlr jar?

> HoS hangs when job is empty
> ---
>
> Key: HIVE-13525
> URL: https://issues.apache.org/jira/browse/HIVE-13525
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13525.1.patch, HIVE-13525.2.patch, 
> HIVE-13525.3.patch
>
>
> Observed in local tests. This should be the cause of HIVE-13402.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command

2016-05-07 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275291#comment-15275291
 ] 

Ashutosh Chauhan commented on HIVE-13425:
-

I am not sure if this is a right fix. e.g., if table is partitioned by (a int, 
b int) then for tbl/a=1/b=2 partspec will become (b=2/a=1) instead of (a=1/b=2) 
That would be incorrect. 
Also, can you take a look at HIVE-13703 where we might end up with dir 
structure after executing union query on Tez. I presume patch from here will 
also fix that. If so, can you add a test case for that too?

> Fix partition addition in MSCK REPAIR TABLE command
> ---
>
> Key: HIVE-13425
> URL: https://issues.apache.org/jira/browse/HIVE-13425
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch, 
> HIVE-13425.3.patch
>
>
> I create a table in following HiveQL.
> {code}
> hive> create table example (name string) partitioned by (id int);
> {code}
> And, I make some directories about this table in HDFS.
> {code}
> [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example
> drwxr-xr-x   - root hadoop  0 2016-04-05 22:21 
> /user/hive/warehouse/example/id=1
> drwxr-xr-x   - root hadoop  0 2016-04-05 22:22 
> /user/hive/warehouse/example/id=1/id=2
> -rw-r--r--   1 root hadoop  8 2016-04-05 22:22 
> /user/hive/warehouse/example/id=1/id=2/example.txt
> {code}
> Next I executed MSCK REPAIR TABLE command and added a partition. And this 
> result became as follows.
> {code}
> [root@hadoop ~]# hive -e 'msck repair table example'
> OK
> Partitions not in metastore:  example:id=1/id=2
> Repair: Added partition to metastore example:id=1/id=2
> Time taken: 1.243 seconds, Fetched: 2 row(s)
> [root@hadoop ~]# hive -e 'show partitions example'
> OK
> id=2
> {code}
> "id=1" should be a partition, but "id=2" was added. I will fix this problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13352) Seems unnecessary for HBase tests to call QTestUtil.tearDown to close zookeeper and others.

2016-05-07 Thread Ashutosh Chauhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-13352:

   Resolution: Fixed
Fix Version/s: 2.1.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks, [~baluchicken]

> Seems unnecessary for HBase tests to call QTestUtil.tearDown to close 
> zookeeper and others.
> ---
>
> Key: HIVE-13352
> URL: https://issues.apache.org/jira/browse/HIVE-13352
> Project: Hive
>  Issue Type: Improvement
>  Components: Testing Infrastructure
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Balint Molnar
> Fix For: 2.1.0
>
> Attachments: HIVE-13352.patch
>
>
> HBase tests TestHBaseCliDriver.java right now call QTestUtil.tearDown to turn 
> off Zookeeper and others after each test. Seems we can reuse them for all the 
> tests while we just need to clear all the test data similar to TestCliDriver.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13352) Seems unnecessary for HBase tests to call QTestUtil.tearDown to close zookeeper and others.

2016-05-07 Thread Ashutosh Chauhan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-13352:

Component/s: (was: Test)
 Testing Infrastructure

> Seems unnecessary for HBase tests to call QTestUtil.tearDown to close 
> zookeeper and others.
> ---
>
> Key: HIVE-13352
> URL: https://issues.apache.org/jira/browse/HIVE-13352
> Project: Hive
>  Issue Type: Improvement
>  Components: Testing Infrastructure
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Balint Molnar
> Fix For: 2.1.0
>
> Attachments: HIVE-13352.patch
>
>
> HBase tests TestHBaseCliDriver.java right now call QTestUtil.tearDown to turn 
> off Zookeeper and others after each test. Seems we can reuse them for all the 
> tests while we just need to clear all the test data similar to TestCliDriver.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275300#comment-15275300
 ] 

Hive QA commented on HIVE-13449:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802541/HIVE-13449.03.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9938 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more 
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testThreadPoolMetrics
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/205/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/205/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-205/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12802541 - PreCommit-HIVE-MASTER-Build

> LLAP: HS2 should get the token directly, rather than from LLAP
> --
>
> Key: HIVE-13449
> URL: https://issues.apache.org/jira/browse/HIVE-13449
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13449.01.patch, HIVE-13449.02.WIP.patch, 
> HIVE-13449.02.patch, HIVE-13449.03.patch, HIVE-13449.patch
>
>
> HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager 
> directly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13351) Support drop Primary Key/Foreign Key constraints

2016-05-07 Thread Sushanth Sowmyan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275342#comment-15275342
 ] 

Sushanth Sowmyan commented on HIVE-13351:
-

[~hsubramaniyan], the commit for this patch introduced a new test failure, in 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/200/testReport/org.apache.hadoop.hive.ql.security.authorization.plugin/TestHiveOperationType/checkHiveOperationTypeMatch/

(Note that this test failure report is from another jira, but was caused by the 
commit for this one)

{noformat}
Error Message

Unable to find corresponding type in HiveOperationType for 
ALTERTABLE_DROPCONSTRAINT : java.lang.IllegalArgumentException: No enum 
constant 
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveOperationType.ALTERTABLE_DROPCONSTRAINT
Stacktrace

java.lang.AssertionError: Unable to find corresponding type in 
HiveOperationType for ALTERTABLE_DROPCONSTRAINT : 
java.lang.IllegalArgumentException: No enum constant 
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveOperationType.ALTERTABLE_DROPCONSTRAINT
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch(TestHiveOperationType.java:40)
{noformat}

Basically, TestHiveOperationType tries to ensure that for each new 
HiveOperation we add, we make sure that we add a corresponding authorization 
definition in HiveOperationType, so that external security frameworks like 
Apache Ranger can use it. Please create a followup jira to add an entry in 
HiveOperationType, and in conjunction, you will want to add an entry in 
Operation2Privilege as well.


> Support drop Primary Key/Foreign Key constraints
> 
>
> Key: HIVE-13351
> URL: https://issues.apache.org/jira/browse/HIVE-13351
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>  Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch
>
>
> ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME;
> The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there 
> should be atleast 1 table column of TABLENAME participating in the constraint.
> Otherwise, we should throw an error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13670) Improve Beeline reconnect semantics

2016-05-07 Thread Sushanth Sowmyan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275351#comment-15275351
 ] 

Sushanth Sowmyan commented on HIVE-13670:
-

The test failures reported on 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/200/testReport/
 are not related to this patch, and are not regressions (they have been failing 
for the past runs as well), and the new tests added for this patch have 
succeeded.

> Improve Beeline reconnect semantics
> ---
>
> Key: HIVE-13670
> URL: https://issues.apache.org/jira/browse/HIVE-13670
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-13670.2.patch, HIVE-13670.3.patch, HIVE-13670.patch
>
>
> For most users of beeline, chances are that they will be using it with a 
> single HS2 instance most of the time. In this scenario, having them type out 
> a jdbc uri for HS2 every single time to !connect can get tiresome. Thus, we 
> should improve semantics so that if a user does a successful !connect, then 
> we must store the last-connected-to-url, so that if they do a !close, and 
> then a !reconnect, then !reconnect should attempt to connect to the last 
> successfully used url.
> Also, if they then do a !save, then that last-successfully-used url must be 
> saved, so that in subsequent sessions, they can simply do !reconnect rather 
> than specifying a url for !connect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13670) Improve Beeline connect/reconnect semantics

2016-05-07 Thread Sushanth Sowmyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-13670:

Summary: Improve Beeline connect/reconnect semantics  (was: Improve Beeline 
reconnect semantics)

> Improve Beeline connect/reconnect semantics
> ---
>
> Key: HIVE-13670
> URL: https://issues.apache.org/jira/browse/HIVE-13670
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-13670.2.patch, HIVE-13670.3.patch, HIVE-13670.patch
>
>
> For most users of beeline, chances are that they will be using it with a 
> single HS2 instance most of the time. In this scenario, having them type out 
> a jdbc uri for HS2 every single time to !connect can get tiresome. Thus, we 
> should improve semantics so that if a user does a successful !connect, then 
> we must store the last-connected-to-url, so that if they do a !close, and 
> then a !reconnect, then !reconnect should attempt to connect to the last 
> successfully used url.
> Also, if they then do a !save, then that last-successfully-used url must be 
> saved, so that in subsequent sessions, they can simply do !reconnect rather 
> than specifying a url for !connect.
> In addition, it would be useful to introduce a new way of doing !connect that 
> does involve typing out a jdbc url every time (since this is highly likely to 
> be error-prone)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13670) Improve Beeline reconnect semantics

2016-05-07 Thread Sushanth Sowmyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-13670:

Description: 
For most users of beeline, chances are that they will be using it with a single 
HS2 instance most of the time. In this scenario, having them type out a jdbc 
uri for HS2 every single time to !connect can get tiresome. Thus, we should 
improve semantics so that if a user does a successful !connect, then we must 
store the last-connected-to-url, so that if they do a !close, and then a 
!reconnect, then !reconnect should attempt to connect to the last successfully 
used url.

Also, if they then do a !save, then that last-successfully-used url must be 
saved, so that in subsequent sessions, they can simply do !reconnect rather 
than specifying a url for !connect.

In addition, it would be useful to introduce a new way of doing !connect that 
does involve typing out a jdbc url every time (since this is highly likely to 
be error-prone)

  was:
For most users of beeline, chances are that they will be using it with a single 
HS2 instance most of the time. In this scenario, having them type out a jdbc 
uri for HS2 every single time to !connect can get tiresome. Thus, we should 
improve semantics so that if a user does a successful !connect, then we must 
store the last-connected-to-url, so that if they do a !close, and then a 
!reconnect, then !reconnect should attempt to connect to the last successfully 
used url.

Also, if they then do a !save, then that last-successfully-used url must be 
saved, so that in subsequent sessions, they can simply do !reconnect rather 
than specifying a url for !connect.


> Improve Beeline reconnect semantics
> ---
>
> Key: HIVE-13670
> URL: https://issues.apache.org/jira/browse/HIVE-13670
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-13670.2.patch, HIVE-13670.3.patch, HIVE-13670.patch
>
>
> For most users of beeline, chances are that they will be using it with a 
> single HS2 instance most of the time. In this scenario, having them type out 
> a jdbc uri for HS2 every single time to !connect can get tiresome. Thus, we 
> should improve semantics so that if a user does a successful !connect, then 
> we must store the last-connected-to-url, so that if they do a !close, and 
> then a !reconnect, then !reconnect should attempt to connect to the last 
> successfully used url.
> Also, if they then do a !save, then that last-successfully-used url must be 
> saved, so that in subsequent sessions, they can simply do !reconnect rather 
> than specifying a url for !connect.
> In addition, it would be useful to introduce a new way of doing !connect that 
> does involve typing out a jdbc url every time (since this is highly likely to 
> be error-prone)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13390) HiveServer2: Add more test to ZK service discovery using MiniHS2

2016-05-07 Thread Sushanth Sowmyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushanth Sowmyan updated HIVE-13390:

Fix Version/s: 2.1.0
   1.2.2
   1.3.0

> HiveServer2: Add more test to ZK service discovery using MiniHS2
> 
>
> Key: HIVE-13390
> URL: https://issues.apache.org/jira/browse/HIVE-13390
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 1.3.0, 1.2.2, 2.1.0, 2.0.1
>
> Attachments: HIVE-13390.1.patch, HIVE-13390.1.patch, 
> HIVE-13390.2.patch, HIVE-13390.3.patch, HIVE-13390.branch-1.2.patch, 
> HIVE-13390.branch-1.patch, keystore.jks, keystore_exampledotcom.jks, 
> truststore.jks
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13390) HiveServer2: Add more test to ZK service discovery using MiniHS2

2016-05-07 Thread Sushanth Sowmyan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275357#comment-15275357
 ] 

Sushanth Sowmyan commented on HIVE-13390:
-

Thanks for the spot, [~leftylev], I've added them.

> HiveServer2: Add more test to ZK service discovery using MiniHS2
> 
>
> Key: HIVE-13390
> URL: https://issues.apache.org/jira/browse/HIVE-13390
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Fix For: 1.3.0, 1.2.2, 2.1.0, 2.0.1
>
> Attachments: HIVE-13390.1.patch, HIVE-13390.1.patch, 
> HIVE-13390.2.patch, HIVE-13390.3.patch, HIVE-13390.branch-1.2.patch, 
> HIVE-13390.branch-1.patch, keystore.jks, keystore_exampledotcom.jks, 
> truststore.jks
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13699) Make JavaDataModel#get thread safe for parallel compilation

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275371#comment-15275371
 ] 

Hive QA commented on HIVE-13699:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802547/HIVE-13699.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 9938 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more 
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/206/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/206/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-206/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12802547 - PreCommit-HIVE-MASTER-Build

> Make JavaDataModel#get thread safe for parallel compilation
> ---
>
> Key: HIVE-13699
> URL: https://issues.apache.org/jira/browse/HIVE-13699
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, storage-api
>Affects Versions: 2.0.0
>Reporter: Peter Slawski
>Assignee: Peter Slawski
>Priority: Minor
> Attachments: HIVE-13699.1.patch
>
>
> The class JavaDataModel has a static method, #get, that is not thread safe. 
> This may be an issue when parallel query compilation is enabled because two 
> threads may attempt to call JavaDataModel#get at the same time, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13351) Support drop Primary Key/Foreign Key constraints

2016-05-07 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275387#comment-15275387
 ] 

Hari Sankar Sivarama Subramaniyan commented on HIVE-13351:
--

[~sushanth] HIVE-13700 is the jira, I have uploaded a patch there, can you 
please review it.

Thanks
Hari


> Support drop Primary Key/Foreign Key constraints
> 
>
> Key: HIVE-13351
> URL: https://issues.apache.org/jira/browse/HIVE-13351
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>  Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch
>
>
> ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME;
> The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there 
> should be atleast 1 table column of TABLENAME participating in the constraint.
> Otherwise, we should throw an error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13700) TestHiveOperationType is failing on master

2016-05-07 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-13700:
-
Attachment: HIVE-13700.1.patch

> TestHiveOperationType is failing on master
> --
>
> Key: HIVE-13700
> URL: https://issues.apache.org/jira/browse/HIVE-13700
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.1.0
>Reporter: Ashutosh Chauhan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13700.1.patch
>
>
> Presumably be broken by HIVE-13351



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13700) TestHiveOperationType is failing on master

2016-05-07 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-13700:
-
Status: Patch Available  (was: Open)

> TestHiveOperationType is failing on master
> --
>
> Key: HIVE-13700
> URL: https://issues.apache.org/jira/browse/HIVE-13700
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.1.0
>Reporter: Ashutosh Chauhan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13700.1.patch
>
>
> Presumably be broken by HIVE-13351



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table

2016-05-07 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-13598:
-
Status: Patch Available  (was: Open)

> Describe extended table should show the primary keys/foreign keys associated 
> with the table
> ---
>
> Key: HIVE-13598
> URL: https://issues.apache.org/jira/browse/HIVE-13598
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, 
> HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch
>
>
> After HIVE-13290 is committed, we need to show the constraints as part of 
> table description when extended label is used. Currently, the constraints 
> would not be shown as part of table description since Constraint is a 
> separate entity.
> The purpose of the jira is to modify Hive.describeTable() to enable the user 
> to view the constraints associated with the table when the user does a 
> "describe extended table";



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table

2016-05-07 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-13598:
-
Attachment: (was: HIVE-13598.5.patch)

> Describe extended table should show the primary keys/foreign keys associated 
> with the table
> ---
>
> Key: HIVE-13598
> URL: https://issues.apache.org/jira/browse/HIVE-13598
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, 
> HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch
>
>
> After HIVE-13290 is committed, we need to show the constraints as part of 
> table description when extended label is used. Currently, the constraints 
> would not be shown as part of table description since Constraint is a 
> separate entity.
> The purpose of the jira is to modify Hive.describeTable() to enable the user 
> to view the constraints associated with the table when the user does a 
> "describe extended table";



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table

2016-05-07 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-13598:
-
Attachment: HIVE-13598.5.patch

> Describe extended table should show the primary keys/foreign keys associated 
> with the table
> ---
>
> Key: HIVE-13598
> URL: https://issues.apache.org/jira/browse/HIVE-13598
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, 
> HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch
>
>
> After HIVE-13290 is committed, we need to show the constraints as part of 
> table description when extended label is used. Currently, the constraints 
> would not be shown as part of table description since Constraint is a 
> separate entity.
> The purpose of the jira is to modify Hive.describeTable() to enable the user 
> to view the constraints associated with the table when the user does a 
> "describe extended table";



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table

2016-05-07 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Sankar Sivarama Subramaniyan updated HIVE-13598:
-
Status: Open  (was: Patch Available)

> Describe extended table should show the primary keys/foreign keys associated 
> with the table
> ---
>
> Key: HIVE-13598
> URL: https://issues.apache.org/jira/browse/HIVE-13598
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, 
> HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch
>
>
> After HIVE-13290 is committed, we need to show the constraints as part of 
> table description when extended label is used. Currently, the constraints 
> would not be shown as part of table description since Constraint is a 
> separate entity.
> The purpose of the jira is to modify Hive.describeTable() to enable the user 
> to view the constraints associated with the table when the user does a 
> "describe extended table";



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275415#comment-15275415
 ] 

Hive QA commented on HIVE-11793:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802556/HIVE-11793.6.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 9939 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more 
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/207/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/207/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-207/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12802556 - PreCommit-HIVE-MASTER-Build

> SHOW LOCKS with DbTxnManager ignores filter options
> ---
>
> Key: HIVE-11793
> URL: https://issues.apache.org/jira/browse/HIVE-11793
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Minor
> Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, 
> HIVE-11793.3.patch, HIVE-11793.4.patch, HIVE-11793.5.patch, HIVE-11793.6.patch
>
>
> https://cwiki.apache.org/confluence/display/Hive/Locking and 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks
>  list various options that can be used with SHOW LOCKS, e.g. 
> When ACID is enabled, all these options are ignored and a full list is 
> returned.
> (also only ext lock id is shown, int lock id is not).
> see DDLTask.showLocks() and TxnHandler.showLocks()
> requires extending ShowLocksRequest which is a Thrift object



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11417) Create shims for the row by row read path that is backed by VectorizedRowBatch

2016-05-07 Thread Prasanth Jayachandran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275430#comment-15275430
 ] 

Prasanth Jayachandran commented on HIVE-11417:
--

Can you post the patch to RB? I cannot fully see the diff in github. It says 
too large to display. Atleast RB will paginate big patches. So far mostly the 
looks good. I cannot see the diff for around 10 files at the last.

> Create shims for the row by row read path that is backed by VectorizedRowBatch
> --
>
> Key: HIVE-11417
> URL: https://issues.apache.org/jira/browse/HIVE-11417
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 2.1.0
>
> Attachments: HIVE-11417.patch, HIVE-11417.patch, HIVE-11417.patch, 
> HIVE-11417.patch, HIVE-11417.patch
>
>
> I'd like to make the default path for reading and writing ORC files to be 
> vectorized. To ensure that Hive can still read row by row, we'll need shims 
> to support the old API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13696) Validate jobs submitted to fair-scheduler

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275457#comment-15275457
 ] 

Hive QA commented on HIVE-13696:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802507/HIVE-13696.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 9922 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more 
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveAuthorizerCheckInvocation.org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveAuthorizerCheckInvocation
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveAuthorizerShowFilters.org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveAuthorizerShowFilters
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/208/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/208/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-208/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12802507 - PreCommit-HIVE-MASTER-Build

> Validate jobs submitted to fair-scheduler
> -
>
> Key: HIVE-13696
> URL: https://issues.apache.org/jira/browse/HIVE-13696
> Project: Hive
>  Issue Type: Improvement
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
> Attachments: HIVE-13696.01.patch
>
>
> Ensure that jobs are placed into the correct queue according to 
> {{fair-scheduler.xml}}. Jobs should be placed into the correct queue, and 
> users should not be able to submit jobs to queues they do not have access to.
> This patch builds on the existing functionality in {{FairSchedulerShim}} to 
> route jobs to user-specific queue based on {{fair-scheduler.xml}} 
> configuration (leveraging the Yarn {{QueuePlacementPolicy}} class). In 
> addition to configuring job routing at session connect (current behavior), 
> the routing is validated per submission to yarn (when impersonation is off). 
> A {{FileSystemWatcher}} class is included to monitor changes in the 
> {{fair-scheduler.xml}} file (so updates are automatically reloaded when the 
> file pointed to by {{yarn.scheduler.fair.allocation.file}} is changed).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13700) TestHiveOperationType is failing on master

2016-05-07 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275461#comment-15275461
 ] 

Ashutosh Chauhan commented on HIVE-13700:
-

+1

> TestHiveOperationType is failing on master
> --
>
> Key: HIVE-13700
> URL: https://issues.apache.org/jira/browse/HIVE-13700
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.1.0
>Reporter: Ashutosh Chauhan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13700.1.patch
>
>
> Presumably be broken by HIVE-13351



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table

2016-05-07 Thread Ashutosh Chauhan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275462#comment-15275462
 ] 

Ashutosh Chauhan commented on HIVE-13598:
-

+1

> Describe extended table should show the primary keys/foreign keys associated 
> with the table
> ---
>
> Key: HIVE-13598
> URL: https://issues.apache.org/jira/browse/HIVE-13598
> Project: Hive
>  Issue Type: Bug
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, 
> HIVE-13598.3.patch, HIVE-13598.4.patch, HIVE-13598.5.patch
>
>
> After HIVE-13290 is committed, we need to show the constraints as part of 
> table description when extended label is used. Currently, the constraints 
> would not be shown as part of table description since Constraint is a 
> separate entity.
> The purpose of the jira is to modify Hive.describeTable() to enable the user 
> to view the constraints associated with the table when the user does a 
> "describe extended table";



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13525) HoS hangs when job is empty

2016-05-07 Thread Rui Li (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275463#comment-15275463
 ] 

Rui Li commented on HIVE-13525:
---

Thanks [~szehon] and [~xuefuz] for the review.
What triggers the deserialization error is {{NoClassDefFoundError: 
org/antlr/runtime/tree/CommonTree}}. It only happens for local-cluster mode. 
I'm not sure why the class is needed, but adding the antlr jar to driver's 
class path can workaround the issue. Otherwise, any failed task may make the 
job hanging.

> HoS hangs when job is empty
> ---
>
> Key: HIVE-13525
> URL: https://issues.apache.org/jira/browse/HIVE-13525
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13525.1.patch, HIVE-13525.2.patch, 
> HIVE-13525.3.patch
>
>
> Observed in local tests. This should be the cause of HIVE-13402.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13525) HoS hangs when job is empty

2016-05-07 Thread Xuefu Zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275468#comment-15275468
 ] 

Xuefu Zhang commented on HIVE-13525:


Thanks for the explanation, [~lirui]. +1.

> HoS hangs when job is empty
> ---
>
> Key: HIVE-13525
> URL: https://issues.apache.org/jira/browse/HIVE-13525
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13525.1.patch, HIVE-13525.2.patch, 
> HIVE-13525.3.patch
>
>
> Observed in local tests. This should be the cause of HIVE-13402.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options

2016-05-07 Thread Wei Zheng (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275475#comment-15275475
 ] 

Wei Zheng commented on HIVE-11793:
--

No new failures. [~ekoifman] Can you take another look?

> SHOW LOCKS with DbTxnManager ignores filter options
> ---
>
> Key: HIVE-11793
> URL: https://issues.apache.org/jira/browse/HIVE-11793
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Minor
> Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, 
> HIVE-11793.3.patch, HIVE-11793.4.patch, HIVE-11793.5.patch, HIVE-11793.6.patch
>
>
> https://cwiki.apache.org/confluence/display/Hive/Locking and 
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks
>  list various options that can be used with SHOW LOCKS, e.g. 
> When ACID is enabled, all these options are ignored and a full list is 
> returned.
> (also only ext lock id is shown, int lock id is not).
> see DDLTask.showLocks() and TxnHandler.showLocks()
> requires extending ShowLocksRequest which is a Thrift object



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

2016-05-07 Thread Matt McCline (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13713:

Attachment: HIVE-13713.01.patch

> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13713.01.patch
>
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

2016-05-07 Thread Matt McCline (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275482#comment-15275482
 ] 

Matt McCline commented on HIVE-13713:
-

(Running tests on internal PTest cluster)

> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13713.01.patch
>
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions

2016-05-07 Thread Hive QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275484#comment-15275484
 ] 

Hive QA commented on HIVE-13249:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12802606/HIVE-13249.7.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 9939 tests 
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did 
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
 - did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
 - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
 - did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more 
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.ql.security.authorization.plugin.TestHiveOperationType.checkHiveOperationTypeMatch
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}

Test results: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/210/testReport
Console output: 
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/210/console
Test logs: 
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-210/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12802606 - PreCommit-HIVE-MASTER-Build

> Hard upper bound on number of open transactions
> ---
>
> Key: HIVE-13249
> URL: https://issues.apache.org/jira/browse/HIVE-13249
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13249.1.patch, HIVE-13249.2.patch, 
> HIVE-13249.3.patch, HIVE-13249.4.patch, HIVE-13249.5.patch, 
> HIVE-13249.6.patch, HIVE-13249.7.patch
>
>
> We need to have a safeguard by adding an upper bound for open transactions to 
> avoid huge number of open-transaction requests, usually due to improper 
> configuration of clients such as Storm.
> Once that limit is reached, clients will start failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

2016-05-07 Thread Matt McCline (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13713:

Description: 
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}

It was producing a stack trace like:

{code}
Vector aggregate not implemented: "count" for type: "NONE (reduce-merge-partial 
= true)
{code}

  was:
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}

It was throwing an error like:

{code}
Vector aggregate not implemented: "count" for type: "NONE (reduce-merge-partial 
= true)
{code}


> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13713.01.patch
>
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}
> It was producing a stack trace like:
> {code}
> Vector aggregate not implemented: "count" for type: "NONE 
> (reduce-merge-partial = true)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

2016-05-07 Thread Matt McCline (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13713:

Description: 
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}

It was throwing an error like:

{code}
Vector aggregate not implemented: "count" for type: "NONE (reduce-merge-partial 
= true)
{code}

  was:
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}


> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13713.01.patch
>
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}
> It was throwing an error like:
> {code}
> Vector aggregate not implemented: "count" for type: "NONE 
> (reduce-merge-partial = true)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

2016-05-07 Thread Matt McCline (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13713:

Description: 
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}

It was producing a stack trace with this error...

{code}
Vector aggregate not implemented: "count" for type: "NONE (reduce-merge-partial 
= true)
{code}

  was:
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}

It was producing a stack trace like:

{code}
Vector aggregate not implemented: "count" for type: "NONE (reduce-merge-partial 
= true)
{code}


> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13713.01.patch
>
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}
> It was producing a stack trace with this error...
> {code}
> Vector aggregate not implemented: "count" for type: "NONE 
> (reduce-merge-partial = true)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-13713) We miss vectorization in a case of count(*) when aggregation mode is COMPLETE

2016-05-07 Thread Matt McCline (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-13713:

Description: 
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}

It was producing a stack trace with this error... when trying to vectorize the 
COMPLETE mode GROUP BY operator.

{code}
Vector aggregate not implemented: "count" for type: "NONE (reduce-merge-partial 
= true)
{code}

  was:
E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:

{code}
"select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
order by ctinyint limit 20"
{code}

It was producing a stack trace with this error...

{code}
Vector aggregate not implemented: "count" for type: "NONE (reduce-merge-partial 
= true)
{code}


> We miss vectorization in a case of count(*) when aggregation mode is COMPLETE
> -
>
> Key: HIVE-13713
> URL: https://issues.apache.org/jira/browse/HIVE-13713
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13713.01.patch
>
>
> E.g. vectorization_limit.q doesn't vectorize Reducer 2 for the query:
> {code}
> "select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint 
> order by ctinyint limit 20"
> {code}
> It was producing a stack trace with this error... when trying to vectorize 
> the COMPLETE mode GROUP BY operator.
> {code}
> Vector aggregate not implemented: "count" for type: "NONE 
> (reduce-merge-partial = true)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)