[jira] [Commented] (HIVE-9448) Merge spark to trunk 1/23/15

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290497#comment-14290497
 ] 

Szehon Ho commented on HIVE-9448:
-

Failure looks like due to HIVE-9341, basically just some output diff.  
[~brocknoland] if you have time can you help regenerate the output file in 
these few days?  If not, I can look on Monday.

> Merge spark to trunk 1/23/15
> 
>
> Key: HIVE-9448
> URL: https://issues.apache.org/jira/browse/HIVE-9448
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 0.15.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-9448.2.patch, HIVE-9448.patch
>
>
> Merging latest spark changes to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-9443:
--
Attachment: HIVE-9443.3.patch

Reuploading.

> ORC PPD - fix fuzzy case evaluation of IS_NULL
> --
>
> Key: HIVE-9443
> URL: https://issues.apache.org/jira/browse/HIVE-9443
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.15.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.15.0
>
> Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch, HIVE-9443.3.patch
>
>
> CLEAR LIBRARY CACHE
> ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
> The code-flow in effect should be
> {code}
> if (min == null) {
>   // all nulls
>   return YES;
> } else {
>   if (hasNull == true) {
>  // some nulls
>  return YES_NO; //maybe
>   }
>   // no nulls
>   return NO;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-9443:
--
Status: Patch Available  (was: Open)

> ORC PPD - fix fuzzy case evaluation of IS_NULL
> --
>
> Key: HIVE-9443
> URL: https://issues.apache.org/jira/browse/HIVE-9443
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.15.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.15.0
>
> Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch, HIVE-9443.3.patch
>
>
> CLEAR LIBRARY CACHE
> ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
> The code-flow in effect should be
> {code}
> if (min == null) {
>   // all nulls
>   return YES;
> } else {
>   if (hasNull == true) {
>  // some nulls
>  return YES_NO; //maybe
>   }
>   // no nulls
>   return NO;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-9443:
--
Description: 
CLEAR LIBRARY CACHE

ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.

The code-flow in effect should be

{code}
if (min == null) {
  // all nulls
  return YES;
} else {
  if (hasNull == true) {
 // some nulls
 return YES_NO; //maybe
  }
  // no nulls
  return NO;
}
{code}

  was:
ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.

The code-flow in effect should be

{code}
if (min == null) {
  // all nulls
  return YES;
} else {
  if (hasNull == true) {
 // some nulls
 return YES_NO; //maybe
  }
  // no nulls
  return NO;
}
{code}


> ORC PPD - fix fuzzy case evaluation of IS_NULL
> --
>
> Key: HIVE-9443
> URL: https://issues.apache.org/jira/browse/HIVE-9443
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.15.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.15.0
>
> Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch, HIVE-9443.3.patch
>
>
> CLEAR LIBRARY CACHE
> ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
> The code-flow in effect should be
> {code}
> if (min == null) {
>   // all nulls
>   return YES;
> } else {
>   if (hasNull == true) {
>  // some nulls
>  return YES_NO; //maybe
>   }
>   // no nulls
>   return NO;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-9443:
--
Status: Open  (was: Patch Available)

> ORC PPD - fix fuzzy case evaluation of IS_NULL
> --
>
> Key: HIVE-9443
> URL: https://issues.apache.org/jira/browse/HIVE-9443
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.15.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.15.0
>
> Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch
>
>
> ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
> The code-flow in effect should be
> {code}
> if (min == null) {
>   // all nulls
>   return YES;
> } else {
>   if (hasNull == true) {
>  // some nulls
>  return YES_NO; //maybe
>   }
>   // no nulls
>   return NO;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9416) Get rid of Extract Operator

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290477#comment-14290477
 ] 

Hive QA commented on HIVE-9416:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694291/HIVE-9416.4.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2503/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2503/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2503/

Messages:
{noformat}
 This message was trimmed, see log for full details 
Reverted 'ql/src/test/results/clientpositive/spark/auto_join31.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/skewjoinopt2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_reorder3.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/outer_join_ppr.q.java1.7.out'
Reverted 'ql/src/test/results/clientpositive/spark/union_remove_20.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/tez_joins_explain.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join10.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/vectorization_13.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_cond_pushdown_2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/mapjoin_mapjoin.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/ppd_transform.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/skewjoinopt14.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union_remove_9.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_hive_626.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/router_join_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join7.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_join28.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union5.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join18_multi_distinct.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union_remove_4.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/mergejoins_mixed.q.out'
Reverted 
'ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer3.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/ppd_outer_join3.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/skewjoinopt8.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/having.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_vc.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_join23.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union15.q.out'
Reverted 
'ql/src/test/results/clientpositive/spark/join_cond_pushdown_unqual2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join16.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_join32.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/skewjoinopt3.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/ppd_join.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union_remove_21.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/ppd_join2.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union33.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join11.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/vectorization_14.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join34.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join_cond_pushdown_3.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/groupby10.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join20.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/multi_insert_mixed.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/groupby_cube1.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join8.q.out'
Reverted 
'ql/src/test/results/clientpositive/spark/reduce_deduplicate_exclude_join.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_join29.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/tez_join_tests.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out'
Reverted 
'ql/src/test/results/clientpositive/spark/limit_partition_metadataonly.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union_remove_18.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/join0.q.java1.7.out'
Reverted 'ql/src/test/results/clientpositive/spark/mapreduce1.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/union_remove_5.q.out'
Reverted 'ql/src/test/results/clientpositive/spark/auto_join15.q.out'
Reverted 'ql/src/test/results/client

[jira] [Commented] (HIVE-9448) Merge spark to trunk 1/23/15

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290476#comment-14290476
 ] 

Hive QA commented on HIVE-9448:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694228/HIVE-9448.2.patch

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 7375 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap_auto
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ptf_streaming
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2502/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2502/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2502/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12694228 - PreCommit-HIVE-TRUNK-Build

> Merge spark to trunk 1/23/15
> 
>
> Key: HIVE-9448
> URL: https://issues.apache.org/jira/browse/HIVE-9448
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 0.15.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-9448.2.patch, HIVE-9448.patch
>
>
> Merging latest spark changes to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9453) Initial patch [hbase-metastore branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290443#comment-14290443
 ] 

Hive QA commented on HIVE-9453:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694218/HIVE-9453.patch

{color:red}ERROR:{color} -1 due to 64 failed/errored test(s), 7409 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriv

[jira] [Updated] (HIVE-9039) Support Union Distinct

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-9039:
--
Status: Open  (was: Patch Available)

> Support Union Distinct
> --
>
> Key: HIVE-9039
> URL: https://issues.apache.org/jira/browse/HIVE-9039
> Project: Hive
>  Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, 
> HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, 
> HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch, 
> HIVE-9039.09.patch, HIVE-9039.10.patch, HIVE-9039.11.patch, 
> HIVE-9039.12.patch, HIVE-9039.13.patch, HIVE-9039.14.patch, 
> HIVE-9039.15.patch, HIVE-9039.16.patch, HIVE-9039.17.patch, 
> HIVE-9039.18.patch, HIVE-9039.19.patch, HIVE-9039.20.patch, HIVE-9039.21.patch
>
>
> CLEAR LIBRARY CACHE
> Current version (Hive 0.14) does not support union (or union distinct). It 
> only supports union all. In this patch, we try to add this new feature by 
> rewriting union distinct to union all followed by group by.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9039) Support Union Distinct

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-9039:
--
Status: Patch Available  (was: Open)

> Support Union Distinct
> --
>
> Key: HIVE-9039
> URL: https://issues.apache.org/jira/browse/HIVE-9039
> Project: Hive
>  Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, 
> HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, 
> HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch, 
> HIVE-9039.09.patch, HIVE-9039.10.patch, HIVE-9039.11.patch, 
> HIVE-9039.12.patch, HIVE-9039.13.patch, HIVE-9039.14.patch, 
> HIVE-9039.15.patch, HIVE-9039.16.patch, HIVE-9039.17.patch, 
> HIVE-9039.18.patch, HIVE-9039.19.patch, HIVE-9039.20.patch, HIVE-9039.21.patch
>
>
> CLEAR LIBRARY CACHE
> Current version (Hive 0.14) does not support union (or union distinct). It 
> only supports union all. In this patch, we try to add this new feature by 
> rewriting union distinct to union all followed by group by.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9039) Support Union Distinct

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-9039:
--
Attachment: HIVE-9039.21.patch

rebase patch

> Support Union Distinct
> --
>
> Key: HIVE-9039
> URL: https://issues.apache.org/jira/browse/HIVE-9039
> Project: Hive
>  Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, 
> HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, 
> HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch, 
> HIVE-9039.09.patch, HIVE-9039.10.patch, HIVE-9039.11.patch, 
> HIVE-9039.12.patch, HIVE-9039.13.patch, HIVE-9039.14.patch, 
> HIVE-9039.15.patch, HIVE-9039.16.patch, HIVE-9039.17.patch, 
> HIVE-9039.18.patch, HIVE-9039.19.patch, HIVE-9039.20.patch, HIVE-9039.21.patch
>
>
> CLEAR LIBRARY CACHE
> Current version (Hive 0.14) does not support union (or union distinct). It 
> only supports union all. In this patch, we try to add this new feature by 
> rewriting union distinct to union all followed by group by.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-6617:
--
Status: Open  (was: Patch Available)

> Reduce ambiguity in grammar
> ---
>
> Key: HIVE-6617
> URL: https://issues.apache.org/jira/browse/HIVE-6617
> Project: Hive
>  Issue Type: Task
>Reporter: Ashutosh Chauhan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, 
> HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, 
> HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch
>
>
> CLEAR LIBRARY CACHE
> As of today, antlr reports 214 warnings. Need to bring down this number, 
> ideally to 0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-6617:
--
Status: Patch Available  (was: Open)

> Reduce ambiguity in grammar
> ---
>
> Key: HIVE-6617
> URL: https://issues.apache.org/jira/browse/HIVE-6617
> Project: Hive
>  Issue Type: Task
>Reporter: Ashutosh Chauhan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, 
> HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, 
> HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch
>
>
> CLEAR LIBRARY CACHE
> As of today, antlr reports 214 warnings. Need to bring down this number, 
> ideally to 0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-6617:
--
Attachment: HIVE-6617.08.patch

rebase the patch

> Reduce ambiguity in grammar
> ---
>
> Key: HIVE-6617
> URL: https://issues.apache.org/jira/browse/HIVE-6617
> Project: Hive
>  Issue Type: Task
>Reporter: Ashutosh Chauhan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, 
> HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, 
> HIVE-6617.06.patch, HIVE-6617.07.patch, HIVE-6617.08.patch
>
>
> CLEAR LIBRARY CACHE
> As of today, antlr reports 214 warnings. Need to bring down this number, 
> ideally to 0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-6617) Reduce ambiguity in grammar

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-6617:
--
Description: 
CLEAR LIBRARY CACHE

As of today, antlr reports 214 warnings. Need to bring down this number, 
ideally to 0.

  was:As of today, antlr reports 214 warnings. Need to bring down this number, 
ideally to 0.


> Reduce ambiguity in grammar
> ---
>
> Key: HIVE-6617
> URL: https://issues.apache.org/jira/browse/HIVE-6617
> Project: Hive
>  Issue Type: Task
>Reporter: Ashutosh Chauhan
>Assignee: Pengcheng Xiong
> Attachments: HIVE-6617.01.patch, HIVE-6617.02.patch, 
> HIVE-6617.03.patch, HIVE-6617.04.patch, HIVE-6617.05.patch, 
> HIVE-6617.06.patch, HIVE-6617.07.patch
>
>
> CLEAR LIBRARY CACHE
> As of today, antlr reports 214 warnings. Need to bring down this number, 
> ideally to 0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9039) Support Union Distinct

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-9039:
--
Description: 
CLEAR LIBRARY CACHE

Current version (Hive 0.14) does not support union (or union distinct). It only 
supports union all. In this patch, we try to add this new feature by rewriting 
union distinct to union all followed by group by.

  was:Current version (Hive 0.14) does not support union (or union distinct). 
It only supports union all. In this patch, we try to add this new feature by 
rewriting union distinct to union all followed by group by.


> Support Union Distinct
> --
>
> Key: HIVE-9039
> URL: https://issues.apache.org/jira/browse/HIVE-9039
> Project: Hive
>  Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, 
> HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, 
> HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch, 
> HIVE-9039.09.patch, HIVE-9039.10.patch, HIVE-9039.11.patch, 
> HIVE-9039.12.patch, HIVE-9039.13.patch, HIVE-9039.14.patch, 
> HIVE-9039.15.patch, HIVE-9039.16.patch, HIVE-9039.17.patch, 
> HIVE-9039.18.patch, HIVE-9039.19.patch, HIVE-9039.20.patch
>
>
> CLEAR LIBRARY CACHE
> Current version (Hive 0.14) does not support union (or union distinct). It 
> only supports union all. In this patch, we try to add this new feature by 
> rewriting union distinct to union all followed by group by.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290419#comment-14290419
 ] 

Hive QA commented on HIVE-9443:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694213/HIVE-9443.2.patch

{color:red}ERROR:{color} -1 due to 73 failed/errored test(s), 7365 tests 
executed
*Failed tests:*
{noformat}
TestMarkPartition - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hi

[jira] [Commented] (HIVE-5107) Change hive's build to maven

[phanikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290418#comment-14290418
 ] 

phanikumar commented on HIVE-5107:
--

Hi I am trying to compile the code using maven but i am getting an error can 
someone help me

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) 
on project hive-exec: Compilation failure
[ERROR] 
/home/datafreak/Downloads/hive-trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:[2402,15]
 error: cannot find symbol
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hive-exec


> Change hive's build to maven
> 
>
> Key: HIVE-5107
> URL: https://issues.apache.org/jira/browse/HIVE-5107
> Project: Hive
>  Issue Type: Task
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
> Fix For: 0.13.0
>
>
> I can not cope with hive's build infrastructure any more. I have started 
> working on porting the project to maven. When I have some solid progess i 
> will github the entire thing for review. Then we can talk about switching the 
> project somehow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: setting up a branch for testing

[Szehon]

Yea but for precommit testing it would need a cluster setup that runs an 
instance of Ptest server.  We only have spark branch setup for that other than 
trunk, we have one cluster running spark and another running trunk.

Setup is doable (actually just need to setup master) but takes some steps, and 
physical machines.

Thanks
Szehon

> On Jan 23, 2015, at 6:04 PM, Sergey Shelukhin  wrote:
> 
> Hi.
> Hive dev doc mentions that patches can be tested by HiveQA against the
> branch by supplying the branch name in the patch name.
> However, as far as I understand this requires some setup for each specific
> branch.
> 
> Is it possible to set up "llap" branch for HiveQA testing?
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.

[jira] [Commented] (HIVE-9402) Create GREATEST and LEAST udf

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290412#comment-14290412
 ] 

Lefty Leverenz commented on HIVE-9402:
--

Doc note:  GREATEST and LEAST should be documented in the "Conditional 
Functions" section of the UDFs doc, with version information and a link to this 
issue.

* [Hive Operators and UDFs -- Conditional Functions | 
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-ConditionalFunctions]

> Create GREATEST and LEAST udf
> -
>
> Key: HIVE-9402
> URL: https://issues.apache.org/jira/browse/HIVE-9402
> Project: Hive
>  Issue Type: Task
>  Components: UDF
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>  Labels: TODOC15
> Fix For: 0.15.0
>
> Attachments: HIVE-9402.1.patch, HIVE-9402.2.patch, HIVE-9402.3.patch, 
> HIVE-9402.4.patch, HIVE-9402.4.patch, HIVE-9402.5.patch, HIVE-9402.5.patch, 
> HIVE-9402.6.patch, HIVE-9402.7.patch
>
>
> GREATEST function returns the greatest value in a list of values
> Signature: T greatest(T v1, T v2, ...)
> all values should be the same type (like in COALESCE)
> LEAST returns the least value in a list of values
> Signature: T least(T v1, T v2, ...)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Created branch 1.0

[Sergey Shelukhin]

I think the way it is done in Hadoop space is better for Hadoop space (and
better wrt consistency, us being in the Hadoop space).
Because no single company or QA process controls or covers all the changes
to the product, and some changes go unseen by every actor, stabilization
period is a must...


And anyway enterprise software on trunk model does not cut releases
immediately off trunk and ship them. With enterprise software there's
lengthy QA, with Hadoop there's lengthy "cutting edge" release.
How about we cut 1.0 with stable version 0.14.1, and instead of 0.15 do
2.0, like HBase did?
We can maintain 1.0 as maintenance release; with 2.0 we can add new
unstable stuff, and also remove all the old paths we don't care about (old
Hadoop support, HiveCLI(?), old Java version support) etc.


On Fri, Jan 23, 2015 at 11:40 AM, Szehon Ho  wrote:

> Wherever I've seen in enterprise software, the trunk-based development
> model has been the standard where all release branches are cut from trunk
> and short-lived.  I've never heard of a case where a branch originally
> designated for 0.14 (minor release) is cut again to become 1.0 (major
> release), and I dont think if you ask anyone they will expect it either.
> There was also no announced plan when cutting 0.14 branch that it was
> eventually going to be 1.0.  As Brock pointed out in the beginning, Hadoop
> branch/versioning is the only exception and an anti-pattern, and all the
> confusion like why 0.xx has features not in 1.0 would not be there if it
> followed this.  I would really hate to see the same anti-pattern happen to
> Hive, so my vote is also against this.  Also this standard release
> branching practice has been in Hive throughout its history, you wouldn't
> make 0.14 out of 0.13 branch, would you?
>
> From the stability and long-term support use-cases that is very definitely
> > the wrong thing to do - to cram code into a 1.0 release.
>
> Major release is supposed to be stable.
>
>
> I also don't see how cutting 1.0 from trunk precludes it from stabilizing.
> Also I don't think those arguments of 0.14 as most stable that can be
> backed up, what constitutes stability?  Bug fixes are just one part, in
> that case there are always more bug fixes in later Hive versions than
> earlier ones, so probably API stability is a more measure-able term and
> should be more important to consider.
>
> Thanks,
> Szehon
>
>
> On Fri, Jan 23, 2015 at 10:42 AM, Gopal V  wrote:
>
> > On 1/23/15, 6:59 AM, Xuefu Zhang wrote:
> >
> >> While it's true that a release isn't going to include everything from
> >> trunk, proposed 1.0 release is branched off 0.14, which was again
> branched
> >> from trunk long time ago. If you compare the code base, you will see the
> >> huge difference.
> >>
> >
> > From the stability and long-term support use-cases that is very
> definitely
> > the wrong thing to do - to cram code into a 1.0 release.
> >
> > The "huge difference" is *THE* really worrying red-flag.
> >
> > Or is the thought behind "everything from trunk" that 1.0 just a number?
> >
> >  0.14.1 in terms of functionality and stability will be much clearer,
> >> meeting the all expectations for a major release.
> >>
> >
> > Just to be clear, when hive-14 was released, it was actually a major
> > release.
> >
> > That branch kicked off in Sept and has been updated since then with a
> > known set of critical fixes, giving it pedigree and has already seen
> > customer time.
> >
> > In all this discussion, it doesn't sound like you consider 0.15 to be a
> > major release - that gives me no confidence in your approach.
> >
> > Cheers,
> > Gopal
> >
> >  On Thu, Jan 22, 2015 at 3:08 PM, Thejas Nair 
> >> wrote:
> >>
> >>  On Thu, Jan 22, 2015 at 12:31 PM, Xuefu Zhang 
> >>> wrote:
> >>> > Hi Thejas/Alan,
> >>> >
> >>> > From all the argument, I think there was an assumption that the
> >>> proposed
> >>> > 1.0 release will be imminent and 0.15 will happen far after that.
> Based
> >>> on
> >>> > that assumption, 0.15 will become 1.1, which is greater in scope than
> >>> 1.0.
> >>> > However, this assumption may not be true. The confusion will be
> >>> significant
> >>> > if 0.15 is released early as 0.15 before 0.14.1 is released as 1.0.
> >>>
> >>> Yes, the assumption is that 1.0 will be out very soon,  before 0.15
> >>> line is ready, and that 0.15 can become 1.1 .
> >>> Do you think that assumption won't hold true ? (In previous emails in
> >>> this thread, I talk about reasons why this assumption is reliable).
> >>> I agree that it does not make sense to release 1.0 as proposed in this
> >>> thread if that does not hold true.
> >>>
> >>> > Another concern is that, the proposed release of 1.0 is a subset of
> of
> >>> > Hive's functionality, and for major releases users are expecting
> major
> >>> > improvement in functionality as well as stability. Mutating from
> 0.14.1
> >>> > release seems falling short in that expectation.
> >>>
> >>> Every release of hive has been a subset of ti

Re: Created branch 1.0

[Sergey Shelukhin]

We really didn't define 1.0 very well.
The way HBase did this (correct me if I'm wrong) is that 1.0 was the stable
release (stable APIs and code).
Then 1.1, 1.2 etc. become patch releases, and 2.0 becomes the release for
all the new crazy stuff...

On Fri, Jan 23, 2015 at 4:01 PM, Lefty Leverenz 
wrote:

> Setting aside branching specifics for a moment, what are the pros & cons of
> doing release 1.0 now or later?
>
>- From a documentation point of view, planning a later 1.0 release could
>give us time to catch up on the backlog of doc tasks, or at least
>prioritize the backlog and address the most important tasks.
>- On the other hand, releasing 1.0 sooner *might* increase the pressure
>to bring documentation up to par.  (Italics added by my inner cynic.)
>
> What else is at stake here?
>
>
>
> -- Lefty
>
> On Fri, Jan 23, 2015 at 11:40 AM, Szehon Ho  wrote:
>
> > Wherever I've seen in enterprise software, the trunk-based development
> > model has been the standard where all release branches are cut from trunk
> > and short-lived.  I've never heard of a case where a branch originally
> > designated for 0.14 (minor release) is cut again to become 1.0 (major
> > release), and I dont think if you ask anyone they will expect it either.
> > There was also no announced plan when cutting 0.14 branch that it was
> > eventually going to be 1.0.  As Brock pointed out in the beginning,
> Hadoop
> > branch/versioning is the only exception and an anti-pattern, and all the
> > confusion like why 0.xx has features not in 1.0 would not be there if it
> > followed this.  I would really hate to see the same anti-pattern happen
> to
> > Hive, so my vote is also against this.  Also this standard release
> > branching practice has been in Hive throughout its history, you wouldn't
> > make 0.14 out of 0.13 branch, would you?
> >
> > From the stability and long-term support use-cases that is very
> definitely
> > > the wrong thing to do - to cram code into a 1.0 release.
> >
> > Major release is supposed to be stable.
> >
> >
> > I also don't see how cutting 1.0 from trunk precludes it from
> stabilizing.
> > Also I don't think those arguments of 0.14 as most stable that can be
> > backed up, what constitutes stability?  Bug fixes are just one part, in
> > that case there are always more bug fixes in later Hive versions than
> > earlier ones, so probably API stability is a more measure-able term and
> > should be more important to consider.
> >
> > Thanks,
> > Szehon
> >
> >
> > On Fri, Jan 23, 2015 at 10:42 AM, Gopal V  wrote:
> >
> > > On 1/23/15, 6:59 AM, Xuefu Zhang wrote:
> > >
> > >> While it's true that a release isn't going to include everything from
> > >> trunk, proposed 1.0 release is branched off 0.14, which was again
> > branched
> > >> from trunk long time ago. If you compare the code base, you will see
> the
> > >> huge difference.
> > >>
> > >
> > > From the stability and long-term support use-cases that is very
> > definitely
> > > the wrong thing to do - to cram code into a 1.0 release.
> > >
> > > The "huge difference" is *THE* really worrying red-flag.
> > >
> > > Or is the thought behind "everything from trunk" that 1.0 just a
> number?
> > >
> > >  0.14.1 in terms of functionality and stability will be much clearer,
> > >> meeting the all expectations for a major release.
> > >>
> > >
> > > Just to be clear, when hive-14 was released, it was actually a major
> > > release.
> > >
> > > That branch kicked off in Sept and has been updated since then with a
> > > known set of critical fixes, giving it pedigree and has already seen
> > > customer time.
> > >
> > > In all this discussion, it doesn't sound like you consider 0.15 to be a
> > > major release - that gives me no confidence in your approach.
> > >
> > > Cheers,
> > > Gopal
> > >
> > >  On Thu, Jan 22, 2015 at 3:08 PM, Thejas Nair 
> > >> wrote:
> > >>
> > >>  On Thu, Jan 22, 2015 at 12:31 PM, Xuefu Zhang 
> > >>> wrote:
> > >>> > Hi Thejas/Alan,
> > >>> >
> > >>> > From all the argument, I think there was an assumption that the
> > >>> proposed
> > >>> > 1.0 release will be imminent and 0.15 will happen far after that.
> > Based
> > >>> on
> > >>> > that assumption, 0.15 will become 1.1, which is greater in scope
> than
> > >>> 1.0.
> > >>> > However, this assumption may not be true. The confusion will be
> > >>> significant
> > >>> > if 0.15 is released early as 0.15 before 0.14.1 is released as 1.0.
> > >>>
> > >>> Yes, the assumption is that 1.0 will be out very soon,  before 0.15
> > >>> line is ready, and that 0.15 can become 1.1 .
> > >>> Do you think that assumption won't hold true ? (In previous emails in
> > >>> this thread, I talk about reasons why this assumption is reliable).
> > >>> I agree that it does not make sense to release 1.0 as proposed in
> this
> > >>> thread if that does not hold true.
> > >>>
> > >>> > Another concern is that, the proposed release of 1.0 is a subset of
> > of
> > >>> > Hive's functionality,

setting up a branch for testing

[Sergey Shelukhin]

Hi.
Hive dev doc mentions that patches can be tested by HiveQA against the
branch by supplying the branch name in the patch name.
However, as far as I understand this requires some setup for each specific
branch.

Is it possible to set up "llap" branch for HiveQA testing?

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (HIVE-9103) Support backup task for join related optimization [Spark Branch]

[Chao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290361#comment-14290361
 ] 

Chao commented on HIVE-9103:


If task doesn't fail, backup task won't be triggered.

> Support backup task for join related optimization [Spark Branch]
> 
>
> Key: HIVE-9103
> URL: https://issues.apache.org/jira/browse/HIVE-9103
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Chao
>Priority: Blocker
>
> In MR, backup task can be executed if the original task, which probably 
> contains certain (join) optimization fails. This JIRA is to track this topic 
> for Spark. We need to determine if we need this and implement if necessary.
> This is a followup of HIVE-9099.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-3280) Make HiveMetaStoreClient a public API

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290351#comment-14290351
 ] 

Alan Gates commented on HIVE-3280:
--

+1, evolving makes sense for this since we seem to change it almost every 
release.

> Make HiveMetaStoreClient a public API
> -
>
> Key: HIVE-3280
> URL: https://issues.apache.org/jira/browse/HIVE-3280
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Carl Steinbach
>Assignee: Thejas M Nair
>  Labels: api-addition
> Attachments: HIVE-3280.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8966:
-
Attachment: HIVE-8966.6.patch

Final version of the patch.  Moved ValidCompactorTxnList per Owen's request.  
Also made small changes to StreamingIntegrationTester to make it work properly 
in cases where you want it to go slowly.

> Delta files created by hive hcatalog streaming cannot be compacted
> --
>
> Key: HIVE-8966
> URL: https://issues.apache.org/jira/browse/HIVE-8966
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
> Environment: hive
>Reporter: Jihong Liu
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.1
>
> Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.4.patch, 
> HIVE-8966.5.patch, HIVE-8966.6.patch, HIVE-8966.patch
>
>
> hive hcatalog streaming will also create a file like bucket_n_flush_length in 
> each delta directory. Where "n" is the bucket number. But the 
> compactor.CompactorMR think this file also needs to compact. However this 
> file of course cannot be compacted, so compactor.CompactorMR will not 
> continue to do the compaction. 
> Did a test, after removed the bucket_n_flush_length file, then the "alter 
> table partition compact" finished successfully. If don't delete that file, 
> nothing will be compacted. 
> This is probably a very severity bug. Both 0.13 and 0.14 have this issue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8966:
-
Status: Patch Available  (was: Open)

> Delta files created by hive hcatalog streaming cannot be compacted
> --
>
> Key: HIVE-8966
> URL: https://issues.apache.org/jira/browse/HIVE-8966
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
> Environment: hive
>Reporter: Jihong Liu
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.1
>
> Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.4.patch, 
> HIVE-8966.5.patch, HIVE-8966.6.patch, HIVE-8966.patch
>
>
> hive hcatalog streaming will also create a file like bucket_n_flush_length in 
> each delta directory. Where "n" is the bucket number. But the 
> compactor.CompactorMR think this file also needs to compact. However this 
> file of course cannot be compacted, so compactor.CompactorMR will not 
> continue to do the compaction. 
> Did a test, after removed the bucket_n_flush_length file, then the "alter 
> table partition compact" finished successfully. If don't delete that file, 
> nothing will be compacted. 
> This is probably a very severity bug. Both 0.13 and 0.14 have this issue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-8966) Delta files created by hive hcatalog streaming cannot be compacted

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-8966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-8966:
-
Status: Open  (was: Patch Available)

> Delta files created by hive hcatalog streaming cannot be compacted
> --
>
> Key: HIVE-8966
> URL: https://issues.apache.org/jira/browse/HIVE-8966
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Affects Versions: 0.14.0
> Environment: hive
>Reporter: Jihong Liu
>Assignee: Alan Gates
>Priority: Critical
> Fix For: 0.14.1
>
> Attachments: HIVE-8966.2.patch, HIVE-8966.3.patch, HIVE-8966.4.patch, 
> HIVE-8966.5.patch, HIVE-8966.patch
>
>
> hive hcatalog streaming will also create a file like bucket_n_flush_length in 
> each delta directory. Where "n" is the bucket number. But the 
> compactor.CompactorMR think this file also needs to compact. However this 
> file of course cannot be compacted, so compactor.CompactorMR will not 
> continue to do the compaction. 
> Did a test, after removed the bucket_n_flush_length file, then the "alter 
> table partition compact" finished successfully. If don't delete that file, 
> nothing will be compacted. 
> This is probably a very severity bug. Both 0.13 and 0.14 have this issue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9361) Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290338#comment-14290338
 ] 

Jason Dere commented on HIVE-9361:
--

+1

> Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable
> -
>
> Key: HIVE-9361
> URL: https://issues.apache.org/jira/browse/HIVE-9361
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-9361.patch
>
>
> it's happening at 
> {noformat}
> MetaStoreUtils.updateUnpartitionedTableStatsFast(newtCopy,
> wh.getFileStatusesForSD(newtCopy.getSd()), false, true);
> {noformat}
> other methods in this class call getWh() to get Warehouse so this likely 
> explains why it's intermittent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9361) Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-9361:
-
Attachment: HIVE-9361.patch

> Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable
> -
>
> Key: HIVE-9361
> URL: https://issues.apache.org/jira/browse/HIVE-9361
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-9361.patch
>
>
> it's happening at 
> {noformat}
> MetaStoreUtils.updateUnpartitionedTableStatsFast(newtCopy,
> wh.getFileStatusesForSD(newtCopy.getSd()), false, true);
> {noformat}
> other methods in this class call getWh() to get Warehouse so this likely 
> explains why it's intermittent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9361) Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-9361:
-
Status: Patch Available  (was: Open)

> Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable
> -
>
> Key: HIVE-9361
> URL: https://issues.apache.org/jira/browse/HIVE-9361
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-9361.patch
>
>
> it's happening at 
> {noformat}
> MetaStoreUtils.updateUnpartitionedTableStatsFast(newtCopy,
> wh.getFileStatusesForSD(newtCopy.getSd()), false, true);
> {noformat}
> other methods in this class call getWh() to get Warehouse so this likely 
> explains why it's intermittent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9404) NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290327#comment-14290327
 ] 

Eugene Koifman commented on HIVE-9404:
--

Committed to 0.14 and 1.0

> NPE in 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()
> -
>
> Key: HIVE-9404
> URL: https://issues.apache.org/jira/browse/HIVE-9404
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.15.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.15.0, 0.14.1, 1.0.0
>
> Attachments: HIVE-9404.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct(TxnHandler.java:1015)
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:906)
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:238)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_open_txns(HiveMetaStore.java:5321)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9390) Enhance retry logic wrt DB access in TxnHandler

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290328#comment-14290328
 ] 

Eugene Koifman commented on HIVE-9390:
--

Committed to 0.14 and 1.0

> Enhance retry logic wrt DB access in TxnHandler
> ---
>
> Key: HIVE-9390
> URL: https://issues.apache.org/jira/browse/HIVE-9390
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.15.0, 0.14.1, 1.0.0
>
> Attachments: HIVE-9390.patch
>
>
> example of error
> {noformat}
> 2015-01-13 16:09:21,148 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(141)) - org.apache.thrift.TException: 
> MetaException(message:Unable to get jdbc connection from pool, Read timed out)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_open_txns(HiveMetaStore.java:5324)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
>   at com.sun.proxy.$Proxy11.get_open_txns(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getValidTxns(HiveMetaStoreClient.java:1696)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
>   at com.sun.proxy.$Proxy12.getValidTxns(Unknown Source)
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidTxns(DbTxnManager.java:289)
>   at org.apache.hadoop.hive.ql.Driver.recordValidTxns(Driver.java:882)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:399)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1067)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1061)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:100)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:171)
>   at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:256)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:363)
>   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
>   at com.sun.proxy.$Proxy21.executeStatementAsync(Unknown Source)
>   at 
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:247)
>   at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:401)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>   at 
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHtt

[jira] [Updated] (HIVE-9404) NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-9404:
-
Fix Version/s: 1.0.0
   0.14.1

> NPE in 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()
> -
>
> Key: HIVE-9404
> URL: https://issues.apache.org/jira/browse/HIVE-9404
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.15.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.15.0, 0.14.1, 1.0.0
>
> Attachments: HIVE-9404.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct(TxnHandler.java:1015)
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:906)
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:238)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_open_txns(HiveMetaStore.java:5321)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-3280) Make HiveMetaStoreClient a public API

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3280:

Attachment: HIVE-3280.1.patch

> Make HiveMetaStoreClient a public API
> -
>
> Key: HIVE-3280
> URL: https://issues.apache.org/jira/browse/HIVE-3280
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Carl Steinbach
>Assignee: Thejas M Nair
>  Labels: api-addition
> Attachments: HIVE-3280.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-3280) Make HiveMetaStoreClient a public API

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-3280:

Status: Patch Available  (was: Open)

> Make HiveMetaStoreClient a public API
> -
>
> Key: HIVE-3280
> URL: https://issues.apache.org/jira/browse/HIVE-3280
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Carl Steinbach
>Assignee: Thejas M Nair
>  Labels: api-addition
> Attachments: HIVE-3280.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-3280) Make HiveMetaStoreClient a public API

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair reassigned HIVE-3280:
---

Assignee: Thejas M Nair

> Make HiveMetaStoreClient a public API
> -
>
> Key: HIVE-3280
> URL: https://issues.apache.org/jira/browse/HIVE-3280
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Reporter: Carl Steinbach
>Assignee: Thejas M Nair
>  Labels: api-addition
> Attachments: HIVE-3280.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HIVE-9208) MetaStore DB schema inconsistent for MS SQL Server in use of varchar/nvarchar

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou resolved HIVE-9208.
-
Resolution: Won't Fix

HIVE-9456 is going to change all varchar to nvarchar in 0.15 and provide 
upgrade script form 0.14 to 0.15, so there's no consistency issue like the 
case. 

Marked it as won't fix.

> MetaStore DB schema inconsistent for MS SQL Server in use of varchar/nvarchar
> -
>
> Key: HIVE-9208
> URL: https://issues.apache.org/jira/browse/HIVE-9208
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Xiaobing Zhou
> Attachments: HIVE-9208.1.patch, HIVE-9208.2.patch
>
>
> hive-schema-0.15.0.mssql.sql has PARTITIONS.PART_NAME as NVARCHAR but 
> COMPLETED_TXN_COMPONENTS.CTC_PARTITON, COMPACTION_QUEUE.CQ_PARTITION, 
> HIVE_LOCKS.HL_PARTITION, TXN_COMPONENTS.TC_PARTITION all use VARCHAR.  This 
> cannot be right since they all store the same value.
> the same is true of hive-schema-0.14.0.mssql.sql and the two corresponding 
> hvie-txn-schema-... files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9456) Make Hive support unicode with MSSQL as Metastore backend

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HIVE-9456:

Attachment: HIVE-9456.1.patch

Made a patch. Could anyone have it reviewed? Thanks!

> Make Hive support unicode with MSSQL as Metastore backend
> -
>
> Key: HIVE-9456
> URL: https://issues.apache.org/jira/browse/HIVE-9456
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HIVE-9456.1.patch
>
>
> There are significant issues when Hive uses MSSQL as metastore backend to 
> support unicode, since MSSQL handles varchar and nvarchar datatypes 
> differently. Hive 0.14 metastore mssql script DDL was using varchar as 
> datatype, which can't handle multi-bytes/unicode characters, e.g., Chinese 
> chars. This JIRA is going to track implementation of unicode support in that 
> case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9390) Enhance retry logic wrt DB access in TxnHandler

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-9390:
-
Fix Version/s: 1.0.0
   0.14.1

> Enhance retry logic wrt DB access in TxnHandler
> ---
>
> Key: HIVE-9390
> URL: https://issues.apache.org/jira/browse/HIVE-9390
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.14.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.15.0, 0.14.1, 1.0.0
>
> Attachments: HIVE-9390.patch
>
>
> example of error
> {noformat}
> 2015-01-13 16:09:21,148 ERROR metastore.RetryingHMSHandler 
> (RetryingHMSHandler.java:invoke(141)) - org.apache.thrift.TException: 
> MetaException(message:Unable to get jdbc connection from pool, Read timed out)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_open_txns(HiveMetaStore.java:5324)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:102)
>   at com.sun.proxy.$Proxy11.get_open_txns(Unknown Source)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getValidTxns(HiveMetaStoreClient.java:1696)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)
>   at com.sun.proxy.$Proxy12.getValidTxns(Unknown Source)
>   at 
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getValidTxns(DbTxnManager.java:289)
>   at org.apache.hadoop.hive.ql.Driver.recordValidTxns(Driver.java:882)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:399)
>   at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303)
>   at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1067)
>   at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1061)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:100)
>   at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:171)
>   at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:256)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:376)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:363)
>   at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>   at 
> org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:536)
>   at 
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:60)
>   at com.sun.proxy.$Proxy21.executeStatementAsync(Unknown Source)
>   at 
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:247)
>   at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:401)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
>   at 
> org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:101)
>   at jav

[jira] [Updated] (HIVE-9456) Make Hive support unicode with MSSQL as Metastore backend

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HIVE-9456:

Description: There are significant issues when Hive uses MSSQL as metastore 
backend to support unicode, since MSSQL handles varchar and nvarchar datatypes 
differently. Hive 0.14 metastore mssql script DDL was using varchar as 
datatype, which can't handle multi-bytes/unicode characters, e.g., Chinese 
chars. This JIRA is going to track implementation of unicode support in that 
case.  (was: There are significant issues when Hive uses MSSQL as )

> Make Hive support unicode with MSSQL as Metastore backend
> -
>
> Key: HIVE-9456
> URL: https://issues.apache.org/jira/browse/HIVE-9456
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> There are significant issues when Hive uses MSSQL as metastore backend to 
> support unicode, since MSSQL handles varchar and nvarchar datatypes 
> differently. Hive 0.14 metastore mssql script DDL was using varchar as 
> datatype, which can't handle multi-bytes/unicode characters, e.g., Chinese 
> chars. This JIRA is going to track implementation of unicode support in that 
> case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9456) Make Hive support unicode with MSSQL as Metastore backend

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HIVE-9456:

Description: There are significant issues when Hive uses MSSQL as   (was: 
hive-schema-0.15.0.mssql.sql has PARTITIONS.PART_NAME as NVARCHAR but 
COMPLETED_TXN_COMPONENTS.CTC_PARTITON, COMPACTION_QUEUE.CQ_PARTITION, 
HIVE_LOCKS.HL_PARTITION, TXN_COMPONENTS.TC_PARTITION all use VARCHAR.  This 
cannot be right since they all store the same value.

the same is true of hive-schema-0.14.0.mssql.sql and the two corresponding 
hvie-txn-schema-... files.)

> Make Hive support unicode with MSSQL as Metastore backend
> -
>
> Key: HIVE-9456
> URL: https://issues.apache.org/jira/browse/HIVE-9456
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
>
> There are significant issues when Hive uses MSSQL as 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9456) Make Hive support unicode with MSSQL as Metastore backend

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HIVE-9456:
---

 Summary: Make Hive support unicode with MSSQL as Metastore backend
 Key: HIVE-9456
 URL: https://issues.apache.org/jira/browse/HIVE-9456
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.14.0
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou


hive-schema-0.15.0.mssql.sql has PARTITIONS.PART_NAME as NVARCHAR but 
COMPLETED_TXN_COMPONENTS.CTC_PARTITON, COMPACTION_QUEUE.CQ_PARTITION, 
HIVE_LOCKS.HL_PARTITION, TXN_COMPONENTS.TC_PARTITION all use VARCHAR.  This 
cannot be right since they all store the same value.

the same is true of hive-schema-0.14.0.mssql.sql and the two corresponding 
hvie-txn-schema-... files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9416) Get rid of Extract Operator

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290275#comment-14290275
 ] 

Hive QA commented on HIVE-9416:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694191/HIVE-9416.3.patch

{color:red}ERROR:{color} -1 due to 70 failed/errored test(s), 7346 tests 
executed
*Failed tests:*
{noformat}
TestHs2Hooks - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.

[jira] [Created] (HIVE-9455) MapJoin task shouldn't start if HashTableSink task failed [Spark Branch]

[Chao (JIRA)]

Chao created HIVE-9455:
--

 Summary: MapJoin task shouldn't start if HashTableSink task failed 
[Spark Branch] 
 Key: HIVE-9455
 URL: https://issues.apache.org/jira/browse/HIVE-9455
 Project: Hive
  Issue Type: Sub-task
Affects Versions: spark-branch
Reporter: Chao


While playing with {{auto_join25.q}}, I noticed that even though the task for 
hash table sink failed, HOS will still continue launch the task for map join. 
This is not the desired result. Instead, like MR, we should abandon the second 
task.

Console output:

{code}
Total jobs = 2
Launching Job 1 out of 2
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=

Query Hive on Spark job[0] stages:
0

Status: Running (Hive on Spark job[0])
Job Progress Format
CurrentTime StageId_StageAttemptId: 
SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount 
[StageCost]
2015-01-23 16:18:14,604 Stage-0_0: 0/1
2015-01-23 04:18:14 Processing rows:4   Hashtable size: 3   
Memory usage:   119199408   percentage: 0.25
2015-01-23 16:18:15,611 Stage-0_0: 0(+0,-1)/1
Status: Finished successfully in 1.07 seconds
Launching Job 2 out of 2
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
2015-01-23 16:22:27,854 Stage-1_0: 0(+0,-1)/1
Status: Finished successfully in 1.01 seconds
Loading data to table default.dest1
Table default.dest1 stats: [numFiles=0, numRows=0, totalSize=0, rawDataSize=0]
OK
Time taken: 311.979 seconds
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-5021) Add Interval data type

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-5021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere reassigned HIVE-5021:


Assignee: Jason Dere

> Add Interval data type
> --
>
> Key: HIVE-5021
> URL: https://issues.apache.org/jira/browse/HIVE-5021
> Project: Hive
>  Issue Type: Bug
>  Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
>
> If we want to support arithmetic operations involving date/timestamp types, 
> we should add support for the interval data type where the result is an 
> offset of years/months/days/hours/minutes/seconds. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9446) JDBC DatabaseMetadata.getColumns() does not work for temporary tables

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-9446:
-
   Resolution: Fixed
Fix Version/s: 0.15.0
   Status: Resolved  (was: Patch Available)

Committed to trunk

> JDBC DatabaseMetadata.getColumns() does not work for temporary tables
> -
>
> Key: HIVE-9446
> URL: https://issues.apache.org/jira/browse/HIVE-9446
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.14.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Fix For: 0.15.0
>
> Attachments: HIVE-9446.1.patch
>
>
> After creating temporary table, calling DatabaseMetadData.getColumns() hits 
> error "UnknownTableException(message:default.tmp_07 table not found)"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9402) Create GREATEST and LEAST udf

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-9402:
-
Labels: TODOC15  (was: )

> Create GREATEST and LEAST udf
> -
>
> Key: HIVE-9402
> URL: https://issues.apache.org/jira/browse/HIVE-9402
> Project: Hive
>  Issue Type: Task
>  Components: UDF
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
>  Labels: TODOC15
> Fix For: 0.15.0
>
> Attachments: HIVE-9402.1.patch, HIVE-9402.2.patch, HIVE-9402.3.patch, 
> HIVE-9402.4.patch, HIVE-9402.4.patch, HIVE-9402.5.patch, HIVE-9402.5.patch, 
> HIVE-9402.6.patch, HIVE-9402.7.patch
>
>
> GREATEST function returns the greatest value in a list of values
> Signature: T greatest(T v1, T v2, ...)
> all values should be the same type (like in COALESCE)
> LEAST returns the least value in a list of values
> Signature: T least(T v1, T v2, ...)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9341) Apply ColumnPrunning for noop PTFs

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-9341:
---
   Resolution: Fixed
Fix Version/s: 0.15.0
   Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Navis!

> Apply ColumnPrunning for noop PTFs
> --
>
> Key: HIVE-9341
> URL: https://issues.apache.org/jira/browse/HIVE-9341
> Project: Hive
>  Issue Type: Improvement
>  Components: PTF-Windowing
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Fix For: 0.15.0
>
> Attachments: HIVE-9341.1.patch.txt, HIVE-9341.2.patch.txt, 
> HIVE-9341.3.patch.txt, HIVE-9341.4.patch.txt
>
>
> Currently, PTF disables CP optimization, which can make a huge burden. For 
> example,
> {noformat}
> select p_mfgr, p_name, p_size,
> rank() over (partition by p_mfgr order by p_name) as r,
> dense_rank() over (partition by p_mfgr order by p_name) as dr,
> sum(p_retailprice) over (partition by p_mfgr order by p_name rows between 
> unbounded preceding and current row) as s1
> from noop(on part 
>   partition by p_mfgr
>   order by p_name
>   );
> STAGE PLANS:
>   Stage: Stage-1
> Map Reduce
>   Map Operator Tree:
>   TableScan
> alias: part
> Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
> Column stats: NONE
> Reduce Output Operator
>   key expressions: p_mfgr (type: string), p_name (type: string)
>   sort order: ++
>   Map-reduce partition columns: p_mfgr (type: string)
>   Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE 
> Column stats: NONE
>   value expressions: p_partkey (type: int), p_name (type: 
> string), p_mfgr (type: string), p_brand (type: string), p_type (type: 
> string), p_size (type: int), p_container (type: string), p_retailprice (type: 
> double), p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE (type: 
> bigint), INPUT__FILE__NAME (type: string), ROW__ID (type: 
> struct)
> ...
> {noformat}
> There should be a generic way to discern referenced columns but before that, 
> we know CP can be safely applied to noop functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9402) Create GREATEST and LEAST udf

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-9402:
-
   Resolution: Fixed
Fix Version/s: 0.15.0
   Status: Resolved  (was: Patch Available)

I've committed this to trunk. Thanks Alexander!

> Create GREATEST and LEAST udf
> -
>
> Key: HIVE-9402
> URL: https://issues.apache.org/jira/browse/HIVE-9402
> Project: Hive
>  Issue Type: Task
>  Components: UDF
>Reporter: Alexander Pivovarov
>Assignee: Alexander Pivovarov
> Fix For: 0.15.0
>
> Attachments: HIVE-9402.1.patch, HIVE-9402.2.patch, HIVE-9402.3.patch, 
> HIVE-9402.4.patch, HIVE-9402.4.patch, HIVE-9402.5.patch, HIVE-9402.5.patch, 
> HIVE-9402.6.patch, HIVE-9402.7.patch
>
>
> GREATEST function returns the greatest value in a list of values
> Signature: T greatest(T v1, T v2, ...)
> all values should be the same type (like in COALESCE)
> LEAST returns the least value in a list of values
> Signature: T least(T v1, T v2, ...)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-7270) SerDe Properties are not considered by show create table Command

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-7270:
-
Fix Version/s: 0.14.1

Committed to 0.14/1.0 branches

> SerDe Properties are not considered by show create table Command
> 
>
> Key: HIVE-7270
> URL: https://issues.apache.org/jira/browse/HIVE-7270
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.13.1
>Reporter: R J
>Assignee: Navis
>Priority: Minor
> Fix For: 0.15.0, 0.14.1
>
> Attachments: HIVE-7270.1.patch.txt, HIVE-7270.2.patch.txt, 
> HIVE-7270.3.patch.txt
>
>
> The HIVE table DDl generated by show create table  command does 
> not contain SerDe properties of the target table even though it contain 
> specific SerDe properties. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Created branch 1.0

[Lefty Leverenz]

Setting aside branching specifics for a moment, what are the pros & cons of
doing release 1.0 now or later?

   - From a documentation point of view, planning a later 1.0 release could
   give us time to catch up on the backlog of doc tasks, or at least
   prioritize the backlog and address the most important tasks.
   - On the other hand, releasing 1.0 sooner *might* increase the pressure
   to bring documentation up to par.  (Italics added by my inner cynic.)

What else is at stake here?



-- Lefty

On Fri, Jan 23, 2015 at 11:40 AM, Szehon Ho  wrote:

> Wherever I've seen in enterprise software, the trunk-based development
> model has been the standard where all release branches are cut from trunk
> and short-lived.  I've never heard of a case where a branch originally
> designated for 0.14 (minor release) is cut again to become 1.0 (major
> release), and I dont think if you ask anyone they will expect it either.
> There was also no announced plan when cutting 0.14 branch that it was
> eventually going to be 1.0.  As Brock pointed out in the beginning, Hadoop
> branch/versioning is the only exception and an anti-pattern, and all the
> confusion like why 0.xx has features not in 1.0 would not be there if it
> followed this.  I would really hate to see the same anti-pattern happen to
> Hive, so my vote is also against this.  Also this standard release
> branching practice has been in Hive throughout its history, you wouldn't
> make 0.14 out of 0.13 branch, would you?
>
> From the stability and long-term support use-cases that is very definitely
> > the wrong thing to do - to cram code into a 1.0 release.
>
> Major release is supposed to be stable.
>
>
> I also don't see how cutting 1.0 from trunk precludes it from stabilizing.
> Also I don't think those arguments of 0.14 as most stable that can be
> backed up, what constitutes stability?  Bug fixes are just one part, in
> that case there are always more bug fixes in later Hive versions than
> earlier ones, so probably API stability is a more measure-able term and
> should be more important to consider.
>
> Thanks,
> Szehon
>
>
> On Fri, Jan 23, 2015 at 10:42 AM, Gopal V  wrote:
>
> > On 1/23/15, 6:59 AM, Xuefu Zhang wrote:
> >
> >> While it's true that a release isn't going to include everything from
> >> trunk, proposed 1.0 release is branched off 0.14, which was again
> branched
> >> from trunk long time ago. If you compare the code base, you will see the
> >> huge difference.
> >>
> >
> > From the stability and long-term support use-cases that is very
> definitely
> > the wrong thing to do - to cram code into a 1.0 release.
> >
> > The "huge difference" is *THE* really worrying red-flag.
> >
> > Or is the thought behind "everything from trunk" that 1.0 just a number?
> >
> >  0.14.1 in terms of functionality and stability will be much clearer,
> >> meeting the all expectations for a major release.
> >>
> >
> > Just to be clear, when hive-14 was released, it was actually a major
> > release.
> >
> > That branch kicked off in Sept and has been updated since then with a
> > known set of critical fixes, giving it pedigree and has already seen
> > customer time.
> >
> > In all this discussion, it doesn't sound like you consider 0.15 to be a
> > major release - that gives me no confidence in your approach.
> >
> > Cheers,
> > Gopal
> >
> >  On Thu, Jan 22, 2015 at 3:08 PM, Thejas Nair 
> >> wrote:
> >>
> >>  On Thu, Jan 22, 2015 at 12:31 PM, Xuefu Zhang 
> >>> wrote:
> >>> > Hi Thejas/Alan,
> >>> >
> >>> > From all the argument, I think there was an assumption that the
> >>> proposed
> >>> > 1.0 release will be imminent and 0.15 will happen far after that.
> Based
> >>> on
> >>> > that assumption, 0.15 will become 1.1, which is greater in scope than
> >>> 1.0.
> >>> > However, this assumption may not be true. The confusion will be
> >>> significant
> >>> > if 0.15 is released early as 0.15 before 0.14.1 is released as 1.0.
> >>>
> >>> Yes, the assumption is that 1.0 will be out very soon,  before 0.15
> >>> line is ready, and that 0.15 can become 1.1 .
> >>> Do you think that assumption won't hold true ? (In previous emails in
> >>> this thread, I talk about reasons why this assumption is reliable).
> >>> I agree that it does not make sense to release 1.0 as proposed in this
> >>> thread if that does not hold true.
> >>>
> >>> > Another concern is that, the proposed release of 1.0 is a subset of
> of
> >>> > Hive's functionality, and for major releases users are expecting
> major
> >>> > improvement in functionality as well as stability. Mutating from
> 0.14.1
> >>> > release seems falling short in that expectation.
> >>>
> >>> Every release of hive has been a subset of tip of the trunk (we branch
> >>> for release while trunk moves ahead), and super set of changes of
> >>> every previous release. So every release so far has had a subset of
> >>> functionality of hive trunk branch (if that is what you are referring
> >>> to).
> >>> With the 1.0 proposed base

[jira] [Updated] (HIVE-9416) Get rid of Extract Operator

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-9416:
---
Status: Open  (was: Patch Available)

> Get rid of Extract Operator
> ---
>
> Key: HIVE-9416
> URL: https://issues.apache.org/jira/browse/HIVE-9416
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9416.1.patch, HIVE-9416.2.patch, HIVE-9416.3.patch, 
> HIVE-9416.4.patch, HIVE-9416.patch
>
>
> {{Extract Operator}} has been there for legacy reasons. But there is no 
> functionality it provides which cant be provided by {{Select Operator}} 
> Instead of having two operators, one being subset of another we should just 
> get rid of {{Extract}} and simplify our codebase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9416) Get rid of Extract Operator

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-9416:
---
Status: Patch Available  (was: Open)

> Get rid of Extract Operator
> ---
>
> Key: HIVE-9416
> URL: https://issues.apache.org/jira/browse/HIVE-9416
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9416.1.patch, HIVE-9416.2.patch, HIVE-9416.3.patch, 
> HIVE-9416.4.patch, HIVE-9416.patch
>
>
> {{Extract Operator}} has been there for legacy reasons. But there is no 
> functionality it provides which cant be provided by {{Select Operator}} 
> Instead of having two operators, one being subset of another we should just 
> get rid of {{Extract}} and simplify our codebase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9416) Get rid of Extract Operator

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-9416:
---
Attachment: HIVE-9416.4.patch

Deleted ExtractOperator.java & friends.

> Get rid of Extract Operator
> ---
>
> Key: HIVE-9416
> URL: https://issues.apache.org/jira/browse/HIVE-9416
> Project: Hive
>  Issue Type: Task
>  Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-9416.1.patch, HIVE-9416.2.patch, HIVE-9416.3.patch, 
> HIVE-9416.4.patch, HIVE-9416.patch
>
>
> {{Extract Operator}} has been there for legacy reasons. But there is no 
> functionality it provides which cant be provided by {{Select Operator}} 
> Instead of having two operators, one being subset of another we should just 
> get rid of {{Extract}} and simplify our codebase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9277) Hybrid Hybrid Grace Hash Join

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290217#comment-14290217
 ] 

Lefty Leverenz commented on HIVE-9277:
--

Good update, thanks [~wzheng].  Now I understand that the diagram was correct 
all along.

> Hybrid Hybrid Grace Hash Join
> -
>
> Key: HIVE-9277
> URL: https://issues.apache.org/jira/browse/HIVE-9277
> Project: Hive
>  Issue Type: New Feature
>  Components: Physical Optimizer
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>  Labels: join
> Attachments: High-leveldesignforHybridHybridGraceHashJoinv1.0.pdf
>
>
> We are proposing an enhanced hash join algorithm called “hybrid hybrid grace 
> hash join”. We can benefit from this feature as illustrated below:
> o The query will not fail even if the estimated memory requirement is 
> slightly wrong
> o Expensive garbage collection overhead can be avoided when hash table grows
> o Join execution using a Map join operator even though the small table 
> doesn't fit in memory as spilling some data from the build and probe sides 
> will still be cheaper than having to shuffle the large fact table
> The design was based on Hadoop’s parallel processing capability and 
> significant amount of memory available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9454) Test failures due to new Calcite version

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290207#comment-14290207
 ] 

Julian Hyde commented on HIVE-9454:
---

I've put back the old snapshot (from Dec 19th). So, these changes should go 
away.

These differences are a preview of what will happen when Hive moves to Calcite 
1.0.0. Calcite's RC1 vote is going on right now, and closes at 10pm Pacific on 
Sunday.

Can someone from Hive please evaluate the test failures and cast a 
(non-binding) vote? If you discover a show-stopper cast a -1 vote. I don't want 
to make a release that cannot be used. If issues are discovered after the vote 
has closed we won't be inclined to make a make a new release in a hurry. :)

> Test failures due to new Calcite version
> 
>
> Key: HIVE-9454
> URL: https://issues.apache.org/jira/browse/HIVE-9454
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>
> A bunch of failures have started appearing in patches which seen unrelated. I 
> am thinking we've picked up a new version of Calcite. E.g.:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_auto_join12/
> {noformat}
> Running: diff -a 
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
>  
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
> 32c32
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 35c35
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> 39c39
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 54c54
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9450) [Parquet] Check all data types work for Parquet in Group By operator

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290175#comment-14290175
 ] 

Hive QA commented on HIVE-9450:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694180/HIVE-9450.patch

{color:red}ERROR:{color} -1 due to 64 failed/errored test(s), 7347 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCl

Re: Tests errors in precommit build

[Jason Dere]

Looks like this was an issue with 1.0.0-incubating-SNAPSHOT. On my local build 
I removed the existing org/apache/calcite/ from my local Maven repository and 
looks like things are working again.


On Jan 23, 2015, at 4:01 AM, Jesus Camacho Rodriguez 
 wrote:

> Hi,
> 
> For the last few builds we are getting many tests errors that were not
> there before:
> 
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2486/
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2487/
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2489/
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2490/
> 
> I checkout the code from trunk, compile it as it is, and these tests failed
> in my Linux box.
> 
> I don't know the exact cause, any idea when these fails have been
> introduced?
> 
> Thanks,
> Jesús
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (HIVE-9454) Test failures due to new Calcite version

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290167#comment-14290167
 ] 

Sushanth Sowmyan commented on HIVE-9454:


I saw a whole bunch of failures on HIVE-9436's run.

A couple are as in : 
https://issues.apache.org/jira/browse/HIVE-9436?focusedCommentId=14289984

> Test failures due to new Calcite version
> 
>
> Key: HIVE-9454
> URL: https://issues.apache.org/jira/browse/HIVE-9454
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>
> A bunch of failures have started appearing in patches which seen unrelated. I 
> am thinking we've picked up a new version of Calcite. E.g.:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_auto_join12/
> {noformat}
> Running: diff -a 
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
>  
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
> 32c32
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 35c35
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> 39c39
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 54c54
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7270) SerDe Properties are not considered by show create table Command

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290142#comment-14290142
 ] 

Vikram Dixit K commented on HIVE-7270:
--

+1. Go ahead with the branch commits.

> SerDe Properties are not considered by show create table Command
> 
>
> Key: HIVE-7270
> URL: https://issues.apache.org/jira/browse/HIVE-7270
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.13.1
>Reporter: R J
>Assignee: Navis
>Priority: Minor
> Fix For: 0.15.0
>
> Attachments: HIVE-7270.1.patch.txt, HIVE-7270.2.patch.txt, 
> HIVE-7270.3.patch.txt
>
>
> The HIVE table DDl generated by show create table  command does 
> not contain SerDe properties of the target table even though it contain 
> specific SerDe properties. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-7270) SerDe Properties are not considered by show create table Command

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290138#comment-14290138
 ] 

Jason Dere commented on HIVE-7270:
--

Would like to add this to 0.14.1/1.0 branch

> SerDe Properties are not considered by show create table Command
> 
>
> Key: HIVE-7270
> URL: https://issues.apache.org/jira/browse/HIVE-7270
> Project: Hive
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 0.13.1
>Reporter: R J
>Assignee: Navis
>Priority: Minor
> Fix For: 0.15.0
>
> Attachments: HIVE-7270.1.patch.txt, HIVE-7270.2.patch.txt, 
> HIVE-7270.3.patch.txt
>
>
> The HIVE table DDl generated by show create table  command does 
> not contain SerDe properties of the target table even though it contain 
> specific SerDe properties. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9454) Test failures due to new Calcite version

[Laljo John Pullokkaran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290129#comment-14290129
 ] 

Laljo John Pullokkaran commented on HIVE-9454:
--

Most likely introduced by new version of Calcite snapshot.
[~julianhyde] i thought was going to push old snapshot back.

We would need to analyze these failures separately before bringing in the new 
calcite version.

> Test failures due to new Calcite version
> 
>
> Key: HIVE-9454
> URL: https://issues.apache.org/jira/browse/HIVE-9454
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>
> A bunch of failures have started appearing in patches which seen unrelated. I 
> am thinking we've picked up a new version of Calcite. E.g.:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_auto_join12/
> {noformat}
> Running: diff -a 
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
>  
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
> 32c32
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 35c35
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> 39c39
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 54c54
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9454) Test failures due to new Calcite version

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290109#comment-14290109
 ] 

Brock Noland commented on HIVE-9454:


FYI [~jpullokkaran] [~ashutoshc] [~julianhyde]

> Test failures due to new Calcite version
> 
>
> Key: HIVE-9454
> URL: https://issues.apache.org/jira/browse/HIVE-9454
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>
> A bunch of failures have started appearing in patches which seen unrelated. I 
> am thinking we've picked up a new version of Calcite. E.g.:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_auto_join12/
> {noformat}
> Running: diff -a 
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
>  
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
> 32c32
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 35c35
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> 39c39
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 54c54
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9454) Test failures due to new Calcite version

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-9454:
---
Description: 
A bunch of failures have started appearing in patches which seen unrelated. I 
am thinking we've picked up a new version of Calcite. E.g.:

http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_auto_join12/

{noformat}
Running: diff -a 
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
 
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
32c32
< $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
---
> $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
35c35
< $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
---
> $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
39c39
< $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
---
> $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
54c54
< $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
---
> $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
{noformat}

  was:
A bunch of failures have started appearing in patches which seen unrelated. I 
am thinking we've picked up a new version of Calcite.

{noformat}
Running: diff -a 
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
 
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
32c32
< $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
---
> $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
35c35
< $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
---
> $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
39c39
< $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
---
> $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
54c54
< $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
---
> $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
{noformat}


> Test failures due to new Calcite version
> 
>
> Key: HIVE-9454
> URL: https://issues.apache.org/jira/browse/HIVE-9454
> Project: Hive
>  Issue Type: Bug
>Reporter: Brock Noland
>
> A bunch of failures have started appearing in patches which seen unrelated. I 
> am thinking we've picked up a new version of Calcite. E.g.:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2488/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_auto_join12/
> {noformat}
> Running: diff -a 
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
>  
> /home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
> 32c32
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 35c35
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> 39c39
> < $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
> 54c54
> < $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
> ---
> > $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-9454) Test failures due to new Calcite version

[Brock Noland (JIRA)]

Brock Noland created HIVE-9454:
--

 Summary: Test failures due to new Calcite version
 Key: HIVE-9454
 URL: https://issues.apache.org/jira/browse/HIVE-9454
 Project: Hive
  Issue Type: Bug
Reporter: Brock Noland


A bunch of failures have started appearing in patches which seen unrelated. I 
am thinking we've picked up a new version of Calcite.

{noformat}
Running: diff -a 
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
 
/home/hiveptest/54.147.202.89-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
32c32
< $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
---
> $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
35c35
< $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
---
> $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
39c39
< $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
---
> $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
54c54
< $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
---
> $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland updated HIVE-9445:
---
Summary: Revert HIVE-5700 - enforce single date format for partition column 
storage  (was: Revert HIVE-5700 - schemaTool failure when date partition has 
non-date value)

> Revert HIVE-5700 - enforce single date format for partition column storage
> --
>
> Key: HIVE-9445
> URL: https://issues.apache.org/jira/browse/HIVE-9445
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1
>Reporter: Brock Noland
>Assignee: Brock Noland
>Priority: Blocker
> Attachments: HIVE-9445.1.patch
>
>
> HIVE-5700 has the following issues:
> * HIVE-8730 - fails mysql upgrades
> * Does not upgrade all metadata, e.g. {{PARTITIONS.PART_NAME}} See comments 
> in HIVE-5700.
> * Completely corrupts postgres, see below.
> With a postgres metastore on 0.12, I executed the following:
> {noformat}
> CREATE TABLE HIVE5700_DATE_PARTED (line string) PARTITIONED BY (ddate date);
> CREATE TABLE HIVE5700_STRING_PARTED (line string) PARTITIONED BY (ddate 
> string);
> ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='NOT_DATE');
> ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150121');
> ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150122');
> ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='2015-01-23');
> ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='NOT_DATE');
> ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150121');
> ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150122');
> ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='2015-01-23');
> LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
> HIVE5700_DATE_PARTED PARTITION (ddate='NOT_DATE');
> LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
> HIVE5700_DATE_PARTED PARTITION (ddate='20150121');
> LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
> HIVE5700_DATE_PARTED PARTITION (ddate='20150122');
> LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
> HIVE5700_DATE_PARTED PARTITION (ddate='2015-01-23');
> LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
> HIVE5700_STRING_PARTED PARTITION (ddate='NOT_DATE');
> LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
> HIVE5700_STRING_PARTED PARTITION (ddate='20150121');
> LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
> HIVE5700_STRING_PARTED PARTITION (ddate='20150122');
> LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
> HIVE5700_STRING_PARTED PARTITION (ddate='2015-01-23');
> hive> show partitions HIVE5700_DATE_PARTED;  
> OK
> ddate=20150121
> ddate=20150122
> ddate=2015-01-23
> ddate=NOT_DATE
> Time taken: 0.052 seconds, Fetched: 4 row(s)
> hive> show partitions HIVE5700_STRING_PARTED;
> OK
> ddate=20150121
> ddate=20150122
> ddate=2015-01-23
> ddate=NOT_DATE
> Time taken: 0.051 seconds, Fetched: 4 row(s)
> {noformat}
> I then took a dump of the database named {{postgres-pre-upgrade.sql}} and the 
> data in the dump looks good:
> {noformat}
> [root@hive5700-1-1 ~]# egrep -A9 '^COPY "PARTITIONS"|^COPY 
> "PARTITION_KEY_VALS"' postgres-pre-upgrade.sql 
> COPY "PARTITIONS" ("PART_ID", "CREATE_TIME", "LAST_ACCESS_TIME", "PART_NAME", 
> "SD_ID", "TBL_ID") FROM stdin;
> 3 1421943647  0   ddate=NOT_DATE  6   2
> 4 1421943647  0   ddate=20150121  7   2
> 5 1421943648  0   ddate=20150122  8   2
> 6 1421943664  0   ddate=NOT_DATE  9   3
> 7 1421943664  0   ddate=20150121  10  3
> 8 1421943665  0   ddate=20150122  11  3
> 9 1421943694  0   ddate=2015-01-2312  2
> 101421943695  0   ddate=2015-01-2313  3
> \.
> --
> COPY "PARTITION_KEY_VALS" ("PART_ID", "PART_KEY_VAL", "INTEGER_IDX") FROM 
> stdin;
> 3 NOT_DATE0
> 4 201501210
> 5 201501220
> 6 NOT_DATE0
> 7 201501210
> 8 201501220
> 9 2015-01-23  0
> 102015-01-23  0
> \.
> {noformat}
> I then upgraded to 0.13 and subsequently upgraded the MS with the following 
> command: {{schematool -dbType postgres -upgradeSchema -verbose}}
> The file {{postgres-post-upgrade.sql}} is the post-upgrade db dump. As you 
> can see the data is completely corrupt.
> {noformat}
> [root@hive5700-1-1 ~]# egrep -A9 '^COPY "PARTITIONS"|^COPY 
> "PARTITION_KEY_VALS"' postgres-post-upgrade.sql 
> COPY "PARTITIONS" ("PART_ID", "CREATE_TIME", "LAST_ACCESS_TIME", "PART_NAME", 
> "SD_ID", "TBL_ID") FROM stdin;
> 3 1421943647  0   ddate=NOT_DATE  6   2
> 4 1421943647  0   ddat

[jira] [Commented] (HIVE-8485) HMS on Oracle incompatibility

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290065#comment-14290065
 ] 

Sushanth Sowmyan commented on HIVE-8485:


[~ctang.ma] : Thanks for your comment! I considered auto-enabling this flag if 
we detect oracle, but I think we will wind up having too many corner cases of 
if-this-then-that in our code if we go down that path, and having a config 
variable that we simply ask all oracle users to set is cleaner, I thought. 
Also, with this patch, Oracle also does "work" out of the box, it just chooses 
the strategy of trimming by default, as is done in all other dbs.

Also, if we wanted to support nulls inside the maps, that would be a change in 
hive behaviour (for the better, I'd think, but a behaviour change nonetheless) 
if we supported nulls. We'd have to handle escaping over thrift maps, but it 
could be done.

Multiple spaces already work and is valid, just maybe not for a delim 
parameter, but there's nothing stopping a user from having an arbitrary other 
parameter be multiples of spaces, either in oracle or elsewhere.

> HMS on Oracle incompatibility
> -
>
> Key: HIVE-8485
> URL: https://issues.apache.org/jira/browse/HIVE-8485
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
> Environment: Oracle as metastore DB
>Reporter: Ryan Pridgeon
>Assignee: Chaoyu Tang
> Attachments: HIVE-8485.2.patch, HIVE-8485.patch
>
>
> Oracle does not distinguish between empty strings and NULL,which proves 
> problematic for DataNucleus.
> In the event a user creates a table with some property stored as an empty 
> string the table will no longer be accessible.
> i.e. TBLPROPERTIES ('serialization.null.format'='')
> If they try to select, describe, drop, etc the client prints the following 
> exception.
> ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found 
> 
> The work around for this was to go into the hive metastore on the Oracle 
> database and replace NULL with some other string. Users could then drop the 
> tables or alter their data to use the new null format they just set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9434) Shim the method Path.getPathWithoutSchemeAndAuthority

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-9434:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

I have committed this to trunk. Thanks [~dongc] and [~brocknoland]!

> Shim the method Path.getPathWithoutSchemeAndAuthority
> -
>
> Key: HIVE-9434
> URL: https://issues.apache.org/jira/browse/HIVE-9434
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 0.15.0
>Reporter: Brock Noland
>Assignee: Dong Chen
> Fix For: 0.15.0
>
> Attachments: HIVE-9434.patch
>
>
> Since Hadoop 1 does not have the method 
> {{Path.getPathWithoutSchemeAndAuthority}} we need to shim it out.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9446) JDBC DatabaseMetadata.getColumns() does not work for temporary tables

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290030#comment-14290030
 ] 

Jason Dere commented on HIVE-9446:
--

Failures do not look related to the patch. udaf_covar_pop.q does not fail 
locally, and the other 62 failures appear to have been failing recently during 
other precommit tests as well.

> JDBC DatabaseMetadata.getColumns() does not work for temporary tables
> -
>
> Key: HIVE-9446
> URL: https://issues.apache.org/jira/browse/HIVE-9446
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 0.14.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-9446.1.patch
>
>
> After creating temporary table, calling DatabaseMetadData.getColumns() hits 
> error "UnknownTableException(message:default.tmp_07 table not found)"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9228) Problem with subquery using windowing functions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289997#comment-14289997
 ] 

Hive QA commented on HIVE-9228:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694092/HIVE-9228.1.patch.txt

{color:red}ERROR:{color} -1 due to 68 failed/errored test(s), 7347 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqualcolumnrefs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynam

[jira] [Commented] (HIVE-9436) RetryingMetaStoreClient does not retry JDOExceptions

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289961#comment-14289961
 ] 

Sushanth Sowmyan commented on HIVE-9436:


[~thejas]/[~hsubramaniyan] : I have a couple of thoughts about moving 
JDOException retries solely to the metastore:

a) Firstly, we have had cases so far where a JDOException invalidates the 
connection on the metastore side, and retrying from the metastore has not 
helped. Retrying from the client-side, though, causes a fresh openTransaction() 
that clears the connection and all history, sometimes by hitting a different 
HMSHandler, and this causes the retry from client to be more successful than a 
retry from server. Admittedly, this is more likely because we need to clean up 
our metastore code to make sure that the retry from the metastore-side handles 
this properly, and thus, is something we should attempt to improve.
b) Second, from a perspective of a loaded metastore, having a metastore thread 
do retries, thus using up valuable metastore resources/time is more wasteful 
than having the client do retries. We thus tend to keep our metastore-side 
retries to a low amount, but the fact that we have client-side retries as well 
gives us an ability to be fail-fast on the metastore, but retry a large number 
of times in particular clients if we find the need to do so. Particularly, in 
HA configurations, I've seen a large number of retries and longer 
retry-intervals on the client side that allow a connection to go through 
despite metastore HUPs.
c) Thirdly, speaking of HA, retrying on the client-side allows us to hit 
alternate metastores as well, if configured, if we have scenarios where one 
metastore is getting bogged down. As you mention, client should ideally only be 
retrying connection exceptions, but JDOExceptions are frequently the result of 
connection exceptions raised by the connection pool from the metastore to the 
db.

There is definitely scope for refactoring and improvement in all this, I will 
look into it further, but for now, this is a simpler bugfix to enable the 
already-existing regex to work correctly.

> RetryingMetaStoreClient does not retry JDOExceptions
> 
>
> Key: HIVE-9436
> URL: https://issues.apache.org/jira/browse/HIVE-9436
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-9436.2.patch, HIVE-9436.patch
>
>
> RetryingMetaStoreClient has a bug in the following bit of code:
> {code}
> } else if ((e.getCause() instanceof MetaException) &&
> e.getCause().getMessage().matches("JDO[a-zA-Z]*Exception")) {
>   caughtException = (MetaException) e.getCause();
> } else {
>   throw e.getCause();
> }
> {code}
> The bug here is that java String.matches matches the entire string to the 
> regex, and thus, that match will fail if the message contains anything before 
> or after JDO[a-zA-Z]\*Exception. The solution, however, is very simple, we 
> should match .\*JDO[a-zA-Z]\*Exception.\*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9436) RetryingMetaStoreClient does not retry JDOExceptions

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289984#comment-14289984
 ] 

Sushanth Sowmyan commented on HIVE-9436:


As to the precommit tests, it looks like the majority of those tests were 
already failing before this build:

{noformat}
Test Result (66 failures / +3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
...
{noformat}

This indicates that tests were already flaky (probably due to an earlier 
commit?) and there were a total of 3 reported regressions. Diffing between this 
build(2495) and the previous one where tests ran (2493) to find changes, I get:

{noformat}
< org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket5
---
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
> org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore
> org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testMetastoreProxyUser
> org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}

For 
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric:

{noformat}
Running: diff -a 
/home/hiveptest/54.205.215.38-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/udaf_histogram_numeric.q.out
 
/home/hiveptest/54.205.215.38-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/udaf_histogram_numeric.q.out
9c9
< [{"x":139.16078431372554,"y":255.0},{"x":386.1428571428572,"y":245.0}]
---
> [{"x":135.0284552845532,"y":246.0},{"x":381.39370078740143,"y":254.0}]
{noformat}

This does not look connected to this issue.

As for 
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore 
and 

{noformat}
java.lang.NullPointerException: null
at 
org.apache.hadoop.hive.metastore.HiveMetaStore.getDelegationToken(HiveMetaStore.java:5596)
at 
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.getDelegationTokenStr(TestHadoop20SAuthBridge.java:318)
at 
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.obtainTokenAndAddIntoUGI(TestHadoop20SAuthBridge.java:339)
at 
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.testSaslWithHiveMetaStore(TestHadoop20SAuthBridge.java:231)
{noformat}
{noformat}
java.lang.NullPointerException: null
at 
org.apache.hadoop.hive.metastore.HiveMetaStore.getDelegationToken(HiveMetaStore.java:5596)
at 
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.getDelegationTokenStr(TestHadoop20SAuthBridge.java:318)
at 
org.apache.hadoop.hive.thrift.TestHadoop20SAuthBridge.access$100(TestHadoop20SAuthBridge.java:62)
{noformat}

Both of these seem to be errors getting a delegation token, again something not 
connected with this connection retry issue.

As to the last one, 
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection, that 
looks like an issue with derby setup? I'm not certain. It does look like a 
valid issue in and of itself, but again, not related to this patch.

{noformat}
java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
at 
org.apache.derby.impl.sql.catalog.DataDictionaryImpl.duplicateDescriptorException(Unknown
 Source)
at 
org.apache.derby.impl.sql.catalog.DataDictionaryImpl.addDescriptor(Unknown 
Source)
at 
org.apache.derby.impl.sql.execute.CreateTableConstantAction.executeConstantAction(Unknown
 Source)
at org.apache.derby.impl.sql.execute.MiscResultSet.open(Unknown Source)
at 
org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown 
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown 
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at 
org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:72)
at 
org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:131)
at 
org.apache.hive.hcatalog.streaming.TestStreaming.(TestStreaming.java:157)
{noformat}


> RetryingMetaStoreClient does not retry JDOExceptions
> 
>
> Key: HIVE-9436
> URL: https://issues.apache.org/jira/browse/HIVE-9436
> Project

[jira] [Commented] (HIVE-8485) HMS on Oracle incompatibility

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289869#comment-14289869
 ] 

Sergey Shelukhin commented on HIVE-8485:


+1

> HMS on Oracle incompatibility
> -
>
> Key: HIVE-8485
> URL: https://issues.apache.org/jira/browse/HIVE-8485
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
> Environment: Oracle as metastore DB
>Reporter: Ryan Pridgeon
>Assignee: Chaoyu Tang
> Attachments: HIVE-8485.2.patch, HIVE-8485.patch
>
>
> Oracle does not distinguish between empty strings and NULL,which proves 
> problematic for DataNucleus.
> In the event a user creates a table with some property stored as an empty 
> string the table will no longer be accessible.
> i.e. TBLPROPERTIES ('serialization.null.format'='')
> If they try to select, describe, drop, etc the client prints the following 
> exception.
> ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found 
> 
> The work around for this was to go into the hive metastore on the Oracle 
> database and replace NULL with some other string. Users could then drop the 
> tables or alter their data to use the new null format they just set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9277) Hybrid Hybrid Grace Hash Join

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289866#comment-14289866
 ] 

Wei Zheng commented on HIVE-9277:
-

Thanks [~leftylev] for your review! I've updated the wording. Hope this time 
it's better explained.

> Hybrid Hybrid Grace Hash Join
> -
>
> Key: HIVE-9277
> URL: https://issues.apache.org/jira/browse/HIVE-9277
> Project: Hive
>  Issue Type: New Feature
>  Components: Physical Optimizer
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>  Labels: join
> Attachments: High-leveldesignforHybridHybridGraceHashJoinv1.0.pdf
>
>
> We are proposing an enhanced hash join algorithm called “hybrid hybrid grace 
> hash join”. We can benefit from this feature as illustrated below:
> o The query will not fail even if the estimated memory requirement is 
> slightly wrong
> o Expensive garbage collection overhead can be avoided when hash table grows
> o Join execution using a Map join operator even though the small table 
> doesn't fit in memory as spilling some data from the build and probe sides 
> will still be cheaper than having to shuffle the large fact table
> The design was based on Hadoop’s parallel processing capability and 
> significant amount of memory available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9404) NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289844#comment-14289844
 ] 

Eugene Koifman commented on HIVE-9404:
--

[~vikram.dixit] yes but only if you are also including HIVE-9390 

> NPE in 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()
> -
>
> Key: HIVE-9404
> URL: https://issues.apache.org/jira/browse/HIVE-9404
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.15.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.15.0
>
> Attachments: HIVE-9404.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct(TxnHandler.java:1015)
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:906)
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:238)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_open_txns(HiveMetaStore.java:5321)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9436) RetryingMetaStoreClient does not retry JDOExceptions

[Sushanth Sowmyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289829#comment-14289829
 ] 

Sushanth Sowmyan commented on HIVE-9436:


Randomly sampling some of the test failures:

{noformat}
source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/udaf_histogram_numeric.q.out
 
/home/hiveptest/54.205.215.38-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/udaf_histogram_numeric.q.out
9c9
< [{"x":139.16078431372554,"y":255.0},{"x":386.1428571428572,"y":245.0}]
---
> [{"x":135.0284552845532,"y":246.0},{"x":381.39370078740143,"y":254.0}]
{noformat}

{noformat}
Running: diff -a 
/home/hiveptest/54.159.206.72-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../itests/qtest/target/qfile-results/clientpositive/auto_join12.q.out
 
/home/hiveptest/54.159.206.72-hiveptest-1/apache-svn-trunk-source/itests/qtest/../../ql/src/test/results/clientpositive/auto_join12.q.out
32c32
< $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
---
> $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
35c35
< $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
---
> $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
39c39
< $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
---
> $hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:$hdt$_0:src 
54c54
< $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:src 
---
> $hdt$_0:$hdt$_0:$hdt$_1:$hdt$_1:$hdt$_1:$hdt$_1:src 
{noformat}

Neither of those have anything whatsoever to do with retries.

Also, I see some other errors reporting "junit.framework.AssertionFailedError: 
Client Execution failed with error code = 4 running", which is similar to 
what's reported in HIVE-8997. Is there a test flakiness issue right now in 
trunk?

> RetryingMetaStoreClient does not retry JDOExceptions
> 
>
> Key: HIVE-9436
> URL: https://issues.apache.org/jira/browse/HIVE-9436
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-9436.2.patch, HIVE-9436.patch
>
>
> RetryingMetaStoreClient has a bug in the following bit of code:
> {code}
> } else if ((e.getCause() instanceof MetaException) &&
> e.getCause().getMessage().matches("JDO[a-zA-Z]*Exception")) {
>   caughtException = (MetaException) e.getCause();
> } else {
>   throw e.getCause();
> }
> {code}
> The bug here is that java String.matches matches the entire string to the 
> regex, and thus, that match will fail if the message contains anything before 
> or after JDO[a-zA-Z]\*Exception. The solution, however, is very simple, we 
> should match .\*JDO[a-zA-Z]\*Exception.\*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9404) NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289830#comment-14289830
 ] 

Vikram Dixit K commented on HIVE-9404:
--

[~ekoifman] Does this need to ported to branch 1.0?

> NPE in 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct()
> -
>
> Key: HIVE-9404
> URL: https://issues.apache.org/jira/browse/HIVE-9404
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 0.15.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.15.0
>
> Attachments: HIVE-9404.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct(TxnHandler.java:1015)
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.checkRetryable(TxnHandler.java:906)
>   at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getOpenTxns(TxnHandler.java:238)
>   at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_open_txns(HiveMetaStore.java:5321)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9448) Merge spark to trunk 1/23/15

[Szehon Ho (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szehon Ho updated HIVE-9448:

Attachment: HIVE-9448.2.patch

> Merge spark to trunk 1/23/15
> 
>
> Key: HIVE-9448
> URL: https://issues.apache.org/jira/browse/HIVE-9448
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Affects Versions: 0.15.0
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-9448.2.patch, HIVE-9448.patch
>
>
> Merging latest spark changes to trunk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289823#comment-14289823
 ] 

Hive QA commented on HIVE-9152:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694198/HIVE-9152.2-spark.patch

{color:red}ERROR:{color} -1 due to 65 failed/errored test(s), 7356 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_spark_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cl

Re: Created branch 1.0

[Szehon Ho]

Wherever I've seen in enterprise software, the trunk-based development
model has been the standard where all release branches are cut from trunk
and short-lived.  I've never heard of a case where a branch originally
designated for 0.14 (minor release) is cut again to become 1.0 (major
release), and I dont think if you ask anyone they will expect it either.
There was also no announced plan when cutting 0.14 branch that it was
eventually going to be 1.0.  As Brock pointed out in the beginning, Hadoop
branch/versioning is the only exception and an anti-pattern, and all the
confusion like why 0.xx has features not in 1.0 would not be there if it
followed this.  I would really hate to see the same anti-pattern happen to
Hive, so my vote is also against this.  Also this standard release
branching practice has been in Hive throughout its history, you wouldn't
make 0.14 out of 0.13 branch, would you?

>From the stability and long-term support use-cases that is very definitely
> the wrong thing to do - to cram code into a 1.0 release.

Major release is supposed to be stable.


I also don't see how cutting 1.0 from trunk precludes it from stabilizing.
Also I don't think those arguments of 0.14 as most stable that can be
backed up, what constitutes stability?  Bug fixes are just one part, in
that case there are always more bug fixes in later Hive versions than
earlier ones, so probably API stability is a more measure-able term and
should be more important to consider.

Thanks,
Szehon


On Fri, Jan 23, 2015 at 10:42 AM, Gopal V  wrote:

> On 1/23/15, 6:59 AM, Xuefu Zhang wrote:
>
>> While it's true that a release isn't going to include everything from
>> trunk, proposed 1.0 release is branched off 0.14, which was again branched
>> from trunk long time ago. If you compare the code base, you will see the
>> huge difference.
>>
>
> From the stability and long-term support use-cases that is very definitely
> the wrong thing to do - to cram code into a 1.0 release.
>
> The "huge difference" is *THE* really worrying red-flag.
>
> Or is the thought behind "everything from trunk" that 1.0 just a number?
>
>  0.14.1 in terms of functionality and stability will be much clearer,
>> meeting the all expectations for a major release.
>>
>
> Just to be clear, when hive-14 was released, it was actually a major
> release.
>
> That branch kicked off in Sept and has been updated since then with a
> known set of critical fixes, giving it pedigree and has already seen
> customer time.
>
> In all this discussion, it doesn't sound like you consider 0.15 to be a
> major release - that gives me no confidence in your approach.
>
> Cheers,
> Gopal
>
>  On Thu, Jan 22, 2015 at 3:08 PM, Thejas Nair 
>> wrote:
>>
>>  On Thu, Jan 22, 2015 at 12:31 PM, Xuefu Zhang 
>>> wrote:
>>> > Hi Thejas/Alan,
>>> >
>>> > From all the argument, I think there was an assumption that the
>>> proposed
>>> > 1.0 release will be imminent and 0.15 will happen far after that. Based
>>> on
>>> > that assumption, 0.15 will become 1.1, which is greater in scope than
>>> 1.0.
>>> > However, this assumption may not be true. The confusion will be
>>> significant
>>> > if 0.15 is released early as 0.15 before 0.14.1 is released as 1.0.
>>>
>>> Yes, the assumption is that 1.0 will be out very soon,  before 0.15
>>> line is ready, and that 0.15 can become 1.1 .
>>> Do you think that assumption won't hold true ? (In previous emails in
>>> this thread, I talk about reasons why this assumption is reliable).
>>> I agree that it does not make sense to release 1.0 as proposed in this
>>> thread if that does not hold true.
>>>
>>> > Another concern is that, the proposed release of 1.0 is a subset of of
>>> > Hive's functionality, and for major releases users are expecting major
>>> > improvement in functionality as well as stability. Mutating from 0.14.1
>>> > release seems falling short in that expectation.
>>>
>>> Every release of hive has been a subset of tip of the trunk (we branch
>>> for release while trunk moves ahead), and super set of changes of
>>> every previous release. So every release so far has had a subset of
>>> functionality of hive trunk branch (if that is what you are referring
>>> to).
>>> With the 1.0 proposed based on 0.14 maintenance branch, this still
>>> holds true. (Again, this is with the assumption you called out about
>>> about timeline of 1.0 and 0.15).
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from you

[jira] [Commented] (HIVE-6617) Reduce ambiguity in grammar

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289812#comment-14289812
 ] 

Hive QA commented on HIVE-6617:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694082/HIVE-6617.07.patch

{color:red}ERROR:{color} -1 due to 81 failed/errored test(s), 7348 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_set_show_current_role
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_10_0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonreserved_keywords_input37
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonreserved_keywords_insert_into1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_vectorization_ppd
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unset_table_view_property
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_10_0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_navfn
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_prunin

[jira] [Updated] (HIVE-9440) Folders may not be pruned for Hadoop 2

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-9440:
--
Fix Version/s: spark-branch

Also merged to Spark branch.

> Folders may not be pruned for Hadoop 2
> --
>
> Key: HIVE-9440
> URL: https://issues.apache.org/jira/browse/HIVE-9440
> Project: Hive
>  Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: spark-branch, 0.15.0
>
> Attachments: HIVE-9440.1-spark.patch, HIVE-9440.1.patch
>
>
> HIVE-9367 is not a complete fix. It fixed for Hadoop 1. For Hadoop2, this 
> method is not invoked.
> {noformat}
> protected FileStatus[] listStatus(JobConf job) throws IOException;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9235) Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR

[Vikram Dixit K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vikram Dixit K updated HIVE-9235:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, 
> TIMESTAMP, CHAR, and VARCHAR
> -
>
> Key: HIVE-9235
> URL: https://issues.apache.org/jira/browse/HIVE-9235
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-9235.01.patch, HIVE-9235.02.patch
>
>
> Title was: Make Parquet Vectorization of these data types work: DECIMAL, 
> DATE, TIMESTAMP, CHAR, and VARCHAR
> Support for doing vector column assign is missing for some data types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9235) Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR

[Vikram Dixit K (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289810#comment-14289810
 ] 

Vikram Dixit K commented on HIVE-9235:
--

Committed to trunk and branches.

> Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, 
> TIMESTAMP, CHAR, and VARCHAR
> -
>
> Key: HIVE-9235
> URL: https://issues.apache.org/jira/browse/HIVE-9235
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-9235.01.patch, HIVE-9235.02.patch
>
>
> Title was: Make Parquet Vectorization of these data types work: DECIMAL, 
> DATE, TIMESTAMP, CHAR, and VARCHAR
> Support for doing vector column assign is missing for some data types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9453) Initial patch [hbase-metastore branch]

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-9453:
-
Status: Patch Available  (was: Open)

> Initial patch [hbase-metastore branch]
> --
>
> Key: HIVE-9453
> URL: https://issues.apache.org/jira/browse/HIVE-9453
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-9453.patch
>
>
> This initial patch has several important features:
> # HBaseStore, a new implementation of RawStore that stores the data in HBase.
> # Subclasses of the thrift metastore objects to remove the massive 
> duplication of data where every partition contains a nearly identical storage 
> descriptor.
> # Caches for catalog objects and statistics so that repeated metastore calls 
> don't result in repeated calls against HBase.
> Currently this "works" to the point that "load table" and "select" work.  I 
> have not tested any other statements, and I suspect most fail.  There is no 
> security, no authorization, and a no a lot of other things.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9453) Initial patch [hbase-metastore branch]

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-9453:
-
Attachment: HIVE-9453.patch

> Initial patch [hbase-metastore branch]
> --
>
> Key: HIVE-9453
> URL: https://issues.apache.org/jira/browse/HIVE-9453
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-9453.patch
>
>
> This initial patch has several important features:
> # HBaseStore, a new implementation of RawStore that stores the data in HBase.
> # Subclasses of the thrift metastore objects to remove the massive 
> duplication of data where every partition contains a nearly identical storage 
> descriptor.
> # Caches for catalog objects and statistics so that repeated metastore calls 
> don't result in repeated calls against HBase.
> Currently this "works" to the point that "load table" and "select" work.  I 
> have not tested any other statements, and I suspect most fail.  There is no 
> security, no authorization, and a no a lot of other things.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Creating a branch for hbase metastore work

[Alan Gates]

I've created a new branch hive-metastore for this work and filed 
HIVE-9452 as the umbrella JIRA for this work.


Alan.


Edward Capriolo 
January 23, 2015 at 8:59
A while back the datastax people did this for brisk + cassandra. We should
probably make this pluggable so it works with nosql beyond hbase.


Nick Dimiduk 
January 22, 2015 at 22:48
+1


Brock Noland 
January 22, 2015 at 21:58
+1
Alan Gates 
January 22, 2015 at 18:19
I've been working on a prototype of using HBase to store Hive's 
metadata.  Basically I've built a new implementation of RawStore that 
writes to HBase rather than DataNucleus.  I want to see if I can build 
something that has a much more straightforward schema than DN and that 
is much faster.


I'd like to get this out in public so other can look at it and 
contribute, but it's no where near ready for real time.  So I propose 
to create a branch and put the code there.  Any objections?


Alan.


--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HIVE-9453) Initial patch [hbase-metastore branch]

[Alan Gates (JIRA)]

Alan Gates created HIVE-9453:


 Summary: Initial patch [hbase-metastore branch]
 Key: HIVE-9453
 URL: https://issues.apache.org/jira/browse/HIVE-9453
 Project: Hive
  Issue Type: Sub-task
Reporter: Alan Gates
Assignee: Alan Gates


This initial patch has several important features:
# HBaseStore, a new implementation of RawStore that stores the data in HBase.
# Subclasses of the thrift metastore objects to remove the massive duplication 
of data where every partition contains a nearly identical storage descriptor.
# Caches for catalog objects and statistics so that repeated metastore calls 
don't result in repeated calls against HBase.

Currently this "works" to the point that "load table" and "select" work.  I 
have not tested any other statements, and I suspect most fail.  There is no 
security, no authorization, and a no a lot of other things.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9431) CBO (Calcite Return Path): Removing AST from ParseContext

[Laljo John Pullokkaran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289743#comment-14289743
 ] 

Laljo John Pullokkaran commented on HIVE-9431:
--

[~jcamachorodriguez] seems like we have unit test failures.

> CBO (Calcite Return Path): Removing AST from ParseContext
> -
>
> Key: HIVE-9431
> URL: https://issues.apache.org/jira/browse/HIVE-9431
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 0.15.0
>
> Attachments: HIVE-9431.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HIVE-6308) COLUMNS_V2 Metastore table not populated for tables created without an explicit column list.

[Yongzhi Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yongzhi Chen reassigned HIVE-6308:
--

Assignee: Yongzhi Chen

> COLUMNS_V2 Metastore table not populated for tables created without an 
> explicit column list.
> 
>
> Key: HIVE-6308
> URL: https://issues.apache.org/jira/browse/HIVE-6308
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 0.10.0
>Reporter: Alexander Behm
>Assignee: Yongzhi Chen
>
> Consider this example table:
> CREATE TABLE avro_test
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED as INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> TBLPROPERTIES (
> 'avro.schema.url'='file:///path/to/the/schema/test_serializer.avsc');
> When I try to run an ANALYZE TABLE for computing column stats on any of the 
> columns, then I get:
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> NoSuchObjectException(message:Column o_orderpriority for which stats 
> gathering is requested doesn't exist.)
> at 
> org.apache.hadoop.hive.ql.metadata.Hive.updateTableColumnStatistics(Hive.java:2280)
> at 
> org.apache.hadoop.hive.ql.exec.ColumnStatsTask.persistTableStats(ColumnStatsTask.java:331)
> at 
> org.apache.hadoop.hive.ql.exec.ColumnStatsTask.execute(ColumnStatsTask.java:343)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
> at 
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:66)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1383)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:982)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> The root cause appears to be that the COLUMNS_V2 table in the Metastore isn't 
> populated properly during the table creation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-9443:
--
Status: Patch Available  (was: Open)

Updated testcases.

> ORC PPD - fix fuzzy case evaluation of IS_NULL
> --
>
> Key: HIVE-9443
> URL: https://issues.apache.org/jira/browse/HIVE-9443
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.15.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.15.0
>
> Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch
>
>
> ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
> The code-flow in effect should be
> {code}
> if (min == null) {
>   // all nulls
>   return YES;
> } else {
>   if (hasNull == true) {
>  // some nulls
>  return YES_NO; //maybe
>   }
>   // no nulls
>   return NO;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-9443:
--
Attachment: HIVE-9443.2.patch

> ORC PPD - fix fuzzy case evaluation of IS_NULL
> --
>
> Key: HIVE-9443
> URL: https://issues.apache.org/jira/browse/HIVE-9443
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.15.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.15.0
>
> Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch
>
>
> ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
> The code-flow in effect should be
> {code}
> if (min == null) {
>   // all nulls
>   return YES;
> } else {
>   if (hasNull == true) {
>  // some nulls
>  return YES_NO; //maybe
>   }
>   // no nulls
>   return NO;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9443) ORC PPD - fix fuzzy case evaluation of IS_NULL

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-9443:
--
Status: Open  (was: Patch Available)

> ORC PPD - fix fuzzy case evaluation of IS_NULL
> --
>
> Key: HIVE-9443
> URL: https://issues.apache.org/jira/browse/HIVE-9443
> Project: Hive
>  Issue Type: Bug
>  Components: File Formats
>Affects Versions: 0.15.0
>Reporter: Gopal V
>Assignee: Gopal V
> Fix For: 0.15.0
>
> Attachments: HIVE-9443.1.patch, HIVE-9443.2.patch
>
>
> ORC PPD returns IS_NULL wrong for the fuzz case of some-nulls.
> The code-flow in effect should be
> {code}
> if (min == null) {
>   // all nulls
>   return YES;
> } else {
>   if (hasNull == true) {
>  // some nulls
>  return YES_NO; //maybe
>   }
>   // no nulls
>   return NO;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Created branch 1.0

[Gopal V]

On 1/23/15, 6:59 AM, Xuefu Zhang wrote:

While it's true that a release isn't going to include everything from
trunk, proposed 1.0 release is branched off 0.14, which was again branched
from trunk long time ago. If you compare the code base, you will see the
huge difference.


From the stability and long-term support use-cases that is very 
definitely the wrong thing to do - to cram code into a 1.0 release.


The "huge difference" is *THE* really worrying red-flag.

Or is the thought behind "everything from trunk" that 1.0 just a number?


0.14.1 in terms of functionality and stability will be much clearer,
meeting the all expectations for a major release.


Just to be clear, when hive-14 was released, it was actually a major 
release.


That branch kicked off in Sept and has been updated since then with a 
known set of critical fixes, giving it pedigree and has already seen 
customer time.


In all this discussion, it doesn't sound like you consider 0.15 to be a 
major release - that gives me no confidence in your approach.


Cheers,
Gopal


On Thu, Jan 22, 2015 at 3:08 PM, Thejas Nair  wrote:


On Thu, Jan 22, 2015 at 12:31 PM, Xuefu Zhang  wrote:
> Hi Thejas/Alan,
>
> From all the argument, I think there was an assumption that the proposed
> 1.0 release will be imminent and 0.15 will happen far after that. Based
on
> that assumption, 0.15 will become 1.1, which is greater in scope than
1.0.
> However, this assumption may not be true. The confusion will be
significant
> if 0.15 is released early as 0.15 before 0.14.1 is released as 1.0.

Yes, the assumption is that 1.0 will be out very soon,  before 0.15
line is ready, and that 0.15 can become 1.1 .
Do you think that assumption won't hold true ? (In previous emails in
this thread, I talk about reasons why this assumption is reliable).
I agree that it does not make sense to release 1.0 as proposed in this
thread if that does not hold true.

> Another concern is that, the proposed release of 1.0 is a subset of of
> Hive's functionality, and for major releases users are expecting major
> improvement in functionality as well as stability. Mutating from 0.14.1
> release seems falling short in that expectation.

Every release of hive has been a subset of tip of the trunk (we branch
for release while trunk moves ahead), and super set of changes of
every previous release. So every release so far has had a subset of
functionality of hive trunk branch (if that is what you are referring
to).
With the 1.0 proposed based on 0.14 maintenance branch, this still
holds true. (Again, this is with the assumption you called out about
about timeline of 1.0 and 0.15).

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.

[jira] [Created] (HIVE-9452) Use HBase to store Hive metadata

[Alan Gates (JIRA)]

Alan Gates created HIVE-9452:


 Summary: Use HBase to store Hive metadata
 Key: HIVE-9452
 URL: https://issues.apache.org/jira/browse/HIVE-9452
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Alan Gates
Assignee: Alan Gates


This is an umbrella JIRA for a project to explore using HBase to store the Hive 
data catalog (ie the metastore).  This project has several goals:
# The current metastore implementation is slow when tables have thousands or 
more partitions.  With Tez and Spark engines we are pushing Hive to a point 
where queries only take a few seconds to run.  But planning the query can take 
as long as running it.  Much of this time is spent in metadata operations.
# Due to scale limitations we have never allowed tasks to communicate directly 
with the metastore.  However, with the development of LLAP this requirement 
will have to be relaxed.  If we can relax this there are other use cases that 
could benefit from this.  
# Eating our own dogfood.  Rather than using external systems to store our 
metadata there are benefits to using other components in the Hadoop system.

The proposal is to create a new branch and work on the prototype there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Created branch 1.0

[Sergey Shelukhin]

I don't think that 1.0 release is expected to have major increase in
functionality, in fact I strongly disagree with this.
Major release is supposed to be stable.

With 0.14, we have a good evidence of use both in the community, and
releases within the platforms by major vendors that caused bugs (including
critical ones) to be found and fixed.
That makes 0.14.1 patch release a very good candidate for 1.0 release.
We should not release 1.0 off 0.15 until there's* similarly good evidence *of
use by the larger community, and bug fixing period.

If we cannot agree on making 0.14.1 (2/3...) 1.0 (for which there's no real
reason IMHO, confusion argument is completely bogus both because there
isn't any - we will spell loud and clear that 1.0 is a the stable release
and 0.15 can becomes 1.1, and because users don't follow version
intricacies that closely anyway), then I feel we will need to go to
0.15.1/2/3.
But we'd need evidence of use and stabilization before we proceed with
15-based 1.0.







However, imho cutting 1.0 from 0.14.1 is completely acceptable.

On Thu, Jan 22, 2015 at 12:31 PM, Xuefu Zhang  wrote:

> Hi Thejas/Alan,
>
> From all the argument, I think there was an assumption that the proposed
> 1.0 release will be imminent and 0.15 will happen far after that. Based on
> that assumption, 0.15 will become 1.1, which is greater in scope than 1.0.
> However, this assumption may not be true. The confusion will be significant
> if 0.15 is released early as 0.15 before 0.14.1 is released as 1.0.
>
> Another concern is that, the proposed release of 1.0 is a subset of of
> Hive's functionality, and for major releases users are expecting major
> improvement in functionality as well as stability. Mutating from 0.14.1
> release seems falling short in that expectation.
>
> Having said that, I'd think it makes more sense to release 0.15 as 0.15,
> and later we release 1.0 as the major release that supersedes any previous
> releases. That will fulfill the expectations of a major release.
>
> Thanks,
> Xuefu
>
> On Thu, Jan 22, 2015 at 12:12 PM, Alan Gates 
> wrote:
>
> > I had one clarifying question for Brock and Xuefu.  Was your proposal to
> > still call the branch from trunk you are planning in a few days 0.15 (and
> > hence release it as 0.15) and have 1.0 be a later release?  Or did you
> want
> > to call what is now 0.15 1.0?  If you wanted 1.0 to be post 0.15, are you
> > ok with stipulating that the next release from trunk after 0.15 (what
> would
> > have been 0.16) is 1.0?
> >
> > Alan.
> >
> >   Thejas Nair 
> >  January 22, 2015 at 12:04
> > Brock, Xuefu,
> >
> > We seem to have trouble reaching to a consensus here. (Please see my
> > arguments why I don't see this causing confusions, and let me know if
> > it changes your opinion).
> > How should we move forward ? Do you think we need to go through a
> > formal vote regarding the release plan as per hive by-laws ?
> >
> >
> >   Thejas Nair 
> >  January 22, 2015 at 10:38
> > I don't see any reasons for confusion.
> > From a user perspective, 1.0 is going to have a super set of changes of
> > 0.14.
> > 1.1 (based on planned 0.15 release) will have a super set of changes in
> > 1.0 .
> >
> >
> >   Xuefu Zhang 
> >  January 21, 2015 at 22:47
> > I strongly believe that the concept of 1.0 out of a branch as proposed is
> > creating the greatest confusion in the community. If for any reason that
> > 1.0 cannot be cut from the trunk, that means that we are not ready and so
> > shall wait until so before considering such a release. Thus, I'd -1 on
> this
> > proposal.
> >
> > Thanks,
> > Xuefu
> >
> >
> >   Gopal V 
> >  January 21, 2015 at 22:29
> > On 1/21/15, 7:09 PM, Brock Noland wrote:
> >
> > Too be clear I strongly feel creating 1.0 from 0.14 will be confusing. In
> > fact it's already crrated confusion amongst folks on this list.
> > Furthermore
> > 1.0 should be created from trunk and be a superset of previous releases.
> >
> >
> > I don't think there is any confusion over that - 1.0 is a long-term
> > maintenance which is going to be a super-set of all *critical fixes* made
> > from here on (emphasis).
> >
> > In fact, a long-term support release should be released off an actively
> > updated maintenance branch, which has been baked-in and never from the
> > trunk.
> >
> > Those who have followed the earlier mails would realize that the most
> > important "feature" about this branch is to stick to only long term
> > maintenance - which in effect is adopting HBase's successful idea.
> >
> > That is just plain solid engineering.
> >
> > Anyway, it would be in the best interests of the larger community, to
> find
> > out who else finds that approach confusing.
> >
> > Brock, I'm not sure whether you are confused or whether you think other
> > people will be confused (and if so, why?).
> >
> > Cheers,
> > Gopal
> >
> > On Wed, Jan 21, 2015 at 6:05 PM, Vikram Dixit K 
> > 
> > wrote:
> >
> > @Brock,
> >
> > I created this branch from 0.14. I created 

[jira] [Commented] (HIVE-9439) merge ORC disk ranges as we go when reading RGs

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289635#comment-14289635
 ] 

Sergey Shelukhin commented on HIVE-9439:


I don't think these are related...

> merge ORC disk ranges as we go when reading RGs
> ---
>
> Key: HIVE-9439
> URL: https://issues.apache.org/jira/browse/HIVE-9439
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HIVE-9439.01.patch, HIVE-9439.patch
>
>
> Currently we get ranges for all the RGs individually, then merge them. We can 
> do some (probably most of) the merging as we go.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-9436) RetryingMetaStoreClient does not retry JDOExceptions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-9436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289628#comment-14289628
 ] 

Hive QA commented on HIVE-9436:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694069/HIVE-9436.2.patch

{color:red}ERROR:{color} -1 due to 67 failed/errored test(s), 7346 tests 
executed
*Failed tests:*
{noformat}
TestCustomAuthentication - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_column_access_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_skew_1_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataOnlyOptimizer
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_gby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_join_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_25
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union24
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union28
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_null
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_6_subq
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_filter_join_breaktask2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pru

[jira] [Updated] (HIVE-9152) Dynamic Partition Pruning [Spark Branch]

[Chao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao updated HIVE-9152:
---
Attachment: HIVE-9152.2-spark.patch

> Dynamic Partition Pruning [Spark Branch]
> 
>
> Key: HIVE-9152
> URL: https://issues.apache.org/jira/browse/HIVE-9152
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Affects Versions: spark-branch
>Reporter: Brock Noland
>Assignee: Chao
> Attachments: HIVE-9152.1-spark.patch, HIVE-9152.2-spark.patch
>
>
> Tez implemented dynamic partition pruning in HIVE-7826. This is a nice 
> optimization and we should implement the same in HOS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)