[jira] [Updated] (HIVE-15556) Replicate views

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-15556:

Status: In Progress  (was: Patch Available)

> Replicate views
> ---
>
> Key: HIVE-15556
> URL: https://issues.apache.org/jira/browse/HIVE-15556
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Vaibhav Gumashta
>Assignee: Sankar Hariappan
> Attachments: HIVE-15556.01.patch, HIVE-15556.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16119) HiveMetaStoreChecker: remove singleThread logic duplication

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-16119:

Summary: HiveMetaStoreChecker: remove singleThread logic duplication  (was: 
HiveMetaStoreChecker - singleThread/parallel logic duplication)

> HiveMetaStoreChecker: remove singleThread logic duplication
> ---
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch, HIVE-16119.2.patch, 
> HIVE-16119.3.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading 
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special 
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16104) LLAP: preemption may be too aggressive if the pre-empted task doesn't die immediately

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902605#comment-15902605
 ] 

Hive QA commented on HIVE-16104:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856901/HIVE-16104.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10335 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4040/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4040/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4040/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856901 - PreCommit-HIVE-Build

> LLAP: preemption may be too aggressive if the pre-empted task doesn't die 
> immediately
> -
>
> Key: HIVE-16104
> URL: https://issues.apache.org/jira/browse/HIVE-16104
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16104.01.patch, HIVE-16104.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16135) Vectorization: unhandled constant type for scalar argument

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902566#comment-15902566
 ] 

Matt McCline commented on HIVE-16135:
-

Committed to master.

> Vectorization: unhandled constant type for scalar argument
> --
>
> Key: HIVE-16135
> URL: https://issues.apache.org/jira/browse/HIVE-16135
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Fix For: 2.2.0
>
> Attachments: HIVE-16135.1.patch, HIVE-16135.2.patch, 
> HIVE-16135.3.patch
>
>
> Map: Predicate expression for FILTER operator: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Udf: unhandled constant 
> type for scalar argument. Expecting string.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16135) Vectorization: unhandled constant type for scalar argument

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-16135:

Fix Version/s: 2.2.0

> Vectorization: unhandled constant type for scalar argument
> --
>
> Key: HIVE-16135
> URL: https://issues.apache.org/jira/browse/HIVE-16135
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Fix For: 2.2.0
>
> Attachments: HIVE-16135.1.patch, HIVE-16135.2.patch, 
> HIVE-16135.3.patch
>
>
> Map: Predicate expression for FILTER operator: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Udf: unhandled constant 
> type for scalar argument. Expecting string.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16135) Vectorization: unhandled constant type for scalar argument

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-16135:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Vectorization: unhandled constant type for scalar argument
> --
>
> Key: HIVE-16135
> URL: https://issues.apache.org/jira/browse/HIVE-16135
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Fix For: 2.2.0
>
> Attachments: HIVE-16135.1.patch, HIVE-16135.2.patch, 
> HIVE-16135.3.patch
>
>
> Map: Predicate expression for FILTER operator: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Udf: unhandled constant 
> type for scalar argument. Expecting string.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902555#comment-15902555
 ] 

Hive QA commented on HIVE-16133:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856897/HIVE-16133.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 380 failed/errored test(s), 9758 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] 
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery] 
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_partition]
 (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_project]
 (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete] 
(batchId=76)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete_own_table]
 (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update_own_table]
 (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_serde] (batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_serde] (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_join2] 
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_non_partitioned]
 (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned] 
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_orig_table] 
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table] 
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_no_match] 
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_non_partitioned]
 (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_partitioned]
 (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition] 
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[implicit_cast_during_insert]
 (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_dynamic_partition]
 (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_not_bucketed]
 (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_nonacid_from_acid]
 (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_orig_table] 
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_update_delete] 
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_acid_not_bucketed]
 (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_dynamic_partitioned]
 (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_non_partitioned]
 (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table]
 (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_partitioned]
 (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table] 
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_reader] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_acid_no_masking] 
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_authorization_sqlstd]
 (batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create]
 (batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_multi_db]
 (batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_describe]
 (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_createas1] 
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_dictionary_threshold]
 (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_diff_part_cols2] 
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_diff_part_cols] 

[jira] [Commented] (HIVE-15983) Support the named columns join

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902514#comment-15902514
 ] 

Hive QA commented on HIVE-15983:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856896/HIVE-15983.01.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 10252 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_with_constraints] 
(batchId=63)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_constraint_duplicate_pk]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_constraint_invalid_fk_col1]
 (batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_constraint_invalid_fk_col2]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_constraint_invalid_fk_tbl1]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_constraint_invalid_fk_tbl2]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_constraint_invalid_pk_col]
 (batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_table_constraint_invalid_pk_tbl]
 (batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_with_constraints_duplicate_name]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_with_constraints_enable]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[create_with_constraints_validate]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_invalid_constraint1]
 (batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_invalid_constraint2]
 (batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_invalid_constraint3]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[drop_invalid_constraint4]
 (batchId=87)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[join_alt_syntax_comma_on]
 (batchId=86)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct]
 (batchId=107)
org.apache.hive.jdbc.TestJdbcDriver2.org.apache.hive.jdbc.TestJdbcDriver2 
(batchId=217)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4038/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4038/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4038/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856896 - PreCommit-HIVE-Build

> Support the named columns join
> --
>
> Key: HIVE-15983
> URL: https://issues.apache.org/jira/browse/HIVE-15983
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15983.01.patch
>
>
> The named columns join is a common shortcut allowing joins on identically 
> named keys. Example: select * from t1 join t2 using c1 is equivalent to 
> select * from t1 join t2 on t1.c1 = t2.c1. SQL standard reference: Section 7.7



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14864) Distcp is not called from MoveTask when src is a directory

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902460#comment-15902460
 ] 

Sahil Takiar commented on HIVE-14864:
-

[~spena] I worked on this some more, and think a unit test may be better suited 
for this patch rather than a qtest. There are a number of different queries 
that could invoke this method (e.g. IMPORT queries use this method too), and 
more may be added in the future. I added some integration tests that run 
against a mini HDFS cluster, and some unit tests that just rely on mocking.

[~ste...@apache.org] I agree, calling {{getContentSummary}} on S3 will be very 
slow. I've thought about filing a JIRA for the optimization you mentioned a few 
times, but never got around to doing it. Fortunately, this specific code won't 
be hit for S3, only for HDFS.

> Distcp is not called from MoveTask when src is a directory
> --
>
> Key: HIVE-14864
> URL: https://issues.apache.org/jira/browse/HIVE-14864
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Sahil Takiar
> Attachments: HIVE-14864.1.patch, HIVE-14864.2.patch, 
> HIVE-14864.3.patch, HIVE-14864.4.patch, HIVE-14864.patch
>
>
> In FileUtils.java the following code does not get executed even when src 
> directory size is greater than HIVE_EXEC_COPYFILE_MAXSIZE because 
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We 
> should use srcFS.getContentSummary(src).getLength() instead.
> {noformat}
> /* Run distcp if source file/dir is too big */
> if (srcFS.getUri().getScheme().equals("hdfs") &&
> srcFS.getFileStatus(src).getLen() > 
> conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE)) {
>   LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes. 
> (MAX: " + conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) + 
> ")");
>   LOG.info("Launch distributed copy (distcp) job.");
>   HiveConfUtil.updateJobCredentialProviders(conf);
>   copied = shims.runDistCp(src, dst, conf);
>   if (copied && deleteSource) {
> srcFS.delete(src, true);
>   }
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16107) JDBC: HttpClient should retry one more time on NoHttpResponseException

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902458#comment-15902458
 ] 

Hive QA commented on HIVE-16107:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856857/HIVE-16107.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10336 tests 
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery 
(batchId=219)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4037/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4037/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4037/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856857 - PreCommit-HIVE-Build

> JDBC: HttpClient should retry one more time on NoHttpResponseException
> --
>
> Key: HIVE-16107
> URL: https://issues.apache.org/jira/browse/HIVE-16107
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-16107.1.patch
>
>
> Hive's JDBC client in HTTP transport mode doesn't retry on 
> NoHttpResponseException. We've seen the exception being thrown to the JDBC 
> end user when used with Knox as the proxy, when Knox upgraded its jetty 
> version, which has a smaller value for jetty connector idletimeout, and as a 
> result closes the HTTP connection on server side. The next jdbc query on the 
> client, throws a NoHttpResponseException. However, subsequent queries 
> reconnect, but the JDBC driver should ideally handle this by retrying.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14864) Distcp is not called from MoveTask when src is a directory

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-14864:

Attachment: HIVE-14864.4.patch

> Distcp is not called from MoveTask when src is a directory
> --
>
> Key: HIVE-14864
> URL: https://issues.apache.org/jira/browse/HIVE-14864
> Project: Hive
>  Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Sahil Takiar
> Attachments: HIVE-14864.1.patch, HIVE-14864.2.patch, 
> HIVE-14864.3.patch, HIVE-14864.4.patch, HIVE-14864.patch
>
>
> In FileUtils.java the following code does not get executed even when src 
> directory size is greater than HIVE_EXEC_COPYFILE_MAXSIZE because 
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We 
> should use srcFS.getContentSummary(src).getLength() instead.
> {noformat}
> /* Run distcp if source file/dir is too big */
> if (srcFS.getUri().getScheme().equals("hdfs") &&
> srcFS.getFileStatus(src).getLen() > 
> conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE)) {
>   LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes. 
> (MAX: " + conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) + 
> ")");
>   LOG.info("Launch distributed copy (distcp) job.");
>   HiveConfUtil.updateJobCredentialProviders(conf);
>   copied = shims.runDistCp(src, dst, conf);
>   if (copied && deleteSource) {
> srcFS.delete(src, true);
>   }
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16091) Support subqueries in project/select

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-16091:
---
Status: Patch Available  (was: Open)

First patch contain changes to support subqueries in SELECT. More testings 
remain pending.

> Support subqueries in project/select
> 
>
> Key: HIVE-16091
> URL: https://issues.apache.org/jira/browse/HIVE-16091
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16091.1.patch
>
>
> Currently scalar subqueries are supported in filter only (WHERE/HAVING). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16071) Spark remote driver misuses the timeout in RPC handshake

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902455#comment-15902455
 ] 

Xuefu Zhang commented on HIVE-16071:


{quote}
One improvement may be to call onError unconditionally in channelInactive - 
current code depends on dispose to call onError:
{quote}
I think this makes sense. However, given that we don't really have a concrete 
case to justify, we probably just need to keep an eye on this.

What seems more certain is that we can fix cancelTask timeout value using 
client.connnect.timeout. If the default value of 1s is inadequate, we can pop 
up it as well.

> Spark remote driver misuses the timeout in RPC handshake
> 
>
> Key: HIVE-16071
> URL: https://issues.apache.org/jira/browse/HIVE-16071
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-16071.patch
>
>
> Based on its property description in HiveConf and the comments in HIVE-12650 
> (https://issues.apache.org/jira/browse/HIVE-12650?focusedCommentId=15128979=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15128979),
>  hive.spark.client.connect.timeout is the timeout when the spark remote 
> driver makes a socket connection (channel) to RPC server. But currently it is 
> also used by the remote driver for RPC client/server handshaking, which is 
> not right. Instead, hive.spark.client.server.connect.timeout should be used 
> and it has already been used by the RPCServer in the handshaking.
> The error like following is usually caused by this issue, since the default 
> hive.spark.client.connect.timeout value (1000ms) used by remote driver for 
> handshaking is a little too short.
> {code}
> 17/02/20 08:46:08 ERROR yarn.ApplicationMaster: User class threw exception: 
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
> at 
> org.apache.hive.spark.client.RemoteDriver.(RemoteDriver.java:156)
> at 
> org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
> Caused by: javax.security.sasl.SaslException: Client closed before SASL 
> negotiation finished.
> at 
> org.apache.hive.spark.client.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:453)
> at 
> org.apache.hive.spark.client.rpc.SaslHandler.channelInactive(SaslHandler.java:90)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16091) Support subqueries in project/select

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-16091:
---
Attachment: HIVE-16091.1.patch

> Support subqueries in project/select
> 
>
> Key: HIVE-16091
> URL: https://issues.apache.org/jira/browse/HIVE-16091
> Project: Hive
>  Issue Type: Sub-task
>  Components: Logical Optimizer
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-16091.1.patch
>
>
> Currently scalar subqueries are supported in filter only (WHERE/HAVING). 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HIVE-16150) LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory pressure

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran resolved HIVE-16150.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

Committed to master. Thanks Rajesh for the review!

> LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory 
> pressure
> ---
>
> Key: HIVE-16150
> URL: https://issues.apache.org/jira/browse/HIVE-16150
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Fix For: 2.2.0
>
> Attachments: HIVE-16150.1.patch, Screen Shot 2017-02-22 at 2.42.12 
> PM.png
>
>
> Easiest: following needs to be fixed.
> LOG.debug("Found spec for " + hsplit.getPath() + " " + part + " from " + 
> pathToPartitionInfo);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16150) LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory pressure

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902432#comment-15902432
 ] 

Prasanth Jayachandran commented on HIVE-16150:
--

Added the isInfoEnabled check wherever it was missing. Some places had it and 
some places didn't. Will commit the patch shortly.

> LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory 
> pressure
> ---
>
> Key: HIVE-16150
> URL: https://issues.apache.org/jira/browse/HIVE-16150
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16150.1.patch, Screen Shot 2017-02-22 at 2.42.12 
> PM.png
>
>
> Easiest: following needs to be fixed.
> LOG.debug("Found spec for " + hsplit.getPath() + " " + part + " from " + 
> pathToPartitionInfo);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16103) LLAP: Scheduler timeout monitor never stops with slot nodes

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902430#comment-15902430
 ] 

Prasanth Jayachandran commented on HIVE-16103:
--

Scheduled another ptest run. Not sure if the failure is related.

> LLAP: Scheduler timeout monitor never stops with slot nodes
> ---
>
> Key: HIVE-16103
> URL: https://issues.apache.org/jira/browse/HIVE-16103
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16103.1.patch, HIVE-16103.2.patch, 
> HIVE-16103.3.patch
>
>
> The scheduler timeout monitor is started when node count becomes 0 and 
> stopped when node count becomes 1. For node count, we were relying on the 
> paths under llap namespace. With addition of slot znodes, every node creates 
> 2 paths (worker and slot). As a result, the size of the instances cache will 
> never be 1 (always multiple of 2) which leads to condition where timeout 
> monitor is not stopped. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15407) add distcp to classpath by default, because hive depends on it.

[Fei Hui (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902427#comment-15902427
 ] 

Fei Hui commented on HIVE-15407:


[~sershe] could you please take a look? should we add distcp to hive classpath?

> add distcp to classpath by default, because hive depends on it. 
> 
>
> Key: HIVE-15407
> URL: https://issues.apache.org/jira/browse/HIVE-15407
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, CLI
>Affects Versions: 2.2.0
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-15407.1.patch
>
>
> when i run hive queries, i get errors as follow
> java.lang.NoClassDefFoundError: org/apache/hadoop/tools/DistCpOptions
> ...
> I dig into code, and find that hive depends on distcp ,but distcp is not in 
> classpath by default.
> I think if adding distcp to hadoop classpath by default in hadoop project, 
> but hadoop committers will not do that. discussions in HADOOP-13865 . They 
> propose that Resolving this problem on HIVE
> So i add distcp to classpath on HIVE



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16071) Spark remote driver misuses the timeout in RPC handshake

[Rui Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902425#comment-15902425
 ] 

Rui Li commented on HIVE-16071:
---

Hi [~xuefuz], your understanding is inline with mine. Not sure if 
SaslHandler.onError has to close the channel. If it's called via 
channelInactive, it means the channel is already closed. If it's called via 
exceptionCaught, the method itself will close the channel.
One improvement may be to call onError unconditionally in channelInactive - 
current code depends on dispose to call onError:
{code}
  @Override
  public void channelInactive(ChannelHandlerContext ctx) throws Exception {
dispose();
super.channelInactive(ctx);
  }
{code}

> Spark remote driver misuses the timeout in RPC handshake
> 
>
> Key: HIVE-16071
> URL: https://issues.apache.org/jira/browse/HIVE-16071
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-16071.patch
>
>
> Based on its property description in HiveConf and the comments in HIVE-12650 
> (https://issues.apache.org/jira/browse/HIVE-12650?focusedCommentId=15128979=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15128979),
>  hive.spark.client.connect.timeout is the timeout when the spark remote 
> driver makes a socket connection (channel) to RPC server. But currently it is 
> also used by the remote driver for RPC client/server handshaking, which is 
> not right. Instead, hive.spark.client.server.connect.timeout should be used 
> and it has already been used by the RPCServer in the handshaking.
> The error like following is usually caused by this issue, since the default 
> hive.spark.client.connect.timeout value (1000ms) used by remote driver for 
> handshaking is a little too short.
> {code}
> 17/02/20 08:46:08 ERROR yarn.ApplicationMaster: User class threw exception: 
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: 
> Client closed before SASL negotiation finished.
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37)
> at 
> org.apache.hive.spark.client.RemoteDriver.(RemoteDriver.java:156)
> at 
> org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:556)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
> Caused by: javax.security.sasl.SaslException: Client closed before SASL 
> negotiation finished.
> at 
> org.apache.hive.spark.client.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:453)
> at 
> org.apache.hive.spark.client.rpc.SaslHandler.channelInactive(SaslHandler.java:90)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14804) HPLSQL multiple db connection does not switch back to Hive

[Fei Hui (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902422#comment-15902422
 ] 

Fei Hui commented on HIVE-14804:


[~alangates] thanks for review and commit

> HPLSQL multiple db connection does not switch back to Hive
> --
>
> Key: HIVE-14804
> URL: https://issues.apache.org/jira/browse/HIVE-14804
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Reporter: Dmitry Kozlov
>Assignee: Fei Hui
>Priority: Blocker
> Fix For: 2.0.2, 2.1.2
>
> Attachments: HIVE-14804.1-branch-2.0.patch, 
> HIVE-14804.1-branch-2.1.patch, HIVE-14804.2-branch-2.0.patch, 
> HIVE-14804.2-branch-2.1.patch
>
>
> I have a problem with a multi database connection. I have 3 environments that 
> I would like to connect in my HPLSQL code Hive, DB2 and MySql. As soon as I 
> map any table either from DB2 or MySQL my code stops to recognize Hive 
> tables. Actually it starts to think that it is a table from the same database 
> (DB2 or MySql) that was mapped the last. It means your example 
> http://www.hplsql.org/map-object works only one way from Hive to MySQL and it 
> is not possible to go back to Hive.  
> Here is a simple piece of code.
> declare cnt int;
> begin
> /*
> PRINT 'Start MySQL';
> MAP OBJECT tbls TO hive.TBLS AT mysqlconn;
> select count(*)
> into cnt
> from tbls;
> PRINT cnt;
> PRINT 'Start Db2';
> MAP OBJECT exch TO DBDEV2.TEST_EXCHANGE AT db2conn;
> select count(1) 
> into cnt
> from exch;
> PRINT cnt;*/
> PRINT 'Check Hive';
> SELECT count(1) 
> into cnt
> FROM dev.test_sqoop;
> PRINT cnt;
> end;
> It has three blocks. One select from MySQL, second from DB2 and third from 
> Hive ORC table.
> When first two blocks are commented then block 3 works. See below
> Check Hive
> 16/09/20 18:08:08 INFO jdbc.Utils: Supplied authorities: localhost:1
> 16/09/20 18:08:08 INFO jdbc.Utils: Resolved authority: localhost:1
> 16/09/20 18:08:08 INFO jdbc.HiveConnection: Will try to open client transport 
> with JDBC Uri: jdbc:hive2://localhost:1
> Open connection: jdbc:hive2://localhost:1 (497 ms)
> Starting query
> Query executed successfully (177 ms)
> 82
> When I try to uncomment any of those blocks then block 3 stops working. For 
> example, if I uncomment block 1 I get this output. It is now assumes that 
> dev.test_sqoop is a MySQL table. Contrarily to your example
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (489 ms)
> Starting query
> Query executed successfully (4 ms)
> 539
> Check Hive
> Starting query
> Unhandled exception in HPL/SQL
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'dev.test_sqoop' doesn't exist
> If I comment the second block then it starts to assume that dev.test_sqoop is 
> a DB2 table. See below. So switch between DB2 and MySQL is working, however, 
> the hive table is still not working
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (485 ms)
> Starting query
> Query executed successfully (5 ms)
> 539
> Start Db2
> Open connection: jdbc:db2://10.11.12.141:5/WM (227 ms)
> Starting query
> Query executed successfully (48 ms)
> 0
> Check Hive
> Starting query
> Unhandled exception in HPL/SQL
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, 
> SQLSTATE=42704, SQLERRMC=DEV.TEST_SQOOP, DRIVER=4.16.53
> Could you, please, provide your feedback on this finding. In addition, I 
> would like to check if it would be possible to insert into a DB2 table 
> records that were selected from a Hive with one statement as soon as DB2 
> table is properly mapped. Please, explain.
> Looking forward to hearing from you soon.
> Regards,
> Dmitry Kozlov
> Daisy Intelligence   



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-1626) stop using java.util.Stack

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902419#comment-15902419
 ] 

Hive QA commented on HIVE-1626:
---



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856829/HIVE-1626.3.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4036/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4036/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4036/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-09 03:30:39.148
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4036/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 03:30:39.151
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 348a592 HIVE-16114 : NullPointerException in 
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey 
Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 348a592 HIVE-16114 : NullPointerException in 
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey 
Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 03:30:40.027
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java:37
error: 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java:
 patch does not apply
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java:20
error: ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java: 
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856829 - PreCommit-HIVE-Build

> stop using java.util.Stack
> --
>
> Key: HIVE-1626
> URL: https://issues.apache.org/jira/browse/HIVE-1626
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.7.0
>Reporter: John Sichi
>Assignee: Teddy Choi
> Attachments: HIVE-1626.2.patch, HIVE-1626.2.patch, HIVE-1626.3.patch, 
> HIVE-1626.3.patch, HIVE-1626.3.patch
>
>
> We currently use Stack as part of the generic node walking library.  Stack 
> should not be used for this since its inheritance from Vector incurs 
> superfluous synchronization overhead.
> Most projects end up adding an ArrayStack implementation and using that 
> instead.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902418#comment-15902418
 ] 

Siddharth Seth commented on HIVE-16133:
---

The old default was 10K entries. Do we know how much memory that would take, on 
average anyway. Don't want to end up with the 500MB default being larger than 
what 10K records would typically take.

> Footer cache in Tez AM can take too much memory
> ---
>
> Key: HIVE-16133
> URL: https://issues.apache.org/jira/browse/HIVE-16133
> Project: Hive
>  Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16133.01.patch, HIVE-16133.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16127) Separate database initialization from actual query run in TestBeeLineDriver

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902415#comment-15902415
 ] 

Hive QA commented on HIVE-16127:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856822/HIVE-16127.02.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4035/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4035/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4035/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-09 03:29:25.890
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4035/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 03:29:25.893
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 348a592 HIVE-16114 : NullPointerException in 
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey 
Shelukhin)
+ git clean -f -d
Removing 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectFilterTransposeRule.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 348a592 HIVE-16114 : NullPointerException in 
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey 
Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 03:29:26.841
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
beeline/src/java/org/apache/hive/beeline/util/QFileClient.java:1
error: beeline/src/java/org/apache/hive/beeline/util/QFileClient.java: patch 
does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856822 - PreCommit-HIVE-Build

> Separate database initialization from actual query run in TestBeeLineDriver
> ---
>
> Key: HIVE-16127
> URL: https://issues.apache.org/jira/browse/HIVE-16127
> Project: Hive
>  Issue Type: Improvement
>  Components: Testing Infrastructure
>Affects Versions: 2.2.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-16127.01.patch, HIVE-16127.02.patch
>
>
> Improve the TestBeeLineDriver, so when running multiple tests, then reuse the 
> default database for multiple runs. This helps to keep the runtimes in check.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15619) Column pruner should handle DruidQuery

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902408#comment-15902408
 ] 

Hive QA commented on HIVE-15619:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856817/HIVE-15619.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 204 failed/errored test(s), 10335 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_excludeHadoop20] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[archive_multi] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join9] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrotblsjoin] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1] 
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2] 
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3] 
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
 (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
 (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_const] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantPropagateForSubQuery]
 (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_3] 
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog2] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog3] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_partitioner] 
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_5] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_1] (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cte_mat_2] (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[deleteAnalyze] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_rdd_cache] 
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_logical] 
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_join_breaktask] 
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_case] (batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fouter_join_ppr] 
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_sets6] 
(batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_map_ppr] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_map_ppr_multi_distinct]
 (batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr] (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr_multi_distinct]
 (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_1_23] 
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_6] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_skew_1_23] 
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[having2] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input23] (batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part1] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join28] (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join32] (batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join33] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join35] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join42] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join9] (batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_1] 
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_3] 
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual1]
 (batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_cond_pushdown_unqual3]
 (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_filters_overlap] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_view] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_14] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_1] 
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_2] 
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_3] 
(batchId=13)

[jira] [Updated] (HIVE-15556) Replicate views

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-15556:

Attachment: HIVE-15556.02.patch

> Replicate views
> ---
>
> Key: HIVE-15556
> URL: https://issues.apache.org/jira/browse/HIVE-15556
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Vaibhav Gumashta
>Assignee: Sankar Hariappan
> Attachments: HIVE-15556.01.patch, HIVE-15556.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-15556) Replicate views

[Sankar Hariappan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900944#comment-15900944
 ] 

Sankar Hariappan edited comment on HIVE-15556 at 3/9/17 2:44 AM:
-

Supported bootstrap and incremental replication for both virtual and 
materialized views.
Request [~sushanth], [~thejas], [~anishek] to review the patch 
HIVE-15556.02.patch


was (Author: sankarh):
Supported bootstrap and incremental replication for both virtual and 
materialized views.
Request [~sushanth], [~thejas], [~anishek] to review the patch 
HIVE-15556.01.patch

> Replicate views
> ---
>
> Key: HIVE-15556
> URL: https://issues.apache.org/jira/browse/HIVE-15556
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Vaibhav Gumashta
>Assignee: Sankar Hariappan
> Attachments: HIVE-15556.01.patch, HIVE-15556.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15556) Replicate views

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902378#comment-15902378
 ] 

ASF GitHub Bot commented on HIVE-15556:
---

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/156

HIVE-15556: Replicate Views

Support replication on views

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-15556

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #156


commit b09b777ced1fb23006cdc76d0348a57f083b5a7b
Author: Sankar Hariappan 
Date:   2017-03-06T08:23:16Z

HIVE-15556: Replicate Views




> Replicate views
> ---
>
> Key: HIVE-15556
> URL: https://issues.apache.org/jira/browse/HIVE-15556
> Project: Hive
>  Issue Type: Sub-task
>  Components: repl
>Reporter: Vaibhav Gumashta
>Assignee: Sankar Hariappan
> Attachments: HIVE-15556.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15988) LLAP: cache ORC ACID delete deltas

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902359#comment-15902359
 ] 

Hive QA commented on HIVE-15988:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856810/HIVE-15988.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4033/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4033/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4033/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:24:44.545
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4033/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:24:44.548
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 348a592 HIVE-16114 : NullPointerException in 
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey 
Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 348a592 HIVE-16114 : NullPointerException in 
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey 
Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:24:45.439
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: ql/src/test/results/clientpositive/perf/query14.q.out:1
error: ql/src/test/results/clientpositive/perf/query14.q.out: patch does not 
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856810 - PreCommit-HIVE-Build

> LLAP: cache ORC ACID delete deltas
> --
>
> Key: HIVE-15988
> URL: https://issues.apache.org/jira/browse/HIVE-15988
> Project: Hive
>  Issue Type: Improvement
>  Components: llap, Transactions
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Attachments: HIVE-15988.1.patch
>
>
> HIVE-12631 implemented ORC ACID table support in LLAP. If we cache ACID 
> delete delta in LLAP, it will be faster.
> The cache is currently low-level (CB-level in ORC), so we could just use it 
> to read bases and deltas (deltas should be cached with higher priority) and 
> merge as usual. We could also cache merged representation in future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15988) LLAP: cache ORC ACID delete deltas

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902356#comment-15902356
 ] 

Hive QA commented on HIVE-15988:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856810/HIVE-15988.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4032/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4032/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4032/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:23:28.426
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4032/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:23:28.429
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   bfe930c..348a592  master -> origin/master
+ git reset --hard HEAD
HEAD is now at bfe930c HIVE-16072: LLAP: Add some additional jvm metrics for 
hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 348a592 HIVE-16114 : NullPointerException in 
TezSessionPoolManager when getting the session (Zhihua Deng, reviewed by Sergey 
Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 02:23:29.671
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: ql/src/test/results/clientpositive/perf/query14.q.out:1
error: ql/src/test/results/clientpositive/perf/query14.q.out: patch does not 
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856810 - PreCommit-HIVE-Build

> LLAP: cache ORC ACID delete deltas
> --
>
> Key: HIVE-15988
> URL: https://issues.apache.org/jira/browse/HIVE-15988
> Project: Hive
>  Issue Type: Improvement
>  Components: llap, Transactions
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Attachments: HIVE-15988.1.patch
>
>
> HIVE-12631 implemented ORC ACID table support in LLAP. If we cache ACID 
> delete delta in LLAP, it will be faster.
> The cache is currently low-level (CB-level in ORC), so we could just use it 
> to read bases and deltas (deltas should be cached with higher priority) and 
> merge as usual. We could also cache merged representation in future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16119) HiveMetaStoreChecker - singleThread/parallel logic duplication

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902346#comment-15902346
 ] 

Hive QA commented on HIVE-16119:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856809/HIVE-16119.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10335 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=141)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4031/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4031/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4031/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856809 - PreCommit-HIVE-Build

> HiveMetaStoreChecker - singleThread/parallel logic duplication
> --
>
> Key: HIVE-16119
> URL: https://issues.apache.org/jira/browse/HIVE-16119
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-16119.1.patch, HIVE-16119.2.patch, 
> HIVE-16119.3.patch
>
>
> It looks to me that the main logic is duplicated, because of multithereading 
> support:
> * {{HiveMetaStoreChecker#PathDepthInfoCallable#processPathDepthInfo}}
> * {{HiveMetaStoreChecker#checkPartitionDirsSingleThreaded}}
> It might be possible to remove the singleThreaded methods by using a special 
> executor for single thread support: {{MoreExecutors.sameThreadExecutor()}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16104) LLAP: preemption may be too aggressive if the pre-empted task doesn't die immediately

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902341#comment-15902341
 ] 

Siddharth Seth commented on HIVE-16104:
---

Thank you. Still some parts which should not have changed... Anyway. Actual 
review comments this time

Thread.sleep(PREEMPTION_KILL_GRACE_SLEEP_MS); - Think it is possible to replace 
this with a timed wait. Similar to the way the main scheduling loop waits for a 
fragment to complete or get scheduled before it tries to schedule the next one. 
That way we won't actually be sleeping for the entire 100ms.
Another nice to have bit would be to allow the element to go back into the 
queue, and fetch the latest from the queue - which could be at a higher 
priority. Don't think this is critical though.

In TaskRunnerCallable - killTask needs a minor change. shouldRunTask = false 
needs to be set independent of the (taskRunner) condition that it is under. 
Having a fragment wait outside the queue increases the possibility of an issue 
like this being hit. (The nice to have about putting the fragment back in the 
queue and waiting for the next schedule attempt would limit the possibility of 
other such issues, if they exist, as well)

Any chance of adding tests? The test class is setup with some controlled 
scheduling constructs to help with this. Suspect it'll need more work with the 
timed wait though.

System.nanoTime - replace with the Clock instance already used in the class. 
(Helps with unit tests to simulate time changes, may not be plugged in yet).

Nit: lastKillTimeNs - Long(null) vs long (-1/special value for unset)?

Not required.
{code}
// TODO: this can all be replaced by a Thread with a Runnable and a catch block
{code}

> LLAP: preemption may be too aggressive if the pre-empted task doesn't die 
> immediately
> -
>
> Key: HIVE-16104
> URL: https://issues.apache.org/jira/browse/HIVE-16104
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16104.01.patch, HIVE-16104.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-16104) LLAP: preemption may be too aggressive if the pre-empted task doesn't die immediately

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902341#comment-15902341
 ] 

Siddharth Seth edited comment on HIVE-16104 at 3/9/17 2:10 AM:
---

Thank you. Still some parts which should not have changed... Anyway. Actual 
review comments this time

- Thread.sleep(PREEMPTION_KILL_GRACE_SLEEP_MS); - Think it is possible to 
replace this with a timed wait. Similar to the way the main scheduling loop 
waits for a fragment to complete or get scheduled before it tries to schedule 
the next one. That way we won't actually be sleeping for the entire 100ms.
- Another nice to have bit would be to allow the element to go back into the 
queue, and fetch the latest from the queue - which could be at a higher 
priority. Don't think this is critical though.

- In TaskRunnerCallable - killTask needs a minor change. shouldRunTask = false 
needs to be set independent of the (taskRunner) condition that it is under. 
Having a fragment wait outside the queue increases the possibility of an issue 
like this being hit. (The nice to have about putting the fragment back in the 
queue and waiting for the next schedule attempt would limit the possibility of 
other such issues, if they exist, as well)

- Any chance of adding tests? The test class is setup with some controlled 
scheduling constructs to help with this. Suspect it'll need more work with the 
timed wait though.

- System.nanoTime - replace with the Clock instance already used in the class. 
(Helps with unit tests to simulate time changes, may not be plugged in yet).

- Nit: lastKillTimeNs - Long(null) vs long (-1/special value for unset)?

- Not required.
{code}
// TODO: this can all be replaced by a Thread with a Runnable and a catch block
{code}


was (Author: sseth):
Thank you. Still some parts which should not have changed... Anyway. Actual 
review comments this time

Thread.sleep(PREEMPTION_KILL_GRACE_SLEEP_MS); - Think it is possible to replace 
this with a timed wait. Similar to the way the main scheduling loop waits for a 
fragment to complete or get scheduled before it tries to schedule the next one. 
That way we won't actually be sleeping for the entire 100ms.
Another nice to have bit would be to allow the element to go back into the 
queue, and fetch the latest from the queue - which could be at a higher 
priority. Don't think this is critical though.

In TaskRunnerCallable - killTask needs a minor change. shouldRunTask = false 
needs to be set independent of the (taskRunner) condition that it is under. 
Having a fragment wait outside the queue increases the possibility of an issue 
like this being hit. (The nice to have about putting the fragment back in the 
queue and waiting for the next schedule attempt would limit the possibility of 
other such issues, if they exist, as well)

Any chance of adding tests? The test class is setup with some controlled 
scheduling constructs to help with this. Suspect it'll need more work with the 
timed wait though.

System.nanoTime - replace with the Clock instance already used in the class. 
(Helps with unit tests to simulate time changes, may not be plugged in yet).

Nit: lastKillTimeNs - Long(null) vs long (-1/special value for unset)?

Not required.
{code}
// TODO: this can all be replaced by a Thread with a Runnable and a catch block
{code}

> LLAP: preemption may be too aggressive if the pre-empted task doesn't die 
> immediately
> -
>
> Key: HIVE-16104
> URL: https://issues.apache.org/jira/browse/HIVE-16104
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16104.01.patch, HIVE-16104.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16114:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the patch!

> NullPointerException in TezSessionPoolManager when getting the session
> --
>
> Key: HIVE-16114
> URL: https://issues.apache.org/jira/browse/HIVE-16114
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-16114.1.patch, HIVE-16114.2.patch, HIVE-16114.patch
>
>
> hive version: apache-hive-2.1.1 
> we use hue(3.11.0) connecting to the HiveServer2.  when hue starts up, it 
> works with no problems, a few hours passed, when we use the same sql, an 
> exception about unable to execute TezTask will come into being.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15468) Enhance the vectorized execution engine to support complex types

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902284#comment-15902284
 ] 

Sergey Shelukhin commented on HIVE-15468:
-

It doesn't vectorize the pipeline... that I guess is the part that is not 
supported. I was just clarifying :) 

> Enhance the vectorized execution engine to support complex types
> 
>
> Key: HIVE-15468
> URL: https://issues.apache.org/jira/browse/HIVE-15468
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Chao Sun
>Assignee: Teddy Choi
>
> Currently Hive's vectorized execution engine only supports scalar types, as 
> documented here: 
> https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution.
> To be complete, we should add support for complex types as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-15468) Enhance the vectorized execution engine to support complex types

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902284#comment-15902284
 ] 

Sergey Shelukhin edited comment on HIVE-15468 at 3/9/17 1:31 AM:
-

It doesn't vectorize the pipeline... that is the part that is not supported. 
ORC vectorized read is translated to rows. I was just clarifying :) 


was (Author: sershe):
It doesn't vectorize the pipeline... that I guess is the part that is not 
supported. I was just clarifying :) 

> Enhance the vectorized execution engine to support complex types
> 
>
> Key: HIVE-15468
> URL: https://issues.apache.org/jira/browse/HIVE-15468
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Chao Sun
>Assignee: Teddy Choi
>
> Currently Hive's vectorized execution engine only supports scalar types, as 
> documented here: 
> https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution.
> To be complete, we should add support for complex types as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16151) BytesBytesHashTable allocates large arrays

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16151:

Attachment: HIVE-16151.patch

The patch. [~mmccline] can you take a look?

> BytesBytesHashTable allocates large arrays
> --
>
> Key: HIVE-16151
> URL: https://issues.apache.org/jira/browse/HIVE-16151
> Project: Hive
>  Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16151.patch
>
>
> These arrays cause GC pressure and also impose key count limitations on the 
> table. Wrt the latter, we won't be able to get rid of it without a 64-bit 
> hash function, but for now we can get rid of the former. If we need the 
> latter we'd add murmur64 and probably account for it differently for resize 
> (we don't want to blow up the hashtable by 4 bytes/key in the common case 
> where #of keys is less than ~1.5B :))



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16151) BytesBytesHashTable allocates large arrays

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16151:

Status: Patch Available  (was: Open)

> BytesBytesHashTable allocates large arrays
> --
>
> Key: HIVE-16151
> URL: https://issues.apache.org/jira/browse/HIVE-16151
> Project: Hive
>  Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16151.patch
>
>
> These arrays cause GC pressure and also impose key count limitations on the 
> table. Wrt the latter, we won't be able to get rid of it without a 64-bit 
> hash function, but for now we can get rid of the former. If we need the 
> latter we'd add murmur64 and probably account for it differently for resize 
> (we don't want to blow up the hashtable by 4 bytes/key in the common case 
> where #of keys is less than ~1.5B :))



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-12631) LLAP: support ORC ACID tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902268#comment-15902268
 ] 

Hive QA commented on HIVE-12631:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856808/HIVE-12631.6.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4030/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4030/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4030/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-09 01:19:18.376
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4030/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 01:19:18.378
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at bfe930c HIVE-16072: LLAP: Add some additional jvm metrics for 
hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
Removing ql/src/test/queries/clientnegative/avro_add_column_extschema.q
Removing ql/src/test/queries/clientpositive/avro_add_column_extschema.q
Removing ql/src/test/results/clientnegative/avro_add_column_extschema.q.out
Removing ql/src/test/results/clientpositive/avro_add_column_extschema.q.out
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at bfe930c HIVE-16072: LLAP: Add some additional jvm metrics for 
hadoop-metrics2 (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-09 01:19:19.309
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: ql/src/test/results/clientpositive/perf/query14.q.out:1
error: ql/src/test/results/clientpositive/perf/query14.q.out: patch does not 
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856808 - PreCommit-HIVE-Build

> LLAP: support ORC ACID tables
> -
>
> Key: HIVE-12631
> URL: https://issues.apache.org/jira/browse/HIVE-12631
> Project: Hive
>  Issue Type: Bug
>  Components: llap, Transactions
>Reporter: Sergey Shelukhin
>Assignee: Teddy Choi
> Attachments: HIVE-12631.1.patch, HIVE-12631.2.patch, 
> HIVE-12631.3.patch, HIVE-12631.4.patch, HIVE-12631.5.patch, HIVE-12631.6.patch
>
>
> LLAP uses a completely separate read path in ORC to allow for caching and 
> parallelization of reads and processing. This path does not support ACID. As 
> far as I remember ACID logic is embedded inside ORC format; we need to 
> refactor it to be on top of some interface, if practical; or just port it to 
> LLAP read path.
> Another consideration is how the logic will work with cache. The cache is 
> currently low-level (CB-level in ORC), so we could just use it to read bases 
> and deltas (deltas should be cached with higher priority) and merge as usual. 
> We could also cache merged representation in future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-13780) Allow user to update AVRO table schema via command even if table's definition was defined through schema file

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902266#comment-15902266
 ] 

Hive QA commented on HIVE-13780:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856802/HIVE-13780.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10337 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4029/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4029/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4029/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856802 - PreCommit-HIVE-Build

> Allow user to update AVRO table schema via command even if table's definition 
> was defined through schema file
> -
>
> Key: HIVE-13780
> URL: https://issues.apache.org/jira/browse/HIVE-13780
> Project: Hive
>  Issue Type: Improvement
>  Components: CLI
>Affects Versions: 2.0.0
>Reporter: Eric Lin
>Assignee: Adam Szita
>Priority: Minor
> Attachments: HIVE-13780.0.patch, HIVE-13780.1.patch, 
> HIVE-13780.3.patch
>
>
> If a table is defined as below:
> {code}
> CREATE TABLE test
> STORED AS AVRO 
> TBLPROPERTIES ('avro.schema.url'='/tmp/schema.json');
> {code}
> if user tries to run command:
> {code}
> ALTER TABLE test CHANGE COLUMN col1 col1 STRING COMMENT 'test comment';
> {code}
> The query will return without any warning, but has no affect to the table.
> It would be good if we can allow user to ALTER table (add/change column, 
> update comment etc) even though the schema is defined through schema file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16135) Vectorization: unhandled constant type for scalar argument

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902240#comment-15902240
 ] 

Matt McCline commented on HIVE-16135:
-

+1 LGTM

> Vectorization: unhandled constant type for scalar argument
> --
>
> Key: HIVE-16135
> URL: https://issues.apache.org/jira/browse/HIVE-16135
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Attachments: HIVE-16135.1.patch, HIVE-16135.2.patch, 
> HIVE-16135.3.patch
>
>
> Map: Predicate expression for FILTER operator: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Udf: unhandled constant 
> type for scalar argument. Expecting string.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14550) HiveServer2: enable ThriftJDBCBinarySerde use by default

[Ziyang Zhao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902239#comment-15902239
 ] 

Ziyang Zhao commented on HIVE-14550:


org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
 (batchId=147)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=224)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in] 
(batchId=119)

Above three test cases are common failures among all recent tests.

org.apache.hive.jdbc.TestJdbcDriver2.testDescribeTable (batchId=216)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=216)
org.apache.hive.jdbc.TestJdbcDriver2.testShowGrant (batchId=216)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testEscapedStrings (batchId=218)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd 
(batchId=218)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testNonAsciiStrings (batchId=218)

I have verified the above six tests locally, and all passed.

> HiveServer2: enable ThriftJDBCBinarySerde use by default
> 
>
> Key: HIVE-14550
> URL: https://issues.apache.org/jira/browse/HIVE-14550
> Project: Hive
>  Issue Type: Sub-task
>  Components: HiveServer2, JDBC, ODBC
>Affects Versions: 2.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Ziyang Zhao
> Attachments: HIVE-14550.1.patch, HIVE-14550.1.patch, 
> HIVE-14550.2.patch
>
>
> We've covered all items in HIVE-12427 and created HIVE-14549 for part2 of the 
> effort. Before closing the umbrella jira, we should enable this feature by 
> default.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15988) LLAP: cache ORC ACID delete deltas

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-15988:
--
Status: Open  (was: Patch Available)

> LLAP: cache ORC ACID delete deltas
> --
>
> Key: HIVE-15988
> URL: https://issues.apache.org/jira/browse/HIVE-15988
> Project: Hive
>  Issue Type: Improvement
>  Components: llap, Transactions
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Attachments: HIVE-15988.1.patch
>
>
> HIVE-12631 implemented ORC ACID table support in LLAP. If we cache ACID 
> delete delta in LLAP, it will be faster.
> The cache is currently low-level (CB-level in ORC), so we could just use it 
> to read bases and deltas (deltas should be cached with higher priority) and 
> merge as usual. We could also cache merged representation in future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16148) Flaky test: schema_evol_text_vec_table

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-16148:
--
Status: Patch Available  (was: Open)

> Flaky test: schema_evol_text_vec_table
> --
>
> Key: HIVE-16148
> URL: https://issues.apache.org/jira/browse/HIVE-16148
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-16148.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-15468) Enhance the vectorized execution engine to support complex types

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902226#comment-15902226
 ] 

Teddy Choi edited comment on HIVE-15468 at 3/9/17 12:33 AM:


[~sershe], thank you. But the following message shows that it's not vectorized 
because of Select expression for SELECT operator: struct is not supported. I 
guess that it supports reading but not vectorization now.

{noformat}
POSTHOOK: query: EXPLAIN VECTORIZATION EXPRESSION
SELECT strct, mp, lst FROM orc_create_complex
POSTHOOK: type: QUERY
Explain
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
 A masked pattern was here 
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: orc_create_complex
  Statistics: Num rows: 3 Data size: 3432 Basic stats: COMPLETE 
Column stats: NONE
  Select Operator
expressions: strct (type: struct), mp 
(type: map), lst (type: array)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: llap
LLAP IO: all inputs
Map Vectorization:
enabled: true
enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
notVectorizedReason: Select expression for SELECT operator: 
Data type struct of Column[strct] not supported
vectorized: false

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink
{noformat}


was (Author: teddy.choi):
[~sershe], thank you. But the following message doesn't match to your 
explanation. It's not vectorized because of Select expression for SELECT 
operator: struct is not supported. Do you have some idea?

{noformat}
POSTHOOK: query: EXPLAIN VECTORIZATION EXPRESSION
SELECT strct, mp, lst FROM orc_create_complex
POSTHOOK: type: QUERY
Explain
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
 A masked pattern was here 
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: orc_create_complex
  Statistics: Num rows: 3 Data size: 3432 Basic stats: COMPLETE 
Column stats: NONE
  Select Operator
expressions: strct (type: struct), mp 
(type: map), lst (type: array)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: llap
LLAP IO: all inputs
Map Vectorization:
enabled: true
enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
notVectorizedReason: Select expression for SELECT operator: 
Data type struct of Column[strct] not supported
vectorized: false

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink
{noformat}

> Enhance the vectorized execution engine to support complex types
> 
>
> Key: HIVE-15468
> URL: 

[jira] [Comment Edited] (HIVE-15468) Enhance the vectorized execution engine to support complex types

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902226#comment-15902226
 ] 

Teddy Choi edited comment on HIVE-15468 at 3/9/17 12:30 AM:


[~sershe], thank you. But the following message doesn't match to your 
explanation. It's not vectorized because of Select expression for SELECT 
operator: struct is not supported. Do you have some idea?

{noformat}
POSTHOOK: query: EXPLAIN VECTORIZATION EXPRESSION
SELECT strct, mp, lst FROM orc_create_complex
POSTHOOK: type: QUERY
Explain
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
 A masked pattern was here 
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: orc_create_complex
  Statistics: Num rows: 3 Data size: 3432 Basic stats: COMPLETE 
Column stats: NONE
  Select Operator
expressions: strct (type: struct), mp 
(type: map), lst (type: array)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: llap
LLAP IO: all inputs
Map Vectorization:
enabled: true
enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
notVectorizedReason: Select expression for SELECT operator: 
Data type struct of Column[strct] not supported
vectorized: false

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink
{noformat}


was (Author: teddy.choi):
[~sershe], thank you. But the following message doesn't match to your 
explanation. It's not vectorized because of Select expression for SELECT 
operator: struct is not supported. Do you have some idea?

{noformat}
Explain
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
 A masked pattern was here 
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: orc_create_complex
  Statistics: Num rows: 3 Data size: 3432 Basic stats: COMPLETE 
Column stats: NONE
  Select Operator
expressions: strct (type: struct), mp 
(type: map), lst (type: array)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: llap
LLAP IO: all inputs
Map Vectorization:
enabled: true
enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
notVectorizedReason: Select expression for SELECT operator: 
Data type struct of Column[strct] not supported
vectorized: false

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink
{noformat}

> Enhance the vectorized execution engine to support complex types
> 
>
> Key: HIVE-15468
> URL: https://issues.apache.org/jira/browse/HIVE-15468
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>

[jira] [Comment Edited] (HIVE-15468) Enhance the vectorized execution engine to support complex types

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902226#comment-15902226
 ] 

Teddy Choi edited comment on HIVE-15468 at 3/9/17 12:30 AM:


[~sershe], thank you. But the following message doesn't match to your 
explanation. It's not vectorized because of Select expression for SELECT 
operator: struct is not supported. Do you have some idea?

{noformat}
Explain
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
 A masked pattern was here 
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: orc_create_complex
  Statistics: Num rows: 3 Data size: 3432 Basic stats: COMPLETE 
Column stats: NONE
  Select Operator
expressions: strct (type: struct), mp 
(type: map), lst (type: array)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: llap
LLAP IO: all inputs
Map Vectorization:
enabled: true
enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
notVectorizedReason: Select expression for SELECT operator: 
Data type struct of Column[strct] not supported
vectorized: false

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink
{noformat}


was (Author: teddy.choi):
[~sershe], thank you. But the following message doesn't match to your 
explanation. It's not vectorized because of Select expression for SELECT 
operator: struct is not supported. Do you have some idea?

{{noformat}}
Explain
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
 A masked pattern was here 
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: orc_create_complex
  Statistics: Num rows: 3 Data size: 3432 Basic stats: COMPLETE 
Column stats: NONE
  Select Operator
expressions: strct (type: struct), mp 
(type: map), lst (type: array)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: llap
LLAP IO: all inputs
Map Vectorization:
enabled: true
enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
notVectorizedReason: Select expression for SELECT operator: 
Data type struct of Column[strct] not supported
vectorized: false

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink
{{noformat}}

> Enhance the vectorized execution engine to support complex types
> 
>
> Key: HIVE-15468
> URL: https://issues.apache.org/jira/browse/HIVE-15468
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Chao Sun
>Assignee: Teddy Choi
>
> Currently Hive's vectorized execution engine only supports 

[jira] [Commented] (HIVE-15468) Enhance the vectorized execution engine to support complex types

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902226#comment-15902226
 ] 

Teddy Choi commented on HIVE-15468:
---

[~sershe], thank you. But the following message doesn't match to your 
explanation. It's not vectorized because of Select expression for SELECT 
operator: struct is not supported. Do you have some idea?

{{noformat}}
Explain
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
 A masked pattern was here 
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: orc_create_complex
  Statistics: Num rows: 3 Data size: 3432 Basic stats: COMPLETE 
Column stats: NONE
  Select Operator
expressions: strct (type: struct), mp 
(type: map), lst (type: array)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 3 Data size: 3432 Basic stats: 
COMPLETE Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: llap
LLAP IO: all inputs
Map Vectorization:
enabled: true
enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
notVectorizedReason: Select expression for SELECT operator: 
Data type struct of Column[strct] not supported
vectorized: false

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink
{{noformat}}

> Enhance the vectorized execution engine to support complex types
> 
>
> Key: HIVE-15468
> URL: https://issues.apache.org/jira/browse/HIVE-15468
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Chao Sun
>Assignee: Teddy Choi
>
> Currently Hive's vectorized execution engine only supports scalar types, as 
> documented here: 
> https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution.
> To be complete, we should add support for complex types as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16151) BytesBytesHashTable allocates large arrays

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16151:

Description: These arrays cause GC pressure and also impose key count 
limitations on the table. Wrt the latter, we won't be able to get rid of it 
without a 64-bit hash function, but for now we can get rid of the former. If we 
need the latter we'd add murmur64 and probably account for it differently for 
resize (we don't want to blow up the hashtable by 4 bytes/key in the common 
case where #of keys is less than ~1.5B :))  (was: These arrays cause GC 
pressure and also impose key count limitations on the table. Wrt the latter, we 
won't be able to get rid of it without a 64-bit hash function, but for now we 
can get rid of the former.)

> BytesBytesHashTable allocates large arrays
> --
>
> Key: HIVE-16151
> URL: https://issues.apache.org/jira/browse/HIVE-16151
> Project: Hive
>  Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>
> These arrays cause GC pressure and also impose key count limitations on the 
> table. Wrt the latter, we won't be able to get rid of it without a 64-bit 
> hash function, but for now we can get rid of the former. If we need the 
> latter we'd add murmur64 and probably account for it differently for resize 
> (we don't want to blow up the hashtable by 4 bytes/key in the common case 
> where #of keys is less than ~1.5B :))



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16151) BytesBytesHashTable allocates large arrays

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-16151:
---


> BytesBytesHashTable allocates large arrays
> --
>
> Key: HIVE-16151
> URL: https://issues.apache.org/jira/browse/HIVE-16151
> Project: Hive
>  Issue Type: Bug
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
>
> These arrays cause GC pressure and also impose key count limitations on the 
> table. Wrt the latter, we won't be able to get rid of it without a 64-bit 
> hash function, but for now we can get rid of the former.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16076) LLAP packaging - include aux libs

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902212#comment-15902212
 ] 

Prasanth Jayachandran commented on HIVE-16076:
--

Utilities.getResourceFiles(conf, SessionState.ResourceType.JAR) will likely be 
a no-op since we won't have added jars in session when llap service driver is 
invoked. I think this can be removed on commit.

+1 otherwise

> LLAP packaging - include aux libs 
> --
>
> Key: HIVE-16076
> URL: https://issues.apache.org/jira/browse/HIVE-16076
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16076.01.patch, HIVE-16076.02.patch, 
> HIVE-16076.patch
>
>
> The old auxlibs (or whatever) should be packaged by default, if present.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16135) Vectorization: unhandled constant type for scalar argument

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902210#comment-15902210
 ] 

Hive QA commented on HIVE-16135:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856791/HIVE-16135.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10335 tests passed

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4028/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4028/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4028/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856791 - PreCommit-HIVE-Build

> Vectorization: unhandled constant type for scalar argument
> --
>
> Key: HIVE-16135
> URL: https://issues.apache.org/jira/browse/HIVE-16135
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Teddy Choi
>Assignee: Teddy Choi
> Attachments: HIVE-16135.1.patch, HIVE-16135.2.patch, 
> HIVE-16135.3.patch
>
>
> Map: Predicate expression for FILTER operator: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Udf: unhandled constant 
> type for scalar argument. Expecting string.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15468) Enhance the vectorized execution engine to support complex types

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902211#comment-15902211
 ] 

Sergey Shelukhin commented on HIVE-15468:
-

LLAP has support for complex types via the vector-to-row converter right now... 
so the reading of vectors for complex types is supported (outside of LLAP too, 
iirc) and then they get converted back to rows.

> Enhance the vectorized execution engine to support complex types
> 
>
> Key: HIVE-15468
> URL: https://issues.apache.org/jira/browse/HIVE-15468
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Chao Sun
>Assignee: Teddy Choi
>
> Currently Hive's vectorized execution engine only supports scalar types, as 
> documented here: 
> https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution.
> To be complete, we should add support for complex types as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15468) Enhance the vectorized execution engine to support complex types

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902209#comment-15902209
 ] 

Teddy Choi commented on HIVE-15468:
---

When I try llap/vector_complex_all.q with EXPLAIN VECTORIZATION EXPRESSION, I 
get following messages.
{noformat}
notVectorizedReason: Select expression for SELECT operator: Data type 
array of Column[lst] not supported
notVectorizedReason: Key expression for GROUPBY operator: Data type 
array of Column[KEY._col0] not supported
notVectorizedReason: Select expression for SELECT operator: Data type 
struct of Column[strct] not supported
notVectorizedReason: Select expression for SELECT operator: Data type 
map of Column[mp] not supported
notVectorizedReason: Key expression for GROUPBY operator: Data type 
map of Column[KEY._col1] not supported
{noformat}

I will start with SELECT operator support for ARRAY, STRUCT, MAP, then GROUPBY 
for ARRAY and MAP. The target for the first phase is to cover most of TPC-DS 
queries in ql/src/test/queries/clientpositive/perf.

> Enhance the vectorized execution engine to support complex types
> 
>
> Key: HIVE-15468
> URL: https://issues.apache.org/jira/browse/HIVE-15468
> Project: Hive
>  Issue Type: Improvement
>  Components: Vectorization
>Reporter: Chao Sun
>Assignee: Teddy Choi
>
> Currently Hive's vectorized execution engine only supports scalar types, as 
> documented here: 
> https://cwiki.apache.org/confluence/display/Hive/Vectorized+Query+Execution.
> To be complete, we should add support for complex types as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16104) LLAP: preemption may be too aggressive if the pre-empted task doesn't die immediately

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16104:

Attachment: HIVE-16104.01.patch

Restoring the amount of nesting and copy-paste that is expected of Tez-related 
code ;)

> LLAP: preemption may be too aggressive if the pre-empted task doesn't die 
> immediately
> -
>
> Key: HIVE-16104
> URL: https://issues.apache.org/jira/browse/HIVE-16104
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16104.01.patch, HIVE-16104.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HIVE-16149) Druid query path fails when using LLAP mode

[slim bouguerra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra resolved HIVE-16149.
---
Resolution: Invalid

> Druid query path fails when using LLAP mode
> ---
>
> Key: HIVE-16149
> URL: https://issues.apache.org/jira/browse/HIVE-16149
> Project: Hive
>  Issue Type: Sub-task
>  Components: Druid integration
>Reporter: slim bouguerra
>Assignee: Ashutosh Chauhan
>
> {code}
> hive> select i_item_desc ,i_category ,i_class ,i_current_price ,i_item_id 
> ,sum(ss_ext_sales_price)
> > as itemrevenue 
> ,sum(ss_ext_sales_price)*100/sum(sum(ss_ext_sales_price)) over (partition by 
> i_class) as revenueratio
> > from tpcds_store_sales_sold_time_1000_day_all
> > where  (i_category ='Jewelry' or  i_category = 'Sports' or i_category 
> ='Books') and `__time` >= cast('2001-01-12' as date) and `__time` <= 
> cast('2001-02-11' as date)
> > group by i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price 
> order by i_category ,i_class ,i_item_id ,i_item_desc ,revenueratio limit 10;
> Query ID = sbouguerra_20170308131436_225330b7-1142-4e4e-a05a-46ef544c8ee8
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id 
> application_1488231257387_1862)
> --
> VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
> FAILED  KILLED
> --
> Map 1 llapINITED  1  001  
>  0   0
> Reducer 2 llapINITED  2  002  
>  0   0
> Reducer 3 llapINITED  1  001  
>  0   0
> --
> VERTICES: 00/03  [>>--] 0%ELAPSED TIME: 59.68 s
> --
> Status: Failed
> Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.
> Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP 
> Daemons are running
> Vertex killed, vertexName=Reducer 3, vertexId=vertex_1488231257387_1862_3_02, 
> diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not 
> succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex 
> vertex_1488231257387_1862_3_02 [Reducer 3] killed/failed due 
> to:DAG_TERMINATED]
> Vertex killed, vertexName=Reducer 2, vertexId=vertex_1488231257387_1862_3_01, 
> diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not 
> succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:2, Vertex 
> vertex_1488231257387_1862_3_01 [Reducer 2] killed/failed due 
> to:DAG_TERMINATED]
> Vertex killed, vertexName=Map 1, vertexId=vertex_1488231257387_1862_3_00, 
> diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not 
> succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex 
> vertex_1488231257387_1862_3_00 [Map 1] killed/failed due to:DAG_TERMINATED]
> DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 
> killedVertices:3
> FAILED: Execution Error, return code 2 from 
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, 
> SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler 
> [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, 
> vertexName=Reducer 3, vertexId=vertex_1488231257387_1862_3_02, 
> diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not 
> succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex 
> vertex_1488231257387_1862_3_02 [Reducer 3] killed/failed due 
> to:DAG_TERMINATED]Vertex killed, vertexName=Reducer 2, 
> vertexId=vertex_1488231257387_1862_3_01, diagnostics=[Vertex received Kill 
> while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, 
> failedTasks:0 killedTasks:2, Vertex vertex_1488231257387_1862_3_01 [Reducer 
> 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, 
> vertexId=vertex_1488231257387_1862_3_00, diagnostics=[Vertex received Kill 
> while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, 
> failedTasks:0 killedTasks:1, Vertex vertex_1488231257387_1862_3_00 [Map 1] 
> killed/failed due to:DAG_TERMINATED]DAG did not succeed due to 
> SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:3
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16133) Footer cache in Tez AM can take too much memory

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16133:

Attachment: HIVE-16133.01.patch

The same patch for tests... doesn't look like it was in the queue

> Footer cache in Tez AM can take too much memory
> ---
>
> Key: HIVE-16133
> URL: https://issues.apache.org/jira/browse/HIVE-16133
> Project: Hive
>  Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16133.01.patch, HIVE-16133.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15983) Support the named columns join

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15983:
---
Attachment: HIVE-15983.01.patch

> Support the named columns join
> --
>
> Key: HIVE-15983
> URL: https://issues.apache.org/jira/browse/HIVE-15983
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15983.01.patch
>
>
> The named columns join is a common shortcut allowing joins on identically 
> named keys. Example: select * from t1 join t2 using c1 is equivalent to 
> select * from t1 join t2 on t1.c1 = t2.c1. SQL standard reference: Section 7.7



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15983) Support the named columns join

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15983:
---
Status: Patch Available  (was: Open)

> Support the named columns join
> --
>
> Key: HIVE-15983
> URL: https://issues.apache.org/jira/browse/HIVE-15983
> Project: Hive
>  Issue Type: Sub-task
>  Components: SQL
>Reporter: Carter Shanklin
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15983.01.patch
>
>
> The named columns join is a common shortcut allowing joins on identically 
> named keys. Example: select * from t1 join t2 using c1 is equivalent to 
> select * from t1 join t2 on t1.c1 = t2.c1. SQL standard reference: Section 7.7



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16148) Flaky test: schema_evol_text_vec_table

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline reassigned HIVE-16148:
---

Assignee: Teddy Choi  (was: Matt McCline)

> Flaky test: schema_evol_text_vec_table
> --
>
> Key: HIVE-16148
> URL: https://issues.apache.org/jira/browse/HIVE-16148
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-16148.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16148) Flaky test: schema_evol_text_vec_table

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902174#comment-15902174
 ] 

Matt McCline commented on HIVE-16148:
-

And assigned it to you.

> Flaky test: schema_evol_text_vec_table
> --
>
> Key: HIVE-16148
> URL: https://issues.apache.org/jira/browse/HIVE-16148
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Teddy Choi
>Priority: Critical
> Attachments: HIVE-16148.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16148) Flaky test: schema_evol_text_vec_table

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-16148:

Attachment: HIVE-16148.1.patch

> Flaky test: schema_evol_text_vec_table
> --
>
> Key: HIVE-16148
> URL: https://issues.apache.org/jira/browse/HIVE-16148
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16148.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16148) Flaky test: schema_evol_text_vec_table

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902171#comment-15902171
 ] 

Matt McCline commented on HIVE-16148:
-

I attached the patch.

> Flaky test: schema_evol_text_vec_table
> --
>
> Key: HIVE-16148
> URL: https://issues.apache.org/jira/browse/HIVE-16148
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-16148.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15166) Provide beeline option to set the jline history max size

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902157#comment-15902157
 ] 

Hive QA commented on HIVE-15166:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856790/HIVE-15166.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10333 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=141)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
 (batchId=232)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4027/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4027/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4027/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856790 - PreCommit-HIVE-Build

> Provide beeline option to set the jline history max size
> 
>
> Key: HIVE-15166
> URL: https://issues.apache.org/jira/browse/HIVE-15166
> Project: Hive
>  Issue Type: Improvement
>  Components: Beeline
>Affects Versions: 2.1.0
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Minor
> Attachments: HIVE-15166.2.patch, HIVE-15166.3.patch, HIVE-15166.patch
>
>
> Currently Beeline does not provide an option to limit the max size for 
> beeline history file, in the case that each query is very big, it will flood 
> the history file and slow down beeline on start up and shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16148) Flaky test: schema_evol_text_vec_table

[Teddy Choi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902148#comment-15902148
 ] 

Teddy Choi commented on HIVE-16148:
---

[~mmccline], I sent an e-mail with a patch to you. Please attach it. Or may I 
assign it to me then proceed?

> Flaky test: schema_evol_text_vec_table
> --
>
> Key: HIVE-16148
> URL: https://issues.apache.org/jira/browse/HIVE-16148
> Project: Hive
>  Issue Type: Sub-task
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15903) Compute table stats when user computes column stats

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15903:
---
Affects Version/s: 2.1.0
   2.1.1

> Compute table stats when user computes column stats
> ---
>
> Key: HIVE-15903
> URL: https://issues.apache.org/jira/browse/HIVE-15903
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15903.01.patch, HIVE-15903.02.patch, 
> HIVE-15903.03.patch, HIVE-15903.04.patch, HIVE-15903.05.patch, 
> HIVE-15903.06.patch, HIVE-15903.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15903) Compute table stats when user computes column stats

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15903:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Compute table stats when user computes column stats
> ---
>
> Key: HIVE-15903
> URL: https://issues.apache.org/jira/browse/HIVE-15903
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15903.01.patch, HIVE-15903.02.patch, 
> HIVE-15903.03.patch, HIVE-15903.04.patch, HIVE-15903.05.patch, 
> HIVE-15903.06.patch, HIVE-15903.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15903) Compute table stats when user computes column stats

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902134#comment-15902134
 ] 

Pengcheng Xiong commented on HIVE-15903:


pushed to master. Thanks [~ashutoshc] for the review.

> Compute table stats when user computes column stats
> ---
>
> Key: HIVE-15903
> URL: https://issues.apache.org/jira/browse/HIVE-15903
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15903.01.patch, HIVE-15903.02.patch, 
> HIVE-15903.03.patch, HIVE-15903.04.patch, HIVE-15903.05.patch, 
> HIVE-15903.06.patch, HIVE-15903.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15903) Compute table stats when user computes column stats

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15903:
---
Fix Version/s: 2.2.0

> Compute table stats when user computes column stats
> ---
>
> Key: HIVE-15903
> URL: https://issues.apache.org/jira/browse/HIVE-15903
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15903.01.patch, HIVE-15903.02.patch, 
> HIVE-15903.03.patch, HIVE-15903.04.patch, HIVE-15903.05.patch, 
> HIVE-15903.06.patch, HIVE-15903.07.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15849) hplsql should add enterGlobalScope func to UDF

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902126#comment-15902126
 ] 

Alan Gates commented on HIVE-15849:
---

Adding a unit test that reproduces the problem and shows this fixes it would be 
good.

> hplsql should add enterGlobalScope func to UDF
> --
>
> Key: HIVE-15849
> URL: https://issues.apache.org/jira/browse/HIVE-15849
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Affects Versions: 2.2.0
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-15849.patch
>
>
> code in Udf.java
> {code:title=Udf.java|borderStyle=solid}
> if (exec == null) {
>   exec = new Exec();
>   String query = queryOI.getPrimitiveJavaObject(arguments[0].get());
>   String[] args = { "-e", query, "-trace" };
>   try {
> exec.setUdfRun(true);
> exec.init(args);
>   } catch (Exception e) {
> throw new HiveException(e.getMessage());
>   }
> }
> if (arguments.length > 1) {
>   setParameters(arguments);
> }
> Var result = exec.run();
> if (result != null) {
>   return result.toString();
> }
> {code}
> Here is my thoughts
> {quote}
> we should add 'exec.enterGlobalScope(); '  between 'exec = new Exec();' and 
> 'setParameters(arguments);'
> Because if we do not call exec.enterGlobalScope(),  setParameters(arguments) 
> will useless. Vars are not added into scope , but exec.run() will use vars 
> which we set. The vars are parameters passed to UDF, [, :1, :2, ...n] which 
> are description in Udf.java
> {quote}
> Before add this function, the result as follow. we get the wrong result, 
> because the result contains  empty string  
> {quote}
> Starting pre-SQL statement
> Starting pre-SQL statement
> Starting pre-SQL statement
> Starting pre-SQL statement
> Starting pre-SQL statement
> Starting query
> Query executed successfully (2.30 sec)
> Ln:8 SELECT completed successfully
> Ln:8 Standalone SELECT executed: 1 columns in the result set
> Hello, !
> Hello, !
> {quote}
> After add this function, we get the right result
> {quote}
> Starting pre-SQL statement
> Starting pre-SQL statement
> Starting pre-SQL statement
> Starting pre-SQL statement
> Starting pre-SQL statement
> Starting query
> Query executed successfully (2.35 sec)
> Ln:8 SELECT completed successfully
> Ln:8 Standalone SELECT executed: 1 columns in the result set
> Hello, fei!
> Hello, fei!
> {quote}
> tests come from http://www.hplsql.org/udf



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16150) LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory pressure

[Rajesh Balamohan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902119#comment-15902119
 ] 

Rajesh Balamohan commented on HIVE-16150:
-

lgtm. +1. Thanks [~prasanth_j].

"LOG.info("Not using LLAP IO because it is not initialized");" may not be 
wrapped in isInfoEnabled.

> LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory 
> pressure
> ---
>
> Key: HIVE-16150
> URL: https://issues.apache.org/jira/browse/HIVE-16150
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16150.1.patch, Screen Shot 2017-02-22 at 2.42.12 
> PM.png
>
>
> Easiest: following needs to be fixed.
> LOG.debug("Found spec for " + hsplit.getPath() + " " + part + " from " + 
> pathToPartitionInfo);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HIVE-14804) HPLSQL multiple db connection does not switch back to Hive

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates resolved HIVE-14804.
---
   Resolution: Fixed
Fix Version/s: 2.1.2
   2.0.2

Checked patch 2 into branches 2.0 and 2.1.  Thank Fei for the patch and for 
your patience.

> HPLSQL multiple db connection does not switch back to Hive
> --
>
> Key: HIVE-14804
> URL: https://issues.apache.org/jira/browse/HIVE-14804
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Reporter: Dmitry Kozlov
>Assignee: Fei Hui
>Priority: Blocker
> Fix For: 2.0.2, 2.1.2
>
> Attachments: HIVE-14804.1-branch-2.0.patch, 
> HIVE-14804.1-branch-2.1.patch, HIVE-14804.2-branch-2.0.patch, 
> HIVE-14804.2-branch-2.1.patch
>
>
> I have a problem with a multi database connection. I have 3 environments that 
> I would like to connect in my HPLSQL code Hive, DB2 and MySql. As soon as I 
> map any table either from DB2 or MySQL my code stops to recognize Hive 
> tables. Actually it starts to think that it is a table from the same database 
> (DB2 or MySql) that was mapped the last. It means your example 
> http://www.hplsql.org/map-object works only one way from Hive to MySQL and it 
> is not possible to go back to Hive.  
> Here is a simple piece of code.
> declare cnt int;
> begin
> /*
> PRINT 'Start MySQL';
> MAP OBJECT tbls TO hive.TBLS AT mysqlconn;
> select count(*)
> into cnt
> from tbls;
> PRINT cnt;
> PRINT 'Start Db2';
> MAP OBJECT exch TO DBDEV2.TEST_EXCHANGE AT db2conn;
> select count(1) 
> into cnt
> from exch;
> PRINT cnt;*/
> PRINT 'Check Hive';
> SELECT count(1) 
> into cnt
> FROM dev.test_sqoop;
> PRINT cnt;
> end;
> It has three blocks. One select from MySQL, second from DB2 and third from 
> Hive ORC table.
> When first two blocks are commented then block 3 works. See below
> Check Hive
> 16/09/20 18:08:08 INFO jdbc.Utils: Supplied authorities: localhost:1
> 16/09/20 18:08:08 INFO jdbc.Utils: Resolved authority: localhost:1
> 16/09/20 18:08:08 INFO jdbc.HiveConnection: Will try to open client transport 
> with JDBC Uri: jdbc:hive2://localhost:1
> Open connection: jdbc:hive2://localhost:1 (497 ms)
> Starting query
> Query executed successfully (177 ms)
> 82
> When I try to uncomment any of those blocks then block 3 stops working. For 
> example, if I uncomment block 1 I get this output. It is now assumes that 
> dev.test_sqoop is a MySQL table. Contrarily to your example
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (489 ms)
> Starting query
> Query executed successfully (4 ms)
> 539
> Check Hive
> Starting query
> Unhandled exception in HPL/SQL
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'dev.test_sqoop' doesn't exist
> If I comment the second block then it starts to assume that dev.test_sqoop is 
> a DB2 table. See below. So switch between DB2 and MySQL is working, however, 
> the hive table is still not working
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (485 ms)
> Starting query
> Query executed successfully (5 ms)
> 539
> Start Db2
> Open connection: jdbc:db2://10.11.12.141:5/WM (227 ms)
> Starting query
> Query executed successfully (48 ms)
> 0
> Check Hive
> Starting query
> Unhandled exception in HPL/SQL
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, 
> SQLSTATE=42704, SQLERRMC=DEV.TEST_SQOOP, DRIVER=4.16.53
> Could you, please, provide your feedback on this finding. In addition, I 
> would like to check if it would be possible to insert into a DB2 table 
> records that were selected from a Hive with one statement as soon as DB2 
> table is properly mapped. Please, explain.
> Looking forward to hearing from you soon.
> Regards,
> Dmitry Kozlov
> Daisy Intelligence   



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16072) LLAP: Add some additional jvm metrics for hadoop-metrics2

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16072:
-
   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed patch to master.

> LLAP: Add some additional jvm metrics for hadoop-metrics2 
> --
>
> Key: HIVE-16072
> URL: https://issues.apache.org/jira/browse/HIVE-16072
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Fix For: 2.2.0
>
> Attachments: HIVE-16072.1.patch, HIVE-16072.2.patch
>
>
> It will be helpful for debugging to expose some metrics like buffer pool, 
> file descriptors etc. that are not exposed via Hadoop's JvmMetrics. We 
> already a /jmx endpoint that gives out these info but we don't know the 
> timestamp of allocations, number file descriptors to correlated with the 
> logs. This will better suited for graphing tools. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16072) LLAP: Add some additional jvm metrics for hadoop-metrics2

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16072:
-
Attachment: HIVE-16072.2.patch

> LLAP: Add some additional jvm metrics for hadoop-metrics2 
> --
>
> Key: HIVE-16072
> URL: https://issues.apache.org/jira/browse/HIVE-16072
> Project: Hive
>  Issue Type: Bug
>  Components: llap
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16072.1.patch, HIVE-16072.2.patch
>
>
> It will be helpful for debugging to expose some metrics like buffer pool, 
> file descriptors etc. that are not exposed via Hadoop's JvmMetrics. We 
> already a /jmx endpoint that gives out these info but we don't know the 
> timestamp of allocations, number file descriptors to correlated with the 
> logs. This will better suited for graphing tools. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-14804) HPLSQL multiple db connection does not switch back to Hive

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-14804:
--
Status: In Progress  (was: Patch Available)

> HPLSQL multiple db connection does not switch back to Hive
> --
>
> Key: HIVE-14804
> URL: https://issues.apache.org/jira/browse/HIVE-14804
> Project: Hive
>  Issue Type: Bug
>  Components: hpl/sql
>Reporter: Dmitry Kozlov
>Assignee: Fei Hui
>Priority: Blocker
> Attachments: HIVE-14804.1-branch-2.0.patch, 
> HIVE-14804.1-branch-2.1.patch, HIVE-14804.2-branch-2.0.patch, 
> HIVE-14804.2-branch-2.1.patch
>
>
> I have a problem with a multi database connection. I have 3 environments that 
> I would like to connect in my HPLSQL code Hive, DB2 and MySql. As soon as I 
> map any table either from DB2 or MySQL my code stops to recognize Hive 
> tables. Actually it starts to think that it is a table from the same database 
> (DB2 or MySql) that was mapped the last. It means your example 
> http://www.hplsql.org/map-object works only one way from Hive to MySQL and it 
> is not possible to go back to Hive.  
> Here is a simple piece of code.
> declare cnt int;
> begin
> /*
> PRINT 'Start MySQL';
> MAP OBJECT tbls TO hive.TBLS AT mysqlconn;
> select count(*)
> into cnt
> from tbls;
> PRINT cnt;
> PRINT 'Start Db2';
> MAP OBJECT exch TO DBDEV2.TEST_EXCHANGE AT db2conn;
> select count(1) 
> into cnt
> from exch;
> PRINT cnt;*/
> PRINT 'Check Hive';
> SELECT count(1) 
> into cnt
> FROM dev.test_sqoop;
> PRINT cnt;
> end;
> It has three blocks. One select from MySQL, second from DB2 and third from 
> Hive ORC table.
> When first two blocks are commented then block 3 works. See below
> Check Hive
> 16/09/20 18:08:08 INFO jdbc.Utils: Supplied authorities: localhost:1
> 16/09/20 18:08:08 INFO jdbc.Utils: Resolved authority: localhost:1
> 16/09/20 18:08:08 INFO jdbc.HiveConnection: Will try to open client transport 
> with JDBC Uri: jdbc:hive2://localhost:1
> Open connection: jdbc:hive2://localhost:1 (497 ms)
> Starting query
> Query executed successfully (177 ms)
> 82
> When I try to uncomment any of those blocks then block 3 stops working. For 
> example, if I uncomment block 1 I get this output. It is now assumes that 
> dev.test_sqoop is a MySQL table. Contrarily to your example
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (489 ms)
> Starting query
> Query executed successfully (4 ms)
> 539
> Check Hive
> Starting query
> Unhandled exception in HPL/SQL
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'dev.test_sqoop' doesn't exist
> If I comment the second block then it starts to assume that dev.test_sqoop is 
> a DB2 table. See below. So switch between DB2 and MySQL is working, however, 
> the hive table is still not working
> Start MySQL
> Open connection: jdbc:mysql://10.11.12.144:3306/hive (485 ms)
> Starting query
> Query executed successfully (5 ms)
> 539
> Start Db2
> Open connection: jdbc:db2://10.11.12.141:5/WM (227 ms)
> Starting query
> Query executed successfully (48 ms)
> 0
> Check Hive
> Starting query
> Unhandled exception in HPL/SQL
> com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, 
> SQLSTATE=42704, SQLERRMC=DEV.TEST_SQOOP, DRIVER=4.16.53
> Could you, please, provide your feedback on this finding. In addition, I 
> would like to check if it would be possible to insert into a DB2 table 
> records that were selected from a Hive with one statement as soon as DB2 
> table is properly mapped. Please, explain.
> Looking forward to hearing from you soon.
> Regards,
> Dmitry Kozlov
> Daisy Intelligence   



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15952) Add blobstore integration test for CREATE LIKE

[Thomas Poepping (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902103#comment-15902103
 ] 

Thomas Poepping commented on HIVE-15952:


hey [~spena] it looks like on the commit, the patch might have been a little 
weird, it was committed outside of the itests directory:

https://github.com/apache/hive/commit/47d06fe0064536e194cb35d5d23f0572a398059d

master on github has a hive-blobstore/ directory in the project root, that 
should be under the itests/ directory. Can we fix this?

> Add blobstore integration test for CREATE LIKE
> --
>
> Key: HIVE-15952
> URL: https://issues.apache.org/jira/browse/HIVE-15952
> Project: Hive
>  Issue Type: Bug
>  Components: Test
>Reporter: Juan Rodríguez Hortalá
>Assignee: Juan Rodríguez Hortalá
> Fix For: 2.2.0
>
> Attachments: HIVE-15952.patch
>
>
> This patch adds a new positive test for the integration with blobstores. The 
> test checks that we can create an external table with LIKE of an existing 
> table, and then drop the new table without affecting the original, with both 
> tables located in a blobstore.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16150) LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory pressure

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16150:
-
Attachment: HIVE-16150.1.patch

[~rajesh.balamohan]/[~sseth] could someone please take a look? small patch

> LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory 
> pressure
> ---
>
> Key: HIVE-16150
> URL: https://issues.apache.org/jira/browse/HIVE-16150
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16150.1.patch, Screen Shot 2017-02-22 at 2.42.12 
> PM.png
>
>
> Easiest: following needs to be fixed.
> LOG.debug("Found spec for " + hsplit.getPath() + " " + part + " from " + 
> pathToPartitionInfo);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16150) LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory pressure

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902097#comment-15902097
 ] 

Prasanth Jayachandran commented on HIVE-16150:
--

Not submitting for ptest as this is logging only change. 

> LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory 
> pressure
> ---
>
> Key: HIVE-16150
> URL: https://issues.apache.org/jira/browse/HIVE-16150
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16150.1.patch, Screen Shot 2017-02-22 at 2.42.12 
> PM.png
>
>
> Easiest: following needs to be fixed.
> LOG.debug("Found spec for " + hsplit.getPath() + " " + part + " from " + 
> pathToPartitionInfo);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16150) LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory pressure

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-16150:
-
Attachment: Screen Shot 2017-02-22 at 2.42.12 PM.png

> LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory 
> pressure
> ---
>
> Key: HIVE-16150
> URL: https://issues.apache.org/jira/browse/HIVE-16150
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: Screen Shot 2017-02-22 at 2.42.12 PM.png
>
>
> Easiest: following needs to be fixed.
> LOG.debug("Found spec for " + hsplit.getPath() + " " + part + " from " + 
> pathToPartitionInfo);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-16150) LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory pressure

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran reassigned HIVE-16150:



> LLAP: HiveInputFormat:getRecordReader: Fix log statements to reduce memory 
> pressure
> ---
>
> Key: HIVE-16150
> URL: https://issues.apache.org/jira/browse/HIVE-16150
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 2.2.0
>Reporter: Rajesh Balamohan
>Assignee: Prasanth Jayachandran
> Attachments: Screen Shot 2017-02-22 at 2.42.12 PM.png
>
>
> Easiest: following needs to be fixed.
> LOG.debug("Found spec for " + hsplit.getPath() + " " + part + " from " + 
> pathToPartitionInfo);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15944) The order of cols is error in ColumnPrunerReduceSinkProc because of sort operator

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902078#comment-15902078
 ] 

Hive QA commented on HIVE-15944:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856787/HIVE-15944.8.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10322 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
 (batchId=101)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4026/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4026/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4026/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856787 - PreCommit-HIVE-Build

> The order of cols is error in ColumnPrunerReduceSinkProc because of sort 
> operator
> -
>
> Key: HIVE-15944
> URL: https://issues.apache.org/jira/browse/HIVE-15944
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 1.1.0, 2.2.0
>Reporter: wan kun
>Assignee: wan kun
> Fix For: 2.2.0
>
> Attachments: HIVE-15944.1.patch, 
> HIVE-15944.2-branch-1.1.1.path.erroroutput, HIVE-15944.3.patch, 
> HIVE-15944.4-branch-1.1.1.patch, HIVE-15944.4.patch, HIVE-15944.5.patch, 
> HIVE-15944.6.patch, HIVE-15944.7.patch, HIVE-15944.8.patch, 
> HIVE-15944-branch-1.1.patch, HIVE-15944.patch, STAGE_DEPENDENCIES
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> If one sql have two job.
> job 1:The order of cols is updated in ColumnPrunerReduceSinkProc because of 
> sort operator.
> job 2 will read error in map operation because the cols order is old.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14848) S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs

[Thomas Poepping (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902075#comment-15902075
 ] 

Thomas Poepping commented on HIVE-14848:


Have we completely dropped this patch?

> S3 creds added to a hidden list by HIVE-14588 are not working on MR jobs
> 
>
> Key: HIVE-14848
> URL: https://issues.apache.org/jira/browse/HIVE-14848
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14848.1.patch, HIVE-14848.1.patch
>
>
> When S3 credentials are included in hive-site.xml, then MR jobs that need to 
> read
> data from S3 cannot use them because S3 values are stripped from the Job 
> configuration
> object before submitting the MR job.
> {noformat}
> @Override
> public void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext 
> driverContext) {
>   ...
>   conf.stripHiddenConfigurations(job);
>   this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
> }
> {noformat}
> A nice to have (available on hadoop 2.9.0) is an MR 
> {{mapreduce.job.redacted-properties}} that can be used to hide this list on 
> the MR side (such as history server UI) to allow MR run the job without 
> issues.
> UPDATE:
> Change the call to stripHiddenConfigurations() in 
> ql/exec/tez/DagUtils.createConfiguration(), because this is currently broken 
> for running hive-blobstore suite against Tez



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16104) LLAP: preemption may be too aggressive if the pre-empted task doesn't die immediately

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902006#comment-15902006
 ] 

Siddharth Seth commented on HIVE-16104:
---

We generally don't refactor sections which interfere with the review. And in 
most cases, unrelated changes are avoided. Don't see any reason to make the 
additional changes here.

> LLAP: preemption may be too aggressive if the pre-empted task doesn't die 
> immediately
> -
>
> Key: HIVE-16104
> URL: https://issues.apache.org/jira/browse/HIVE-16104
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16104.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15556) Replicate views

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15902002#comment-15902002
 ] 

Hive QA commented on HIVE-15556:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856780/HIVE-15556.01.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4025/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4025/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4025/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-08 21:27:16.066
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4025/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-08 21:27:16.068
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   0cc1afa..1321293  master -> origin/master
+ git reset --hard HEAD
HEAD is now at 0cc1afa HIVE-16078 : improve abort checking in Tez/LLAP (Sergey 
Shelukhin, reviewed by Rajesh Balamohan, Siddharth Seth)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 1321293 HIVE-15903: Compute table stats when user computes 
column stats (Pengcheng Xiong, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-08 21:27:17.612
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestReplicationScenarios.java
patching file ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
patching file ql/src/java/org/apache/hadoop/hive/ql/parse/EximUtil.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
patching file ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java
patching file ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/metastore/target/generated-sources/antlr3/org/apache/hadoop/hive/metastore/parser/FilterParser.java
 does not exist: must build 
/data/hiveptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
DataNucleus Enhancer (version 4.1.17) for API "JDO"
DataNucleus Enhancer : Classpath
>>  /usr/share/maven/boot/plexus-classworlds-2.x.jar
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MType
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTable
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MConstraint
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MSerDeInfo
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MOrder
ENHANCED (Persistable) : 
org.apache.hadoop.hive.metastore.model.MColumnDescriptor
ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStringList
ENHANCED (Persistable) : 

[jira] [Commented] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901992#comment-15901992
 ] 

Hive QA commented on HIVE-16114:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856770/HIVE-16114.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10333 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
 (batchId=147)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4024/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4024/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4024/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856770 - PreCommit-HIVE-Build

> NullPointerException in TezSessionPoolManager when getting the session
> --
>
> Key: HIVE-16114
> URL: https://issues.apache.org/jira/browse/HIVE-16114
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Attachments: HIVE-16114.1.patch, HIVE-16114.2.patch, HIVE-16114.patch
>
>
> hive version: apache-hive-2.1.1 
> we use hue(3.11.0) connecting to the HiveServer2.  when hue starts up, it 
> works with no problems, a few hours passed, when we use the same sql, an 
> exception about unable to execute TezTask will come into being.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16142) ATSHook NPE via LLAP

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901964#comment-15901964
 ] 

Pengcheng Xiong commented on HIVE-16142:


[~jdere], i am sorry that I do not get it. This is the write path of RS's 
output operator. And there is another one in Optimizer. We just need to prevent 
the NPE in the read path, which is in the patch.

> ATSHook NPE via LLAP
> 
>
> Key: HIVE-16142
> URL: https://issues.apache.org/jira/browse/HIVE-16142
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-16142.01.patch
>
>
> Exceptions in the log of the form:
> 2017-03-06T15:42:30,046 WARN  [ATS Logger 0]: hooks.ATSHook 
> (ATSHook.java:run(318)) - Failed to submit to ATS for 
> hive_20170306154227_f41bc7cb-1a2f-40f1-a85b-b2bc260a451a
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:608) 
> ~[hive-exec-2.1.0.2.6.0.0-585.jar:2.1.0.2.6.0.0-585]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901938#comment-15901938
 ] 

Gunther Hagleitner commented on HIVE-16133:
---

LGTM +1

> Footer cache in Tez AM can take too much memory
> ---
>
> Key: HIVE-16133
> URL: https://issues.apache.org/jira/browse/HIVE-16133
> Project: Hive
>  Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16133.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HIVE-16001) add test for merge + runtime filtering

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman resolved HIVE-16001.
---
Resolution: Duplicate

done in HIVE-16022

> add test for merge + runtime filtering
> --
>
> Key: HIVE-16001
> URL: https://issues.apache.org/jira/browse/HIVE-16001
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> make sure merge works with HIVE-15802 and HIVE-15269
> add to sqlmerge.q
> cc [~jdere]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16078) improve abort checking in Tez/LLAP

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16078:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the reviews!

> improve abort checking in Tez/LLAP
> --
>
> Key: HIVE-16078
> URL: https://issues.apache.org/jira/browse/HIVE-16078
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-16078.01.patch, HIVE-16078.02.patch, 
> HIVE-16078.03.patch, HIVE-16078.patch
>
>
> Sometimes, a fragment can run for a long time after a query fails. It looks 
> from logs like the abort/interrupt were called correctly on the thread, yet 
> the thread hangs around minutes after, doing the below. Other tasks for the 
> same job appear to have exited correctly, after the same abort logic (at 
> least, the same log lines, fwiw)
> {noformat}
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.copyByValue(VectorCopyRow.java:317)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:263)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> 

[jira] [Commented] (HIVE-16078) improve abort checking in Tez/LLAP

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901924#comment-15901924
 ] 

Sergey Shelukhin commented on HIVE-16078:
-

Forgot to mention Matt on commit message :( sorry for that, there were too many 
comments after.

> improve abort checking in Tez/LLAP
> --
>
> Key: HIVE-16078
> URL: https://issues.apache.org/jira/browse/HIVE-16078
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-16078.01.patch, HIVE-16078.02.patch, 
> HIVE-16078.03.patch, HIVE-16078.patch
>
>
> Sometimes, a fragment can run for a long time after a query fails. It looks 
> from logs like the abort/interrupt were called correctly on the thread, yet 
> the thread hangs around minutes after, doing the below. Other tasks for the 
> same job appear to have exited correctly, after the same abort logic (at 
> least, the same log lines, fwiw)
> {noformat}
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorCopyRow.copyByValue(VectorCopyRow.java:317)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:263)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.generateHashMapResultMultiValue(VectorMapJoinGenerateResultOperator.java:277)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator.finishInner(VectorMapJoinInnerGenerateResultOperator.java:189)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerLongOperator.process(VectorMapJoinInnerLongOperator.java:389)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator.forwardOverflow(VectorMapJoinGenerateResultOperator.java:628)
>   at 
> 

[jira] [Commented] (HIVE-15515) Remove the docs directory

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901923#comment-15901923
 ] 

Hive QA commented on HIVE-15515:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12856771/HIVE-15515.02.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10333 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_table]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=140)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentStatements (batchId=218)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4023/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4023/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4023/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12856771 - PreCommit-HIVE-Build

> Remove the docs directory
> -
>
> Key: HIVE-15515
> URL: https://issues.apache.org/jira/browse/HIVE-15515
> Project: Hive
>  Issue Type: Bug
>  Components: Documentation
>Reporter: Lefty Leverenz
>Assignee: Akira Ajisaka
> Attachments: HIVE-15515.01.patch, HIVE-15515.02.patch
>
>
> Hive xdocs have not been used since 2012.  The docs directory only holds six 
> xml documents, and their contents are in the wiki.
> It's past time to remove the docs directory from the Hive code.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16131) Hive building with Hadoop 3 - additional stuff broken recently

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-16131:

   Resolution: Fixed
Fix Version/s: 2.2.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the reviews!

> Hive building with Hadoop 3 - additional stuff broken recently
> --
>
> Key: HIVE-16131
> URL: https://issues.apache.org/jira/browse/HIVE-16131
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 2.2.0
>
> Attachments: HIVE-16131.01.patch, HIVE-16131.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16133) Footer cache in Tez AM can take too much memory

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901900#comment-15901900
 ] 

Sergey Shelukhin commented on HIVE-16133:
-

[~hagleitn] perhaps you can review?

> Footer cache in Tez AM can take too much memory
> ---
>
> Key: HIVE-16133
> URL: https://issues.apache.org/jira/browse/HIVE-16133
> Project: Hive
>  Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16133.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16104) LLAP: preemption may be too aggressive if the pre-empted task doesn't die immediately

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901897#comment-15901897
 ] 

Sergey Shelukhin commented on HIVE-16104:
-

We don't normally have separate jiras for refactoring so I don't see why not 
perform small refactors in other changes...

> LLAP: preemption may be too aggressive if the pre-empted task doesn't die 
> immediately
> -
>
> Key: HIVE-16104
> URL: https://issues.apache.org/jira/browse/HIVE-16104
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16104.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16142) ATSHook NPE via LLAP

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901878#comment-15901878
 ] 

Jason Dere commented on HIVE-16142:
---

I believe the same fix needs to apply to the changes that were done in 
DynamicPartitionPruningOptimization:

{code}
// for explain purpose
if (parseContext.getContext().getExplainConfig() != null
&& parseContext.getContext().getExplainConfig().isFormatted()) {
  List outputOperators = new ArrayList<>();
  outputOperators.add(ts.getOperatorId());
  rsOpFinal.getConf().setOutputOperators(outputOperators);
}
{code}

> ATSHook NPE via LLAP
> 
>
> Key: HIVE-16142
> URL: https://issues.apache.org/jira/browse/HIVE-16142
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-16142.01.patch
>
>
> Exceptions in the log of the form:
> 2017-03-06T15:42:30,046 WARN  [ATS Logger 0]: hooks.ATSHook 
> (ATSHook.java:run(318)) - Failed to submit to ATS for 
> hive_20170306154227_f41bc7cb-1a2f-40f1-a85b-b2bc260a451a
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hive.ql.exec.ExplainTask.outputPlan(ExplainTask.java:608) 
> ~[hive-exec-2.1.0.2.6.0.0-585.jar:2.1.0.2.6.0.0-585]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16114) NullPointerException in TezSessionPoolManager when getting the session

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901844#comment-15901844
 ] 

Sergey Shelukhin commented on HIVE-16114:
-

+1, thanks for the patch! Will commit later today if noone objects. cc [~sseth]

> NullPointerException in TezSessionPoolManager when getting the session
> --
>
> Key: HIVE-16114
> URL: https://issues.apache.org/jira/browse/HIVE-16114
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Attachments: HIVE-16114.1.patch, HIVE-16114.2.patch, HIVE-16114.patch
>
>
> hive version: apache-hive-2.1.1 
> we use hue(3.11.0) connecting to the HiveServer2.  when hue starts up, it 
> works with no problems, a few hours passed, when we use the same sql, an 
> exception about unable to execute TezTask will come into being.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16107) JDBC: HttpClient should retry one more time on NoHttpResponseException

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-16107:

Attachment: (was: HIVE-16107.1.patch)

> JDBC: HttpClient should retry one more time on NoHttpResponseException
> --
>
> Key: HIVE-16107
> URL: https://issues.apache.org/jira/browse/HIVE-16107
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-16107.1.patch
>
>
> Hive's JDBC client in HTTP transport mode doesn't retry on 
> NoHttpResponseException. We've seen the exception being thrown to the JDBC 
> end user when used with Knox as the proxy, when Knox upgraded its jetty 
> version, which has a smaller value for jetty connector idletimeout, and as a 
> result closes the HTTP connection on server side. The next jdbc query on the 
> client, throws a NoHttpResponseException. However, subsequent queries 
> reconnect, but the JDBC driver should ideally handle this by retrying.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16107) JDBC: HttpClient should retry one more time on NoHttpResponseException

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-16107:

Attachment: HIVE-16107.1.patch

> JDBC: HttpClient should retry one more time on NoHttpResponseException
> --
>
> Key: HIVE-16107
> URL: https://issues.apache.org/jira/browse/HIVE-16107
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-16107.1.patch
>
>
> Hive's JDBC client in HTTP transport mode doesn't retry on 
> NoHttpResponseException. We've seen the exception being thrown to the JDBC 
> end user when used with Knox as the proxy, when Knox upgraded its jetty 
> version, which has a smaller value for jetty connector idletimeout, and as a 
> result closes the HTTP connection on server side. The next jdbc query on the 
> client, throws a NoHttpResponseException. However, subsequent queries 
> reconnect, but the JDBC driver should ideally handle this by retrying.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-16104) LLAP: preemption may be too aggressive if the pre-empted task doesn't die immediately

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901833#comment-15901833
 ] 

Siddharth Seth commented on HIVE-16104:
---

bq. The lock in trySchedule is unnecessary so I removed it and renamed the 
method;
Not required, or relevant to this patch.
bq. preemption was surrounded by a loop because previously, if the first task 
in queue was finishable it would bail without preempting anything even if there 
are more tasks.
I believe this is mostly harmless - the same task will be picked up in the next 
loop. Unrelated to the jira, but probably a relevant change.There's some change 
around switching the condition to return early (negative condition - return, 
instead of the existing conditions met-> execute) - which is unnecessary.
Similar condition changes elsewhere as well.

bq. I can merge updateQueueMetric back into being copy-pasted in 3 places... 
also one if was refactored because it has lots of repetitive code.
bq. Another method was added because something that was previously called in 
one place is now called in 2 places and I didn't want to copy-paste it.
These would be relevant to the jira? - since additional invocations are 
happening because of the change.

> LLAP: preemption may be too aggressive if the pre-empted task doesn't die 
> immediately
> -
>
> Key: HIVE-16104
> URL: https://issues.apache.org/jira/browse/HIVE-16104
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16104.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16107) JDBC: HttpClient should retry one more time on NoHttpResponseException

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-16107:

Attachment: HIVE-16107.1.patch

> JDBC: HttpClient should retry one more time on NoHttpResponseException
> --
>
> Key: HIVE-16107
> URL: https://issues.apache.org/jira/browse/HIVE-16107
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-16107.1.patch
>
>
> Hive's JDBC client in HTTP transport mode doesn't retry on 
> NoHttpResponseException. We've seen the exception being thrown to the JDBC 
> end user when used with Knox as the proxy, when Knox upgraded its jetty 
> version, which has a smaller value for jetty connector idletimeout, and as a 
> result closes the HTTP connection on server side. The next jdbc query on the 
> client, throws a NoHttpResponseException. However, subsequent queries 
> reconnect, but the JDBC driver should ideally handle this by retrying.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-16107) JDBC: HttpClient should retry one more time on NoHttpResponseException

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-16107:

Attachment: (was: HIVE-16107.1.patch)

> JDBC: HttpClient should retry one more time on NoHttpResponseException
> --
>
> Key: HIVE-16107
> URL: https://issues.apache.org/jira/browse/HIVE-16107
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, JDBC
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-16107.1.patch
>
>
> Hive's JDBC client in HTTP transport mode doesn't retry on 
> NoHttpResponseException. We've seen the exception being thrown to the JDBC 
> end user when used with Knox as the proxy, when Knox upgraded its jetty 
> version, which has a smaller value for jetty connector idletimeout, and as a 
> result closes the HTTP connection on server side. The next jdbc query on the 
> client, throws a NoHttpResponseException. However, subsequent queries 
> reconnect, but the JDBC driver should ideally handle this by retrying.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)