[jira] [Commented] (HIVE-12783) fix the unit test failures in TestSparkClient and TestSparkSessionManagerImpl

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090659#comment-15090659
 ] 

Xuefu Zhang commented on HIVE-12783:


Patch looks good. Just one thing I'm not sure of
{code}
+
+  org.eclipse.jetty.orbit
+  javax.servlet
+  3.0.0.v201112011016
 
{code}
Is this required?


> fix the unit test failures in TestSparkClient and TestSparkSessionManagerImpl
> -
>
> Key: HIVE-12783
> URL: https://issues.apache.org/jira/browse/HIVE-12783
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Affects Versions: 2.0.0
>Reporter: Pengcheng Xiong
>Assignee: Owen O'Malley
>Priority: Blocker
> Attachments: HIVE-12783.patch, HIVE-12783.patch
>
>
> This includes
> {code}
> org.apache.hive.spark.client.TestSparkClient.testSyncRpc
> org.apache.hive.spark.client.TestSparkClient.testJobSubmission
> org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
> org.apache.hive.spark.client.TestSparkClient.testCounters
> org.apache.hive.spark.client.TestSparkClient.testRemoteClient
> org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
> org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
> org.apache.hive.spark.client.TestSparkClient.testErrorJob
> org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
> org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
> {code}
> all of them passed on my laptop. cc'ing [~szehon], [~xuefuz], could you 
> please take a look? Shall we ignore them? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12798) CBO: Calcite Operator To Hive Operator (Calcite Return Path): MiniTezCliDriver.vector* queries failures due to NPE in Vectorizer.onExpressionHasNullSafes()

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090546#comment-15090546
 ] 

Hive QA commented on HIVE-12798:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781050/HIVE-12798.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 9961 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6561/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6561/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6561/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781050 - PreCommit-HIVE-TRUNK-Build

> CBO: Calcite Operator To Hive Operator (Calcite Return Path): 
> MiniTezCliDriver.vector* queries failures due to NPE in 
> Vectorizer.onExpressionHasNullSafes()
> ---
>
> Key: HIVE-12798
> URL: https://issues.apache.org/jira/browse/HIVE-12798
> Project: Hive
>  Issue Type: Sub-task
>  Components: CBO
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-12798.1.patch
>
>
> As of 01/04/2016, the following tests fail in the MiniTezCliDriver mode when 
> the cbo return path is enabled. We need to fix them :
> {code}
>  vector_leftsemi_mapjoin
>  vector_join_filters
>  vector_interval_mapjoin
>  vector_left_outer_join
>  vectorized_mapjoin
>  vector_inner_join
>  vectorized_context
>  tez_vector_dynpart_hashjoin_1
>  count
>  auto_sortmerge_join_6
>  skewjoin
>  vector_auto_smb_mapjoin_14
>  auto_join_filters
>  vector_outer_join0
>  vector_outer_join1
>  vector_outer_join2
>  vector_outer_join3
>  vector_outer_join4
>  vector_outer_join5
>  hybridgrace_hashjoin_1
>  vector_mapjoin_reduce
>  vectorized_nested_mapjoin
>  vector_left_outer_join2
>  vector_char_mapjoin1
>  vector_decimal_mapjoin
>  vectorized_dynamic_partition_pruning
>  vector_varchar_mapjoin1
> {code}
> This jira is intended to cover the vectorization issues related to the 
> MiniTezCliDriver failures caused by NPE via nullSafes array as shown below :
> {code}
> private boolean onExpressionHasNullSafes(MapJoinDesc desc) {
>  boolean[] nullSafes = desc.getNullSafes();
>  for (boolean nullSafe : nullSafes) {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11609) Capability to add a filter to hbase scan via composite key doesn't work

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090621#comment-15090621
 ] 

Hive QA commented on HIVE-11609:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781061/HIVE-11609.4.patch.txt

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 9982 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-tez_joins_explain.q-vector_decimal_aggregate.q-vector_groupby_mapjoin.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6562/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6562/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6562/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781061 - PreCommit-HIVE-TRUNK-Build

> Capability to add a filter to hbase scan via composite key doesn't work
> ---
>
> Key: HIVE-11609
> URL: https://issues.apache.org/jira/browse/HIVE-11609
> Project: Hive
>  Issue Type: Bug
>  Components: HBase Handler
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-11609.1.patch.txt, HIVE-11609.2.patch.txt, 
> HIVE-11609.3.patch.txt, HIVE-11609.4.patch.txt
>
>
> It seems like the capability to add filter to an hbase scan which was added 
> as part of HIVE-6411 doesn't work. This is primarily because in the 
> HiveHBaseInputFormat, the filter is added in the getsplits instead of 
> getrecordreader. This works fine for start and stop keys but not for filter 
> because a filter is respected only when an actual scan is performed. This is 
> also related to the initial refactoring that was done as part of HIVE-3420.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12828) Update Spark version to 1.6

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090652#comment-15090652
 ] 

Xuefu Zhang commented on HIVE-12828:


cc: [~lirui]

> Update Spark version to 1.6
> ---
>
> Key: HIVE-12828
> URL: https://issues.apache.org/jira/browse/HIVE-12828
> Project: Hive
>  Issue Type: Task
>  Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-1841) datanucleus.fixedDatastore should be true in hive-default.xml

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090537#comment-15090537
 ] 

Lefty Leverenz commented on HIVE-1841:
--

Thanks Ashutosh.  I removed the TODOC2.0 label.

But after this patch, HIVE-6113 removed datanucleus.fixedDatastore so I updated 
the wiki again:

* [Configuration Properties -- datanucleus.fixedDatastore | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-datanucleus.fixedDatastore]



>  datanucleus.fixedDatastore should be true in hive-default.xml
> --
>
> Key: HIVE-1841
> URL: https://issues.apache.org/jira/browse/HIVE-1841
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration, Metastore
>Affects Versions: 0.6.0
>Reporter: Edward Capriolo
>Assignee: Ashutosh Chauhan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HIVE-1841.1.patch.txt, HIVE-1841.2.patch, HIVE-1841.patch
>
>
> Two datanucleus variables:
> {noformat}
> 
>  datanucleus.autoCreateSchema
>  false
> 
> 
>  datanucleus.fixedDatastore
>  true
> 
> {noformat}
> are dangerous.  We do want the schema to auto-create itself, but we do not 
> want the schema to auto update itself. 
> Someone might accidentally point a trunk at the wrong meta-store and 
> unknowingly update. I believe we should set this to false and possibly trap 
> exceptions stemming from hive wanting to do any update. This way someone has 
> to actively acknowledge the update, by setting this to true and then starting 
> up hive, or leaving it false, removing schema modifies for the user that hive 
> usages, and doing all the time and doing the updates by hand. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9862) Vectorized execution corrupts timestamp values

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-9862:
---
Attachment: HIVE-9862.04.patch

> Vectorized execution corrupts timestamp values
> --
>
> Key: HIVE-9862
> URL: https://issues.apache.org/jira/browse/HIVE-9862
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 1.0.0
>Reporter: Nathan Howell
>Assignee: Matt McCline
> Attachments: HIVE-9862.01.patch, HIVE-9862.02.patch, 
> HIVE-9862.03.patch, HIVE-9862.04.patch
>
>
> Timestamps in the future (year 2250?) and before ~1700 are silently corrupted 
> in vectorized execution mode. Simple repro:
> {code}
> hive> DROP TABLE IF EXISTS test;
> hive> CREATE TABLE test(ts TIMESTAMP) STORED AS ORC;
> hive> INSERT INTO TABLE test VALUES ('-12-31 23:59:59');
> hive> SET hive.vectorized.execution.enabled = false;
> hive> SELECT MAX(ts) FROM test;
> -12-31 23:59:59
> hive> SET hive.vectorized.execution.enabled = true;
> hive> SELECT MAX(ts) FROM test;
> 1816-03-30 05:56:07.066277376
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12823) Thrift and RDBMS schema changes for HIVE-11956

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-12823:
--
Attachment: HIVE-12823.patch

> Thrift and RDBMS schema changes for HIVE-11956
> --
>
> Key: HIVE-12823
> URL: https://issues.apache.org/jira/browse/HIVE-12823
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12823.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12783) fix the unit test failures in TestSparkClient and TestSparkSessionManagerImpl

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090657#comment-15090657
 ] 

Hive QA commented on HIVE-12783:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781318/HIVE-12783.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 3 failed/errored test(s),  tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6563/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6563/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6563/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781318 - PreCommit-HIVE-TRUNK-Build

> fix the unit test failures in TestSparkClient and TestSparkSessionManagerImpl
> -
>
> Key: HIVE-12783
> URL: https://issues.apache.org/jira/browse/HIVE-12783
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test
>Affects Versions: 2.0.0
>Reporter: Pengcheng Xiong
>Assignee: Owen O'Malley
>Priority: Blocker
> Attachments: HIVE-12783.patch, HIVE-12783.patch
>
>
> This includes
> {code}
> org.apache.hive.spark.client.TestSparkClient.testSyncRpc
> org.apache.hive.spark.client.TestSparkClient.testJobSubmission
> org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
> org.apache.hive.spark.client.TestSparkClient.testCounters
> org.apache.hive.spark.client.TestSparkClient.testRemoteClient
> org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
> org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
> org.apache.hive.spark.client.TestSparkClient.testErrorJob
> org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
> org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
> {code}
> all of them passed on my laptop. cc'ing [~szehon], [~xuefuz], could you 
> please take a look? Shall we ignore them? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12750) Vectorization logic not maintaining noNulls flag and isNull array properly causing wrong query results

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090615#comment-15090615
 ] 

Gopal V commented on HIVE-12750:


The patch adds minimal performance impact (~8%), but after getting a simpler 
test-case, I've isolated some code which does not populate the isNull array and 
is triggering this particular issue.

A better approach to fixing this would be to maintain isNull correctly for a 
rare combination of FIL -> MJ -> FIL.

As a top level analysis, fixing HIVE-12824 or HIVE-12809 would end up fixing 
the symptoms again without fixing the root-cause.

I'll file a few more possible bugs which came up during a step through of the 
code.

> Vectorization logic not maintaining noNulls flag and isNull array properly 
> causing wrong query results
> --
>
> Key: HIVE-12750
> URL: https://issues.apache.org/jira/browse/HIVE-12750
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12750.1.patch, HIVE-12750.2.patch, 
> HIVE-12750.3.patch, HIVE-12750.4.patch, HIVE-12750.5.patch, 
> HIVE-12750.6.patch, HIVE-12750.8.patch
>
>
> A very large query produced wrong results due to not maintaining noNulls flag 
> and isNull array properly.  No Q file since input files too large.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12366) Refactor Heartbeater logic for transaction

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090757#comment-15090757
 ] 

Hive QA commented on HIVE-12366:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781097/HIVE-12366.13.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10002 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6565/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6565/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6565/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781097 - PreCommit-HIVE-TRUNK-Build

> Refactor Heartbeater logic for transaction
> --
>
> Key: HIVE-12366
> URL: https://issues.apache.org/jira/browse/HIVE-12366
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12366.1.patch, HIVE-12366.11.patch, 
> HIVE-12366.12.patch, HIVE-12366.13.patch, HIVE-12366.2.patch, 
> HIVE-12366.3.patch, HIVE-12366.4.patch, HIVE-12366.5.patch, 
> HIVE-12366.6.patch, HIVE-12366.7.patch, HIVE-12366.8.patch, HIVE-12366.9.patch
>
>
> Currently there is a gap between the time locks acquisition and the first 
> heartbeat being sent out. Normally the gap is negligible, but when it's big 
> it will cause query fail since the locks are timed out by the time the 
> heartbeat is sent.
> Need to remove this gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12819) Thrift and RDBMS schema changes for HIVE-11957

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090724#comment-15090724
 ] 

Eugene Koifman commented on HIVE-12819:
---

This is missing a change to OpenTxnRequest.  it needs agentInfo filed to pass 
in as part of openTxns() call

> Thrift and RDBMS schema changes for HIVE-11957
> --
>
> Key: HIVE-12819
> URL: https://issues.apache.org/jira/browse/HIVE-12819
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12819.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12832) RDBMS schema changes for HIVE-11388

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-12832:
--
Attachment: HIVE-12382.patch

> RDBMS schema changes for HIVE-11388
> ---
>
> Key: HIVE-12832
> URL: https://issues.apache.org/jira/browse/HIVE-12832
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12382.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12832) RDBMS schema changes for HIVE-11388

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-12832:
--
Attachment: HIVE-12832.uber.patch

This uber patch has all of the recent changes to thrift and RDBMS in one patch.

> RDBMS schema changes for HIVE-11388
> ---
>
> Key: HIVE-12832
> URL: https://issues.apache.org/jira/browse/HIVE-12832
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12382.patch, HIVE-12832.uber.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-10468) Create scripts to do metastore upgrade tests on jenkins for Oracle DB.

[Naveen Gangam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090665#comment-15090665
 ] 

Naveen Gangam commented on HIVE-10468:
--

[~spena] The test failed 
(http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/88/console)
 failed to download the patch with the following error.

{code}
+ cd hive
+ wget 
https://issues.apache.org/jira/secure/attachment/12781004/HIVE-10468.9.patch -O 
hms.patch
bash: line 5: wget: command not found
+ exit 1
{code}



> Create scripts to do metastore upgrade tests on jenkins for Oracle DB.
> --
>
> Key: HIVE-10468
> URL: https://issues.apache.org/jira/browse/HIVE-10468
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-10468.1.patch, HIVE-10468.2.patch, 
> HIVE-10468.3.patch, HIVE-10468.4.patch, HIVE-10468.5.patch, 
> HIVE-10468.6.patch, HIVE-10468.7.patch, HIVE-10468.9.patch, HIVE-10468.patch
>
>
> This JIRA is to isolate the work specific to Oracle DB in HIVE-10239. Because 
> of absence of 64 bit debian packages for oracle-xe, the apt-get install fails 
> on the AWS systems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12819) Thrift and RDBMS schema changes for HIVE-11957

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090742#comment-15090742
 ] 

Eugene Koifman commented on HIVE-12819:
---

Never mind, I can't read the diffs.  This is OK

> Thrift and RDBMS schema changes for HIVE-11957
> --
>
> Key: HIVE-12819
> URL: https://issues.apache.org/jira/browse/HIVE-12819
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12819.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12830) Thrift changes for HIVE-11793

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-12830:
--
Attachment: HIVE-12830.patch

> Thrift changes for HIVE-11793
> -
>
> Key: HIVE-12830
> URL: https://issues.apache.org/jira/browse/HIVE-12830
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Minor
> Attachments: HIVE-12830.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12814) Make thrift and DB changes for HIVE-11444

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090686#comment-15090686
 ] 

Eugene Koifman commented on HIVE-12814:
---

Never mind the comment about COMPACTION_QUEUE/Thrift above

> Make thrift and DB changes for HIVE-11444
> -
>
> Key: HIVE-12814
> URL: https://issues.apache.org/jira/browse/HIVE-12814
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12814.patch
>
>
> This JIRA tracks the Thrift and DB schema changes for HIVE-11444.  It depends 
> on HIVE-12807.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12831) Thrift and RDBMS changes for HIVE-10249

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-12831:
--
Attachment: HIVE-12831.patch

> Thrift and RDBMS changes for HIVE-10249
> ---
>
> Key: HIVE-12831
> URL: https://issues.apache.org/jira/browse/HIVE-12831
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore, Transactions
>Affects Versions: 1.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12831.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9862) Vectorized execution corrupts timestamp values

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-9862:
---
Attachment: (was: HIVE-9862.04.patch)

> Vectorized execution corrupts timestamp values
> --
>
> Key: HIVE-9862
> URL: https://issues.apache.org/jira/browse/HIVE-9862
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 1.0.0
>Reporter: Nathan Howell
>Assignee: Matt McCline
> Attachments: HIVE-9862.01.patch, HIVE-9862.02.patch, 
> HIVE-9862.03.patch
>
>
> Timestamps in the future (year 2250?) and before ~1700 are silently corrupted 
> in vectorized execution mode. Simple repro:
> {code}
> hive> DROP TABLE IF EXISTS test;
> hive> CREATE TABLE test(ts TIMESTAMP) STORED AS ORC;
> hive> INSERT INTO TABLE test VALUES ('-12-31 23:59:59');
> hive> SET hive.vectorized.execution.enabled = false;
> hive> SELECT MAX(ts) FROM test;
> -12-31 23:59:59
> hive> SET hive.vectorized.execution.enabled = true;
> hive> SELECT MAX(ts) FROM test;
> 1816-03-30 05:56:07.066277376
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9862) Vectorized execution corrupts timestamp values

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-9862:
---
Attachment: HIVE-9862.04.patch

> Vectorized execution corrupts timestamp values
> --
>
> Key: HIVE-9862
> URL: https://issues.apache.org/jira/browse/HIVE-9862
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 1.0.0
>Reporter: Nathan Howell
>Assignee: Matt McCline
> Attachments: HIVE-9862.01.patch, HIVE-9862.02.patch, 
> HIVE-9862.03.patch, HIVE-9862.04.patch
>
>
> Timestamps in the future (year 2250?) and before ~1700 are silently corrupted 
> in vectorized execution mode. Simple repro:
> {code}
> hive> DROP TABLE IF EXISTS test;
> hive> CREATE TABLE test(ts TIMESTAMP) STORED AS ORC;
> hive> INSERT INTO TABLE test VALUES ('-12-31 23:59:59');
> hive> SET hive.vectorized.execution.enabled = false;
> hive> SELECT MAX(ts) FROM test;
> -12-31 23:59:59
> hive> SET hive.vectorized.execution.enabled = true;
> hive> SELECT MAX(ts) FROM test;
> 1816-03-30 05:56:07.066277376
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12829) Thrift changes for HIVE-12686

[Alan Gates (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Gates updated HIVE-12829:
--
Attachment: HIVE-12829.patch

> Thrift changes for HIVE-12686
> -
>
> Key: HIVE-12829
> URL: https://issues.apache.org/jira/browse/HIVE-12829
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 1.3.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12829.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12797) Synchronization issues with tez/llap session pool in hs2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090734#comment-15090734
 ] 

Hive QA commented on HIVE-12797:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781112/HIVE-12797.4.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 9985 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6564/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6564/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6564/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781112 - PreCommit-HIVE-TRUNK-Build

> Synchronization issues with tez/llap session pool in hs2
> 
>
> Key: HIVE-12797
> URL: https://issues.apache.org/jira/browse/HIVE-12797
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-12797.1.patch, HIVE-12797.2.patch, 
> HIVE-12797.3.patch, HIVE-12797.4.patch
>
>
> The changes introduced as part of HIVE-12674 causes issues while shutting 
> down hs2 when session pools are used.
> {code}
> java.util.ConcurrentModificationException
> at 
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) 
> ~[?:1.8.0_45]
> at java.util.LinkedList$ListItr.remove(LinkedList.java:921) 
> ~[?:1.8.0_45]
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.stop(TezSessionPoolManager.java:288)
>  ~[hive-exec-2.0.0.2.3.5.0-79.jar:2.0.0.2.3.5.0-79]
> at 
> org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:479) 
> [hive-jdbc-2.0.0.2.3.5.0-79-standalone.jar:2.0.0.2.3.5.0-79]
> at 
> org.apache.hive.service.server.HiveServer2$2.run(HiveServer2.java:183) 
> [hive-jdbc-2.0.0.2.3.5.0-79-standalone.jar:2.0.0.2.3.5.0-79]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12787) Trace improvement - Inconsistent logging upon shutdown-start of the Hive metastore process

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090820#comment-15090820
 ] 

Hive QA commented on HIVE-12787:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781125/HIVE-12787.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 9985 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6566/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6566/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6566/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781125 - PreCommit-HIVE-TRUNK-Build

> Trace improvement - Inconsistent logging upon shutdown-start of the Hive 
> metastore process
> --
>
> Key: HIVE-12787
> URL: https://issues.apache.org/jira/browse/HIVE-12787
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 1.2.1
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
> Attachments: HIVE-12787.1.patch
>
>
> The log at: 
> https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L793
>  logged at the start of the shutdown of the Hive metastore process can be 
> improved to match the finish of the shutdown log at: 
> https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L793
> by rephrasing from: "Shutting down the object store..." to: "Metastore 
> shutdown started...". This will match the shutdown completion log: "Metastore 
> shutdown complete.".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12442) Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks

[Rohit Dholakia (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090821#comment-15090821
 ] 

Rohit Dholakia commented on HIVE-12442:
---

Updated rb with the most recent patch. 

> Refactor/repackage HiveServer2's Thrift code so that it can be used in the 
> tasks
> 
>
> Key: HIVE-12442
> URL: https://issues.apache.org/jira/browse/HIVE-12442
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>Assignee: Rohit Dholakia
>  Labels: HiveServer2, RPC, Thrift
> Fix For: 2.1.0
>
> Attachments: hive-12442.1.patch, hive-12442.2.patch, 
> hive-12442.3.patch, hive-12442.4.patch, hive-12442.5.patch
>
>
> For implementing HIVE-12427, the tasks will need to have knowledge of thrift 
> types from HS2's thrift API. This jira will look at the least invasive way to 
> do that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12366) Refactor Heartbeater logic for transaction

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090822#comment-15090822
 ] 

Wei Zheng commented on HIVE-12366:
--

Test failures are unrelated

> Refactor Heartbeater logic for transaction
> --
>
> Key: HIVE-12366
> URL: https://issues.apache.org/jira/browse/HIVE-12366
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12366.1.patch, HIVE-12366.11.patch, 
> HIVE-12366.12.patch, HIVE-12366.13.patch, HIVE-12366.2.patch, 
> HIVE-12366.3.patch, HIVE-12366.4.patch, HIVE-12366.5.patch, 
> HIVE-12366.6.patch, HIVE-12366.7.patch, HIVE-12366.8.patch, HIVE-12366.9.patch
>
>
> Currently there is a gap between the time locks acquisition and the first 
> heartbeat being sent out. Normally the gap is negligible, but when it's big 
> it will cause query fail since the locks are timed out by the time the 
> heartbeat is sent.
> Need to remove this gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12799) Always use Schema Evolution for ACID

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090837#comment-15090837
 ] 

Hive QA commented on HIVE-12799:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781228/HIVE-12799.02.patch

{color:green}SUCCESS:{color} +1 due to 24 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 1 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6567/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6567/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6567/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781228 - PreCommit-HIVE-TRUNK-Build

> Always use Schema Evolution for ACID
> 
>
> Key: HIVE-12799
> URL: https://issues.apache.org/jira/browse/HIVE-12799
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-12799.01.patch, HIVE-12799.02.patch
>
>
> Always use Schema Evolution for ACID -- ignore hive.exec.schema.evolution 
> setting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12785) View with union type and UDF to `cast` the struct is broken

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-12785:
---
Attachment: HIVE-12785.02.patch

address [~jpullokkaran]'s comments.

> View with union type and UDF to `cast` the struct is broken
> ---
>
> Key: HIVE-12785
> URL: https://issues.apache.org/jira/browse/HIVE-12785
> Project: Hive
>  Issue Type: Bug
>  Components: Parser
>Affects Versions: 1.2.1
> Environment: HDP-2.3.4.0
>Reporter: Benoit Perroud
>Assignee: Pengcheng Xiong
>Priority: Minor
> Attachments: HIVE-12785.01.patch, HIVE-12785.02.patch, 
> StructFromUnionMinimalB.java, data_minimal.avro
>
>
> Unfortunately HIVE-12156 is breaking the following use case:
> I do have a table with a {{uniontype}} of {{struct}} s, such as:
> {code}
> CREATE TABLE `minimal_sample`(
>   `record_type` string,
>   `event` uniontype)
> {code}
> In my case, the table comes from an Avro schema which looks like: 
> {code}  
> 'avro.schema.literal'='{\"type\":\"record\",\"name\":\"Minimal\",\"namespace\":\"org.ver.vkanalas.minimalsamp\",\"fields\":[{\"name\":\"record_type\",\"type\":\"string\"},{\"name\":\"event\",\"type\":[{\"type\":\"record\",\"name\":\"a\",\"fields\":[{\"name\":\"string_value\",\"type\":\"string\"}]},{\"type\":\"record\",\"name\":\"b\",\"fields\":[{\"name\":\"int_value\",\"type\":\"int\"}]}]}]}'
> {code}
> I wrote custom UDF (source attached) to _cast_ the union type to one of the 
> struct to access nested elements, such as {{int_value}} in my example.
> {code}
> CREATE FUNCTION toSint AS 'org.ver.udf.minimal.StructFromUnionMinimalB';
> {code}
> A simple query with the UDF is working fine. But creating a view with the 
> same select is failing when I'm trying to query it:
> {code}
> CREATE OR REPLACE VIEW minimal_sample_viewB AS SELECT toSint(event).int_value 
> FROM minimal_sample WHERE record_type = 'B';
> SELECT * FROM minimal_sample_viewB;
> {code}
> The stack trace is posted below.
> I did try to revert (or exclude) HIVE-12156 from the version I'm running and 
> this use case is working fine.
> {code}
> FAILED: SemanticException Line 0:-1 . Operator is only supported on struct or 
> list of struct types 'int_value' in definition of VIEW minimal_sample_viewb [
> SELECT null.`int_value` FROM `default`.`minimal_sample` WHERE 
> `minimal_sample`.`record_type` = 'B'
> ] used as minimal_sample_viewb at Line 3:14
> 16/01/05 22:49:41 [main]: ERROR ql.Driver: FAILED: SemanticException Line 
> 0:-1 . Operator is only supported on struct or list of struct types 
> 'int_value' in definition of VIEW minimal_sample_viewb [
> SELECT null.`int_value` FROM `default`.`minimal_sample` WHERE 
> `minimal_sample`.`record_type` = 'B'
> ] used as minimal_sample_viewb at Line 3:14
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 . Operator is 
> only supported on struct or list of struct types 'int_value' in definition of 
> VIEW minimal_sample_viewb [
> SELECT null.`int_value` FROM `default`.`minimal_sample` WHERE 
> `minimal_sample`.`record_type` = 'B'
> ] used as minimal_sample_viewb at Line 3:14
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:893)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1321)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:95)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:79)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:133)
>   at 
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:110)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:209)
>   at 
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:153)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10500)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10455)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3822)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3601)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8943)
>   at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8898)
>

[jira] [Commented] (HIVE-12660) HS2 memory leak with .hiverc file use

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090857#comment-15090857
 ] 

ASF GitHub Bot commented on HIVE-12660:
---

GitHub user thejasmn opened a pull request:

https://github.com/apache/hive/pull/64

HIVE-12660



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thejasmn/hive HIVE-12660

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/64.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #64


commit 2b64f6649efe6d1019c8b4b46953f6b9de9b1e39
Author: Thejas Nair 
Date:   2016-01-09T19:15:05Z

close operations for init file

commit a28d49227e129d2036a0a287cb9ffd997cc42358
Author: Thejas Nair 
Date:   2016-01-09T19:15:16Z

test case

commit cc25c3ffe0e0469a448164afc770f6132db1182f
Author: Thejas Nair 
Date:   2016-01-09T21:28:07Z

fix formatting

commit 7e3d82e7b7a96b743d577a419b6273cf24778231
Author: Thejas Nair 
Date:   2016-01-10T02:00:13Z

test comment




> HS2 memory leak with .hiverc file use
> -
>
> Key: HIVE-12660
> URL: https://issues.apache.org/jira/browse/HIVE-12660
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-12660.1.patch
>
>
> The Operation objects created to process .hiverc file in HS2 are not closed.
> In HiveSessionImpl, GlobalHivercFileProcessor calls executeStatementInternal 
> but ignores the OperationHandle it returns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12795) Vectorized execution causes ClassCastException

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090864#comment-15090864
 ] 

Hive QA commented on HIVE-12795:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781225/HIVE-12795.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10001 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6568/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6568/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6568/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781225 - PreCommit-HIVE-TRUNK-Build

> Vectorized execution causes ClassCastException
> --
>
> Key: HIVE-12795
> URL: https://issues.apache.org/jira/browse/HIVE-12795
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-12795.1.patch, HIVE-12795.2.patch
>
>
> In some hive versions, when
> set hive.auto.convert.join=false;
> set hive.vectorized.execution.enabled = true;
> Some join queries fail with ClassCastException:
> The stack:
> {noformat}
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.SettableStringObjectInspector
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.genVectorExpressionWritable(VectorExpressionWriterFactory.java:419)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.processVectorInspector(VectorExpressionWriterFactory.java:1102)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:55)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> {noformat}
> It can not be reproduced in hive 2.0 and 1.3 because of different code path. 
> Reproduce:
> {noformat}
> CREATE TABLE test1
>  (
>id string)
>PARTITIONED BY (
>   cr_year bigint,
>   cr_month bigint)
>  ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
> TBLPROPERTIES (
>   'serialization.null.format'='' );
>   
>   CREATE TABLE test2(
> id string
>   )
>PARTITIONED BY (
>   cr_year bigint,
>   cr_month bigint)
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
> OUTPUTFORMAT
>   

[jira] [Commented] (HIVE-12478) Improve Hive/Calcite Trasitive Predicate inference

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090885#comment-15090885
 ] 

Hive QA commented on HIVE-12478:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781259/HIVE-12478.03.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6571/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6571/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6571/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6571/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 114674b HIVE-12784: Group by SemanticException: Invalid column 
reference (Yongzhi Chen, reviewed by Xuefu Zhang)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 114674b HIVE-12784: Group by SemanticException: Invalid column 
reference (Yongzhi Chen, reviewed by Xuefu Zhang)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781259 - PreCommit-HIVE-TRUNK-Build

> Improve Hive/Calcite Trasitive Predicate inference
> --
>
> Key: HIVE-12478
> URL: https://issues.apache.org/jira/browse/HIVE-12478
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Laljo John Pullokkaran
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12478.01.patch, HIVE-12478.02.patch, 
> HIVE-12478.03.patch, HIVE-12478.patch
>
>
> HiveJoinPushTransitivePredicatesRule does not pull up predicates for 
> transitive inference if they contain more than one column.
> EXPLAIN select * from srcpart join (select ds as ds, ds as `date` from 
> srcpart where  (ds = '2008-04-08' and value=1)) s on (srcpart.ds = s.ds);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-11741) Add a new hook to run before query parse/compile

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-11741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-11741:
-
Assignee: Guilherme Braccialli  (was: Thejas M Nair)

> Add a new hook to run before query parse/compile
> 
>
> Key: HIVE-11741
> URL: https://issues.apache.org/jira/browse/HIVE-11741
> Project: Hive
>  Issue Type: New Feature
>  Components: hpl/sql, Parser, SQL
>Affects Versions: 1.2.1
>Reporter: Guilherme Braccialli
>Assignee: Guilherme Braccialli
>Priority: Minor
>  Labels: patch
> Attachments: HIVE-11741.patch
>
>
> It would be nice to allow developers to extend hive query language, making 
> possible to use custom wildcards on queries. 
> People uses Python or R to iterate over vectors or lists and create SQL 
> commands, this could be implemented directly on sql syntax.
> For example this python script:
> >>> sql = "SELECT state, "
> >>> for i in range(10):
> ...   sql += "   sum(case when type = " + str(i) + " then value end) as 
> sum_of_" + str(i) + " ,"
> ...
> >>> sql += " count(1) as  total FROM table"
> >>> print(sql)
> Could be written directly in extended sql like this:
> SELECT state,
> %for id = 1 to 10%
>sum(case when type = %id% then value end) as sum_of_%id%,
> %end%
> , count(1) as total
> FROM table
> GROUP BY state
> This kind of extensibility can be easily added if we add a new hook after 
> VariableSubstitution call on Driver.compile method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12660) HS2 memory leak with .hiverc file use

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-12660:
-
Attachment: HIVE-12660.1.patch

> HS2 memory leak with .hiverc file use
> -
>
> Key: HIVE-12660
> URL: https://issues.apache.org/jira/browse/HIVE-12660
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-12660.1.patch
>
>
> The Operation objects created to process .hiverc file in HS2 are not closed.
> In HiveSessionImpl, GlobalHivercFileProcessor calls executeStatementInternal 
> but ignores the OperationHandle it returns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12660) HS2 memory leak with .hiverc file use

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090859#comment-15090859
 ] 

Thejas M Nair commented on HIVE-12660:
--

Can you please take a look ? [~vgumashta] [~szehon] [~dongc]




> HS2 memory leak with .hiverc file use
> -
>
> Key: HIVE-12660
> URL: https://issues.apache.org/jira/browse/HIVE-12660
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.14.0, 1.0.0, 1.2.0, 1.1.0, 1.2.1
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
> Attachments: HIVE-12660.1.patch
>
>
> The Operation objects created to process .hiverc file in HS2 are not closed.
> In HiveSessionImpl, GlobalHivercFileProcessor calls executeStatementInternal 
> but ignores the OperationHandle it returns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12795) Vectorized execution causes ClassCastException

[Yongzhi Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090872#comment-15090872
 ] 

Yongzhi Chen commented on HIVE-12795:
-

The failures are not related.

> Vectorized execution causes ClassCastException
> --
>
> Key: HIVE-12795
> URL: https://issues.apache.org/jira/browse/HIVE-12795
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-12795.1.patch, HIVE-12795.2.patch
>
>
> In some hive versions, when
> set hive.auto.convert.join=false;
> set hive.vectorized.execution.enabled = true;
> Some join queries fail with ClassCastException:
> The stack:
> {noformat}
> Caused by: java.lang.ClassCastException: 
> org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector
>  cannot be cast to 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.SettableStringObjectInspector
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.genVectorExpressionWritable(VectorExpressionWriterFactory.java:419)
> at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.processVectorInspector(VectorExpressionWriterFactory.java:1102)
> at 
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:55)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
> at 
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
> at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at 
> org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
> ... 22 more
> {noformat}
> It can not be reproduced in hive 2.0 and 1.3 because of different code path. 
> Reproduce:
> {noformat}
> CREATE TABLE test1
>  (
>id string)
>PARTITIONED BY (
>   cr_year bigint,
>   cr_month bigint)
>  ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
> TBLPROPERTIES (
>   'serialization.null.format'='' );
>   
>   CREATE TABLE test2(
> id string
>   )
>PARTITIONED BY (
>   cr_year bigint,
>   cr_month bigint)
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
> TBLPROPERTIES (
>   'serialization.null.format'=''
>  );
> set hive.auto.convert.join=false;
> set hive.vectorized.execution.enabled = true;
>  SELECT cr.id1 ,
> cr.id2 
> FROM
> (SELECT t1.id id1,
>  t2.id id2
>  from
>  (select * from test1 ) t1
>  left outer join test2  t2
>  on t1.id=t2.id) cr;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12761) Add stack trace servlet to HS2 web ui

[Jimmy Xiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090875#comment-15090875
 ] 

Jimmy Xiang commented on HIVE-12761:


+1

> Add stack trace servlet to HS2 web ui
> -
>
> Key: HIVE-12761
> URL: https://issues.apache.org/jira/browse/HIVE-12761
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HIVE-12761.1.patch, HIVE-12761.2.patch, ScreenShot1.png, 
> ScreenShot2.png
>
>
> To confirm the state of HS2, I add the servlet which prints stack trace. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12590) Repeated UDAFs with literals can produce incorrect result

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090883#comment-15090883
 ] 

Hive QA commented on HIVE-12590:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781266/HIVE-12590.9.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 9998 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6570/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6570/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6570/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781266 - PreCommit-HIVE-TRUNK-Build

> Repeated UDAFs with literals can produce incorrect result
> -
>
> Key: HIVE-12590
> URL: https://issues.apache.org/jira/browse/HIVE-12590
> Project: Hive
>  Issue Type: Bug
>  Components: Query Planning
>Affects Versions: 1.0.1, 1.1.1, 1.2.1, 2.0.0
>Reporter: Laljo John Pullokkaran
>Assignee: Ashutosh Chauhan
>Priority: Critical
> Attachments: HIVE-12590.2.patch, HIVE-12590.3.patch, 
> HIVE-12590.4.patch, HIVE-12590.4.patch, HIVE-12590.5.patch, 
> HIVE-12590.6.patch, HIVE-12590.7.patch, HIVE-12590.8.patch, 
> HIVE-12590.9.patch, HIVE-12590.patch
>
>
> Repeated UDAF with literals could produce wrong result.
> This is not a common use case, nevertheless a bug.
> hive> select max('pants'), max('pANTS') from t1 group by key;
>  Total MapReduce CPU Time Spent: 0 msec
> OK
> pANTS pANTS
> pANTS pANTS
> pANTS pANTS
> pANTS pANTS
> pANTS pANTS
> Time taken: 296.252 seconds, Fetched: 5 row(s)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12834) Fix to accept the arrow keys in BeeLine CLI

[Shinichi Yamashita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shinichi Yamashita updated HIVE-12834:
--
Attachment: HIVE-12834.1.patch

There is the following statement in "bin/beeline".
{code}
if [[ ! $(ps -o stat= -p $$) =~ + ]]; then
{code}

I think this should be changed as follows.
{code}
if [[ ! $(ps -o stat= -p $$) =~ "+" ]]; then
{code}


> Fix to accept the arrow keys in BeeLine CLI
> ---
>
> Key: HIVE-12834
> URL: https://issues.apache.org/jira/browse/HIVE-12834
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
> Environment: CentOS 6.7
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
> Attachments: HIVE-12834.1.patch
>
>
> BeeLine in the master doesn't accept the arrow keys as follows (e.g. ^[[A is 
> up arrow key).
> {code}
> [root@hadoop ~]# beeline
> which: no hbase in 
> (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/hadoop/bin:/usr/local/hive/bin:/usr/pgsql-9.4/bin:/root/bin)
> Beeline version 2.1.0-SNAPSHOT by Apache Hive
> beeline> ^[[A^[[B^[[C^[[D
> {code}
> Because UnsupportedTerminal is set in the same way as background. we can 
> check with the ps command.
> {code}
> [root@hadoop ~]# ps -ef | grep beeline
> root   5799   1433  1 12:05 pts/000:00:02 /usr/lib/jvm/java/bin/java 
> -Xmx256m (snip) -Djline.terminal=jline.UnsupportedTerminal (snip) 
> org.apache.hive.beeline.BeeLine
> {code}
> I think the HIVE-6758 affected this behavior. I will fix to accept the arrow 
> keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12661) StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly

[Pengcheng Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-12661:
---
Attachment: HIVE-12661.08.patch

> StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly
> ---
>
> Key: HIVE-12661
> URL: https://issues.apache.org/jira/browse/HIVE-12661
> Project: Hive
>  Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12661.01.patch, HIVE-12661.02.patch, 
> HIVE-12661.03.patch, HIVE-12661.04.patch, HIVE-12661.05.patch, 
> HIVE-12661.06.patch, HIVE-12661.07.patch, HIVE-12661.08.patch
>
>
> PROBLEM:
> Hive stats are autogathered properly till an 'analyze table [tablename] 
> compute statistics for columns' is run. Then it does not auto-update the 
> stats till the command is run again. repo:
> {code}
> set hive.stats.autogather=true; 
> set hive.stats.atomic=false ; 
> set hive.stats.collect.rawdatasize=true ; 
> set hive.stats.collect.scancols=false ; 
> set hive.stats.collect.tablekeys=false ; 
> set hive.stats.fetch.column.stats=true; 
> set hive.stats.fetch.partition.stats=true ; 
> set hive.stats.reliable=false ; 
> set hive.compute.query.using.stats=true; 
> CREATE TABLE `default`.`calendar` (`year` int) ROW FORMAT SERDE 
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' TBLPROPERTIES ( 
> 'orc.compress'='NONE') ; 
> insert into calendar values (2010), (2011), (2012); 
> select * from calendar; 
> ++--+ 
> | calendar.year | 
> ++--+ 
> | 2010 | 
> | 2011 | 
> | 2012 | 
> ++--+ 
> select max(year) from calendar; 
> | 2012 | 
> insert into calendar values (2013); 
> select * from calendar; 
> ++--+ 
> | calendar.year | 
> ++--+ 
> | 2010 | 
> | 2011 | 
> | 2012 | 
> | 2013 | 
> ++--+ 
> select max(year) from calendar; 
> | 2013 | 
> insert into calendar values (2014); 
> select max(year) from calendar; 
> | 2014 |
> analyze table calendar compute statistics for columns;
> insert into calendar values (2015);
> select max(year) from calendar;
> | 2014 |
> insert into calendar values (2016), (2017), (2018);
> select max(year) from calendar;
> | 2014  |
> analyze table calendar compute statistics for columns;
> select max(year) from calendar;
> | 2018  |
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12807) Thrift and DB Changes for HIVE-12352

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090904#comment-15090904
 ] 

Hive QA commented on HIVE-12807:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781301/HIVE-12708.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 30 failed/errored test(s),  tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_views
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapreduce1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_merge2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_mapwork_table
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_env_var1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join0
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_string_concat
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_distinct_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_parquet
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6572/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6572/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6572/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 30 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781301 - PreCommit-HIVE-TRUNK-Build

> Thrift and DB Changes for HIVE-12352
> 
>
> Key: HIVE-12807
> URL: https://issues.apache.org/jira/browse/HIVE-12807
> Project: Hive
>  Issue Type: Sub-task
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-12708.2.patch, HIVE-12708.patch
>
>
> This ticket just covers the thrift and DB changes necessary for HIVE-12352



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9862) Vectorized execution corrupts timestamp values

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-9862:
---
Attachment: (was: HIVE-9862.04.patch)

> Vectorized execution corrupts timestamp values
> --
>
> Key: HIVE-9862
> URL: https://issues.apache.org/jira/browse/HIVE-9862
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 1.0.0
>Reporter: Nathan Howell
>Assignee: Matt McCline
> Attachments: HIVE-9862.01.patch, HIVE-9862.02.patch, 
> HIVE-9862.03.patch, HIVE-9862.04.patch
>
>
> Timestamps in the future (year 2250?) and before ~1700 are silently corrupted 
> in vectorized execution mode. Simple repro:
> {code}
> hive> DROP TABLE IF EXISTS test;
> hive> CREATE TABLE test(ts TIMESTAMP) STORED AS ORC;
> hive> INSERT INTO TABLE test VALUES ('-12-31 23:59:59');
> hive> SET hive.vectorized.execution.enabled = false;
> hive> SELECT MAX(ts) FROM test;
> -12-31 23:59:59
> hive> SET hive.vectorized.execution.enabled = true;
> hive> SELECT MAX(ts) FROM test;
> 1816-03-30 05:56:07.066277376
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-9862) Vectorized execution corrupts timestamp values

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-9862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-9862:
---
Attachment: HIVE-9862.04.patch

> Vectorized execution corrupts timestamp values
> --
>
> Key: HIVE-9862
> URL: https://issues.apache.org/jira/browse/HIVE-9862
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Affects Versions: 1.0.0
>Reporter: Nathan Howell
>Assignee: Matt McCline
> Attachments: HIVE-9862.01.patch, HIVE-9862.02.patch, 
> HIVE-9862.03.patch, HIVE-9862.04.patch
>
>
> Timestamps in the future (year 2250?) and before ~1700 are silently corrupted 
> in vectorized execution mode. Simple repro:
> {code}
> hive> DROP TABLE IF EXISTS test;
> hive> CREATE TABLE test(ts TIMESTAMP) STORED AS ORC;
> hive> INSERT INTO TABLE test VALUES ('-12-31 23:59:59');
> hive> SET hive.vectorized.execution.enabled = false;
> hive> SELECT MAX(ts) FROM test;
> -12-31 23:59:59
> hive> SET hive.vectorized.execution.enabled = true;
> hive> SELECT MAX(ts) FROM test;
> 1816-03-30 05:56:07.066277376
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-3764) Support metastore version consistency check

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090494#comment-15090494
 ] 

Lefty Leverenz commented on HIVE-3764:
--

Doc note:  This adds two configuration parameters to HiveConf.java in 0.12.0.  
They're documented in the wiki here:

* [Configuration Properties -- datanucleus.fixedDatastore | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-datanucleus.fixedDatastore]
* [Configuration Properties -- hive.metastore.schema.verification | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.schema.verification]

Their default values are changed in release 2.0.0 by HIVE-1841 and HIVE-12436, 
respectively.

> Support metastore version consistency check
> ---
>
> Key: HIVE-3764
> URL: https://issues.apache.org/jira/browse/HIVE-3764
> Project: Hive
>  Issue Type: Improvement
>  Components: Metastore
>Affects Versions: 0.8.0, 0.9.0, 0.10.0, 0.11.0
>Reporter: Prasad Mujumdar
>Assignee: Prasad Mujumdar
> Fix For: 0.12.0
>
> Attachments: HIVE-3764-12.3.patch, HIVE-3764-trunk.3.patch, 
> HIVE-3764.1.patch, HIVE-3764.2.patch
>
>
> Today there's no version/compatibility information stored in hive metastore. 
> Also the datanucleus configuration property to automatically create missing 
> tables is enabled by default. If you happen to start an older or newer hive 
> or don't run the correct upgrade scripts during migration, the metastore 
> would end up corrupted. The autoCreate schema is not always sufficient to 
> upgrade metastore when migrating to newer release. It's not supported with 
> all databases. Besides the migration often involves altering existing table, 
> changing or moving data etc.
> Hence it's very useful to have some consistency check to make sure that hive 
> is using correct metastore and for production systems the schema is not 
> automatically by running hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12436) Default hive.metastore.schema.verification to true

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090513#comment-15090513
 ] 

Lefty Leverenz commented on HIVE-12436:
---

HIVE-6113 changes the default for hive.metastore.schema.verification back to 
false in 2.0.0, so this issue had no effect on that default value.

> Default hive.metastore.schema.verification to true
> --
>
> Key: HIVE-12436
> URL: https://issues.apache.org/jira/browse/HIVE-12436
> Project: Hive
>  Issue Type: Task
>  Components: Metastore
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
>  Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12436.patch
>
>
> It enforces metastore schema version consistency



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12758) Parallel compilation: Operator::resetId() is not thread-safe

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090522#comment-15090522
 ] 

Hive QA commented on HIVE-12758:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781023/HIVE-12758.03.patch

{color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 25 failed/errored test(s),  tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cross_product_check_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_multi_insert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.testOutputFormat
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6559/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6559/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6559/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781023 - PreCommit-HIVE-TRUNK-Build

> Parallel compilation: Operator::resetId() is not thread-safe
> 
>
> Key: HIVE-12758
> URL: https://issues.apache.org/jira/browse/HIVE-12758
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12758.01.patch, HIVE-12758.02.patch, 
> HIVE-12758.03.patch, HIVE-12758.patch
>
>
> {code}
>   private static AtomicInteger seqId;
> ...
>   public Operator() {
> this(String.valueOf(seqId.getAndIncrement()));
>   }
>   public static void resetId() {
> seqId.set(0);
>   }
> {code}
> Potential race-condition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-12436) Default hive.metastore.schema.verification to true

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-12436:
--
Labels:   (was: TODOC2.0)

> Default hive.metastore.schema.verification to true
> --
>
> Key: HIVE-12436
> URL: https://issues.apache.org/jira/browse/HIVE-12436
> Project: Hive
>  Issue Type: Task
>  Components: Metastore
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.0.0
>
> Attachments: HIVE-12436.patch
>
>
> It enforces metastore schema version consistency



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12436) Default hive.metastore.schema.verification to true

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090524#comment-15090524
 ] 

Lefty Leverenz commented on HIVE-12436:
---

Thanks Ashutosh, I'm removing the TODOC2.0 label.

> Default hive.metastore.schema.verification to true
> --
>
> Key: HIVE-12436
> URL: https://issues.apache.org/jira/browse/HIVE-12436
> Project: Hive
>  Issue Type: Task
>  Components: Metastore
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Fix For: 2.0.0
>
> Attachments: HIVE-12436.patch
>
>
> It enforces metastore schema version consistency



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12442) Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks

[Vaibhav Gumashta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090525#comment-15090525
 ] 

Vaibhav Gumashta commented on HIVE-12442:
-

[~rohitdholakia] Can you also update the rb?

> Refactor/repackage HiveServer2's Thrift code so that it can be used in the 
> tasks
> 
>
> Key: HIVE-12442
> URL: https://issues.apache.org/jira/browse/HIVE-12442
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 1.2.1
>Reporter: Vaibhav Gumashta
>Assignee: Rohit Dholakia
>  Labels: HiveServer2, RPC, Thrift
> Fix For: 2.1.0
>
> Attachments: hive-12442.1.patch, hive-12442.2.patch, 
> hive-12442.3.patch, hive-12442.4.patch, hive-12442.5.patch
>
>
> For implementing HIVE-12427, the tasks will need to have knowledge of thrift 
> types from HS2's thrift API. This jira will look at the least invasive way to 
> do that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HIVE-1841) datanucleus.fixedDatastore should be true in hive-default.xml

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-1841:
-
Labels:   (was: TODOC2.0)

>  datanucleus.fixedDatastore should be true in hive-default.xml
> --
>
> Key: HIVE-1841
> URL: https://issues.apache.org/jira/browse/HIVE-1841
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration, Metastore
>Affects Versions: 0.6.0
>Reporter: Edward Capriolo
>Assignee: Ashutosh Chauhan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HIVE-1841.1.patch.txt, HIVE-1841.2.patch, HIVE-1841.patch
>
>
> Two datanucleus variables:
> {noformat}
> 
>  datanucleus.autoCreateSchema
>  false
> 
> 
>  datanucleus.fixedDatastore
>  true
> 
> {noformat}
> are dangerous.  We do want the schema to auto-create itself, but we do not 
> want the schema to auto update itself. 
> Someone might accidentally point a trunk at the wrong meta-store and 
> unknowingly update. I believe we should set this to false and possibly trap 
> exceptions stemming from hive wanting to do any update. This way someone has 
> to actively acknowledge the update, by setting this to true and then starting 
> up hive, or leaving it false, removing schema modifies for the user that hive 
> usages, and doing all the time and doing the updates by hand. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-12788) Setting hive.optimize.union.remove to TRUE will break UNION ALL with aggregate functions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-12788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090936#comment-15090936
 ] 

Hive QA commented on HIVE-12788:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12781288/HIVE-12788.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10001 tests 
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_remove_26
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6573/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6573/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6573/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12781288 - PreCommit-HIVE-TRUNK-Build

> Setting hive.optimize.union.remove to TRUE will break UNION ALL with 
> aggregate functions
> 
>
> Key: HIVE-12788
> URL: https://issues.apache.org/jira/browse/HIVE-12788
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 1.1.1
>Reporter: Eric Lin
>Assignee: Chaoyu Tang
> Attachments: HIVE-12788.patch
>
>
> See the test case below:
> {code}
> 0: jdbc:hive2://localhost:1/default> create table test (a int);
> 0: jdbc:hive2://localhost:1/default> insert overwrite table test values 
> (1);
> 0: jdbc:hive2://localhost:1/default> set hive.optimize.union.remove=true;
> No rows affected (0.01 seconds)
> 0: jdbc:hive2://localhost:1/default> set 
> hive.mapred.supports.subdirectories=true;
> No rows affected (0.007 seconds)
> 0: jdbc:hive2://localhost:1/default> SELECT COUNT(1) FROM test UNION ALL 
> SELECT COUNT(1) FROM test;
> +--+--+
> | _u1._c0  |
> +--+--+
> +--+--+
> {code}
> UNION ALL without COUNT function will work as expected:
> {code}
> 0: jdbc:hive2://localhost:1/default> select * from test UNION ALL SELECT 
> * FROM test;
> ++--+
> | _u1.a  |
> ++--+
> | 1  |
> | 1  |
> ++--+
> {code}
> Run the same query without setting hive.mapred.supports.subdirectories and 
> hive.optimize.union.remove to true will give correct result:
> {code}
> 0: jdbc:hive2://localhost:1/default> set hive.optimize.union.remove;
> +---+--+
> |set|
> +---+--+
> | hive.optimize.union.remove=false  |
> +---+--+
> 0: jdbc:hive2://localhost:1/default> SELECT COUNT(1) FROM test UNION ALL 
> SELECT COUNT(1) FROM test;
> +--+--+
> | _u1._c0  |
> +--+--+
> | 1|
> | 1|
> +--+--+
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)