[jira] [Commented] (HIVE-21126) Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call
[
https://issues.apache.org/jira/browse/HIVE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743716#comment-16743716
]
Hive QA commented on HIVE-21126:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955042/HIVE-21126.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15694 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=190)
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=220)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15636/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15636/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15636/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955042 - PreCommit-HIVE-Build
> Allow session level queries in LlapBaseInputFormat#getSplits() before actual
> get_splits() call
> --
>
> Key: HIVE-21126
> URL: https://issues.apache.org/jira/browse/HIVE-21126
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21126.1.patch
>
>
> Facilitate execution of session level queries before \{{select get_splits()}}
> call. This will allow us to set params like \{{tez.grouping.split-count}}
> which can be taken into consideration while splits calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Attachment: HIVE-21078.05.patch Status: Patch Available (was: In Progress) Patch with fix for previous test failures. When an UpdateTableColStat messages is processed for dump the underlying table may have been dropped. In this case, Utils.shouldReplicate() checks will fail with "no table found" error. But we require the Table object for checks in Utils.shouldReplicate(). Hence include the table whose column statistics is being updated in the event message. > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch, HIVE-21078.04.patch, HIVE-21078.05.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Status: In Progress (was: Patch Available) > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch, HIVE-21078.04.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21126) Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call
[
https://issues.apache.org/jira/browse/HIVE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743683#comment-16743683
]
Hive QA commented on HIVE-21126:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} llap-ext-client generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:llap-ext-client |
| | org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(JobConf, int)
passes a nonconstant String to an execute or addBatch method on an SQL
statement At LlapBaseInputFormat.java:String to an execute or addBatch method
on an SQL statement At LlapBaseInputFormat.java:[line 272] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15636/dev-support/hive-personality.sh
|
| git revision | master / a3aa074 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15636/yetus/new-findbugs-llap-ext-client.html
|
| modules | C: llap-ext-client U: llap-ext-client |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15636/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow session level queries in LlapBaseInputFormat#getSplits() before actual
> get_splits() call
> --
>
> Key: HIVE-21126
> URL: https://issues.apache.org/jira/browse/HIVE-21126
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21126.1.patch
>
>
> Facilitate execution of session level queries before \{{select get_splits()}}
> call. This will allow us to set params like \{{tez.grouping.split-count}}
> which can be taken into consideration while splits calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-21126) Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call
[
https://issues.apache.org/jira/browse/HIVE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-21126 started by Shubham Chaurasia.
> Allow session level queries in LlapBaseInputFormat#getSplits() before actual
> get_splits() call
> --
>
> Key: HIVE-21126
> URL: https://issues.apache.org/jira/browse/HIVE-21126
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
>
> Facilitate execution of session level queries before \{{select get_splits()}}
> call. This will allow us to set params like \{{tez.grouping.split-count}}
> which can be taken into consideration while splits calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21126) Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call
[
https://issues.apache.org/jira/browse/HIVE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia updated HIVE-21126:
-
Attachment: HIVE-21126.1.patch
Status: Patch Available (was: In Progress)
> Allow session level queries in LlapBaseInputFormat#getSplits() before actual
> get_splits() call
> --
>
> Key: HIVE-21126
> URL: https://issues.apache.org/jira/browse/HIVE-21126
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21126.1.patch
>
>
> Facilitate execution of session level queries before \{{select get_splits()}}
> call. This will allow us to set params like \{{tez.grouping.split-count}}
> which can be taken into consideration while splits calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21126) Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call
[
https://issues.apache.org/jira/browse/HIVE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21126:
--
Labels: pull-request-available (was: )
> Allow session level queries in LlapBaseInputFormat#getSplits() before actual
> get_splits() call
> --
>
> Key: HIVE-21126
> URL: https://issues.apache.org/jira/browse/HIVE-21126
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
>
> Facilitate execution of session level queries before \{{select get_splits()}}
> call. This will allow us to set params like \{{tez.grouping.split-count}}
> which can be taken into consideration while splits calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21126) Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call
[
https://issues.apache.org/jira/browse/HIVE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743644#comment-16743644
]
ASF GitHub Bot commented on HIVE-21126:
---
GitHub user ShubhamChaurasia opened a pull request:
https://github.com/apache/hive/pull/515
HIVE-21126: Allow session level queries in LlapBaseInputFormat#getSplits
Provides a way to pass queries to execute before get_splits() is executed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ShubhamChaurasia/hive HIVE-21126
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/515.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #515
commit 4cf1bf4d95d6eec534c5ae4f56073faad3d12bd2
Author: ShubhamChaurasia
Date: 2019-01-16T05:45:16Z
HIVE-21126: Allow session level queries in LlapBaseInputFormat#getSplits()
before actual get_splits() call
> Allow session level queries in LlapBaseInputFormat#getSplits() before actual
> get_splits() call
> --
>
> Key: HIVE-21126
> URL: https://issues.apache.org/jira/browse/HIVE-21126
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
>
> Facilitate execution of session level queries before \{{select get_splits()}}
> call. This will allow us to set params like \{{tez.grouping.split-count}}
> which can be taken into consideration while splits calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21126) Allow session level queries in LlapBaseInputFormat#getSplits() before actual get_splits() call
[
https://issues.apache.org/jira/browse/HIVE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia reassigned HIVE-21126:
> Allow session level queries in LlapBaseInputFormat#getSplits() before actual
> get_splits() call
> --
>
> Key: HIVE-21126
> URL: https://issues.apache.org/jira/browse/HIVE-21126
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>
> Facilitate execution of session level queries before \{{select get_splits()}}
> call. This will allow us to set params like \{{tez.grouping.split-count}}
> which can be taken into consideration while splits calculation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[
https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743603#comment-16743603
]
Hive QA commented on HIVE-17751:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12930763/HIVE-17751.15.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15635/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15635/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15635/
Messages:
{noformat}
This message was trimmed, see log for full details
error:
standalone-metastore/src/main/sql/derby/upgrade-1.2.0-to-2.0.0.derby.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/derby/upgrade-2.0.0-to-2.1.0.derby.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/derby/upgrade-2.1.0-to-2.2.0.derby.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/derby/upgrade-2.2.0-to-2.3.0.derby.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/derby/upgrade-3.1.0-to-4.0.0.derby.sql: does
not exist in index
error: standalone-metastore/src/main/sql/derby/upgrade.order.derby: does not
exist in index
error: standalone-metastore/src/main/sql/mssql/create-user.mssql.sql: does not
exist in index
error: standalone-metastore/src/main/sql/mssql/hive-schema-1.2.0.mssql.sql:
does not exist in index
error: standalone-metastore/src/main/sql/mssql/hive-schema-3.0.0.mssql.sql:
does not exist in index
error: standalone-metastore/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql:
does not exist in index
error: standalone-metastore/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/src/main/sql/mssql/upgrade-1.2.0-to-2.0.0.mssql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mssql/upgrade-2.0.0-to-2.1.0.mssql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mssql/upgrade-2.1.0-to-2.2.0.mssql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mssql/upgrade-2.2.0-to-2.3.0.mssql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mssql/upgrade-3.1.0-to-4.0.0.mssql.sql: does
not exist in index
error: standalone-metastore/src/main/sql/mssql/upgrade.order.mssql: does not
exist in index
error: standalone-metastore/src/main/sql/mysql/create-user.mysql.sql: does not
exist in index
error: standalone-metastore/src/main/sql/mysql/hive-schema-1.2.0.mysql.sql:
does not exist in index
error: standalone-metastore/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql:
does not exist in index
error: standalone-metastore/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql:
does not exist in index
error: standalone-metastore/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/src/main/sql/mysql/upgrade-1.2.0-to-2.0.0.mysql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mysql/upgrade-2.0.0-to-2.1.0.mysql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mysql/upgrade-2.1.0-to-2.2.0.mysql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mysql/upgrade-2.2.0-to-2.3.0.mysql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql: does
not exist in index
error:
standalone-metastore/src/main/sql/mysql/upgrade-3.1.0-to-4.0.0.mysql.sql: does
not exist in index
error: standalone-metastore/src/main/sql/mysql/upgrade.order.mysql: does not
exist in index
error: standalone-metastore/src/main/sql/oracle/create-user.oracle.sql: does
not exist in index
error: standalone-metastore/src/main/sql/oracle/hive-schema-1.2.0.oracle.sql:
does not exist in index
error: standalone-metastore/src/main/sql/oracle/hive-schema-3.0.0.oracle.sql:
does not exist in index
error: standalone-metastore/src/main/sql/oracle/hive-schema-3.1.0.oracle.sql:
does not exist in index
error: standalone-metastore/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql:
does not exist in index
error:
standalone-metastore/src/main/sql/oracle/upgrade-1.2.0-to-2.0.0.oracle.sql:
does not exist in index
error:
standalone-metastore/src/main/sql/oracle/upgrade-2.0.0-to-2.1.0.oracle.sql:
does not exist
[jira] [Commented] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743602#comment-16743602
]
Hive QA commented on HIVE-19253:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12943121/HIVE-19253.12.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15634/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15634/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15634/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-16 04:22:42.559
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15634/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-16 04:22:42.562
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-client/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-16 04:22:43.591
+ rm -rf ../yetus_PreCommit-HIVE-Build-15634
+ mkdir ../yetus_PreCommit-HIVE-Build-15634
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15634
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15634/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/hcatalog/webhcat/java-client/src/main/java/org/apache/hive/hcatalog/api/HCatTable.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java: does not
exist in index
error:
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java:
does not exist in index
error:
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java:
does not exist in index
error: patch failed:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java:86
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java:86
Falling back to three-way merge...
[jira] [Commented] (HIVE-20189) Separate metastore client code into its own module
[
https://issues.apache.org/jira/browse/HIVE-20189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743599#comment-16743599
]
Hive QA commented on HIVE-20189:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944709/HIVE-20189.06.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15694 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15633/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15633/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15633/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944709 - PreCommit-HIVE-Build
> Separate metastore client code into its own module
> --
>
> Key: HIVE-20189
> URL: https://issues.apache.org/jira/browse/HIVE-20189
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20189.01.patch, HIVE-20189.03.patch,
> HIVE-20189.04.patch, HIVE-20189.05.patch, HIVE-20189.06.patch
>
>
> The goal of this JIRA is to split HiveMetastoreClient code out of
> metastore-common. This is a pom-only change that does not require any changes
> in the code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20189) Separate metastore client code into its own module
[
https://issues.apache.org/jira/browse/HIVE-20189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743588#comment-16743588
]
Hive QA commented on HIVE-20189:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
7s{color} | {color:blue} standalone-metastore/metastore-common in master has 29
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
1s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} llap-server in master has 81 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch standalone-metastore passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} standalone-metastore/metastore-client: The patch
generated 2226 new + 0 unchanged - 0 fixed = 2226 total (was 0) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} llap-server: The patch generated 0 new + 8 unchanged
- 1 fixed = 8 total (was 9) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
15s{color} | {color:red} standalone-metastore/metastore-common generated 4 new
+ 25 unchanged - 4 fixed = 29 total (was 29) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
4s{color} | {color:red} standalone-metastore generated 1 new + 67 unchanged - 1
fixed = 68 total (was 68) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} standalone-metastore_metastore-common generated 0
new + 16 unchanged - 1 fixed = 16 total (was 17) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} standalone-metastore_metastore-client generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} metastore in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} metastore-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color
[jira] [Assigned] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov reassigned HIVE-19253:
-
Assignee: Vihang Karajgaonkar (was: Alexander Kolbasov)
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: newbie
> Attachments: HIVE-19253.01.patch, HIVE-19253.02.patch,
> HIVE-19253.03.patch, HIVE-19253.03.patch, HIVE-19253.04.patch,
> HIVE-19253.05.patch, HIVE-19253.06.patch, HIVE-19253.07.patch,
> HIVE-19253.08.patch, HIVE-19253.09.patch, HIVE-19253.10.patch,
> HIVE-19253.11.patch, HIVE-19253.12.patch
>
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20189) Separate metastore client code into its own module
[ https://issues.apache.org/jira/browse/HIVE-20189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned HIVE-20189: - Assignee: Vihang Karajgaonkar (was: Alexander Kolbasov) > Separate metastore client code into its own module > -- > > Key: HIVE-20189 > URL: https://issues.apache.org/jira/browse/HIVE-20189 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Affects Versions: 4.0.0, 3.2.0 >Reporter: Alexander Kolbasov >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20189.01.patch, HIVE-20189.03.patch, > HIVE-20189.04.patch, HIVE-20189.05.patch, HIVE-20189.06.patch > > > The goal of this JIRA is to split HiveMetastoreClient code out of > metastore-common. This is a pom-only change that does not require any changes > in the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20404) Split HMS security classes into client and server parts
[ https://issues.apache.org/jira/browse/HIVE-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned HIVE-20404: - Assignee: Vihang Karajgaonkar (was: Alexander Kolbasov) > Split HMS security classes into client and server parts > --- > > Key: HIVE-20404 > URL: https://issues.apache.org/jira/browse/HIVE-20404 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alexander Kolbasov >Assignee: Vihang Karajgaonkar >Priority: Major > > Currently many security-related classes handle both client and server side > together. We would like to separate these into server and client components > so that clients do not need server-related code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned HIVE-17751: - Assignee: Peter Vary (was: Alexander Kolbasov) > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-17751.01.patch, HIVE-17751.02.patch, > HIVE-17751.03.patch, HIVE-17751.04.patch, HIVE-17751.07.patch, > HIVE-17751.08.patch, HIVE-17751.09.patch, HIVE-17751.10.patch, > HIVE-17751.11.patch, HIVE-17751.13.patch, HIVE-17751.14.patch, > HIVE-17751.15.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20198) Constant time table drops/renames
[ https://issues.apache.org/jira/browse/HIVE-20198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743514#comment-16743514 ] Vihang Karajgaonkar commented on HIVE-20198: This somehow slipped through the cracks. I will assign it myself and see if I can take a first stab at it. > Constant time table drops/renames > - > > Key: HIVE-20198 > URL: https://issues.apache.org/jira/browse/HIVE-20198 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Alexander Kolbasov >Priority: Major > > Currently table drops and table renames have O(P) performance (where P is the > number of partitions). When a managed table is deleted, the implementation > deletes table metadata and then deletes all partitions in HDFS. HDFS > operations are optimized and only do a sequential deletes for partitions > outside of table prefix. This operation is O(P)where Pis the number of > partitions. > Table rename goes through the list of partitions and modifies table name (and > potentially db name) in each partition. It also modifies each partition > location to match the new db/table name and renames directories (which is a > non-atomic and slow operation on S3). This is O(P) operation where P is the > number of partitions. > Basic idea is to do the following: > # Assign unique ID to each table > # Create directory name based on unique ID rather then the name > # Table rename then becomes metadata-only operation - there is no need to > change any location information. > # Table drop can become an asynchronous operation where the table is marked > as "deleted". Subsequent public metadata APIs should skip such tables. A > background cleaner thread may then go and clean up directories. > Since the table location is unique for each table, new tables will not reuse > existing locations. This change isn't compatible with the current behavior > where there is an assumption that table location is based on table name. We > can get around this by providing "opt-in" mechanism - special table property > that tells that the table can have such new behavior, so the improvement will > initially work for new tables created with this feature enabled. We may later > provide some tool to convert existing tables to the new scheme. > One complication is there in case where impersonation is enabled - the FS > operations should be performed using client UGI rather then server's, so the > cleaner thread should be able to use client UGIs. > Initially we can punt on this and do standard table drops when impersonation > is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21124) HPL/SQL does not support the CREATE TABLE LIKE statement
[ https://issues.apache.org/jira/browse/HIVE-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743530#comment-16743530 ] Baoning He commented on HIVE-21124: --- HI，[~daijy] could you please review ? Thank you! > HPL/SQL does not support the CREATE TABLE LIKE statement > > > Key: HIVE-21124 > URL: https://issues.apache.org/jira/browse/HIVE-21124 > Project: Hive > Issue Type: Bug > Components: hpl/sql >Reporter: Baoning He >Assignee: Baoning He >Priority: Major > Attachments: HIVE-21124.1.patch > > > Hive supports the CREATE TABLE LIKE statement but HPL/SQL does not support. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21115) Add support for object versions in metastore
[
https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743500#comment-16743500
]
Vihang Karajgaonkar commented on HIVE-21115:
Thanks [~odraese]. You may a fair point. I took a look the you pointed above
and here are lines which are interesting with respect to generating the version
numbers using transaction handler.
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L589
{code}
String s = sqlGenerator.addForUpdateClause("select ntxn_next from
NEXT_TXN_ID");
LOG.debug("Going to execute query <" + s + ">");
rs = stmt.executeQuery(s);
if (!rs.next()) {
throw new MetaException("Transaction database not properly " +
"configured, can't find next transaction id.");
}
long first = rs.getLong(1);
s = "update NEXT_TXN_ID set ntxn_next = " + (first + numTxns);
LOG.debug("Going to execute update <" + s + ">");
stmt.executeUpdate(s);
{code}
Although in theory we can use this same logic to generate object versions but
this is different which can have potentially severe performance degradation.
Note that the transaction_ids are unique across all the objects and instances
of metastores. This means all the transactions are serialized to generate these
ids in metastore. While this could be used to generate a globally incrementing
version number it may have some adverse performance implications. Having a
version at a per-object level instead reduces the scope of the lock and only
the transactions which are operating on the same object will be serialized. I
think as a mid-way we can implement the versioning for non-acid enabled tables.
If the table is ACID enabled, it should use the transaction ids for its
versioning. I am hesitant to use this logic above for all the objects which
indirectly is going to serialize all the DML transactions in metastore. Perhaps
someone more familiar with this code can comment on my thoughts above.
[~ekoifman] [~alangates]?
> Add support for object versions in metastore
>
>
> Key: HIVE-21115
> URL: https://issues.apache.org/jira/browse/HIVE-21115
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Currently, metastore objects are identified uniquely by their names (eg.
> catName, dbName and tblName for a table is unique). Once a table or partition
> is created it could be altered in many ways. There is no good way currently
> to identify the version of the object once it is altered. For example,
> suppose there are two clients (Hive and Impala) using the same metastore.
> Once some alter operations are performed by a client, another client which
> wants to do a alter operation has no good way to know if the object which it
> has is the same as the one stored in metastore. Metastore updates the
> {{transient_lastDdlTime}} every time there is a DDL operation on the object.
> However, this value cannot be relied for all the clients since after
> HIVE-1768 metastore updates the value only when it is not set in the
> parameters. It is possible that a client which alters the object state, does
> not remove the {{transient_lastDdlTime}} and metastore will not update it.
> Secondly, if there is a clock skew between multiple HMS instances when HMS-HA
> is configured, time values cannot be relied on to find out the sequence of
> alter operations on a given object.
> This JIRA propose to use JDO versioning support by Datanucleus
> http://www.datanucleus.org/products/accessplatform_4_2/jdo/versioning.html to
> generate a incrementing sequence number every time a object is altered. The
> value of this object can be set as one of the values in the parameters. The
> advantage of using Datanucleus the versioning can be done across HMS
> instances as part of the database transaction and it should work for all the
> supported databases.
> In theory such a version can be used to detect if the client is presenting a
> object which is "stale" when issuing a alter request. Metastore can choose to
> reject such a alter request since the client may be caching a old version of
> the object and any alter operation on such stale object can potentially
> overwrite previous operations. However, this is can be done in a separate
> JIRA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20198) Constant time table drops/renames
[ https://issues.apache.org/jira/browse/HIVE-20198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-20198: -- Assignee: Vihang Karajgaonkar > Constant time table drops/renames > - > > Key: HIVE-20198 > URL: https://issues.apache.org/jira/browse/HIVE-20198 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Vihang Karajgaonkar >Priority: Major > > Currently table drops and table renames have O(P) performance (where P is the > number of partitions). When a managed table is deleted, the implementation > deletes table metadata and then deletes all partitions in HDFS. HDFS > operations are optimized and only do a sequential deletes for partitions > outside of table prefix. This operation is O(P)where Pis the number of > partitions. > Table rename goes through the list of partitions and modifies table name (and > potentially db name) in each partition. It also modifies each partition > location to match the new db/table name and renames directories (which is a > non-atomic and slow operation on S3). This is O(P) operation where P is the > number of partitions. > Basic idea is to do the following: > # Assign unique ID to each table > # Create directory name based on unique ID rather then the name > # Table rename then becomes metadata-only operation - there is no need to > change any location information. > # Table drop can become an asynchronous operation where the table is marked > as "deleted". Subsequent public metadata APIs should skip such tables. A > background cleaner thread may then go and clean up directories. > Since the table location is unique for each table, new tables will not reuse > existing locations. This change isn't compatible with the current behavior > where there is an assumption that table location is based on table name. We > can get around this by providing "opt-in" mechanism - special table property > that tells that the table can have such new behavior, so the improvement will > initially work for new tables created with this feature enabled. We may later > provide some tool to convert existing tables to the new scheme. > One complication is there in case where impersonation is enabled - the FS > operations should be performed using client UGI rather then server's, so the > cleaner thread should be able to use client UGIs. > Initially we can punt on this and do standard table drops when impersonation > is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21115) Add support for object versions in metastore
[
https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743500#comment-16743500
]
Vihang Karajgaonkar edited comment on HIVE-21115 at 1/16/19 1:12 AM:
-
Thanks [~odraese]. You make a fair point. I took a look at the code you pointed
above and here are lines which are interesting with respect to generating the
version numbers using transaction handler.
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L589
{code}
String s = sqlGenerator.addForUpdateClause("select ntxn_next from
NEXT_TXN_ID");
LOG.debug("Going to execute query <" + s + ">");
rs = stmt.executeQuery(s);
if (!rs.next()) {
throw new MetaException("Transaction database not properly " +
"configured, can't find next transaction id.");
}
long first = rs.getLong(1);
s = "update NEXT_TXN_ID set ntxn_next = " + (first + numTxns);
LOG.debug("Going to execute update <" + s + ">");
stmt.executeUpdate(s);
{code}
Although in theory we can use this same logic to generate object versions but
this is different which can have potentially severe performance degradation.
Note that the transaction_ids are unique across all the objects and instances
of metastores. This means all the transactions are serialized to generate these
ids in metastore. While this could be used to generate a globally incrementing
version number it may have some adverse performance implications. Having a
version at a per-object level instead reduces the scope of the lock and only
the transactions which are operating on the same object will be serialized. I
think as a mid-way we can implement the versioning for non-acid enabled tables.
If the table is ACID enabled, it should use the transaction ids for its
versioning. I am hesitant to use this logic above for all the objects which
indirectly is going to serialize all the DML transactions in metastore. Perhaps
someone more familiar with this code can comment on my thoughts above.
[~ekoifman] [~alangates]?
was (Author: vihangk1):
Thanks [~odraese]. You make a fair point. I took a look the you pointed above
and here are lines which are interesting with respect to generating the version
numbers using transaction handler.
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L589
{code}
String s = sqlGenerator.addForUpdateClause("select ntxn_next from
NEXT_TXN_ID");
LOG.debug("Going to execute query <" + s + ">");
rs = stmt.executeQuery(s);
if (!rs.next()) {
throw new MetaException("Transaction database not properly " +
"configured, can't find next transaction id.");
}
long first = rs.getLong(1);
s = "update NEXT_TXN_ID set ntxn_next = " + (first + numTxns);
LOG.debug("Going to execute update <" + s + ">");
stmt.executeUpdate(s);
{code}
Although in theory we can use this same logic to generate object versions but
this is different which can have potentially severe performance degradation.
Note that the transaction_ids are unique across all the objects and instances
of metastores. This means all the transactions are serialized to generate these
ids in metastore. While this could be used to generate a globally incrementing
version number it may have some adverse performance implications. Having a
version at a per-object level instead reduces the scope of the lock and only
the transactions which are operating on the same object will be serialized. I
think as a mid-way we can implement the versioning for non-acid enabled tables.
If the table is ACID enabled, it should use the transaction ids for its
versioning. I am hesitant to use this logic above for all the objects which
indirectly is going to serialize all the DML transactions in metastore. Perhaps
someone more familiar with this code can comment on my thoughts above.
[~ekoifman] [~alangates]?
> Add support for object versions in metastore
>
>
> Key: HIVE-21115
> URL: https://issues.apache.org/jira/browse/HIVE-21115
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Currently, metastore objects are identified uniquely by their names (eg.
> catName, dbName and tblName for a table is unique). Once a table or partition
> is created it could be altered in many ways. There is no good way currently
> to identify the version of the object once it is altered. For example,
> suppose there are two clients (Hive and Impala) using the same metastore.
> Once some alter operations are performed by a clie
[jira] [Comment Edited] (HIVE-21115) Add support for object versions in metastore
[
https://issues.apache.org/jira/browse/HIVE-21115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743500#comment-16743500
]
Vihang Karajgaonkar edited comment on HIVE-21115 at 1/16/19 1:04 AM:
-
Thanks [~odraese]. You make a fair point. I took a look the you pointed above
and here are lines which are interesting with respect to generating the version
numbers using transaction handler.
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L589
{code}
String s = sqlGenerator.addForUpdateClause("select ntxn_next from
NEXT_TXN_ID");
LOG.debug("Going to execute query <" + s + ">");
rs = stmt.executeQuery(s);
if (!rs.next()) {
throw new MetaException("Transaction database not properly " +
"configured, can't find next transaction id.");
}
long first = rs.getLong(1);
s = "update NEXT_TXN_ID set ntxn_next = " + (first + numTxns);
LOG.debug("Going to execute update <" + s + ">");
stmt.executeUpdate(s);
{code}
Although in theory we can use this same logic to generate object versions but
this is different which can have potentially severe performance degradation.
Note that the transaction_ids are unique across all the objects and instances
of metastores. This means all the transactions are serialized to generate these
ids in metastore. While this could be used to generate a globally incrementing
version number it may have some adverse performance implications. Having a
version at a per-object level instead reduces the scope of the lock and only
the transactions which are operating on the same object will be serialized. I
think as a mid-way we can implement the versioning for non-acid enabled tables.
If the table is ACID enabled, it should use the transaction ids for its
versioning. I am hesitant to use this logic above for all the objects which
indirectly is going to serialize all the DML transactions in metastore. Perhaps
someone more familiar with this code can comment on my thoughts above.
[~ekoifman] [~alangates]?
was (Author: vihangk1):
Thanks [~odraese]. You may a fair point. I took a look the you pointed above
and here are lines which are interesting with respect to generating the version
numbers using transaction handler.
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L589
{code}
String s = sqlGenerator.addForUpdateClause("select ntxn_next from
NEXT_TXN_ID");
LOG.debug("Going to execute query <" + s + ">");
rs = stmt.executeQuery(s);
if (!rs.next()) {
throw new MetaException("Transaction database not properly " +
"configured, can't find next transaction id.");
}
long first = rs.getLong(1);
s = "update NEXT_TXN_ID set ntxn_next = " + (first + numTxns);
LOG.debug("Going to execute update <" + s + ">");
stmt.executeUpdate(s);
{code}
Although in theory we can use this same logic to generate object versions but
this is different which can have potentially severe performance degradation.
Note that the transaction_ids are unique across all the objects and instances
of metastores. This means all the transactions are serialized to generate these
ids in metastore. While this could be used to generate a globally incrementing
version number it may have some adverse performance implications. Having a
version at a per-object level instead reduces the scope of the lock and only
the transactions which are operating on the same object will be serialized. I
think as a mid-way we can implement the versioning for non-acid enabled tables.
If the table is ACID enabled, it should use the transaction ids for its
versioning. I am hesitant to use this logic above for all the objects which
indirectly is going to serialize all the DML transactions in metastore. Perhaps
someone more familiar with this code can comment on my thoughts above.
[~ekoifman] [~alangates]?
> Add support for object versions in metastore
>
>
> Key: HIVE-21115
> URL: https://issues.apache.org/jira/browse/HIVE-21115
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Priority: Major
>
> Currently, metastore objects are identified uniquely by their names (eg.
> catName, dbName and tblName for a table is unique). Once a table or partition
> is created it could be altered in many ways. There is no good way currently
> to identify the version of the object once it is altered. For example,
> suppose there are two clients (Hive and Impala) using the same metastore.
> Once some alter operations are performed by a client, anoth
[jira] [Updated] (HIVE-21125) Hive does not check for dependent materialized views when issuing a DROP TABLE command
[
https://issues.apache.org/jira/browse/HIVE-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Taylor Cox updated HIVE-21125:
--
Description:
Dropping a table leads to undefined behavior when that table is the source of
an existing materialized view. The following behavior is observed:
* Table still appears in 'show tables' despite not being in metastore
* Actions on table hang and then display a "could not fetch table" error
* Rebuilding any dependent materialized view has same error
It seems that the root cause is the fact that users are allowed to issue a DROP
TABLE command against a table even if there is a materialized view using this
table at the time. This is not something I have seen other query languages
permit.
Repro steps: Launch these commands from any Hive 3 client:
{code:java}
create table footable (id int); insert into footable values (1), (2), (3);
create materialized view mv_footable as select count(*) from footable;
drop table footable;
--These lines have unexpected behavior
show tables;
select * from footable;
alter materialized view mv_footable rebuild;{code}
was:
Dropping a table leads to undefined behavior when that table is the source of
an existing materialized view. The following behavior is observed:
* Table still appears in 'show tables' despite not being in metastore
* Actions on table hang and then display a "could not fetch table" error
* Rebuilding any dependent materialized view has same error
It seems that the root cause is the fact that users are allowed to issue a DROP
TABLE command against a table even if there is a materialized view using this
table at the time. This is not something I have seen other query languages
permit.
Repro steps: Launch these commands from any Hive 3 client:
{code:java}
create table footable (id int); insert into footable values (1), (2), (3);
create materialized view mv_footable as select count(*) from footable;
drop table footable;
--These lines have unexpected behavior
show tables;
select * from footable;
alter materialized view mv_footable rebuild;{code}
> Hive does not check for dependent materialized views when issuing a DROP
> TABLE command
> --
>
> Key: HIVE-21125
> URL: https://issues.apache.org/jira/browse/HIVE-21125
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 3.1.0
>Reporter: Taylor Cox
>Priority: Major
>
> Dropping a table leads to undefined behavior when that table is the source of
> an existing materialized view. The following behavior is observed:
>
> * Table still appears in 'show tables' despite not being in metastore
> * Actions on table hang and then display a "could not fetch table" error
> * Rebuilding any dependent materialized view has same error
> It seems that the root cause is the fact that users are allowed to issue a
> DROP TABLE command against a table even if there is a materialized view using
> this table at the time. This is not something I have seen other query
> languages permit.
> Repro steps: Launch these commands from any Hive 3 client:
> {code:java}
> create table footable (id int); insert into footable values (1), (2), (3);
> create materialized view mv_footable as select count(*) from footable;
> drop table footable;
> --These lines have unexpected behavior
> show tables;
> select * from footable;
> alter materialized view mv_footable rebuild;{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743453#comment-16743453
]
Hive QA commented on HIVE-21052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955010/HIVE-21052.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15699 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15632/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15632/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15632/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955010 - PreCommit-HIVE-Build
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21107) Cannot find field" error during dynamically partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-21107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743450#comment-16743450 ] Gopal V commented on HIVE-21107: LGTM - +1 Can you add a comment to the line to say that the joins always use the key columns for the keys (so don't read VALUES). > Cannot find field" error during dynamically partitioned hash join > - > > Key: HIVE-21107 > URL: https://issues.apache.org/jira/browse/HIVE-21107 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21107.1.patch, HIVE-21107.2.patch, > HIVE-21107.3.patch > > > This occurs in non-CBO path with dynamic partitioned join + constant > propagation ON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743444#comment-16743444
]
Hive QA commented on HIVE-21052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
18s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
14s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
0s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} shims/common: The patch generated 1 new + 95 unchanged
- 0 fixed = 96 total (was 95) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/0.23: The patch generated 5 new + 69 unchanged -
0 fixed = 74 total (was 69) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 12 new + 575 unchanged - 7
fixed = 587 total (was 582) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 149
unchanged - 0 fixed = 159 total (was 149) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 188 unchanged - 0 fixed = 190 total (was 188) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
54s{color} | {color:red} ql generated 1 new + 2309 unchanged - 1 fixed = 2310
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean()
may fail to close PreparedStatement At
Comp
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: HIVE-21052.7.patch Status: Patch Available (was: Open) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Status: Open (was: Patch Available) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743384#comment-16743384
]
Hive QA commented on HIVE-21052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12955005/HIVE-21052.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15699 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.client.TestCatalogs.alterChangeName[Embedded]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestCatalogs.alterChangeName[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestCatalogs.alterNonExistentCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestCatalogs.catalogOperations[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestCatalogs.dropCatalogWithNonEmptyDefaultDb[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestCatalogs.dropHiveCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestCatalogs.dropNonEmptyCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestCatalogs.dropNonExistentCatalog[Remote]
(batchId=222)
org.apache.hadoop.hive.metastore.client.TestCatalogs.getNonExistentCatalog[Remote]
(batchId=222)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=264)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=264)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15631/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15631/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15631/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12955005 - PreCommit-HIVE-Build
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch,
> HIVE-21052.5.patch, HIVE-21052.6.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743371#comment-16743371
]
Hive QA commented on HIVE-21052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
18s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
18s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
12s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
0s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/common: The patch generated 1 new + 95 unchanged
- 0 fixed = 96 total (was 95) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/0.23: The patch generated 5 new + 69 unchanged -
0 fixed = 74 total (was 69) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 12 new + 575 unchanged - 7
fixed = 587 total (was 582) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 149
unchanged - 0 fixed = 159 total (was 149) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 188 unchanged - 0 fixed = 190 total (was 188) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
6s{color} | {color:red} ql generated 1 new + 2309 unchanged - 1 fixed = 2310
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean()
may fail to close PreparedStatement At
Comp
[jira] [Commented] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743334#comment-16743334
]
BELUGA BEHR commented on HIVE-20238:
[~pvary] [~ngangam] Please consider for inclusion into project.
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch,
> HIVE-20238.3.patch, HIVE-20238.4.patch, HIVE-20238.5.patch,
> HIVE-20238.6.patch, HIVE-20238.7.patch, HIVE-20238.8.patch
>
>
> Deprecate the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: HIVE-21052.6.patch Status: Patch Available (was: Open) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Status: Open (was: Patch Available) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21077) Database and catalogs should have creation time
[
https://issues.apache.org/jira/browse/HIVE-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743320#comment-16743320
]
Bharathkrishna Guruvayoor Murali commented on HIVE-21077:
-
LGTM +1.
> Database and catalogs should have creation time
> ---
>
> Key: HIVE-21077
> URL: https://issues.apache.org/jira/browse/HIVE-21077
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-21077.01.patch, HIVE-21077.02.patch,
> HIVE-21077.03.patch, HIVE-21077.04.patch, HIVE-21077.05.patch,
> HIVE-21077.06.patch, HIVE-21077.07.patch
>
>
> Currently, database do not have creation time like we have for tables and
> partitions.
> {noformat}
> // namespace for tables
> struct Database {
> 1: string name,
> 2: string description,
> 3: string locationUri,
> 4: map parameters, // properties associated with the
> database
> 5: optional PrincipalPrivilegeSet privileges,
> 6: optional string ownerName,
> 7: optional PrincipalType ownerType,
> 8: optional string catalogName
> }
> {noformat}
> Currently, without creationTime there is no way to identify if the copy of
> Database which a client has is the same as the one on the server if the name
> is same. Without object ids creationTime value is the only way currently to
> identify uniquely a instance of metastore object. It would be good to have
> Database creation time as well.
> Same applies for catalogs as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743314#comment-16743314
]
Hive QA commented on HIVE-20238:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954998/HIVE-20238.8.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15694 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15630/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15630/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15630/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954998 - PreCommit-HIVE-Build
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch,
> HIVE-20238.3.patch, HIVE-20238.4.patch, HIVE-20238.5.patch,
> HIVE-20238.6.patch, HIVE-20238.7.patch, HIVE-20238.8.patch
>
>
> Deprecate the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743302#comment-16743302
]
Hive QA commented on HIVE-20238:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
15s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} serde in master has 198 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} cli in master has 13 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} ql: The patch generated 0 new + 1863 unchanged - 23
fixed = 1863 total (was 1886) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} cli: The patch generated 0 new + 31 unchanged - 2
fixed = 31 total (was 33) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch druid-handler passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
23s{color} | {color:green} the patch
[jira] [Updated] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20238:
---
Status: Patch Available (was: Open)
Fixed unit tests
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch,
> HIVE-20238.3.patch, HIVE-20238.4.patch, HIVE-20238.5.patch,
> HIVE-20238.6.patch, HIVE-20238.7.patch, HIVE-20238.8.patch
>
>
> Deprecate the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743257#comment-16743257
]
Hive QA commented on HIVE-21052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954993/HIVE-21052.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15699 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDynamicPartitionInsert
(batchId=326)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMerge3Way01
(batchId=326)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMerge3Way02
(batchId=326)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergePartitioned01
(batchId=326)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergePartitioned02
(batchId=326)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testTableProperties
(batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15629/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15629/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15629/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954993 - PreCommit-HIVE-Build
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, HIVE-21052.5.patch
>
>
> If the transaction is aborted between openTxn and addPartitions and data has
> been written on the table the transaction manager will think it's an empty
> transaction and no cleaning will be done.
> This is currently an issue in the streaming API and in micromanaged tables.
> As proposed by [~ekoifman] this can be solved by:
> * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and
> when addPartitions is called remove this entry from TXN_COMPONENTS and add
> the corresponding partition entry to TXN_COMPONENTS.
> * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that
> specifies that a transaction was opened and it was aborted it must generate
> jobs for the worker for every possible partition available.
> cc [~ewohlstadter]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21119) String UDAF and count distinct in the same select give error
[
https://issues.apache.org/jira/browse/HIVE-21119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Shetye updated HIVE-21119:
---
Labels: (was: plannin)
> String UDAF and count distinct in the same select give error
>
>
> Key: HIVE-21119
> URL: https://issues.apache.org/jira/browse/HIVE-21119
> Project: Hive
> Issue Type: Bug
>Reporter: Ravi Shetye
>Priority: Major
> Attachments: MaxUDA.java, run.log
>
>
> With the attached UDAF the following query crashes on hive.
> CRASHES
> {noformat}
> select rs_max(genderkey),count(distinct genderkey) from
> as_adventure.dimgender;
> {noformat}
> WORKS
> {noformat}
> select rs_max(genderkey) from as_adventure.dimgender;
> {noformat}
> The table looks like
> {noformat}
> 0: jdbc:hive2://localhost:1> select * from dimgender;
> OK
> INFO : Compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:dimgender.genderkey, type:string,
> comment:null), FieldSchema(name:dimgender.gendername, type:string,
> comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.2 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Completed executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.004 seconds
> INFO : OK
> INFO : Concurrency mode is disabled, not creating a lock manager
> +--+---+
> | dimgender.genderkey | dimgender.gendername |
> +--+---+
> | M| Male |
> | F| Female|
> | U| Unisex|
> +--+---+
> {noformat}
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1547169244949_0024_2_01,
> diagnostics=[Task failed, taskId=task_1547169244949_0024_2_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1547169244949_0024_2_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"_col0":"F"},"value":{"_col0":"F"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> {noformat}
> ...
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> execute method public boolean
> com.sample.MaxUDA$Evaluator.merge(java.lang.String) with arguments
> {F}:argument type mismatch
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.merge(GenericUDAFBridge.java:176)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:216)
> {noformat}
> PLAN
> {noformat}
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Stage-1|
> | Reducer 3|
> | File Output Operator [FS_6] |
> | Group By Operator [GBY_12] (rows=1 width=368) |
> |
[jira] [Updated] (HIVE-21119) String UDAF and count distinct in the same select give error
[
https://issues.apache.org/jira/browse/HIVE-21119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Shetye updated HIVE-21119:
---
Component/s: Query Processor
Query Planning
> String UDAF and count distinct in the same select give error
>
>
> Key: HIVE-21119
> URL: https://issues.apache.org/jira/browse/HIVE-21119
> Project: Hive
> Issue Type: Bug
> Components: Query Planning, Query Processor
>Reporter: Ravi Shetye
>Priority: Major
> Attachments: MaxUDA.java, run.log
>
>
> With the attached UDAF the following query crashes on hive.
> CRASHES
> {noformat}
> select rs_max(genderkey),count(distinct genderkey) from
> as_adventure.dimgender;
> {noformat}
> WORKS
> {noformat}
> select rs_max(genderkey) from as_adventure.dimgender;
> {noformat}
> The table looks like
> {noformat}
> 0: jdbc:hive2://localhost:1> select * from dimgender;
> OK
> INFO : Compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:dimgender.genderkey, type:string,
> comment:null), FieldSchema(name:dimgender.gendername, type:string,
> comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.2 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Completed executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.004 seconds
> INFO : OK
> INFO : Concurrency mode is disabled, not creating a lock manager
> +--+---+
> | dimgender.genderkey | dimgender.gendername |
> +--+---+
> | M| Male |
> | F| Female|
> | U| Unisex|
> +--+---+
> {noformat}
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1547169244949_0024_2_01,
> diagnostics=[Task failed, taskId=task_1547169244949_0024_2_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1547169244949_0024_2_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"_col0":"F"},"value":{"_col0":"F"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> {noformat}
> ...
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> execute method public boolean
> com.sample.MaxUDA$Evaluator.merge(java.lang.String) with arguments
> {F}:argument type mismatch
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.merge(GenericUDAFBridge.java:176)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:216)
> {noformat}
> PLAN
> {noformat}
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Stage-1|
> | Reducer 3|
> | File Output Operator [FS_
[jira] [Updated] (HIVE-21119) String UDAF and count distinct in the same select give error
[
https://issues.apache.org/jira/browse/HIVE-21119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Shetye updated HIVE-21119:
---
Labels: plannin (was: wrongresults)
> String UDAF and count distinct in the same select give error
>
>
> Key: HIVE-21119
> URL: https://issues.apache.org/jira/browse/HIVE-21119
> Project: Hive
> Issue Type: Bug
>Reporter: Ravi Shetye
>Priority: Major
> Labels: plannin
> Attachments: MaxUDA.java, run.log
>
>
> With the attached UDAF the following query crashes on hive.
> CRASHES
> {noformat}
> select rs_max(genderkey),count(distinct genderkey) from
> as_adventure.dimgender;
> {noformat}
> WORKS
> {noformat}
> select rs_max(genderkey) from as_adventure.dimgender;
> {noformat}
> The table looks like
> {noformat}
> 0: jdbc:hive2://localhost:1> select * from dimgender;
> OK
> INFO : Compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:dimgender.genderkey, type:string,
> comment:null), FieldSchema(name:dimgender.gendername, type:string,
> comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.2 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7):
> select * from dimgender
> INFO : Completed executing
> command(queryId=hive_20190111225125_486e6e6b-97fa-4dda-9688-a733180bcfe7);
> Time taken: 0.004 seconds
> INFO : OK
> INFO : Concurrency mode is disabled, not creating a lock manager
> +--+---+
> | dimgender.genderkey | dimgender.gendername |
> +--+---+
> | M| Male |
> | F| Female|
> | U| Unisex|
> +--+---+
> {noformat}
> {noformat}
> Vertex failed, vertexName=Reducer 2, vertexId=vertex_1547169244949_0024_2_01,
> diagnostics=[Task failed, taskId=task_1547169244949_0024_2_01_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1547169244949_0024_2_01_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"_col0":"F"},"value":{"_col0":"F"}}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> {noformat}
> ...
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> execute method public boolean
> com.sample.MaxUDA$Evaluator.merge(java.lang.String) with arguments
> {F}:argument type mismatch
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.merge(GenericUDAFBridge.java:176)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:216)
> {noformat}
> PLAN
> {noformat}
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Reducer 2 <- Map 1 (SIMPLE_EDGE) |
> | Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)|
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Stage-1|
> | Reducer 3|
> | File Output Operator [FS_6] |
> | Group By Opera
[jira] [Updated] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20238:
---
Attachment: HIVE-20238.8.patch
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch,
> HIVE-20238.3.patch, HIVE-20238.4.patch, HIVE-20238.5.patch,
> HIVE-20238.6.patch, HIVE-20238.7.patch, HIVE-20238.8.patch
>
>
> Deprecate the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20238) Remove stringifyException Method
[
https://issues.apache.org/jira/browse/HIVE-20238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20238:
---
Status: Open (was: Patch Available)
> Remove stringifyException Method
>
>
> Key: HIVE-20238
> URL: https://issues.apache.org/jira/browse/HIVE-20238
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Labels: newbie, noob
> Fix For: 4.0.0
>
> Attachments: HIVE-20238.1.patch, HIVE-20238.2.patch,
> HIVE-20238.3.patch, HIVE-20238.4.patch, HIVE-20238.5.patch,
> HIVE-20238.6.patch, HIVE-20238.7.patch
>
>
> Deprecate the method {{stringifyException}}
> https://github.com/apache/hive/blob/c2940a07cf0891e922672782b73ec22551a7eedd/common/src/java/org/apache/hive/common/util/HiveStringUtils.java#L146
> The code already exists in Hadoop proper:
> https://github.com/apache/hadoop/blob/2b2399d623539ab68e71a38fa9fbfc9a405bddb8/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java#L86
> And beyond that, I was told on the Hadoop dev mailing list that this function
> should not be used anymore. Developers should just be using the SLF4J
> facilities and not this home-grown thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743242#comment-16743242
]
Hive QA commented on HIVE-21052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
16s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
18s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/common: The patch generated 1 new + 95 unchanged
- 0 fixed = 96 total (was 95) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} shims/0.23: The patch generated 5 new + 69 unchanged -
0 fixed = 74 total (was 69) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 5 new + 150 unchanged - 0
fixed = 155 total (was 150) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} itests/hive-unit: The patch generated 10 new + 149
unchanged - 0 fixed = 159 total (was 149) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
17s{color} | {color:red} standalone-metastore/metastore-server generated 2 new
+ 188 unchanged - 0 fixed = 190 total (was 188) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
2s{color} | {color:red} ql generated 1 new + 2309 unchanged - 1 fixed = 2310
total (was 2310) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean()
may fail to close PreparedStatement At
Compa
[jira] [Commented] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743223#comment-16743223
]
Jesus Camacho Rodriguez commented on HIVE-21095:
+1, thanks [~klcopp]
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch, HIVE-21095.3.patch, HIVE-21095.4.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
> {code}
> This was discussed in the community doc [Consistent timestamp types in Hadoop
> SQL
> engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: HIVE-21052.5.patch Status: Patch Available (was: Open) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, HIVE-21052.5.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Status: Open (was: Patch Available) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, HIVE-21052.5.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: HIVE-21052.4.patch Status: Patch Available (was: Open) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743201#comment-16743201
]
Hive QA commented on HIVE-21052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954992/HIVE-21052.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15628/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15628/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15628/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-15 16:31:10.094
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15628/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-15 16:31:10.098
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-15 16:31:10.798
+ rm -rf ../yetus_PreCommit-HIVE-Build-15628
+ mkdir ../yetus_PreCommit-HIVE-Build-15628
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15628
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15628/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:28
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java' with
conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:1481: trailing whitespace.
tmpMap.put(_Fields.WRITE_IDS, new
org.apache.thrift.meta_data.FieldMetaData("writeIds",
org.apache.thrift.TFieldRequirementType.OPTIONAL,
/data/hiveptest/working/scratch/build.patch:1482: trailing whitespace.
new
org.apache.thrift.meta_data.SetMetaData(org.apache.thrift.protocol.TType.SET,
/data/hiveptest/working/scratch/build.patch:1665: trailing whitespace.
} else {
/data/hiveptest/working/scratch/build.patch:16191: trailing whitespace.
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:28
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java' with
conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
warning: 4 lines add whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-15628
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954992 - PreCommit-HIVE-Build
> Make sure transactions get cleaned if they are aborted before addPartitions
> is called
> -
>
> Key: HIVE-21052
> URL: https://issues.apache.org/jira/browse/HIVE-21052
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Critical
> Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch,
> HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Status: Open (was: Patch Available) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21095) 'Show create table' should not display a time zone for timestamp with local time zone
[
https://issues.apache.org/jira/browse/HIVE-21095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743183#comment-16743183
]
Karen Coppage commented on HIVE-21095:
--
Hi [~jcamachorodriguez], this issue was touched on in your discussion with
Zoltan Ivanfi, since this doesn't have to do with UTC-normalization :) would
you be willing to review this patch?
> 'Show create table' should not display a time zone for timestamp with local
> time zone
> -
>
> Key: HIVE-21095
> URL: https://issues.apache.org/jira/browse/HIVE-21095
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-21095.1.patch, HIVE-21095.2.patch,
> HIVE-21095.2.patch, HIVE-21095.2.patch, HIVE-21095.3.patch, HIVE-21095.4.patch
>
>
> SHOW CREATE TABLE shows the time zone that the table was created in (if it
> contains a TIMESTAMPTZ column). This is also misleading, since it has nothing
> to do with the actual data or server or user time zone.
> e.g.
> {code:java}
> hive> set time zone America/Los_Angeles;
> hive> create table text_local (ts timestamp with local time zone) stored as
> textfile;
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone('America/Los_Angeles'))
> {code}
> should be:
> {code:java}
> hive> show create table text_local;
> CREATE TABLE `text_local`(
> `ts` timestamp with local time zone)
> {code}
> This was discussed in the community doc [Consistent timestamp types in Hadoop
> SQL
> engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21094) Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone
[ https://issues.apache.org/jira/browse/HIVE-21094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743180#comment-16743180 ] Karen Coppage commented on HIVE-21094: -- Patch 1 doesn't work with Druid, and it was hacky anyway. The reason was because both writing and reading TimestampTZs require converting a String to a TimestampTz(Writable) back to a String, and I wasn't able to differentiate between writing (when we want to convert to GMT) and reading (when we want to convert to server time). Since this fix doesn't really change behavior – it might change performance based on the use case –, I'm putting this back on the shelf. IMO either the inner representation of TimestampTZ should be Instant (not ZonedDateTime), since it behaves like Instant; or a better solution will need to be found. > Store TIMESTAMP WITH LOCAL TIME ZONE in UTC instead of writer's time zone > - > > Key: HIVE-21094 > URL: https://issues.apache.org/jira/browse/HIVE-21094 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-21094.1.patch > > > TIMESTAMP WITH LOCAL TIME ZONE (aka TIMESTAMPTZ) is stored in writer's local > time, and the writer's zone is stored with it. When reading, the timestamp in > reader local time + reader zone is displayed. This is misleading for the > user, since it looks like all the data was written in the reader's time zone. > TIMESTAMPTZ should be stored in UTC time and be displayed in reader local > time (as it was before) but should not display the reader's time zone. > This was discussed in the community doc [Consistent timestamp types in Hadoop > SQL > engines|https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit?usp=sharing] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[
https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743169#comment-16743169
]
Hive QA commented on HIVE-21052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954984/HIVE-21052.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15627/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15627/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15627/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-01-15 16:00:38.574
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15627/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-01-15 16:00:38.577
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a3aa074 HIVE-21113: For HPL/SQL that contains boolean expression
with NOT, incorrect SQL may be generated (Baoning He, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-01-15 16:00:39.260
+ rm -rf ../yetus_PreCommit-HIVE-Build-15627
+ mkdir ../yetus_PreCommit-HIVE-Build-15627
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15627
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15627/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java:999
error: repository lacks the necessary blob to fall back on 3-way merge.
error:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java:
patch does not apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java:2541
error: repository lacks the necessary blob to fall back on 3-way merge.
error: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: patch does not
apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java:395
error: repository lacks the necessary blob to fall back on 3-way merge.
error: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java: patch
does not apply
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java:43
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AllocateTableWriteIdsRequest.java'
cleanly.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java:50
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php:16520
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php'
with conflicts.
error: patch failed:
standalone-metastore/
[jira] [Commented] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[
https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743164#comment-16743164
]
Hive QA commented on HIVE-21116:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954979/HIVE-21116.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15694 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15626/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15626/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15626/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954979 - PreCommit-HIVE-Build
> HADOOP_CREDSTORE_PASSWORD is not populated under
> yarn.app.mapreduce.am.admin.user.env
> --
>
> Key: HIVE-21116
> URL: https://issues.apache.org/jira/browse/HIVE-21116
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21116.1.patch, HIVE-21116.2.patch,
> HIVE-21116.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21121) Cast date to timestamp incorrect interpretation
[ https://issues.apache.org/jira/browse/HIVE-21121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] paco87 updated HIVE-21121: -- Affects Version/s: 2.1.0 > Cast date to timestamp incorrect interpretation > --- > > Key: HIVE-21121 > URL: https://issues.apache.org/jira/browse/HIVE-21121 > Project: Hive > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.1, 2.1.0 >Reporter: paco87 >Priority: Major > Attachments: jira_replicate_issue.txt > > > Hive is returning timestamp with current time when casting date to timestamp, > where it should be 00:00:00.0 . > This issue is wired. It seems that this happens only on specific date: > '1900-01-01'. > +++ > |ab| > +++ > |1900-01-01 11:28:46.869| > |1900-01-01 11:28:46.869| > |1900-01-01 11:28:46.869| > |1900-01-01 11:28:46.869| > |1890-01-01 00:00:00.0| > |1901-01-01 00:00:00.0| > |1900-01-01 11:28:46.869| > +---++ > I didin't notice this on any other date. > This might be connected to old issue: HIVE-10488 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Status: Open (was: Patch Available) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-21052: --- Attachment: HIVE-21052.3.patch Status: Patch Available (was: Open) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch > > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[
https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743119#comment-16743119
]
Hive QA commented on HIVE-21116:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15626/dev-support/hive-personality.sh
|
| git revision | master / a3aa074 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15626/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HADOOP_CREDSTORE_PASSWORD is not populated under
> yarn.app.mapreduce.am.admin.user.env
> --
>
> Key: HIVE-21116
> URL: https://issues.apache.org/jira/browse/HIVE-21116
> Project: Hive
> Issue Type: Bug
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-21116.1.patch, HIVE-21116.2.patch,
> HIVE-21116.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21123) LLAP : same sql and TPCDS data occurs different results using textfile fileformat
[ https://issues.apache.org/jira/browse/HIVE-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743114#comment-16743114 ] zhangbutao commented on HIVE-21123: --- The sql result of text fileformat would be right if i set hive.llap.io.enabled=false. Can someone explain why it is okay to close the cache? Is lllap cache only valid for orc? > LLAP : same sql and TPCDS data occurs different results using textfile > fileformat > --- > > Key: HIVE-21123 > URL: https://issues.apache.org/jira/browse/HIVE-21123 > Project: Hive > Issue Type: Bug > Components: File Formats, llap, ORC >Affects Versions: 3.1.0 >Reporter: zhangbutao >Priority: Major > Attachments: llap_tpcds_text_500_query53.sql, > tez_tpcds_text_500_query53.sql > > > I found that llap execution occurs different result using textfile fileformat > and same sql when i test llap with TPCDS textfile data. And the orc > fileformat is ok. The attachment(llap_tpcds_text_500_query53.sql) are TPCDS > query53.sql execution results with data tpcds_text_500(500GB textfile), and > you can find the results is different every time. The attachment > (tez_tpcds_text_500_query53.sql) is the tez container execution mode and it > is the right result, which is same as llap with orc textfile fileformat. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21116) HADOOP_CREDSTORE_PASSWORD is not populated under yarn.app.mapreduce.am.admin.user.env
[ https://issues.apache.org/jira/browse/HIVE-21116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-21116: -- Attachment: HIVE-21116.3.patch > HADOOP_CREDSTORE_PASSWORD is not populated under > yarn.app.mapreduce.am.admin.user.env > -- > > Key: HIVE-21116 > URL: https://issues.apache.org/jira/browse/HIVE-21116 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-21116.1.patch, HIVE-21116.2.patch, > HIVE-21116.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21072) NPE when running partitioned CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-21072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743089#comment-16743089
]
Jesus Camacho Rodriguez commented on HIVE-21072:
[~b.maidics], have you verified whether HIVE-20707 fixes the issue ([~viirya]'s
patch seems to be a part of it)? AFAIK it could be backported if needed, there
is a branch-3 patch in the JIRA but apparently it was pushed.
> NPE when running partitioned CTAS statements
>
>
> Key: HIVE-21072
> URL: https://issues.apache.org/jira/browse/HIVE-21072
> Project: Hive
> Issue Type: Bug
>Reporter: Liang-Chi Hsieh
>Priority: Major
> Labels: pull-request-available
>
> HIVE-20241 adds support of partitioned CTAS statements:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;{code}
>
> However, I've tried this feature by checking out latest branch-3, and
> encountered NPE:
> {code:java}
> hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
> FAILED: NullPointerException null
> {code}
> I also ran the query test partition_ctas.q. The test passes when using
> TestMiniLlapLocalCliDriver, but when I go to test it with TestCliDriver
> manually, it also throws NullPointerException:
> {code:java}
> 2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
> at
> org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
> at
> org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21072) NPE when running partitioned CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-21072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743070#comment-16743070
]
Barnabas Maidics commented on HIVE-21072:
-
I've also tried running partitioned CTAS and ran into the same error. As I saw,
if the execution engine is TEZ, it works perfectly (that is why partition_ctas
worked with TestMiniLlapLocalCliDriver).
Using MR, the NPE was thrown because Hive tried to create a Map-only merge job
(_GenMapRedUtils.createMRWorkForMergingFiles_), but the _tableInfo_ of the
_FileSinkDesc_ doesn't contain an entry with the key of "partition_columns" and
we try to call split on a null.
{code:java}
String[] partNames = properties.getProperty(
org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_PARTITION_COLUMNS)
.split("/");
{code}
A possible quick fix is to *set _hive.merge.mapfiles_ to _false_* (as a
default, it is true) so these steps will be skipped
(_GenMapRedUtils.__isMergeRequired_ will return _false_).
But maybe I miss something about this feature. [~jcamachorodriguez] what do you
think the long-term fix would be for this?
> NPE when running partitioned CTAS statements
>
>
> Key: HIVE-21072
> URL: https://issues.apache.org/jira/browse/HIVE-21072
> Project: Hive
> Issue Type: Bug
>Reporter: Liang-Chi Hsieh
>Priority: Major
> Labels: pull-request-available
>
> HIVE-20241 adds support of partitioned CTAS statements:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;{code}
>
> However, I've tried this feature by checking out latest branch-3, and
> encountered NPE:
> {code:java}
> hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
> FAILED: NullPointerException null
> {code}
> I also ran the query test partition_ctas.q. The test passes when using
> TestMiniLlapLocalCliDriver, but when I go to test it with TestCliDriver
> manually, it also throws NullPointerException:
> {code:java}
> 2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
> at
> org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
> at
> org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21072) NPE when running partitioned CTAS statements
[
https://issues.apache.org/jira/browse/HIVE-21072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barnabas Maidics updated HIVE-21072:
Description:
HIVE-20241 adds support of partitioned CTAS statements:
{code:sql}
CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
SELECT value, key FROM src where key > 200 and key < 300;{code}
However, I've tried this feature by checking out latest branch-3, and
encountered NPE:
{code:java}
hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
FAILED: NullPointerException null
{code}
I also ran the query test partition_ctas.q. The test passes when using
TestMiniLlapLocalCliDriver, but when I go to test it with TestCliDriver
manually, it also throws NullPointerException:
{code:java}
2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main]
ql.Driver: FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
at
org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
at
org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at
org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
at
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
{code}
was:
HIVE-20241 adds support of partitioned CTAS statements:
{code:sql}
CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
SELECT value, key FROM src where key > 200 and key < 300;{code}
However, I've tried this feature by checking out latest branch-3, and
encountered NPE:
{code:java}
hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
FAILED: NullPointerException null
{code}
I also ran the query test partition_ctas.q. The test passes when using
TestMiniLlapLocalCliDriver, but when I go to test it with TestCliDriver
manually, it also throws NullPointerException:
{code}
2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main]
ql.Driver: FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
at
org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
at
org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
at
org.apache.hadoop.hive.ql.lib.De
[jira] [Commented] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[
https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743029#comment-16743029
]
Hive QA commented on HIVE-21078:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954963/HIVE-21078.04.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15700 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testIncrementalInsertDropPartitionedTable
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testIncrementalInsertDropUnpartitionedTable
(batchId=244)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertDropPartitionedTable
(batchId=246)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertDropUnpartitionedTable
(batchId=246)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15625/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15625/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15625/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954963 - PreCommit-HIVE-Build
> Replicate column and table level statistics for unpartitioned Hive tables
> -
>
> Key: HIVE-21078
> URL: https://issues.apache.org/jira/browse/HIVE-21078
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch,
> HIVE-21078.03.patch, HIVE-21078.04.patch
>
>
> This task is for replicating column and table level statistics for
> unpartitioned tables. The same for partitioned tables will be worked upon in
> a separate sub-task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[
https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743018#comment-16743018
]
Hive QA commented on HIVE-21078:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
14s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
3s{color} | {color:blue} standalone-metastore/metastore-server in master has
188 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
39s{color} | {color:blue} ql in master has 2310 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 21 new + 759 unchanged - 1
fixed = 780 total (was 760) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 771
unchanged - 0 fixed = 774 total (was 771) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} metastore-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore_metastore-server generated 0
new + 49 unchanged - 1 fixed = 49 total (was 50) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} hive-unit in the patch passed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15625/dev-support/hive-personality.sh
|
| git revision | master / a3aa074 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15625/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15625/yetus/diff-checkstyle-itests_hive-unit.txt
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Status: In Progress (was: Patch Available) > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Attachment: HIVE-21078.04.patch Status: Patch Available (was: In Progress) Patch with fixes for checkstyle, whitespace notices and test failures in last ptest run. Updated PR. > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch, HIVE-21078.04.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Status: In Progress (was: Patch Available) > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Attachment: (was: HIVE-21078.04.patch) > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21078) Replicate column and table level statistics for unpartitioned Hive tables
[ https://issues.apache.org/jira/browse/HIVE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21078: -- Attachment: HIVE-21078.04.patch Status: Patch Available (was: In Progress) Patch updated with fixes for checkstyle, whitespace notices and test failures in last ptest run. PR updated. > Replicate column and table level statistics for unpartitioned Hive tables > - > > Key: HIVE-21078 > URL: https://issues.apache.org/jira/browse/HIVE-21078 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21078.01.patch, HIVE-21078.02.patch, > HIVE-21078.03.patch, HIVE-21078.04.patch > > > This task is for replicating column and table level statistics for > unpartitioned tables. The same for partitioned tables will be worked upon in > a separate sub-task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742968#comment-16742968 ] Miklos Gergely edited comment on HIVE-16255 at 1/15/19 10:56 AM: - [~ashutoshc] after discussing the issue with Zoltan we came to these conclusions: * The solution that [~abstractdog] created is basically good, though not following the standard syntax defined at [https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions110.htm] and [https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm|https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm.] which would require the introducing of the "WITHIN GROUP clause. Having that as a mandatory clause the elements would arrive in order, so instead of having a multiset, a list could be used to store the elements. * This solution has _O(n)_ memory cost, which may be too much. The only way we could avoid this is to be able to iterate over the elements twice, first to count them, second time to find the appropriate element. This would mean that for such functions there should be a way to configure that the execution to do so. This is obviously a greater task than the scope of this issue. was (Author: mgergely): [~ashutoshc] after discussing the issue with Zoltan we came to these conclusions: * The solution that [~abstractdog] created is basically good, though not following the standard syntax defined at [https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions110.htm] and [https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm|https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm.] which would require the introducing of the "WITHIN GROUP clause. Having that as a mandatory clause the elements would arrive in order, so instead of having a multiset, a list could be used to store the elements. * This solution has O(n) memory cost, which may be too much. The only way we could avoid this is to be able to iterate over the elements twice, first to count them, second time to find the appropriate element. This would mean that for such functions there should be a way to configure that the execution to do so. This is obviously a greater task than the scope of this issue. > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch, > HIVE-16255.03.patch, HIVE-16255.04.patch, HIVE-16255.05.patch > > > Way back in HIVE-259, a percentile function was added that provides a subset > of the standard percentile_cont aggregate function. > The SQL standard provides some additional options and also a percentile_disc > aggregate function with different rules. In the standard you specify an > ordering with arbitrary value expression and the results are drawn from this > value expression. This aggregate functions should be usable as analytic > functions as well (i.e. support the over clause). The current percentile > function is able to be used with an over clause. > The rough outline of how this works is: > percentile_cont(number) within group (order by expression) [ over(window > spec) ] > percentile_disc(number) within group (order by expression) [ over(window > spec) ] > The value of number should be between 0 and 1. The value expression is > evaluated for each row of the group, nulls are discarded, and the remaining > rows are ordered. > — If PERCENTILE_CONT is specified, by considering the pair of consecutive > rows that are indicated by the argument, treated as a fraction of the total > number of rows in the group, and interpolating the value of the value > expression evaluated for these rows. > — If PERCENTILE_DISC is specified, by treating the group as a window > partition of the CUME_DIST window function, using the specified ordering of > the value expression as the window ordering, and returning the first value > expression whose cumulative distribution value is greater than or equal to > the argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-16255) Support percentile_cont / percentile_disc
[
https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742968#comment-16742968
]
Miklos Gergely edited comment on HIVE-16255 at 1/15/19 10:56 AM:
-
[~ashutoshc] after discussing the issue with Zoltan we came to these
conclusions:
* The solution that [~abstractdog] created is basically good, though not
following the standard syntax defined at
[https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions110.htm] and
[https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm|https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm.]
which would require the introducing of the "WITHIN GROUP clause. Having that
as a mandatory clause the elements would arrive in order, so instead of having
a multiset, a list could be used to store the elements.
* This solution has
{noformat}
O(n){noformat}
memory cost, which may be too much. The only way we could avoid this is to be
able to iterate over the elements twice, first to count them, second time to
find the appropriate element. This would mean that for such functions there
should be a way to configure that the execution to do so. This is obviously a
greater task than the scope of this issue.
was (Author: mgergely):
[~ashutoshc] after discussing the issue with Zoltan we came to these
conclusions:
* The solution that [~abstractdog] created is basically good, though not
following the standard syntax defined at
[https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions110.htm] and
[https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm|https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm.]
which would require the introducing of the "WITHIN GROUP clause. Having that
as a mandatory clause the elements would arrive in order, so instead of having
a multiset, a list could be used to store the elements.
* This solution has _O(n)_ memory cost, which may be too much. The only way we
could avoid this is to be able to iterate over the elements twice, first to
count them, second time to find the appropriate element. This would mean that
for such functions there should be a way to configure that the execution to do
so. This is obviously a greater task than the scope of this issue.
> Support percentile_cont / percentile_disc
> -
>
> Key: HIVE-16255
> URL: https://issues.apache.org/jira/browse/HIVE-16255
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Carter Shanklin
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch,
> HIVE-16255.03.patch, HIVE-16255.04.patch, HIVE-16255.05.patch
>
>
> Way back in HIVE-259, a percentile function was added that provides a subset
> of the standard percentile_cont aggregate function.
> The SQL standard provides some additional options and also a percentile_disc
> aggregate function with different rules. In the standard you specify an
> ordering with arbitrary value expression and the results are drawn from this
> value expression. This aggregate functions should be usable as analytic
> functions as well (i.e. support the over clause). The current percentile
> function is able to be used with an over clause.
> The rough outline of how this works is:
> percentile_cont(number) within group (order by expression) [ over(window
> spec) ]
> percentile_disc(number) within group (order by expression) [ over(window
> spec) ]
> The value of number should be between 0 and 1. The value expression is
> evaluated for each row of the group, nulls are discarded, and the remaining
> rows are ordered.
> — If PERCENTILE_CONT is specified, by considering the pair of consecutive
> rows that are indicated by the argument, treated as a fraction of the total
> number of rows in the group, and interpolating the value of the value
> expression evaluated for these rows.
> — If PERCENTILE_DISC is specified, by treating the group as a window
> partition of the CUME_DIST window function, using the specified ordering of
> the value expression as the window ordering, and returning the first value
> expression whose cumulative distribution value is greater than or equal to
> the argument.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16255) Support percentile_cont / percentile_disc
[ https://issues.apache.org/jira/browse/HIVE-16255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742968#comment-16742968 ] Miklos Gergely commented on HIVE-16255: --- [~ashutoshc] after discussing the issue with Zoltan we came to these conclusions: * The solution that [~abstractdog] created is basically good, though not following the standard syntax defined at [https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions110.htm] and [https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm|https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions111.htm.] which would require the introducing of the "WITHIN GROUP clause. Having that as a mandatory clause the elements would arrive in order, so instead of having a multiset, a list could be used to store the elements. * This solution has O(n) memory cost, which may be too much. The only way we could avoid this is to be able to iterate over the elements twice, first to count them, second time to find the appropriate element. This would mean that for such functions there should be a way to configure that the execution to do so. This is obviously a greater task than the scope of this issue. > Support percentile_cont / percentile_disc > - > > Key: HIVE-16255 > URL: https://issues.apache.org/jira/browse/HIVE-16255 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16255.01.patch, HIVE-16255.02.patch, > HIVE-16255.03.patch, HIVE-16255.04.patch, HIVE-16255.05.patch > > > Way back in HIVE-259, a percentile function was added that provides a subset > of the standard percentile_cont aggregate function. > The SQL standard provides some additional options and also a percentile_disc > aggregate function with different rules. In the standard you specify an > ordering with arbitrary value expression and the results are drawn from this > value expression. This aggregate functions should be usable as analytic > functions as well (i.e. support the over clause). The current percentile > function is able to be used with an over clause. > The rough outline of how this works is: > percentile_cont(number) within group (order by expression) [ over(window > spec) ] > percentile_disc(number) within group (order by expression) [ over(window > spec) ] > The value of number should be between 0 and 1. The value expression is > evaluated for each row of the group, nulls are discarded, and the remaining > rows are ordered. > — If PERCENTILE_CONT is specified, by considering the pair of consecutive > rows that are indicated by the argument, treated as a fraction of the total > number of rows in the group, and interpolating the value of the value > expression evaluated for these rows. > — If PERCENTILE_DISC is specified, by treating the group as a window > partition of the CUME_DIST window function, using the specified ordering of > the value expression as the window ordering, and returning the first value > expression whose cumulative distribution value is greater than or equal to > the argument. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-18711) Add percentile_cont and percentile_disc udaf
[ https://issues.apache.org/jira/browse/HIVE-18711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely resolved HIVE-18711. --- Resolution: Duplicate > Add percentile_cont and percentile_disc udaf > > > Key: HIVE-18711 > URL: https://issues.apache.org/jira/browse/HIVE-18711 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Ashutosh Chauhan >Assignee: Miklos Gergely >Priority: Major > > Most common way to implement this is via ordered aggregate which allows users > to specify sort specification with group by clause. Some implementations also > allow to use these with window functions. > Since Hive doesn't have concept of ordered aggregates yet, one possibility is > to support these only for window functions where sort specification is also > taken from window clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21124) HPL/SQL does not support the CREATE TABLE LIKE statement
[
https://issues.apache.org/jira/browse/HIVE-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742878#comment-16742878
]
Hive QA commented on HIVE-21124:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12954937/HIVE-21124.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15695 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15624/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15624/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15624/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12954937 - PreCommit-HIVE-Build
> HPL/SQL does not support the CREATE TABLE LIKE statement
>
>
> Key: HIVE-21124
> URL: https://issues.apache.org/jira/browse/HIVE-21124
> Project: Hive
> Issue Type: Bug
> Components: hpl/sql
>Reporter: Baoning He
>Assignee: Baoning He
>Priority: Major
> Attachments: HIVE-21124.1.patch
>
>
> Hive supports the CREATE TABLE LIKE statement but HPL/SQL does not support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
