[jira] [Commented] (HIVE-19915) master-txnstats branch - support CachedStore
[
https://issues.apache.org/jira/browse/HIVE-19915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551165#comment-16551165
]
Hive QA commented on HIVE-19915:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932354/HIVE-19915.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12740/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12740/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12740/
Messages:
{noformat}
This message was trimmed, see log for full details
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-3.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-3.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-4.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-1.2.0-to-2.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.0.0-to-2.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.1.0-to-2.2.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.2.0-to-2.3.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-3.1.0-to-4.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade.order.derby:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/create-user.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-1.2.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-3.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-1.2.0-to-2.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.0.0-to-2.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.1.0-to-2.2.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.2.0-to-2.3.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-3.1.0-to-4.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade.order.mssql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/create-user.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-1.2.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-1.2.0-to-2.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.0.0-to-2.1.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.1.0-to-2.2.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.2.0-to-2.3.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql:
does not exist in index
error:
[jira] [Commented] (HIVE-20046) [phase 2] remove NUM_FILES check or add a negative test
[
https://issues.apache.org/jira/browse/HIVE-20046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551162#comment-16551162
]
Hive QA commented on HIVE-20046:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932341/HIVE-20046.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12739/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12739/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12739/
Messages:
{noformat}
This message was trimmed, see log for full details
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-3.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-3.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-4.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-1.2.0-to-2.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.0.0-to-2.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.1.0-to-2.2.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.2.0-to-2.3.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-3.1.0-to-4.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade.order.derby:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/create-user.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-1.2.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-3.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-1.2.0-to-2.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.0.0-to-2.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.1.0-to-2.2.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.2.0-to-2.3.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-3.1.0-to-4.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade.order.mssql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/create-user.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-1.2.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-1.2.0-to-2.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.0.0-to-2.1.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.1.0-to-2.2.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.2.0-to-2.3.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql:
does not exist in index
error:
[jira] [Commented] (HIVE-19770) Support for CBO for queries with multiple same columns in select
[
https://issues.apache.org/jira/browse/HIVE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551159#comment-16551159
]
Hive QA commented on HIVE-19770:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932339/HIVE-19770.5.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 14679 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_reflect2]
(batchId=16)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_3]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join5]
(batchId=188)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[ambiguous_col]
(batchId=97)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=120)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12738/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12738/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932339 - PreCommit-HIVE-Build
> Support for CBO for queries with multiple same columns in select
>
>
> Key: HIVE-19770
> URL: https://issues.apache.org/jira/browse/HIVE-19770
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19770.1.patch, HIVE-19770.2.patch,
> HIVE-19770.3.patch, HIVE-19770.4.patch, HIVE-19770.5.patch
>
>
> Currently queries such as {code:sql} select a,a from t1 where b > 10 {code}
> are not supported for CBO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19770) Support for CBO for queries with multiple same columns in select
[
https://issues.apache.org/jira/browse/HIVE-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551121#comment-16551121
]
Hive QA commented on HIVE-19770:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 171 unchanged - 1
fixed = 171 total (was 172) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12738/dev-support/hive-personality.sh
|
| git revision | master / 06a4f98 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12738/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support for CBO for queries with multiple same columns in select
>
>
> Key: HIVE-19770
> URL: https://issues.apache.org/jira/browse/HIVE-19770
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19770.1.patch, HIVE-19770.2.patch,
> HIVE-19770.3.patch, HIVE-19770.4.patch, HIVE-19770.5.patch
>
>
> Currently queries such as {code:sql} select a,a from t1 where b > 10 {code}
> are not supported for CBO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20192) HS2 with embedded metastore is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551100#comment-16551100
]
Vihang Karajgaonkar commented on HIVE-20192:
{quote} The PersistenceManagerFactory object "pmf" is a static object which
keeps references of the allocated PersistenceManager in pmCache Map. That's why
PersistenceManager doesn't get GC'ed and need explicit shutdown for any
exception. In this case we retry instead of closing the thread which overwrites
the pm object and leaks the old one. {quote}
I see. Thanks for the explanation.
{quote}I think, overwriting the entry by cacheThreadLocalRawStore doesn't cause
any leak, because, it overwrites with thread local rawStore which is active in
this thread. If the thread local rawStore is changed, it means, the older one
was already shutdown gracefully before re-create. Also, threadRawStoreMap
shouldn't pile up as we use the same thread id. {quote}
I think you are right. Looks like the model of cleaning up is optimistic in the
sense in case the thread is reused, {{Hive#getInternal}} method does some
checks to make sure if we can reuse this threadlocal rawstore and cleans it up
in case the owner is different or the config is not compatible. So looks like
we are good in case of thread re-use because the object which is being
overwritten in the {{ThreadWithGarbageCleanup.threadRawStoreMap}} is either
replaced with the same object or when the previous one was closed. So that code
path looks good to me. This is all very tricky business and I hope there is no
other code path which is still leaking the rawstore.
This patch looks good to me. +1
> HS2 with embedded metastore is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20192.01.patch
>
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19766) Show the number of rows inserted when execution engine is Spark
[
https://issues.apache.org/jira/browse/HIVE-19766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551084#comment-16551084
]
Hive QA commented on HIVE-19766:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932335/HIVE-19766.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14679 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12737/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12737/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12737/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932335 - PreCommit-HIVE-Build
> Show the number of rows inserted when execution engine is Spark
> ---
>
> Key: HIVE-19766
> URL: https://issues.apache.org/jira/browse/HIVE-19766
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19766.1.patch
>
>
> Currently when insert query is run, the beeline output shows No rows affected.
> The logic to show the number of rows inserted is now present when execution
> engine is MR.
> This Jira is to make this work with Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19990) Query with interval literal in join condition fails
[
https://issues.apache.org/jira/browse/HIVE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19990:
---
Resolution: Fixed
Fix Version/s: 4.0.0
Status: Resolved (was: Patch Available)
Pushed to master
> Query with interval literal in join condition fails
> ---
>
> Key: HIVE-19990
> URL: https://issues.apache.org/jira/browse/HIVE-19990
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-19990.1.patch, HIVE-19990.2.patch,
> HIVE-19990.3.patch, HIVE-19990.4.patch
>
>
> *Reproducer*
> {code:sql}
> > create table date_dim_d1(
> d_week_seqint,
> d_datestring);
> > SELECT
>d1.d_week_seq
> FROM
>date_dim_d1 d1
>JOIN date_dim_d1 d3
> WHERE
>Cast(d3.d_date AS date) > Cast(d1.d_date AS date) + INTERVAL '5' day ;
> {code}
> *Exception*
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: '5 00:00:00.0'
> encountered with 0 children
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2780)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2775)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:3060)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2959)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:9633)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11380)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11285)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:593)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12150)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:658)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1829)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1776)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1771)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:832)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:770)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:694)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20097) Convert standalone-metastore to a submodule
[ https://issues.apache.org/jira/browse/HIVE-20097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-20097: -- Attachment: HIVE-20097.08-branch-3.patch > Convert standalone-metastore to a submodule > --- > > Key: HIVE-20097 > URL: https://issues.apache.org/jira/browse/HIVE-20097 > Project: Hive > Issue Type: Sub-task > Components: Hive, Metastore, Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20097.01.patch, HIVE-20097.02.patch, > HIVE-20097.03.patch, HIVE-20097.04.patch, HIVE-20097.05.patch, > HIVE-20097.06.patch, HIVE-20097.07-branch-3.patch, HIVE-20097.07.patch, > HIVE-20097.08-branch-3.patch > > > This is a subtask to stage HIVE-17751 changes into several smaller phases. > The first part is moving existing code in hive-standalone-metastore to a > sub-module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20218) make sure Statement.executeUpdate() returns number of rows affected
[ https://issues.apache.org/jira/browse/HIVE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551059#comment-16551059 ] Eugene Koifman commented on HIVE-20218: --- [~ashutoshc] could you review please > make sure Statement.executeUpdate() returns number of rows affected > --- > > Key: HIVE-20218 > URL: https://issues.apache.org/jira/browse/HIVE-20218 > Project: Hive > Issue Type: Improvement > Components: JDBC, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20218.01.patch > > > HiveStatement and HivePreparedStatement currently return 0 in all cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20218) make sure Statement.executeUpdate() returns number of rows affected
[ https://issues.apache.org/jira/browse/HIVE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20218: -- Attachment: (was: HIVE-20218.01.patch) > make sure Statement.executeUpdate() returns number of rows affected > --- > > Key: HIVE-20218 > URL: https://issues.apache.org/jira/browse/HIVE-20218 > Project: Hive > Issue Type: Improvement > Components: JDBC, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20218.01.patch > > > HiveStatement and HivePreparedStatement currently return 0 in all cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20218) make sure Statement.executeUpdate() returns number of rows affected
[ https://issues.apache.org/jira/browse/HIVE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20218: -- Attachment: HIVE-20218.01.patch > make sure Statement.executeUpdate() returns number of rows affected > --- > > Key: HIVE-20218 > URL: https://issues.apache.org/jira/browse/HIVE-20218 > Project: Hive > Issue Type: Improvement > Components: JDBC, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20218.01.patch, HIVE-20218.01.patch > > > HiveStatement and HivePreparedStatement currently return 0 in all cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20218) make sure Statement.executeUpdate() returns number of rows affected
[ https://issues.apache.org/jira/browse/HIVE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551054#comment-16551054 ] Eugene Koifman commented on HIVE-20218: --- current ticket fixes the JDBC API need to check what beeline produces - HIVE-8244 > make sure Statement.executeUpdate() returns number of rows affected > --- > > Key: HIVE-20218 > URL: https://issues.apache.org/jira/browse/HIVE-20218 > Project: Hive > Issue Type: Improvement > Components: JDBC, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20218.01.patch > > > HiveStatement and HivePreparedStatement currently return 0 in all cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20218) make sure Statement.executeUpdate() returns number of rows affected
[ https://issues.apache.org/jira/browse/HIVE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20218: -- Status: Patch Available (was: Open) > make sure Statement.executeUpdate() returns number of rows affected > --- > > Key: HIVE-20218 > URL: https://issues.apache.org/jira/browse/HIVE-20218 > Project: Hive > Issue Type: Improvement > Components: JDBC, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20218.01.patch > > > HiveStatement and HivePreparedStatement currently return 0 in all cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20218) make sure Statement.executeUpdate() returns number of rows affected
[ https://issues.apache.org/jira/browse/HIVE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-20218: -- Attachment: HIVE-20218.01.patch > make sure Statement.executeUpdate() returns number of rows affected > --- > > Key: HIVE-20218 > URL: https://issues.apache.org/jira/browse/HIVE-20218 > Project: Hive > Issue Type: Improvement > Components: JDBC, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-20218.01.patch > > > HiveStatement and HivePreparedStatement currently return 0 in all cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20218) make sure Statement.executeUpdate() returns number of rows affected
[ https://issues.apache.org/jira/browse/HIVE-20218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-20218: - > make sure Statement.executeUpdate() returns number of rows affected > --- > > Key: HIVE-20218 > URL: https://issues.apache.org/jira/browse/HIVE-20218 > Project: Hive > Issue Type: Improvement > Components: JDBC, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > HiveStatement and HivePreparedStatement currently return 0 in all cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19766) Show the number of rows inserted when execution engine is Spark
[
https://issues.apache.org/jira/browse/HIVE-19766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551037#comment-16551037
]
Hive QA commented on HIVE-19766:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12737/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12737/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Show the number of rows inserted when execution engine is Spark
> ---
>
> Key: HIVE-19766
> URL: https://issues.apache.org/jira/browse/HIVE-19766
> Project: Hive
> Issue Type: Sub-task
>Reporter: Bharathkrishna Guruvayoor Murali
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19766.1.patch
>
>
> Currently when insert query is run, the beeline output shows No rows affected.
> The logic to show the number of rows inserted is now present when execution
> engine is MR.
> This Jira is to make this work with Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20168) ReduceSinkOperator Logging Hidden
[
https://issues.apache.org/jira/browse/HIVE-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali reassigned HIVE-20168:
---
Assignee: Bharathkrishna Guruvayoor Murali
> ReduceSinkOperator Logging Hidden
> -
>
> Key: HIVE-20168
> URL: https://issues.apache.org/jira/browse/HIVE-20168
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
> Labels: newbie, noob
>
> [https://github.com/apache/hive/blob/ac6b2a3fb195916e22b2e5f465add2ffbcdc7430/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java]
>
> {code:java}
> if (LOG.isTraceEnabled()) {
> if (numRows == cntr) {
> cntr = logEveryNRows == 0 ? cntr * 10 : numRows + logEveryNRows;
> if (cntr < 0 || numRows < 0) {
> cntr = 0;
> numRows = 1;
> }
> LOG.info(toString() + ": records written - " + numRows);
> }
> }
> ...
> if (LOG.isTraceEnabled()) {
> LOG.info(toString() + ": records written - " + numRows);
> }
> {code}
> There are logging guards here checking for TRACE level debugging but the
> logging is actually INFO. This is important logging for detecting data skew.
> Please change guards to check for INFO... or I would prefer that the guards
> are removed altogether since it's very rare that a service is running with
> only WARN level logging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550997#comment-16550997
]
Hive QA commented on HIVE-19532:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932322/HIVE-19532.20.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12736/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12736/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12736/
Messages:
{noformat}
This message was trimmed, see log for full details
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-3.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-3.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-4.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-1.2.0-to-2.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.0.0-to-2.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.1.0-to-2.2.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.2.0-to-2.3.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-2.3.0-to-3.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-3.1.0-to-4.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade.order.derby:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/create-user.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-1.2.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-3.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-1.2.0-to-2.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.0.0-to-2.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.1.0-to-2.2.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.2.0-to-2.3.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-3.1.0-to-4.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade.order.mssql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/create-user.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-1.2.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-3.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-1.2.0-to-2.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.0.0-to-2.1.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.1.0-to-2.2.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.2.0-to-2.3.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql:
does not exist in index
error
[jira] [Commented] (HIVE-20117) schema changes for txn stats
[
https://issues.apache.org/jira/browse/HIVE-20117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550995#comment-16550995
]
Hive QA commented on HIVE-20117:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932324/HIVE-20117.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12735/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12735/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12735/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-20 17:05:56.658
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12735/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-20 17:05:56.660
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 851c8ab HIVE-17896: TopNKey: Create a standalone vectorizable
TopNKey operator (Teddy Choi, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 851c8ab HIVE-17896: TopNKey: Create a standalone vectorizable
TopNKey operator (Teddy Choi, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-20 17:06:03.883
+ rm -rf ../yetus_PreCommit-HIVE-Build-12735
+ mkdir ../yetus_PreCommit-HIVE-Build-12735
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12735
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12735/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-3.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/oracle/hive-schema-3.1.0.oracle.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/oracle/upgrade-3.0.0-to-3.1.0.oracle.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/postgres/hive-schema-3.1.0.postgres.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/postgres/upgrade-3.0.0-to-3.1.0.postgres.sql:
does not exist in index
error: metastore-common/src/main/sql/derby/hive-schema-3.1.0.derby.sql: does
not exist in index
error: metastore-common/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql:
does not exist in index
error: metastore-common/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql: does
not exist in index
error: metastore-common/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql:
does not exist in index
error: metastore-common/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql: does
not exist in index
error: metastore-common/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql:
does not exist in index
error: metastore-common/src/main/sql/oracle/hive-schema-3.1
[jira] [Commented] (HIVE-20213) Upgrade Calcite to 1.17.0
[
https://issues.apache.org/jira/browse/HIVE-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550992#comment-16550992
]
Hive QA commented on HIVE-20213:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932325/HIVE-20213.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 32 failed/errored test(s), 14679 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_intervals]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_join_preds]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_offcbo]
(batchId=47)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_basic2]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_views]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_case]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=186)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[pcr] (batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views]
(batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_case]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query11]
(batchId=264)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query4]
(batchId=264)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query74]
(batchId=264)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query11]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query4]
(batchId=262)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query74]
(batchId=262)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12734/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12734/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12734/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 32 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932325 - PreCommit-HIVE-Build
> Upgrade Calcite to 1.17.0
> -
>
> Key: HIVE-20213
> URL: https://issues.apache.org/jira/browse/HIVE-20213
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20213.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20213) Upgrade Calcite to 1.17.0
[
https://issues.apache.org/jira/browse/HIVE-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550990#comment-16550990
]
Hive QA commented on HIVE-20213:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} jdbc-handler in master has 8 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12734/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: . jdbc-handler ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12734/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade Calcite to 1.17.0
> -
>
> Key: HIVE-20213
> URL: https://issues.apache.org/jira/browse/HIVE-20213
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20213.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[
https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550959#comment-16550959
]
Sahil Takiar commented on HIVE-20032:
-
Attached an updated patch that should fix the unit tests. Since the new
approach is using reflection to create a class in the {{kryo-registrator}}
module, we have to make sure that {{hive-kryo-registrator}} jar file is on the
current classpath. Apparently the {{--jar}} option in {{spark-submit}} does not
guarantee that for the Spark Driver. So I'm using
{{spark.driver.extraClassPath}} to make sure its present when running the
{{SparkPlanGenerator}}.
When {{spark.master}} is {{local}} things are a bit tricker because there is no
separate driver process. So instead I use
{{SparkClientUtilities#addJarToContextLoader}} to add the jar to the current
classpath. The constructor for {{LocalHiveSparkClient}} is already doing this,
but for whatever reason the classloader is getting reset by the time
{{SparkPlanGenerator}} runs. I tried debugging it, but didn't have any luck,
and given that its just a test infra issue it doesn't seem work resolving
unless anyone has suggestions on how to fix it.
> Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
> -
>
> Key: HIVE-20032
> URL: https://issues.apache.org/jira/browse/HIVE-20032
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20032.1.patch, HIVE-20032.2.patch,
> HIVE-20032.3.patch, HIVE-20032.4.patch, HIVE-20032.5.patch, HIVE-20032.6.patch
>
>
> Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles,
> then we don't need to serialize the hashCode when shuffling data in HoS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-20032: Attachment: HIVE-20032.6.patch > Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled > - > > Key: HIVE-20032 > URL: https://issues.apache.org/jira/browse/HIVE-20032 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-20032.1.patch, HIVE-20032.2.patch, > HIVE-20032.3.patch, HIVE-20032.4.patch, HIVE-20032.5.patch, HIVE-20032.6.patch > > > Follow up on HIVE-15104, if we don't enable RDD cacheing or groupByShuffles, > then we don't need to serialize the hashCode when shuffling data in HoS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16668) Hive on Spark generates incorrect plan and result with window function and lateral view
[
https://issues.apache.org/jira/browse/HIVE-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550943#comment-16550943
]
Sahil Takiar commented on HIVE-16668:
-
[~csun] do you know if this is still an issue? Is there anything else that
needs to be done for this JIRA (besides probably a rebase) before merging it?
> Hive on Spark generates incorrect plan and result with window function and
> lateral view
> ---
>
> Key: HIVE-16668
> URL: https://issues.apache.org/jira/browse/HIVE-16668
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Attachments: HIVE-16668.1.patch, HIVE-16668.2.patch,
> HIVE-16668.3.patch
>
>
> To reproduce:
> {code}
> create table t1 (a string);
> create table t2 (a array);
> create table dummy (a string);
> insert into table dummy values ("a");
> insert into t1 values ("1"), ("2");
> insert into t2 select array("1", "2", "3", "4") from dummy;
> set hive.auto.convert.join.noconditionaltask.size=3;
> explain
> with tt1 as (
> select a as id, count(*) over () as count
> from t1
> ),
> tt2 as (
> select id
> from t2
> lateral view outer explode(a) a_tbl as id
> )
> select tt1.count
> from tt1 join tt2 on tt1.id = tt2.id;
> {code}
> For Hive on Spark, the plan is:
> {code}
> STAGE DEPENDENCIES:
> Stage-2 is a root stage
> Stage-1 depends on stages: Stage-2
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-2
> Spark
> Edges:
> Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 3), Map 1 (PARTITION-LEVEL
> SORT, 3)
> DagName: chao_20170515133259_de9e0583-da24-4399-afc8-b881dfef0469:9
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: t1
> Statistics: Num rows: 2 Data size: 2 Basic stats: COMPLETE
> Column stats: NONE
> Reduce Output Operator
> key expressions: 0 (type: int)
> sort order: +
> Map-reduce partition columns: 0 (type: int)
> Statistics: Num rows: 2 Data size: 2 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: a (type: string)
> Reducer 2
> Local Work:
> Map Reduce Local Work
> Reduce Operator Tree:
> Select Operator
> expressions: VALUE._col0 (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 2 Data size: 2 Basic stats: COMPLETE
> Column stats: NONE
> PTF Operator
> Function definitions:
> Input definition
> input alias: ptf_0
> output shape: _col0: string
> type: WINDOWING
> Windowing table definition
> input alias: ptf_1
> name: windowingtablefunction
> order by: 0 ASC NULLS FIRST
> partition by: 0
> raw input shape:
> window functions:
> window function definition
> alias: count_window_0
> name: count
> window function: GenericUDAFCountEvaluator
> window frame: PRECEDING(MAX)~FOLLOWING(MAX)
> isStar: true
> Statistics: Num rows: 2 Data size: 2 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: _col0 is not null (type: boolean)
> Statistics: Num rows: 2 Data size: 2 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> expressions: _col0 (type: string), count_window_0
> (type: bigint)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 2 Data size: 2 Basic stats:
> COMPLETE Column stats: NONE
> Spark HashTable Sink Operator
> keys:
> 0 _col0 (type: string)
> 1 _col0 (type: string)
> Reduce Output Operator
> key expressions: _col0 (type: string)
> sort order: +
> Map-reduce partition columns: _col0 (type: string)
> Statistics: Num rows: 2 Data size: 2 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col1 (type: bigint)
> Stage: Stag
[jira] [Commented] (HIVE-4367) enhance TRUNCATE syntax to drop data of external table
[
https://issues.apache.org/jira/browse/HIVE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550890#comment-16550890
]
Hive QA commented on HIVE-4367:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932314/HIVE-4367.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14680 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12732/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12732/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12732/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932314 - PreCommit-HIVE-Build
> enhance TRUNCATE syntax to drop data of external table
> --
>
> Key: HIVE-4367
> URL: https://issues.apache.org/jira/browse/HIVE-4367
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: caofangkun
>Assignee: caofangkun
>Priority: Minor
> Attachments: HIVE-4367-1.patch, HIVE-4367.2.patch.txt,
> HIVE-4367.3.patch, HIVE-4367.4.patch, HIVE-4367.5.patch, HIVE-4367.6.patch,
> HIVE-4367.7.patch
>
>
> In my use case ,
> sometimes I have to remove data of external tables to free up storage space
> of the cluster .
> So it's necessary for to enhance the syntax like
> "TRUNCATE TABLE srcpart_truncate PARTITION (dt='201130412') FORCE;"
> to remove data from EXTERNAL table.
> And I add a configuration property to enable remove data to Trash
>
> hive.truncate.skiptrash
> false
>
> if true will remove data to trash, else false drop data immediately
>
>
> For example :
> hive (default)> TRUNCATE TABLE external1 partition (ds='11');
> FAILED: Error in semantic analysis: Cannot truncate non-managed table
> external1
> hive (default)> TRUNCATE TABLE external1 partition (ds='11') FORCE;
> [2013-04-16 17:15:52]: Compile Start
> [2013-04-16 17:15:52]: Compile End
> [2013-04-16 17:15:52]: OK
> [2013-04-16 17:15:52]: Time taken: 0.413 seconds
> hive (default)> set hive.truncate.skiptrash;
> hive.truncate.skiptrash=false
> hive (default)> set hive.truncate.skiptrash=true;
> hive (default)> TRUNCATE TABLE external1 partition (ds='12') FORCE;
> [2013-04-16 17:16:21]: Compile Start
> [2013-04-16 17:16:21]: Compile End
> [2013-04-16 17:16:21]: OK
> [2013-04-16 17:16:21]: Time taken: 0.143 seconds
> hive (default)> dfs -ls /user/test/.Trash/Current/;
> Found 1 items
> drwxr-xr-x -test supergroup 0 2013-04-16 17:06 /user/test/.Trash/Current/ds=11
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-4367) enhance TRUNCATE syntax to drop data of external table
[
https://issues.apache.org/jira/browse/HIVE-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550860#comment-16550860
]
Hive QA commented on HIVE-4367:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 3 new + 2277 unchanged - 4 fixed = 2280
total (was 2281) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA227 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46266-46279] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA271 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46616-46629] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA286 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46752-46765] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12732/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12732/yetus/new-findbugs-ql.html
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12732/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> enhance TRUNCATE syntax to drop data of external table
> --
>
> Key: HIVE-4367
> URL: https://issues.apache.org/jira/browse/HIVE-4367
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 0.11.0
>Reporter: caofangkun
>Assignee: caofangkun
>Priority: Minor
> Attachments: HIVE-4367-1.patch, HIVE-4367.2.patch.txt,
> HIVE-4367.3.patch, HIVE-4367.4.patch, HIVE-4367.5.patch, HIVE-4367.6.patch,
> HIVE-4367.7.patch
>
>
> In my use case ,
> sometimes I have to remove data of external tables to free up storage space
[jira] [Commented] (HIVE-20212) Hiveserver2 in http mode emitting metric default.General.open_connections incorrectly
[
https://issues.apache.org/jira/browse/HIVE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550816#comment-16550816
]
Hive QA commented on HIVE-20212:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932317/HIVE-20212.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14674 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=191)
[druidmini_dynamic_partition.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test1.q,druidmini_test_insert.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12731/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12731/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12731/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932317 - PreCommit-HIVE-Build
> Hiveserver2 in http mode emitting metric default.General.open_connections
> incorrectly
> -
>
> Key: HIVE-20212
> URL: https://issues.apache.org/jira/browse/HIVE-20212
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-20212.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20212) Hiveserver2 in http mode emitting metric default.General.open_connections incorrectly
[
https://issues.apache.org/jira/browse/HIVE-20212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550766#comment-16550766
]
Hive QA commented on HIVE-20212:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12731/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12731/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hiveserver2 in http mode emitting metric default.General.open_connections
> incorrectly
> -
>
> Key: HIVE-20212
> URL: https://issues.apache.org/jira/browse/HIVE-20212
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-20212.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[
https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550754#comment-16550754
]
Hive QA commented on HIVE-20203:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932312/HIVE-20203.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14679 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12730/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12730/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12730/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932312 - PreCommit-HIVE-Build
> Arrow SerDe leaks a DirectByteBuffer
>
>
> Key: HIVE-20203
> URL: https://issues.apache.org/jira/browse/HIVE-20203
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch,
> HIVE-20203.3.patch
>
>
> ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task
> that uses the serde.
> The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool.
> This buffer is never closed and leaks about 1K of physical memory for each
> task.
> This patch does three things:
> # Ensure the buffer is closed when the RecordWriter for the task is closed.
> # Adds per-task memory accounting by assigning a ChildAllocator to each task
> from the RootAllocator.
> # Enforces that the ChildAllocator for a task has released all memory
> assigned to it, when the task is completed.
> The patch assumes that close() is always called on the RecordWriter when a
> task is finished (even if there is a failure during task execution).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20203) Arrow SerDe leaks a DirectByteBuffer
[
https://issues.apache.org/jira/browse/HIVE-20203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550737#comment-16550737
]
Hive QA commented on HIVE-20203:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 2 new + 303 unchanged - 0
fixed = 305 total (was 303) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12730/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12730/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12730/yetus/whitespace-eol.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12730/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Arrow SerDe leaks a DirectByteBuffer
>
>
> Key: HIVE-20203
> URL: https://issues.apache.org/jira/browse/HIVE-20203
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Blocker
> Attachments: HIVE-20203.1.patch, HIVE-20203.2.patch,
> HIVE-20203.3.patch
>
>
> ArrowColumnarBatchSerDe allocates an arrow NullableMapVector for each task
> that uses the serde.
> The vector is a DirectByteBuffer allocated from Arrow's off-heap buffer pool.
> This buffer is never closed and leaks about 1K of physical memory for each
> task.
> This patch does three things:
> # Ensure the buffer is closed when the RecordWriter for the task is closed.
> # Adds per-task memory accounting by assigning a ChildAllocator to each task
> from the
[jira] [Commented] (HIVE-20097) Convert standalone-metastore to a submodule
[
https://issues.apache.org/jira/browse/HIVE-20097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550712#comment-16550712
]
Hive QA commented on HIVE-20097:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932308/HIVE-20097.07-branch-3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12729/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12729/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12729/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-20 12:08:01.356
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12729/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-3 ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-20 12:08:01.360
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 851c8ab HIVE-17896: TopNKey: Create a standalone vectorizable
TopNKey operator (Teddy Choi, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout branch-3
Switched to branch 'branch-3'
Your branch is up-to-date with 'origin/branch-3'.
+ git reset --hard origin/branch-3
HEAD is now at cc294d3 HIVE-17896: TopNKey: Create a standalone vectorizable
TopNKey operator (Teddy Choi, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/branch-3
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-20 12:08:03.816
+ rm -rf ../yetus_PreCommit-HIVE-Build-12729
+ mkdir ../yetus_PreCommit-HIVE-Build-12729
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12729
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12729/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git apply: bad git-diff - inconsistent old filename on line 5423
Falling back to three-way merge...
error:
standalone-metastore/metastore-common/src/gen/version/org/apache/hadoop/hive/metastore/annotation/package-info.java:
does not exist in index
error: cannot read the current contents of
'standalone-metastore/metastore-common/src/gen/version/org/apache/hadoop/hive/metastore/annotation/package-info.java'
error:
standalone-metastore/metastore-common/src/gen/version/org/apache/hadoop/hive/metastore/annotation/package-info.java:
patch does not apply
fatal: git diff header lacks filename information when removing 2 leading
pathname components (line 1719)
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12729
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932308 - PreCommit-HIVE-Build
> Convert standalone-metastore to a submodule
> ---
>
> Key: HIVE-20097
> URL: https://issues.apache.org/jira/browse/HIVE-20097
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, Metastore, Standalone Metastore
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20097.01.patch, HIVE-20097.02.patch,
> HIVE-20097.03.patch, HIVE-20097.04.patch, HIVE-20097.05.patch,
> HIVE-20097.06.patch, HIVE-20097.07-branch-3.patch, HIVE-20097.07.patch
>
>
> This is a subtask to stage HIVE-17751 changes into several smaller phases.
> The first part is moving existing code in hive-standalone-metastore to a
> sub-module.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20204) Type conversion during IN () comparisons is using different rules from other comparison operations
[
https://issues.apache.org/jira/browse/HIVE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550708#comment-16550708
]
Hive QA commented on HIVE-20204:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932304/HIVE-20204.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14679 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins]
(batchId=192)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking]
(batchId=192)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12728/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12728/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12728/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932304 - PreCommit-HIVE-Build
> Type conversion during IN () comparisons is using different rules from other
> comparison operations
> --
>
> Key: HIVE-20204
> URL: https://issues.apache.org/jira/browse/HIVE-20204
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20204.1.patch, HIVE-20204.2.patch
>
>
> Noticed this while looking at HIVE-20082.
> The type conversion done during GenericUDFIn (via
> ReturnObjectInspectorResolver) uses FunctionRegistry.getCommonClass(),
> whereas the other comparison operators (=, <, >, <=, >=) use
> FunctionRegistry.getCommonClassForComparison(). As a result, dec_column IN
> ('1.1', '2.2') compares the values as strings, whereas dec_column = '1.1'
> would compare the values as doubles. This makes a difference for HIVE-20082
> since it is related to changing the 0-padding during decimal-to-string
> conversions.
> cc [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20204) Type conversion during IN () comparisons is using different rules from other comparison operations
[
https://issues.apache.org/jira/browse/HIVE-20204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550676#comment-16550676
]
Hive QA commented on HIVE-20204:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 4 new + 153 unchanged - 1
fixed = 157 total (was 154) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12728/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12728/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12728/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Type conversion during IN () comparisons is using different rules from other
> comparison operations
> --
>
> Key: HIVE-20204
> URL: https://issues.apache.org/jira/browse/HIVE-20204
> Project: Hive
> Issue Type: Bug
> Components: Types
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20204.1.patch, HIVE-20204.2.patch
>
>
> Noticed this while looking at HIVE-20082.
> The type conversion done during GenericUDFIn (via
> ReturnObjectInspectorResolver) uses FunctionRegistry.getCommonClass(),
> whereas the other comparison operators (=, <, >, <=, >=) use
> FunctionRegistry.getCommonClassForComparison(). As a result, dec_column IN
> ('1.1', '2.2') compares the values as strings, whereas dec_column = '1.1'
> would compare the values as doubles. This makes a difference for HIVE-20082
> since it is related to changing the 0-padding during decimal-to-string
> conversions.
> cc [~ashutoshc]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17683) Annotate Query Plan with locking information
[ https://issues.apache.org/jira/browse/HIVE-17683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko updated HIVE-17683: - Attachment: HIVE-17683.06.patch > Annotate Query Plan with locking information > > > Key: HIVE-17683 > URL: https://issues.apache.org/jira/browse/HIVE-17683 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Igor Kryvenko >Priority: Critical > Attachments: HIVE-17683.01.patch, HIVE-17683.02.patch, > HIVE-17683.03.patch, HIVE-17683.04.patch, HIVE-17683.05.patch, > HIVE-17683.06.patch > > > Explore if it's possible to add info about what locks will be asked for to > the query plan. > Lock acquisition (for Acid Lock Manager) is done in > DbTxnManager.acquireLocks() which is called once the query starts running. > Would need to refactor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19990) Query with interval literal in join condition fails
[
https://issues.apache.org/jira/browse/HIVE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550653#comment-16550653
]
Hive QA commented on HIVE-19990:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932301/HIVE-19990.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14679 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12727/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12727/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12727/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932301 - PreCommit-HIVE-Build
> Query with interval literal in join condition fails
> ---
>
> Key: HIVE-19990
> URL: https://issues.apache.org/jira/browse/HIVE-19990
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19990.1.patch, HIVE-19990.2.patch,
> HIVE-19990.3.patch, HIVE-19990.4.patch
>
>
> *Reproducer*
> {code:sql}
> > create table date_dim_d1(
> d_week_seqint,
> d_datestring);
> > SELECT
>d1.d_week_seq
> FROM
>date_dim_d1 d1
>JOIN date_dim_d1 d3
> WHERE
>Cast(d3.d_date AS date) > Cast(d1.d_date AS date) + INTERVAL '5' day ;
> {code}
> *Exception*
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: '5 00:00:00.0'
> encountered with 0 children
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2780)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2775)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:3060)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2959)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:9633)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11380)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11285)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:593)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12150)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:658)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1829)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1776)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1771)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:832)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:770)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:694)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19990) Query with interval literal in join condition fails
[
https://issues.apache.org/jira/browse/HIVE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550623#comment-16550623
]
Hive QA commented on HIVE-19990:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 9 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12727/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12727/yetus/whitespace-tabs.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12727/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query with interval literal in join condition fails
> ---
>
> Key: HIVE-19990
> URL: https://issues.apache.org/jira/browse/HIVE-19990
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19990.1.patch, HIVE-19990.2.patch,
> HIVE-19990.3.patch, HIVE-19990.4.patch
>
>
> *Reproducer*
> {code:sql}
> > create table date_dim_d1(
> d_week_seqint,
> d_datestring);
> > SELECT
>d1.d_week_seq
> FROM
>date_dim_d1 d1
>JOIN date_dim_d1 d3
> WHERE
>Cast(d3.d_date AS date) > Cast(d1.d_date AS date) + INTERVAL '5' day ;
> {code}
> *Exception*
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: '5 00:00:00.0'
> encountered with 0 children
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2780)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2775)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:3060)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2959)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:9633)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(Semant
[jira] [Commented] (HIVE-20217) bucket7.q test results should be sorted
[ https://issues.apache.org/jira/browse/HIVE-20217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550618#comment-16550618 ] Adam Szita commented on HIVE-20217: --- +1 pending tests > bucket7.q test results should be sorted > --- > > Key: HIVE-20217 > URL: https://issues.apache.org/jira/browse/HIVE-20217 > Project: Hive > Issue Type: Test > Components: Hive, Test >Affects Versions: 4.0.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-20217.patch > > > The test results should be sorted, so they will be the same on different > configs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-14650) Select fails when ORC file has more columns than table schema
[
https://issues.apache.org/jira/browse/HIVE-14650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tartarus updated HIVE-14650:
Description:
When SELECTing from a Hive ORC table, the following IndexOutOfBoundsException
is thrown if the underlying ORC file has 4 or more columns than the Hive schema
(where N is the number of columns in the ORC file).
{noformat}
Failed with exception java.io.IOException:java.lang.IndexOutOfBoundsException:
toIndex = N
16/08/25 15:22:19 ERROR CliDriver: Failed with exception
java.io.IOException:java.lang.IndexOutOfBoundsException: toIndex = N
java.io.IOException: java.lang.IndexOutOfBoundsException: toIndex = N
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:507)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1686)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IndexOutOfBoundsException: toIndex = 6
at java.util.ArrayList.subListRangeCheck(ArrayList.java:1004)
at java.util.ArrayList.subList(ArrayList.java:996)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202)
at
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:541)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1216)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113)
at
org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:673)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:323)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:445)
... 15 more
{noformat}
This error appears to be related to the patch of HIVE-10591.
Steps to reproduce (Hive QL):
{noformat}
DROP TABLE IF EXISTS orc_drop_column;
CREATE TABLE orc_drop_column (`id` int, `name` string, `description` string,
`somevalue` double, `someflag` boolean, `somedate` timestamp) STORED AS ORC;
INSERT INTO TABLE orc_drop_column select * from (select 1, 'my_name',
'my_desc', 5.5, true, '2016-08-25 06:00:00') a;
ALTER TABLE orc_drop_column SET SERDE
'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe';
ALTER TABLE orc_drop_column REPLACE COLUMNS (
`id` int,
`name` string
);
ALTER TABLE orc_drop_column SET SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde';
SELECT id, name FROM orc_drop_column;
{noformat}
was:
When SELECTing from a Hive ORC table, the following IndexOutOfBoundsException
is thrown if the underlying ORC file has 4 or more columns than the Hive schema
(where N is the number of columns in the ORC file).
{noformat}
Failed with exception java.io.IOException:java.lang.IndexOutOfBoundsException:
toIndex = N
16/08/25 15:22:19 ERROR CliDriver: Failed with exception
java.io.IOException:java.lang.IndexOutOfBoundsException: toIndex = N
java.io.IOException: java.lang.IndexOutOfBoundsException: toIndex = N
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:507)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1686)
at
org.apa
[jira] [Commented] (HIVE-20115) Acid tables should not use footer scan for analyze
[
https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550594#comment-16550594
]
Hive QA commented on HIVE-20115:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932303/HIVE-20115.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14679 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=262)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12726/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12726/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12726/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932303 - PreCommit-HIVE-Build
> Acid tables should not use footer scan for analyze
> --
>
> Key: HIVE-20115
> URL: https://issues.apache.org/jira/browse/HIVE-20115
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics, Transactions
>Affects Versions: 4.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20115.01.patch, HIVE-20115.01.patch,
> HIVE-20115.patch
>
>
> Discovered via incorrect stats in acid_no_buckets test on master-txnstats
> branch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20115) Acid tables should not use footer scan for analyze
[
https://issues.apache.org/jira/browse/HIVE-20115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550550#comment-16550550
]
Hive QA commented on HIVE-20115:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 10 new + 34 unchanged - 0
fixed = 44 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12726/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12726/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12726/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Acid tables should not use footer scan for analyze
> --
>
> Key: HIVE-20115
> URL: https://issues.apache.org/jira/browse/HIVE-20115
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics, Transactions
>Affects Versions: 4.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20115.01.patch, HIVE-20115.01.patch,
> HIVE-20115.patch
>
>
> Discovered via incorrect stats in acid_no_buckets test on master-txnstats
> branch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20217) bucket7.q test results should be sorted
[ https://issues.apache.org/jira/browse/HIVE-20217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550540#comment-16550540 ] Peter Vary commented on HIVE-20217: --- [~szita]: Could you please review? :) > bucket7.q test results should be sorted > --- > > Key: HIVE-20217 > URL: https://issues.apache.org/jira/browse/HIVE-20217 > Project: Hive > Issue Type: Test > Components: Hive, Test >Affects Versions: 4.0.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-20217.patch > > > The test results should be sorted, so they will be the same on different > configs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20217) bucket7.q test results should be sorted
[ https://issues.apache.org/jira/browse/HIVE-20217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20217: -- Status: Patch Available (was: Open) > bucket7.q test results should be sorted > --- > > Key: HIVE-20217 > URL: https://issues.apache.org/jira/browse/HIVE-20217 > Project: Hive > Issue Type: Test > Components: Hive, Test >Affects Versions: 4.0.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-20217.patch > > > The test results should be sorted, so they will be the same on different > configs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20217) bucket7.q test results should be sorted
[ https://issues.apache.org/jira/browse/HIVE-20217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20217: -- Attachment: HIVE-20217.patch > bucket7.q test results should be sorted > --- > > Key: HIVE-20217 > URL: https://issues.apache.org/jira/browse/HIVE-20217 > Project: Hive > Issue Type: Test > Components: Hive, Test >Affects Versions: 4.0.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-20217.patch > > > The test results should be sorted, so they will be the same on different > configs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20202) Add profiler endpoint to HS2 and LLAP
[
https://issues.apache.org/jira/browse/HIVE-20202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550513#comment-16550513
]
Hive QA commented on HIVE-20202:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932302/HIVE-20202.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12725/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12725/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12725/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12932302/HIVE-20202.5.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932302 - PreCommit-HIVE-Build
> Add profiler endpoint to HS2 and LLAP
> -
>
> Key: HIVE-20202
> URL: https://issues.apache.org/jira/browse/HIVE-20202
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-20202.1.patch, HIVE-20202.2.patch,
> HIVE-20202.3.patch, HIVE-20202.4.patch, HIVE-20202.5.patch
>
>
> Add a web endpoint for profiling based on async-profiler. This servlet should
> be added to httpserver so that HS2 and LLAP daemons can output flamegraphs
> when their /prof endpoint is hit. Since this will be based on
> [https://github.com/jvm-profiling-tools/async-profiler] heap allocation, lock
> contentions, HW counters etc. will also be supported in addition to cpu
> profiling. In most cases the profiling overhead is pretty low and is safe to
> run on production. More analysis on CPU and memory overhead here
> [https://github.com/jvm-profiling-tools/async-profiler/issues/14] and
> [https://github.com/jvm-profiling-tools/async-profiler/issues/131]
>
> For the impatient, here is the usage doc and the sample output
> [https://github.com/prasanthj/nightswatch/blob/master/README.md]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19733) RemoteSparkJobStatus#getSparkStageProgress inefficient implementation
[
https://issues.apache.org/jira/browse/HIVE-19733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550511#comment-16550511
]
Hive QA commented on HIVE-19733:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932290/HIVE-19733.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14679 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12724/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12724/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12724/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932290 - PreCommit-HIVE-Build
> RemoteSparkJobStatus#getSparkStageProgress inefficient implementation
> -
>
> Key: HIVE-19733
> URL: https://issues.apache.org/jira/browse/HIVE-19733
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19733.1.patch, HIVE-19733.2.patch,
> HIVE-19733.3.patch
>
>
> The implementation of {{RemoteSparkJobStatus#getSparkStageProgress}} is a bit
> inefficient. There is one RPC call to get the {{SparkJobInfo}} and then for
> every stage there is another RPC call to get each {{SparkStageInfo}}. This
> could all be done in a single RPC call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20217) bucket7.q test results should be sorted
[ https://issues.apache.org/jira/browse/HIVE-20217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-20217: - > bucket7.q test results should be sorted > --- > > Key: HIVE-20217 > URL: https://issues.apache.org/jira/browse/HIVE-20217 > Project: Hive > Issue Type: Test > Components: Hive, Test >Affects Versions: 4.0.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > > The test results should be sorted, so they will be the same on different > configs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19733) RemoteSparkJobStatus#getSparkStageProgress inefficient implementation
[
https://issues.apache.org/jira/browse/HIVE-19733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550371#comment-16550371
]
Hive QA commented on HIVE-19733:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 23 unchanged - 0 fixed
= 24 total (was 23) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12724/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12724/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12724/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> RemoteSparkJobStatus#getSparkStageProgress inefficient implementation
> -
>
> Key: HIVE-19733
> URL: https://issues.apache.org/jira/browse/HIVE-19733
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19733.1.patch, HIVE-19733.2.patch,
> HIVE-19733.3.patch
>
>
> The implementation of {{RemoteSparkJobStatus#getSparkStageProgress}} is a bit
> inefficient. There is one RPC call to get the {{SparkJobInfo}} and then for
> every stage there is another RPC call to get each {{SparkStageInfo}}. This
> could all be done in a single RPC call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20192) HS2 with embedded metastore is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20192:
Attachment: (was: HIVE-20192.01.patch)
> HS2 with embedded metastore is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20192.01.patch
>
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20131) SQL Script changes for creating txn write notification in 3.2.0 files
[ https://issues.apache.org/jira/browse/HIVE-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20131 started by mahesh kumar behera. -- > SQL Script changes for creating txn write notification in 3.2.0 files > --- > > Key: HIVE-20131 > URL: https://issues.apache.org/jira/browse/HIVE-20131 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-20131.01.patch, HIVE-20131.02.patch > > > 1. Change partition name size from 1024 to 767 . (mySQL 5.6 and before that > supports max 767 length keys) > 2. Remove the create txn_write_notification_log table creation from 3.1.0 > scripts and add a new scripts for 3.2.0 > 3. Remove the file 3.1.0-to-4.0.0 and instead add file for 3.2.0-to-4.0.0 and > 3.1.0-to-3.2.0 > 4. Change in metastore init schema xml file to take 4.0.0 instead of 3.1.0 > as current version. > h1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20131) SQL Script changes for creating txn write notification in 3.2.0 files
[ https://issues.apache.org/jira/browse/HIVE-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20131: --- Status: Patch Available (was: In Progress) > SQL Script changes for creating txn write notification in 3.2.0 files > --- > > Key: HIVE-20131 > URL: https://issues.apache.org/jira/browse/HIVE-20131 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-20131.01.patch, HIVE-20131.02.patch > > > 1. Change partition name size from 1024 to 767 . (mySQL 5.6 and before that > supports max 767 length keys) > 2. Remove the create txn_write_notification_log table creation from 3.1.0 > scripts and add a new scripts for 3.2.0 > 3. Remove the file 3.1.0-to-4.0.0 and instead add file for 3.2.0-to-4.0.0 and > 3.1.0-to-3.2.0 > 4. Change in metastore init schema xml file to take 4.0.0 instead of 3.1.0 > as current version. > h1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20131) SQL Script changes for creating txn write notification in 3.2.0 files
[ https://issues.apache.org/jira/browse/HIVE-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20131: --- Status: Open (was: Patch Available) > SQL Script changes for creating txn write notification in 3.2.0 files > --- > > Key: HIVE-20131 > URL: https://issues.apache.org/jira/browse/HIVE-20131 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-20131.01.patch, HIVE-20131.02.patch > > > 1. Change partition name size from 1024 to 767 . (mySQL 5.6 and before that > supports max 767 length keys) > 2. Remove the create txn_write_notification_log table creation from 3.1.0 > scripts and add a new scripts for 3.2.0 > 3. Remove the file 3.1.0-to-4.0.0 and instead add file for 3.2.0-to-4.0.0 and > 3.1.0-to-3.2.0 > 4. Change in metastore init schema xml file to take 4.0.0 instead of 3.1.0 > as current version. > h1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20131) SQL Script changes for creating txn write notification in 3.2.0 files
[ https://issues.apache.org/jira/browse/HIVE-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20131: --- Attachment: HIVE-20131.02.patch > SQL Script changes for creating txn write notification in 3.2.0 files > --- > > Key: HIVE-20131 > URL: https://issues.apache.org/jira/browse/HIVE-20131 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 4.0.0 > > Attachments: HIVE-20131.01.patch, HIVE-20131.02.patch > > > 1. Change partition name size from 1024 to 767 . (mySQL 5.6 and before that > supports max 767 length keys) > 2. Remove the create txn_write_notification_log table creation from 3.1.0 > scripts and add a new scripts for 3.2.0 > 3. Remove the file 3.1.0-to-4.0.0 and instead add file for 3.2.0-to-4.0.0 and > 3.1.0-to-3.2.0 > 4. Change in metastore init schema xml file to take 4.0.0 instead of 3.1.0 > as current version. > h1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20192) HS2 with embedded metastore is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20192:
Status: Open (was: Patch Available)
> HS2 with embedded metastore is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20192.01.patch
>
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20192) HS2 with embedded metastore is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20192:
Status: Patch Available (was: Open)
> HS2 with embedded metastore is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20192.01.patch
>
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20192) HS2 with embedded metastore is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20192:
Attachment: HIVE-20192.01.patch
> HS2 with embedded metastore is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20192.01.patch
>
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20192) HS2 with embedded metastore is leaking JDOPersistenceManager objects.
[
https://issues.apache.org/jira/browse/HIVE-20192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-20192:
Status: Open (was: Patch Available)
> HS2 with embedded metastore is leaking JDOPersistenceManager objects.
> -
>
> Key: HIVE-20192
> URL: https://issues.apache.org/jira/browse/HIVE-20192
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: HiveServer2, pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-20192.01.patch
>
>
> Hiveserver2 instances where crashing every 3-4 days and observed HS2 in on
> unresponsive state. Also, observed that the FGC collection happening regularly
> From JXray report it is seen that pmCache(List of JDOPersistenceManager
> objects) is occupying 84% of the heap and there are around 16,000 references
> of UDFClassLoader.
> {code:java}
> 10,759,230K (84.7%) Object tree for GC root(s) Java Static
> org.apache.hadoop.hive.metastore.ObjectStore.pmf
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.pmCache ↘ 10,744,419K
> (84.6%), 1 reference(s)
> - j.u.Collections$SetFromMap.m ↘ 10,744,419K (84.6%), 1 reference(s)
> - {java.util.concurrent.ConcurrentHashMap}.keys ↘ 10,743,764K (84.5%),
> 16,872 reference(s)
> - org.datanucleus.api.jdo.JDOPersistenceManager.ec ↘ 10,738,831K
> (84.5%), 16,872 reference(s)
> ... 3 more references together retaining 4,933K (< 0.1%)
> - java.util.concurrent.ConcurrentHashMap self 655K (< 0.1%), 1 object(s)
> ... 2 more references together retaining 48b (< 0.1%)
> - org.datanucleus.api.jdo.JDOPersistenceManagerFactory.nucleusContext ↘
> 14,810K (0.1%), 1 reference(s)
> ... 3 more references together retaining 96b (< 0.1%){code}
> When the RawStore object is re-created, it is not allowed to be updated into
> the ThreadWithGarbageCleanup.threadRawStoreMap which leads to the new
> RawStore never gets cleaned-up when the thread exit.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19441) Use LLAP test Driver for Druid test suites
[
https://issues.apache.org/jira/browse/HIVE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550344#comment-16550344
]
Hive QA commented on HIVE-19441:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932286/HIVE-19441.13.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12723/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12723/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12723/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12932286/HIVE-19441.13.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932286 - PreCommit-HIVE-Build
> Use LLAP test Driver for Druid test suites
> --
>
> Key: HIVE-19441
> URL: https://issues.apache.org/jira/browse/HIVE-19441
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19441.10.patch, HIVE-19441.11.patch,
> HIVE-19441.12.patch, HIVE-19441.13.patch, HIVE-19441.2.patch,
> HIVE-19441.3.patch, HIVE-19441.4.patch, HIVE-19441.5.patch,
> HIVE-19441.6.patch, HIVE-19441.7.patch, HIVE-19441.8.patch,
> HIVE-19441.8.patch, HIVE-19441.9.patch, HIVE-19441.patch
>
>
> Adding support to the float kind aggregator.
> Use LLAP as test Driver to reduce execution time of tests from about 2 hours
> to 15 min:
> Although this patches unveiling an issue with timezone, maybe it is fixed by
> [~jcamachorodriguez] upcoming set of patches.
>
> Before
> {code}
> [INFO] Executed tasks
> [INFO]
> [INFO] --- maven-compiler-plugin:3.6.1:testCompile (default-testCompile) @
> hive-it-qfile ---
> [INFO] Compiling 21 source files to
> /Users/sbouguerra/Hdev/hive/itests/qtest/target/test-classes
> [INFO]
> [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hive-it-qfile
> ---
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 6,654.117 s - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 01:51 h
> [INFO] Finished at: 2018-05-04T12:43:19-07:00
> [INFO]
>
> {code}
> After
> {code}
> INFO] Executed tasks
> [INFO]
> [INFO] --- maven-compiler-plugin:3.6.1:testCompile (default-testCompile) @
> hive-it-qfile ---
> [INFO] Compiling 22 source files to
> /Users/sbouguerra/Hdev/hive/itests/qtest/target/test-classes
> [INFO]
> [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hive-it-qfile
> ---
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 907.167 s - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 15:31 min
> [INFO] Finished at: 2018-05-04T13:15:11-07:00
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19927) Last Repl ID set by bootstrap dump is incorrect and may cause data loss if have ACID/MM tables.
[
https://issues.apache.org/jira/browse/HIVE-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550342#comment-16550342
]
Hive QA commented on HIVE-19927:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932104/HIVE-19927.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12722/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12722/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12722/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12932104/HIVE-19927.01.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932104 - PreCommit-HIVE-Build
> Last Repl ID set by bootstrap dump is incorrect and may cause data loss if
> have ACID/MM tables.
> ---
>
> Key: HIVE-19927
> URL: https://issues.apache.org/jira/browse/HIVE-19927
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, Transactions
>Affects Versions: 3.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, pull-request-available, replication
> Attachments: HIVE-19927.01.patch
>
>
> During bootstrap dump of ACID tables, let's consider the below sequence.
> - Current session (REPL DUMP), Open txn (Txn1) - Event-10
> - Another session (Session-2), Open txn (Txn2) - Event-11
> - Session-2 -> Insert data (T1.D1) to ACID table. - Event-12
> - Get lastReplId = last event ID logged. (Event-12)
> - Session-2 -> Commit Txn (Txn2) - Event-13
> - Dump ACID tables based on validTxnList based on Txn1. --> This step skips
> all the data written by txns > Txn1. So, T1.D1 will be missing.
> - Commit Txn (Txn1)
> - REPL LOAD from bootstrap dump will skip T1.D1.
> - Incremental REPL DUMP will start from Event-13 and hence lose Txn2 which is
> opened after Txn1. So, data T1.D1 will be lost for ever.
> Proposed to capture the lastReplId of bootstrap before opening current txn
> (Txn1) and store it in Driver context and use it for dump.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19937) Intern fields in MapWork on deserialization
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550341#comment-16550341
]
Hive QA commented on HIVE-19937:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932285/HIVE-19937.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12721/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12721/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12721/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12932285/HIVE-19937.4.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932285 - PreCommit-HIVE-Build
> Intern fields in MapWork on deserialization
> ---
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch, HIVE-19937.2.patch,
> HIVE-19937.3.patch, HIVE-19937.4.patch, post-patch-report.html, report.html
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19937) Intern fields in MapWork on deserialization
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550340#comment-16550340
]
Hive QA commented on HIVE-19937:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932285/HIVE-19937.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14679 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth (batchId=263)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12720/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12720/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12720/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932285 - PreCommit-HIVE-Build
> Intern fields in MapWork on deserialization
> ---
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch, HIVE-19937.2.patch,
> HIVE-19937.3.patch, HIVE-19937.4.patch, post-patch-report.html, report.html
>
>
> When fixing HIVE-16395, we decided that each new Spark task should clone the
> {{JobConf}} object to prevent any {{ConcurrentModificationException}} from
> being thrown. However, setting this variable comes at a cost of storing a
> duplicate {{JobConf}} object for each Spark task. These objects can take up a
> significant amount of memory, we should intern them so that Spark tasks
> running in the same JVM don't store duplicate copies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20150) TopNKey pushdown
[ https://issues.apache.org/jira/browse/HIVE-20150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550338#comment-16550338 ] Teddy Choi commented on HIVE-20150: --- [~jcamachorodriguez], it looks good. I'll work on its implementation. When I get some more idea, I'll share it here. > TopNKey pushdown > > > Key: HIVE-20150 > URL: https://issues.apache.org/jira/browse/HIVE-20150 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > > TopNKey operator is implemented in HIVE-17896, but it needs more work in > pushdown implementation. So this issue covers TopNKey pushdown implementation > with proper tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19937) Intern fields in MapWork on deserialization
[
https://issues.apache.org/jira/browse/HIVE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550312#comment-16550312
]
Hive QA commented on HIVE-19937:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2281 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 5 new + 298 unchanged - 4
fixed = 303 total (was 302) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 3 new + 2280 unchanged - 1 fixed = 2283
total (was 2281) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Redundant nullcheck of nominal which is known to be null in
org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(Path)
Redundant null check at AbstractMapOperator.java:is known to be null in
org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(Path)
Redundant null check at AbstractMapOperator.java:[line 113] |
| | org.apache.hadoop.hive.ql.exec.SerializationUtilities$MapWorkSerializer
implements Comparator but not Serializable At
SerializationUtilities.java:Serializable At SerializationUtilities.java:[lines
551-562] |
| |
org.apache.hadoop.hive.ql.exec.SerializationUtilities$PartitionDescSerializer
implements Comparator but not Serializable At
SerializationUtilities.java:Serializable At SerializationUtilities.java:[lines
572-585] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12720/dev-support/hive-personality.sh
|
| git revision | master / 851c8ab |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12720/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12720/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12720/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Intern fields in MapWork on deserialization
> ---
>
> Key: HIVE-19937
> URL: https://issues.apache.org/jira/browse/HIVE-19937
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19937.1.patch, HIVE-19937.2.patch,
> HIVE-19937.3.patch, HIVE-19937.4.patch, post-patch-report.html, report.html
>
>
> When fixing HIVE-
