[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669300#comment-16669300 ] Eugene Ilchenko commented on HIVE-20679: [~anishek], found a 26MB single record in the MySQL table now. Seems like we found a root cause - "MSCK repair" for new tables. We have attached the insert statement to the original HW ticket. > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.8.patch, HIVE-20679.9.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663469#comment-16663469 ] anishek commented on HIVE-20679: Thanks [~sankarh] created HIVE-20806 for the same. please review. > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.8.patch, HIVE-20679.9.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662527#comment-16662527 ] Sankar Hariappan commented on HIVE-20679: - [~anishek] Couple of new files added (Serialiser.java and Deserialiser.java) by this patch doesn't have ASF license header. Can you please add it? > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.8.patch, HIVE-20679.9.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658630#comment-16658630
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944936/HIVE-20679.9.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15496 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14604/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14604/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14604/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944936 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch,
> HIVE-20679.6.patch, HIVE-20679.8.patch, HIVE-20679.9.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
> Edit: For notification_log table the message column for all supported
> databases can store messages from 2GB to 4GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656131#comment-16656131
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944493/HIVE-20679.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14556/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14556/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14556/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12944493/HIVE-20679.8.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944493 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch,
> HIVE-20679.6.patch, HIVE-20679.8.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
> Edit: For notification_log table the message column for all supported
> databases can store messages from 2GB to 4GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655846#comment-16655846
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944493/HIVE-20679.8.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15184 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=154)
[intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14550/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14550/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14550/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944493 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch,
> HIVE-20679.6.patch, HIVE-20679.8.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
> Edit: For notification_log table the message column for all supported
> databases can store messages from 2GB to 4GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655835#comment-16655835
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
16s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
2s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
15s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
36s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
32s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
36s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 34 new + 814
unchanged - 36 fixed = 848 total (was 850) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 5 new + 36 unchanged - 5 fixed
= 41 total (was 41) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
37s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} |
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654749#comment-16654749 ] Sankar Hariappan commented on HIVE-20679: - +1, pending tests > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654645#comment-16654645 ] anishek commented on HIVE-20679: [~sankarh] please review > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654641#comment-16654641 ] ASF GitHub Bot commented on HIVE-20679: --- Github user anishek closed the pull request at: https://github.com/apache/hive/pull/446 > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654643#comment-16654643 ] ASF GitHub Bot commented on HIVE-20679: --- GitHub user anishek opened a pull request: https://github.com/apache/hive/pull/449 HIVE-20679: DDL operations on hive might create large messages for DBNotification You can merge this pull request into a Git repository by running: $ git pull https://github.com/anishek/hive HIVE-20679 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/449.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #449 commit 9767cec4e7eb9d05afa6ac6f9279a6358cd50815 Author: Anishek Agarwal Date: 2018-10-03T09:28:40Z HIVE-20679: DDL operations on hive might create large messages for DBNotification > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, > HIVE-20679.6.patch, HIVE-20679.7.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654316#comment-16654316
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12944301/HIVE-20679.6.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15198 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14528/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14528/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14528/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12944301 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch,
> HIVE-20679.6.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
> Edit: For notification_log table the message column for all supported
> databases can store messages from 2GB to 4GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654295#comment-16654295
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
24s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
7s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
38s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
33s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
40s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 40s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} itests/hive-unit: The patch generated 34 new + 814
unchanged - 36 fixed = 848 total (was 850) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 5 new + 36 unchanged - 5 fixed
= 41 total (was 41) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
35s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} |
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650511#comment-16650511 ] Bharathkrishna Guruvayoor Murali commented on HIVE-20679: - Hi, Can you please also add a review board link for the patch? > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650326#comment-16650326
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12943883/HIVE-20679.5.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15112 tests
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=244)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=244)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=244)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely
timed out) (batchId=244)
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=244)
TestReplicationScenariosIncrementalLoadAcidTables - did not produce a
TEST-*.xml file (likely timed out) (batchId=244)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs
(batchId=268)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14474/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14474/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14474/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12943883 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, HIVE-20679.4.patch, HIVE-20679.5.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
> Edit: For notification_log table the message column for all supported
> databases can store messages from 2GB to 4GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650299#comment-16650299
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
24s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
38s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
18s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
44s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 44s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
26s{color} | {color:red} itests/hive-unit: The patch generated 35 new + 812
unchanged - 35 fixed = 847 total (was 847) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 5 new + 36 unchanged - 5 fixed
= 41 total (was 41) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} |
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649194#comment-16649194
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12943617/HIVE-20679.4.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15105 tests
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=243)
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=154)
[intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=243)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=243)
TestReplicationScenariosIncrementalLoadAcidTables - did not produce a
TEST-*.xml file (likely timed out) (batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14420/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14420/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14420/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12943617 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, HIVE-20679.4.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
> Edit: For notification_log table the message column for all supported
> databases can store messages from 2GB to 4GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649188#comment-16649188
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
23s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 2318 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
35s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
32s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
38s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 38s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} itests/hive-unit: The patch generated 39 new + 812
unchanged - 35 fixed = 851 total (was 847) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 5 new + 36 unchanged - 5 fixed
= 41 total (was 41) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
38s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} |
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647635#comment-16647635 ] anishek commented on HIVE-20679: reattached the patch to find out why skip batching is not working. > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645576#comment-16645576 ] Bharathkrishna Guruvayoor Murali commented on HIVE-20679: - Hi [~anishek] , Were you able to run any performance tests with the compressed messages? If so, please share the results. > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645114#comment-16645114 ] Eugene Ilchenko commented on HIVE-20679: [~anishek], I mean 500KB records, not 500.000 records. Single mysql record was half a megabyte large, and we had thousands of them. Too bad we don't have detailed info anymore. > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644922#comment-16644922
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942997/HIVE-20679.4.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 15110 tests
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=243)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=243)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=243)
TestReplicationScenariosIncrementalLoadAcidTables - did not produce a
TEST-*.xml file (likely timed out) (batchId=243)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning]
(batchId=188)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSessionSparkSessionTimeout
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14348/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14348/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14348/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942997 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, HIVE-20679.4.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
> Edit: For notification_log table the message column for all supported
> databases can store messages from 2GB to 4GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644902#comment-16644902
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
27s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 2319 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
5s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
38s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
32s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
39s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 39s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
26s{color} | {color:red} itests/hive-unit: The patch generated 39 new + 812
unchanged - 35 fixed = 851 total (was 847) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 5 new + 36 unchanged - 5 fixed
= 41 total (was 41) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} |
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644457#comment-16644457 ] anishek commented on HIVE-20679: [~diser555] the number of records should not affect this, unless you mean record = partitions ? > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. > Edit: For notification_log table the message column for all supported > databases can store messages from 2GB to 4GB -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643444#comment-16643444 ] Eugene Ilchenko commented on HIVE-20679: [~anishek], we tried to reproduce this 500K record (there were thousands of them at some point, seemed like a bug), still no luck. I'll post an update if we can get it. Thanks for working on this one! > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, > HIVE-20679.3.patch, HIVE-20679.4.patch, a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitification, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641807#comment-16641807
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942776/HIVE-20679.3.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15098 tests
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=243)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=194)
[druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q]
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=243)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=243)
TestReplicationScenariosIncrementalLoadAcidTables - did not produce a
TEST-*.xml file (likely timed out) (batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14312/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14312/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14312/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942776 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch,
> HIVE-20679.3.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641759#comment-16641759
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
33s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
8s{color} | {color:blue} standalone-metastore/metastore-server in master has
182 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
36s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
32s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
42s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} itests/hive-unit: The patch generated 39 new + 812
unchanged - 35 fixed = 851 total (was 847) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 5 new + 36 unchanged - 5 fixed
= 41 total (was 41) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} standalone-metastore/metastore-server generated 5 new
+ 180 unchanged - 2 fixed = 185 total (was 182) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} |
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640565#comment-16640565
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942534/HIVE-20679.2.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 15066 tests
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=243)
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplIncrementalLoadAcidTablesWithJsonMessage - did not produce a TEST-*.xml
file (likely timed out) (batchId=243)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely
timed out) (batchId=243)
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=243)
TestReplicationScenariosIncrementalLoadAcidTables - did not produce a
TEST-*.xml file (likely timed out) (batchId=243)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stat_estimate_related_col]
(batchId=43)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[external_jdbc_table2]
(batchId=172)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIncrementalDumpEmptyDumpDirectory
(batchId=243)
org.apache.hadoop.hive.ql.parse.repl.dump.events.TestEventHandlerFactory.shouldProvideDefaultHandlerWhenNothingRegisteredForThatEvent
(batchId=293)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14265/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14265/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14265/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12942534 - PreCommit-HIVE-Build
> DDL operations on hive might create large messages for DBNotification
> -
>
> Key: HIVE-20679
> URL: https://issues.apache.org/jira/browse/HIVE-20679
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Attachments: HIVE-20679.1.patch, HIVE-20679.2.patch, a.sql, b.sql
>
>
> Certain type of ddl operations might create large messages as part of
> DBNoitification, this might lead to the rdbms throwing an error when storing
> the message since its size is to large. It will also increase the footprint
> of the rdbms space usage.
> We should try store compressed messages to allow handling these situations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640563#comment-16640563
]
Hive QA commented on HIVE-20679:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
26s{color} | {color:blue} standalone-metastore/metastore-common in master has
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/server-extensions in master has 1 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
183 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
37s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
30s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
39s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 39s{color}
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hcatalog/server-extensions: The patch generated 1 new
+ 23 unchanged - 2 fixed = 24 total (was 25) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} itests/hive-unit: The patch generated 39 new + 802
unchanged - 35 fixed = 841 total (was 837) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 5 new + 34 unchanged - 5 fixed
= 39 total (was 39) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} server-extensions in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
39s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} standalone-metastore/metastore-server generated 5 new
+ 181 unchanged - 2 fixed = 186 total (was 183) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} |
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[
https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638947#comment-16638947
]
Hive QA commented on HIVE-20679:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12942355/HIVE-20679.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14236/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14236/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14236/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-10-04 22:06:43.647
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14236/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-10-04 22:06:43.650
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
97f0513..fe3a457 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 97f0513 HIVE-20563: Vectorization: CASE WHEN expression fails
when THEN/ELSE type and result type are different (Matt McCline, reviewed by
Teddy Choi)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at fe3a457 HIVE-20556: Expose an API to retrieve the TBL_ID from
TBLS in the metastore tables (Jaume Marhuenda via Eugene Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-10-04 22:06:45.160
+ rm -rf ../yetus_PreCommit-HIVE-Build-14236
+ mkdir ../yetus_PreCommit-HIVE-Build-14236
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14236
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14236/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git apply: bad git-diff - inconsistent old filename on line 2724
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:514
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java'
cleanly.
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java:1
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageBuilder.java'
with conflicts.
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageFactory.java:20
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageFactory.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:514
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java'
cleanly.
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/json/JSONMessageFactory.java:1
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/messaging/MessageBuilder.java'
with
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637824#comment-16637824 ] anishek commented on HIVE-20679: [~diser555] can you please attach a sample here in with that scale ? > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitificaion, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637529#comment-16637529 ] Eugene Ilchenko commented on HIVE-20679: Entries for tables with deep Avro schemas can get even larger. We have seen 500K entries for a single added partition. > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitificaion, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20679) DDL operations on hive might create large messages for DBNotification
[ https://issues.apache.org/jira/browse/HIVE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636622#comment-16636622 ] anishek commented on HIVE-20679: attached are two create able sql's and since table b is large the message size is about 106 KB for it. > DDL operations on hive might create large messages for DBNotification > - > > Key: HIVE-20679 > URL: https://issues.apache.org/jira/browse/HIVE-20679 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Attachments: a.sql, b.sql > > > Certain type of ddl operations might create large messages as part of > DBNoitificaion, this might lead to the rdbms throwing an error when storing > the message since its size is to large. It will also increase the footprint > of the rdbms space usage. > We should try store compressed messages to allow handling these situations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
