[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[
https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944371#comment-13944371
]
Lefty Leverenz commented on HIVE-6229:
--
*hive.stats.key.prefix.reserve.length* is now documented in the wiki, but its
description needs review and clarification:
{quote}
Reserved length for postfix of statistics key. Currently only meaningful for
counter type statistics which should keep the length of the full statistics key
smaller than the maximum length configured by hive.stats.key.prefix.max.length.
For counter type statistics, it should be bigger than the length of LB spec if
exists.
{quote}
What does "LB spec" mean? All I could find by googling was the Ljung–Box test.
Does "if exists" mean "if the LB spec exists"? And what is "it" in "it should
be bigger" -- this parameter's setting or the length of the full statistics key?
* [Configuration Properties: hive.stats.key.prefix.reserve.length
|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.key.prefix.reserve.length]
> Stats are missing sometimes (regression from HIVE-5936)
> ---
>
> Key: HIVE-6229
> URL: https://issues.apache.org/jira/browse/HIVE-6229
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Navis
>Assignee: Navis
> Fix For: 0.13.0
>
> Attachments: HIVE-6229.1.patch.txt, HIVE-6229.2.patch.txt
>
>
> if prefix length is smaller than hive.stats.key.prefix.max.length but length
> of prefix + postfix is bigger than that, stats are missed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[ https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879786#comment-13879786 ] Lefty Leverenz commented on HIVE-6229: -- Okay, I'll document the other two and there shouldn't be any confusion in the wikidoc because hive.stats.key.prefix won't be documented with them. Thanks. > Stats are missing sometimes (regression from HIVE-5936) > --- > > Key: HIVE-6229 > URL: https://issues.apache.org/jira/browse/HIVE-6229 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Navis >Assignee: Navis > Fix For: 0.13.0 > > Attachments: HIVE-6229.1.patch.txt, HIVE-6229.2.patch.txt > > > if prefix length is smaller than hive.stats.key.prefix.max.length but length > of prefix + postfix is bigger than that, stats are missed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[ https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879480#comment-13879480 ] Navis commented on HIVE-6229: - "hive.stats.key.prefix" is the key for JobConf instance used only for partial scan command. I think the name of config is a little misleading. (It's not related with other two similar-named configurations) > Stats are missing sometimes (regression from HIVE-5936) > --- > > Key: HIVE-6229 > URL: https://issues.apache.org/jira/browse/HIVE-6229 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Navis >Assignee: Navis > Fix For: 0.13.0 > > Attachments: HIVE-6229.1.patch.txt, HIVE-6229.2.patch.txt > > > if prefix length is smaller than hive.stats.key.prefix.max.length but length > of prefix + postfix is bigger than that, stats are missed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[ https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878510#comment-13878510 ] Lefty Leverenz commented on HIVE-6229: -- This adds hive.stats.key.prefix.reserve.length to HiveConf.java and hive-default.xml.template. I can document it in the wiki (for Hive 0.13.0) along with hive.stats.key.prefix.max.length, but hive.stats.key.prefix has an "internal usage only" comment so are these other two hive.stats.key.prefix.* config params also internal use only? > Stats are missing sometimes (regression from HIVE-5936) > --- > > Key: HIVE-6229 > URL: https://issues.apache.org/jira/browse/HIVE-6229 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Navis >Assignee: Navis > Fix For: 0.13.0 > > Attachments: HIVE-6229.1.patch.txt, HIVE-6229.2.patch.txt > > > if prefix length is smaller than hive.stats.key.prefix.max.length but length > of prefix + postfix is bigger than that, stats are missed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[
https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876450#comment-13876450
]
Hive QA commented on HIVE-6229:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12623937/HIVE-6229.2.patch.txt
{color:green}SUCCESS:{color} +1 4943 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/962/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/962/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12623937
> Stats are missing sometimes (regression from HIVE-5936)
> ---
>
> Key: HIVE-6229
> URL: https://issues.apache.org/jira/browse/HIVE-6229
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6229.1.patch.txt, HIVE-6229.2.patch.txt
>
>
> if prefix length is smaller than hive.stats.key.prefix.max.length but length
> of prefix + postfix is bigger than that, stats are missed.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[ https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876155#comment-13876155 ] Ashutosh Chauhan commented on HIVE-6229: I see. Yeah, would be good to fix that too. > Stats are missing sometimes (regression from HIVE-5936) > --- > > Key: HIVE-6229 > URL: https://issues.apache.org/jira/browse/HIVE-6229 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Navis >Assignee: Navis > Attachments: HIVE-6229.1.patch.txt > > > if prefix length is smaller than hive.stats.key.prefix.max.length but length > of prefix + postfix is bigger than that, stats are missed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[ https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876154#comment-13876154 ] Navis commented on HIVE-6229: - [~ashutoshc] SourceTask in StatsWork is not valid when the task is exchanged or removed (auto.convert.join, etc.), which fails to access stats from StatsTask when using counter type. I think it would be better to fix that with this altogether. > Stats are missing sometimes (regression from HIVE-5936) > --- > > Key: HIVE-6229 > URL: https://issues.apache.org/jira/browse/HIVE-6229 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Navis >Assignee: Navis > Attachments: HIVE-6229.1.patch.txt > > > if prefix length is smaller than hive.stats.key.prefix.max.length but length > of prefix + postfix is bigger than that, stats are missed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[ https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875963#comment-13875963 ] Ashutosh Chauhan commented on HIVE-6229: +1 > Stats are missing sometimes (regression from HIVE-5936) > --- > > Key: HIVE-6229 > URL: https://issues.apache.org/jira/browse/HIVE-6229 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Navis >Assignee: Navis > Attachments: HIVE-6229.1.patch.txt > > > if prefix length is smaller than hive.stats.key.prefix.max.length but length > of prefix + postfix is bigger than that, stats are missed. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6229) Stats are missing sometimes (regression from HIVE-5936)
[
https://issues.apache.org/jira/browse/HIVE-6229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875838#comment-13875838
]
Hive QA commented on HIVE-6229:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12623851/HIVE-6229.1.patch.txt
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 4943 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_smb_mapjoin_8
org.apache.hadoop.hive.ql.parse.TestParse.testParse_case_sensitivity
org.apache.hadoop.hive.ql.parse.TestParse.testParse_groupby1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input7
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input9
org.apache.hadoop.hive.ql.parse.TestParse.testParse_input_testsequencefile
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join1
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_join3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample2
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample3
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample4
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample5
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample6
org.apache.hadoop.hive.ql.parse.TestParse.testParse_sample7
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/959/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/959/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12623851
> Stats are missing sometimes (regression from HIVE-5936)
> ---
>
> Key: HIVE-6229
> URL: https://issues.apache.org/jira/browse/HIVE-6229
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-6229.1.patch.txt
>
>
> if prefix length is smaller than hive.stats.key.prefix.max.length but length
> of prefix + postfix is bigger than that, stats are missed.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
