[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548687#comment-16548687 ] Sergey Shelukhin commented on HIVE-20047: - Updated to remove the API arg. Let's see what tests fail on top of the existing failures in the branch. > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[
https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549462#comment-16549462
]
Hive QA commented on HIVE-20047:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932181/HIVE-20047.patch
{color:green}SUCCESS:{color} +1 due to 25 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 14684 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_update_status]
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status_disable_bitvector]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[compute_stats_date]
(batchId=47)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=173)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleanedCleansTxnsAndTxnComponents
(batchId=273)
org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testTxnTable
(batchId=283)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12697/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12697/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12697/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932181 - PreCommit-HIVE-Build
> [phase 1.5] consider removing txnID argument for txn stats methods
> --
>
> Key: HIVE-20047
> URL: https://issues.apache.org/jira/browse/HIVE-20047
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20047.patch
>
>
> Followup from HIVE-19975.
> W.r.t. write IDs and txn IDs, stats validity check currently verifies one of
> two things - that stats write ID is valid for query write ID list, or that
> stats txn ID (derived from write ID) is the same as the query txn ID.
> I'm not sure the latter check is needed; removing it would allow us to make a
> bunch of APIs a little bit simpler.
> [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile)
> observe stats written by the same txn; but in such manner that it doesn't
> have the write ID of the same-txn stats writer, in its valid write ID list?
> I'm assuming it's not possible, e.g. in multi statement txn each query would
> have the previous same-txn writer for the same table in its valid write ID
> list?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[
https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549546#comment-16549546
]
Hive QA commented on HIVE-20047:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
16s{color} | {color:blue} standalone-metastore/metastore-common in master has
218 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} storage-api: The patch generated 1 new + 3 unchanged -
0 fixed = 4 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
0s{color} | {color:red} ql: The patch generated 44 new + 2716 unchanged - 18
fixed = 2760 total (was 2734) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} root: The patch generated 1 new + 426 unchanged - 0
fixed = 427 total (was 426) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hcatalog-unit: The patch generated 2 new + 27
unchanged - 1 fixed = 29 total (was 28) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 388 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
46s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
20s{color} | {color:red} patch/standalone-metastore/metastore-common cannot run
setBugDatabaseInfo from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 6m
58s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
55s{color} | {color:red} patch/itests/hive-unit cannot run setBugDatabaseInfo
from findbugs {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m
10s{color} | {color:red} standalone-metastore generated 11 new + 53 unchanged -
0 fixed = 64 total (was 53) {color} |
| {color:red}-1{color} | {color:red} javadoc {co
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549651#comment-16549651 ] Sergey Shelukhin commented on HIVE-20047: - Same test failures as on the main branch. I will commit this soon. cc [~steveyeom2017] [~ekoifman] this removes the check for the same txn from "isCompliant" method. So, it will not rely entirely on write ID list being correct. Looks like it works. > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549738#comment-16549738 ] Eugene Koifman commented on HIVE-20047: --- "it will not rely entirely" - should 'not' be 'now'? > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549755#comment-16549755 ] Sergey Shelukhin commented on HIVE-20047: - Yeah > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
