[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548687#comment-16548687 ] Sergey Shelukhin commented on HIVE-20047: - Updated to remove the API arg. Let's see what tests fail on top of the existing failures in the branch. > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549462#comment-16549462 ] Hive QA commented on HIVE-20047: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12932181/HIVE-20047.patch {color:green}SUCCESS:{color} +1 due to 25 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 14684 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_update_status] (batchId=94) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status] (batchId=84) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status_disable_bitvector] (batchId=83) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[compute_stats_date] (batchId=47) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets] (batchId=173) org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleanedCleansTxnsAndTxnComponents (batchId=273) org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread.testTxnTable (batchId=283) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/12697/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12697/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12697/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12932181 - PreCommit-HIVE-Build > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549546#comment-16549546 ] Hive QA commented on HIVE-20047: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 56s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 49s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 24s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 34s{color} | {color:blue} common in master has 64 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 16s{color} | {color:blue} standalone-metastore/metastore-common in master has 218 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 17s{color} | {color:blue} ql in master has 2273 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m 52s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 30s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s{color} | {color:red} storage-api: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 0s{color} | {color:red} ql: The patch generated 44 new + 2716 unchanged - 18 fixed = 2760 total (was 2734) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 44s{color} | {color:red} root: The patch generated 1 new + 426 unchanged - 0 fixed = 427 total (was 426) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} itests/hcatalog-unit: The patch generated 2 new + 27 unchanged - 1 fixed = 29 total (was 28) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch has 388 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from findbugs {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from findbugs {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 20s{color} | {color:red} patch/standalone-metastore/metastore-common cannot run setBugDatabaseInfo from findbugs {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 6m 58s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 55s{color} | {color:red} patch/itests/hive-unit cannot run setBugDatabaseInfo from findbugs {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 10s{color} | {color:red} standalone-metastore generated 11 new + 53 unchanged - 0 fixed = 64 total (was 53) {color} | | {color:red}-1{color} | {color:red} javadoc {co
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549651#comment-16549651 ] Sergey Shelukhin commented on HIVE-20047: - Same test failures as on the main branch. I will commit this soon. cc [~steveyeom2017] [~ekoifman] this removes the check for the same txn from "isCompliant" method. So, it will not rely entirely on write ID list being correct. Looks like it works. > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549738#comment-16549738 ] Eugene Koifman commented on HIVE-20047: --- "it will not rely entirely" - should 'not' be 'now'? > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20047) [phase 1.5] consider removing txnID argument for txn stats methods
[ https://issues.apache.org/jira/browse/HIVE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549755#comment-16549755 ] Sergey Shelukhin commented on HIVE-20047: - Yeah > [phase 1.5] consider removing txnID argument for txn stats methods > -- > > Key: HIVE-20047 > URL: https://issues.apache.org/jira/browse/HIVE-20047 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20047.patch > > > Followup from HIVE-19975. > W.r.t. write IDs and txn IDs, stats validity check currently verifies one of > two things - that stats write ID is valid for query write ID list, or that > stats txn ID (derived from write ID) is the same as the query txn ID. > I'm not sure the latter check is needed; removing it would allow us to make a > bunch of APIs a little bit simpler. > [~ekoifman] do you have any feedback? Can any stats reader (e.g. compile) > observe stats written by the same txn; but in such manner that it doesn't > have the write ID of the same-txn stats writer, in its valid write ID list? > I'm assuming it's not possible, e.g. in multi statement txn each query would > have the previous same-txn writer for the same table in its valid write ID > list? -- This message was sent by Atlassian JIRA (v7.6.3#76005)