[jira] [Updated] (HIVE-24494) hive-tez session reuse
[ https://issues.apache.org/jira/browse/HIVE-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hezhang updated HIVE-24494: --- Attachment: HIVE-24494.diff > hive-tez session reuse > -- > > Key: HIVE-24494 > URL: https://issues.apache.org/jira/browse/HIVE-24494 > Project: Hive > Issue Type: Improvement > Components: Hive, Tez >Affects Versions: 3.1.1 >Reporter: hezhang >Priority: Major > Attachments: HIVE-24494.diff > > > allow parallel default sessions reused -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24495) modularize hive metastore server startup logic
[ https://issues.apache.org/jira/browse/HIVE-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cameron Moberg reassigned HIVE-24495: - > modularize hive metastore server startup logic > -- > > Key: HIVE-24495 > URL: https://issues.apache.org/jira/browse/HIVE-24495 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Cameron Moberg >Assignee: Cameron Moberg >Priority: Minor > > In the future there might be cause to have a modular system of > loading/starting a server such that we want to add other servers (such as > grpc/rest), the logic of starting hive server is tightly coupled with > hivemetastore.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24495) modularize hive metastore server startup logic
[ https://issues.apache.org/jira/browse/HIVE-24495?focusedWorklogId=520792&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520792 ] ASF GitHub Bot logged work on HIVE-24495: - Author: ASF GitHub Bot Created on: 06/Dec/20 17:25 Start Date: 06/Dec/20 17:25 Worklog Time Spent: 10m Work Description: Noremac201 opened a new pull request #1747: URL: https://github.com/apache/hive/pull/1747 ### What changes were proposed in this pull request? This change is built on top of #1741 #1740. This extracts out the logic of starting a thrift hive metastore server into a separate class, which will allow for more modular support of additional listeners in the future, while also cleaning up the current hivemetastore.java file. ### Why are the changes needed? This will allow for more modular support of additional listeners in the future (such as grpc) since now there is a common interface to implement that isn't based around thrift solely. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Existing unit tests should cover the changes made here, since this is a refactoring. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 520792) Remaining Estimate: 0h Time Spent: 10m > modularize hive metastore server startup logic > -- > > Key: HIVE-24495 > URL: https://issues.apache.org/jira/browse/HIVE-24495 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Cameron Moberg >Assignee: Cameron Moberg >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > In the future there might be cause to have a modular system of > loading/starting a server such that we want to add other servers (such as > grpc/rest), the logic of starting hive server is tightly coupled with > hivemetastore.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24495) modularize hive metastore server startup logic
[ https://issues.apache.org/jira/browse/HIVE-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24495: -- Labels: pull-request-available (was: ) > modularize hive metastore server startup logic > -- > > Key: HIVE-24495 > URL: https://issues.apache.org/jira/browse/HIVE-24495 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Cameron Moberg >Assignee: Cameron Moberg >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the future there might be cause to have a modular system of > loading/starting a server such that we want to add other servers (such as > grpc/rest), the logic of starting hive server is tightly coupled with > hivemetastore.java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24496) Add support for starting grpc server alongside thrift
[ https://issues.apache.org/jira/browse/HIVE-24496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cameron Moberg reassigned HIVE-24496: - > Add support for starting grpc server alongside thrift > - > > Key: HIVE-24496 > URL: https://issues.apache.org/jira/browse/HIVE-24496 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Cameron Moberg >Assignee: Cameron Moberg >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24496) Add support for starting grpc server alongside thrift
[ https://issues.apache.org/jira/browse/HIVE-24496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24496: -- Labels: pull-request-available (was: ) > Add support for starting grpc server alongside thrift > - > > Key: HIVE-24496 > URL: https://issues.apache.org/jira/browse/HIVE-24496 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Cameron Moberg >Assignee: Cameron Moberg >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24496) Add support for starting grpc server alongside thrift
[ https://issues.apache.org/jira/browse/HIVE-24496?focusedWorklogId=520795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520795 ] ASF GitHub Bot logged work on HIVE-24496: - Author: ASF GitHub Bot Created on: 06/Dec/20 17:49 Start Date: 06/Dec/20 17:49 Worklog Time Spent: 10m Work Description: Noremac201 opened a new pull request #1748: URL: https://github.com/apache/hive/pull/1748 ### What changes were proposed in this pull request? Add support for new variable to start up a custom server in standalone metastore. ### Why are the changes needed? This will allow for support of custom server (such as grpc) when starting up hive metastore by specifying a new configuration variable in metastore-site.xml. ### Does this PR introduce _any_ user-facing change? New configuration variable has been added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 520795) Remaining Estimate: 0h Time Spent: 10m > Add support for starting grpc server alongside thrift > - > > Key: HIVE-24496 > URL: https://issues.apache.org/jira/browse/HIVE-24496 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Cameron Moberg >Assignee: Cameron Moberg >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.2.1
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=520823&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520823 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 06/Dec/20 21:06 Start Date: 06/Dec/20 21:06 Worklog Time Spent: 10m Work Description: belugabehr closed pull request #1742: URL: https://github.com/apache/hive/pull/1742 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 520823) Time Spent: 20m (was: 10m) > Upgrade Hadoop to 3.2.1 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.2.1
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=520824&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520824 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 06/Dec/20 21:10 Start Date: 06/Dec/20 21:10 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1742: URL: https://github.com/apache/hive/pull/1742 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 520824) Time Spent: 0.5h (was: 20m) > Upgrade Hadoop to 3.2.1 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21052) Make sure transactions get cleaned if they are aborted before addPartitions is called
[ https://issues.apache.org/jira/browse/HIVE-21052?focusedWorklogId=520841&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520841 ] ASF GitHub Bot logged work on HIVE-21052: - Author: ASF GitHub Bot Created on: 07/Dec/20 00:50 Start Date: 07/Dec/20 00:50 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #1415: URL: https://github.com/apache/hive/pull/1415#issuecomment-739601932 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 520841) Time Spent: 12h 20m (was: 12h 10m) > Make sure transactions get cleaned if they are aborted before addPartitions > is called > - > > Key: HIVE-21052 > URL: https://issues.apache.org/jira/browse/HIVE-21052 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0, 3.1.1 >Reporter: Jaume M >Assignee: Jaume M >Priority: Critical > Labels: pull-request-available > Attachments: Aborted Txn w_Direct Write.pdf, HIVE-21052.1.patch, > HIVE-21052.10.patch, HIVE-21052.11.patch, HIVE-21052.12.patch, > HIVE-21052.2.patch, HIVE-21052.3.patch, HIVE-21052.4.patch, > HIVE-21052.5.patch, HIVE-21052.6.patch, HIVE-21052.7.patch, > HIVE-21052.8.patch, HIVE-21052.9.patch > > Time Spent: 12h 20m > Remaining Estimate: 0h > > If the transaction is aborted between openTxn and addPartitions and data has > been written on the table the transaction manager will think it's an empty > transaction and no cleaning will be done. > This is currently an issue in the streaming API and in micromanaged tables. > As proposed by [~ekoifman] this can be solved by: > * Writing an entry with a special marker to TXN_COMPONENTS at openTxn and > when addPartitions is called remove this entry from TXN_COMPONENTS and add > the corresponding partition entry to TXN_COMPONENTS. > * If the cleaner finds and entry with a special marker in TXN_COMPONENTS that > specifies that a transaction was opened and it was aborted it must generate > jobs for the worker for every possible partition available. > cc [~ewohlstadter] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24215) Total function count is incorrect in Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-24215?focusedWorklogId=520843&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520843 ] ASF GitHub Bot logged work on HIVE-24215: - Author: ASF GitHub Bot Created on: 07/Dec/20 00:50 Start Date: 07/Dec/20 00:50 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1540: URL: https://github.com/apache/hive/pull/1540 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 520843) Time Spent: 0.5h (was: 20m) > Total function count is incorrect in Replication Metrics > > > Key: HIVE-24215 > URL: https://issues.apache.org/jira/browse/HIVE-24215 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-24215.01.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24195) Avoid reallocation of the arrays in the lateral view explode of complex types
[ https://issues.apache.org/jira/browse/HIVE-24195?focusedWorklogId=520842&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520842 ] ASF GitHub Bot logged work on HIVE-24195: - Author: ASF GitHub Bot Created on: 07/Dec/20 00:50 Start Date: 07/Dec/20 00:50 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1521: URL: https://github.com/apache/hive/pull/1521 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 520842) Time Spent: 40m (was: 0.5h) > Avoid reallocation of the arrays in the lateral view explode of complex types > - > > Key: HIVE-24195 > URL: https://issues.apache.org/jira/browse/HIVE-24195 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Avoid reallocation of the arrays in the lateral view explode of complex types -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24467) ConditionalTask remove tasks that not selected exists thread safety problem
[
https://issues.apache.org/jira/browse/HIVE-24467?focusedWorklogId=520885&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520885
]
ASF GitHub Bot logged work on HIVE-24467:
-
Author: ASF GitHub Bot
Created on: 07/Dec/20 03:57
Start Date: 07/Dec/20 03:57
Worklog Time Spent: 10m
Work Description: gjhkael closed pull request #1743:
URL: https://github.com/apache/hive/pull/1743
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 520885)
Time Spent: 40m (was: 0.5h)
> ConditionalTask remove tasks that not selected exists thread safety problem
> ---
>
> Key: HIVE-24467
> URL: https://issues.apache.org/jira/browse/HIVE-24467
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.4
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When hive execute jobs in parallel(control by “hive.exec.parallel”
> parameter), ConditionalTasks remove the tasks that not selected in parallel,
> because there are thread safety issues, some task may not remove from the
> dependent task tree. This is a very serious bug, which causes some stage task
> not trigger execution.
> In our production cluster, the query run three conditional task in parallel,
> after apply the patch of HIVE-21638, we found Stage-3 is miss and not submit
> to runnable list for his parent Stage-31 is not done. But Stage-31 should
> removed for it not selected.
> Stage dependencies is below:
> {code:java}
> STAGE DEPENDENCIES:
> Stage-41 is a root stage
> Stage-26 depends on stages: Stage-41
> Stage-25 depends on stages: Stage-26 , consists of Stage-39, Stage-40,
> Stage-2
> Stage-39 has a backup stage: Stage-2
> Stage-23 depends on stages: Stage-39
> Stage-3 depends on stages: Stage-2, Stage-12, Stage-16, Stage-20, Stage-23,
> Stage-24, Stage-27, Stage-28, Stage-31, Stage-32, Stage-35, Stage-36
> Stage-8 depends on stages: Stage-3 , consists of Stage-5, Stage-4, Stage-6
> Stage-5
> Stage-0 depends on stages: Stage-5, Stage-4, Stage-7
> Stage-51 depends on stages: Stage-0
> Stage-4
> Stage-6
> Stage-7 depends on stages: Stage-6
> Stage-40 has a backup stage: Stage-2
> Stage-24 depends on stages: Stage-40
> Stage-2
> Stage-44 is a root stage
> Stage-30 depends on stages: Stage-44
> Stage-29 depends on stages: Stage-30 , consists of Stage-42, Stage-43,
> Stage-12
> Stage-42 has a backup stage: Stage-12
> Stage-27 depends on stages: Stage-42
> Stage-43 has a backup stage: Stage-12
> Stage-28 depends on stages: Stage-43
> Stage-12
> Stage-47 is a root stage
> Stage-34 depends on stages: Stage-47
> Stage-33 depends on stages: Stage-34 , consists of Stage-45, Stage-46,
> Stage-16
> Stage-45 has a backup stage: Stage-16
> Stage-31 depends on stages: Stage-45
> Stage-46 has a backup stage: Stage-16
> Stage-32 depends on stages: Stage-46
> Stage-16
> Stage-50 is a root stage
> Stage-38 depends on stages: Stage-50
> Stage-37 depends on stages: Stage-38 , consists of Stage-48, Stage-49,
> Stage-20
> Stage-48 has a backup stage: Stage-20
> Stage-35 depends on stages: Stage-48
> Stage-49 has a backup stage: Stage-20
> Stage-36 depends on stages: Stage-49
> Stage-20
> {code}
> Stage tasks execute log is below, we can see Stage-33 is conditional task and
> it consists of Stage-45, Stage-46, Stage-16, Stage-16 is launched, Stage-45
> and Stage-46 should remove from the dependent tree, Stage-31 is child of
> Stage-45 parent of Stage-3, So, Stage-31 should removed too. As see in the
> below log, we find Stage-31 is still in the parent list of Stage-3, this
> should not happend.
> {code:java}
> 2020-12-03T01:09:50,939 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Launching Job 1 out of 17
> 2020-12-03T01:09:50,940 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Starting task [Stage-26:MAPRED] in parallel
> 2020-12-03T01:09:50,941 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Launching Job 2 out of 17
> 2020-12-03T01:09:50,943 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Starting task [Stage-30:MAPRED] in parallel
> 2020-12-03T01:09:50,943 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Launching Job 3 out of 17
> 2020-12-03T01:09:50,943
[jira] [Work logged] (HIVE-24467) ConditionalTask remove tasks that not selected exists thread safety problem
[
https://issues.apache.org/jira/browse/HIVE-24467?focusedWorklogId=520888&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520888
]
ASF GitHub Bot logged work on HIVE-24467:
-
Author: ASF GitHub Bot
Created on: 07/Dec/20 03:59
Start Date: 07/Dec/20 03:59
Worklog Time Spent: 10m
Work Description: gjhkael opened a new pull request #1749:
URL: https://github.com/apache/hive/pull/1749
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 520888)
Time Spent: 50m (was: 40m)
> ConditionalTask remove tasks that not selected exists thread safety problem
> ---
>
> Key: HIVE-24467
> URL: https://issues.apache.org/jira/browse/HIVE-24467
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.4
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When hive execute jobs in parallel(control by “hive.exec.parallel”
> parameter), ConditionalTasks remove the tasks that not selected in parallel,
> because there are thread safety issues, some task may not remove from the
> dependent task tree. This is a very serious bug, which causes some stage task
> not trigger execution.
> In our production cluster, the query run three conditional task in parallel,
> after apply the patch of HIVE-21638, we found Stage-3 is miss and not submit
> to runnable list for his parent Stage-31 is not done. But Stage-31 should
> removed for it not selected.
> Stage dependencies is below:
> {code:java}
> STAGE DEPENDENCIES:
> Stage-41 is a root stage
> Stage-26 depends on stages: Stage-41
> Stage-25 depends on stages: Stage-26 , consists of Stage-39, Stage-40,
> Stage-2
> Stage-39 has a backup stage: Stage-2
> Stage-23 depends on stages: Stage-39
> Stage-3 depends on stages: Stage-2, Stage-12, Stage-16, Stage-20, Stage-23,
> Stage-24, Stage-27, Stage-28, Stage-31, Stage-32, Stage-35, Stage-36
> Stage-8 depends on stages: Stage-3 , consists of Stage-5, Stage-4, Stage-6
> Stage-5
> Stage-0 depends on stages: Stage-5, Stage-4, Stage-7
> Stage-51 depends on stages: Stage-0
> Stage-4
> Stage-6
> Stage-7 depends on stages: Stage-6
> Stage-40 has a backup stage: Stage-2
> Stage-24 depends on stages: Stage-40
> Stage-2
> Stage-44 is a root stage
> Stage-30 depends on stages: Stage-44
> Stage-29 depends on stages: Stage-30 , consists of Stage-42, Stage-43,
> Stage-12
> Stage-42 has a backup stage: Stage-12
> Stage-27 depends on stages: Stage-42
> Stage-43 has a backup stage: Stage-12
> Stage-28 depends on stages: Stage-43
> Stage-12
> Stage-47 is a root stage
> Stage-34 depends on stages: Stage-47
> Stage-33 depends on stages: Stage-34 , consists of Stage-45, Stage-46,
> Stage-16
> Stage-45 has a backup stage: Stage-16
> Stage-31 depends on stages: Stage-45
> Stage-46 has a backup stage: Stage-16
> Stage-32 depends on stages: Stage-46
> Stage-16
> Stage-50 is a root stage
> Stage-38 depends on stages: Stage-50
> Stage-37 depends on stages: Stage-38 , consists of Stage-48, Stage-49,
> Stage-20
> Stage-48 has a backup stage: Stage-20
> Stage-35 depends on stages: Stage-48
> Stage-49 has a backup stage: Stage-20
> Stage-36 depends on stages: Stage-49
> Stage-20
> {code}
> Stage tasks execute log is below, we can see Stage-33 is conditional task and
> it consists of Stage-45, Stage-46, Stage-16, Stage-16 is launched, Stage-45
> and Stage-46 should remove from the dependent tree, Stage-31 is child of
> Stage-45 parent of Stage-3, So, Stage-31 should removed too. As see in the
> below log, we find Stage-31 is still in the parent list of Stage-3, this
> should not happend.
> {code:java}
> 2020-12-03T01:09:50,939 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Launching Job 1 out of 17
> 2020-12-03T01:09:50,940 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Starting task [Stage-26:MAPRED] in parallel
> 2020-12-03T01:09:50,941 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Launching Job 2 out of 17
> 2020-12-03T01:09:50,943 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Starting task [Stage-30:MAPRED] in parallel
> 2020-12-03T01:09:50,943 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Launching Job 3 out of 17
> 2020-12-03T01:09:50
[jira] [Work logged] (HIVE-24467) ConditionalTask remove tasks that not selected exists thread safety problem
[
https://issues.apache.org/jira/browse/HIVE-24467?focusedWorklogId=520896&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-520896
]
ASF GitHub Bot logged work on HIVE-24467:
-
Author: ASF GitHub Bot
Created on: 07/Dec/20 04:11
Start Date: 07/Dec/20 04:11
Worklog Time Spent: 10m
Work Description: gjhkael commented on pull request #1749:
URL: https://github.com/apache/hive/pull/1749#issuecomment-739650908
@maheshk114 Please review this pr. Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 520896)
Time Spent: 1h (was: 50m)
> ConditionalTask remove tasks that not selected exists thread safety problem
> ---
>
> Key: HIVE-24467
> URL: https://issues.apache.org/jira/browse/HIVE-24467
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.4
>Reporter: guojh
>Assignee: guojh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When hive execute jobs in parallel(control by “hive.exec.parallel”
> parameter), ConditionalTasks remove the tasks that not selected in parallel,
> because there are thread safety issues, some task may not remove from the
> dependent task tree. This is a very serious bug, which causes some stage task
> not trigger execution.
> In our production cluster, the query run three conditional task in parallel,
> after apply the patch of HIVE-21638, we found Stage-3 is miss and not submit
> to runnable list for his parent Stage-31 is not done. But Stage-31 should
> removed for it not selected.
> Stage dependencies is below:
> {code:java}
> STAGE DEPENDENCIES:
> Stage-41 is a root stage
> Stage-26 depends on stages: Stage-41
> Stage-25 depends on stages: Stage-26 , consists of Stage-39, Stage-40,
> Stage-2
> Stage-39 has a backup stage: Stage-2
> Stage-23 depends on stages: Stage-39
> Stage-3 depends on stages: Stage-2, Stage-12, Stage-16, Stage-20, Stage-23,
> Stage-24, Stage-27, Stage-28, Stage-31, Stage-32, Stage-35, Stage-36
> Stage-8 depends on stages: Stage-3 , consists of Stage-5, Stage-4, Stage-6
> Stage-5
> Stage-0 depends on stages: Stage-5, Stage-4, Stage-7
> Stage-51 depends on stages: Stage-0
> Stage-4
> Stage-6
> Stage-7 depends on stages: Stage-6
> Stage-40 has a backup stage: Stage-2
> Stage-24 depends on stages: Stage-40
> Stage-2
> Stage-44 is a root stage
> Stage-30 depends on stages: Stage-44
> Stage-29 depends on stages: Stage-30 , consists of Stage-42, Stage-43,
> Stage-12
> Stage-42 has a backup stage: Stage-12
> Stage-27 depends on stages: Stage-42
> Stage-43 has a backup stage: Stage-12
> Stage-28 depends on stages: Stage-43
> Stage-12
> Stage-47 is a root stage
> Stage-34 depends on stages: Stage-47
> Stage-33 depends on stages: Stage-34 , consists of Stage-45, Stage-46,
> Stage-16
> Stage-45 has a backup stage: Stage-16
> Stage-31 depends on stages: Stage-45
> Stage-46 has a backup stage: Stage-16
> Stage-32 depends on stages: Stage-46
> Stage-16
> Stage-50 is a root stage
> Stage-38 depends on stages: Stage-50
> Stage-37 depends on stages: Stage-38 , consists of Stage-48, Stage-49,
> Stage-20
> Stage-48 has a backup stage: Stage-20
> Stage-35 depends on stages: Stage-48
> Stage-49 has a backup stage: Stage-20
> Stage-36 depends on stages: Stage-49
> Stage-20
> {code}
> Stage tasks execute log is below, we can see Stage-33 is conditional task and
> it consists of Stage-45, Stage-46, Stage-16, Stage-16 is launched, Stage-45
> and Stage-46 should remove from the dependent tree, Stage-31 is child of
> Stage-45 parent of Stage-3, So, Stage-31 should removed too. As see in the
> below log, we find Stage-31 is still in the parent list of Stage-3, this
> should not happend.
> {code:java}
> 2020-12-03T01:09:50,939 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Launching Job 1 out of 17
> 2020-12-03T01:09:50,940 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Starting task [Stage-26:MAPRED] in parallel
> 2020-12-03T01:09:50,941 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Launching Job 2 out of 17
> 2020-12-03T01:09:50,943 INFO [HiveServer2-Background-Pool: Thread-87372]
> ql.Driver: Starting task [Stage-30:MAPRED] in parallel
> 2020-12-03T01:09:50,943 INFO [HiveServer2-Background-Pool: Thread-87372
