[jira] [Created] (TEZ-4252) 期望当基于tez作为计算引擎，对数据倾斜场景的调优
yang created TEZ-4252: - Summary: 期望当基于tez作为计算引擎，对数据倾斜场景的调优 Key: TEZ-4252 URL: https://issues.apache.org/jira/browse/TEZ-4252 Project: Apache Tez Issue Type: Wish Reporter: yang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency
[ https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237342#comment-17237342 ] Emilio Fernandes commented on TEZ-4154: --- +1 for the suggested changes! I hope they will be applied soon! > tez-protobuf: a single module for handling protobuf dependency > -- > > Key: TEZ-4154 > URL: https://issues.apache.org/jira/browse/TEZ-4154 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, > TEZ-4154.04.patch > > > Similarly to what I've seen in TEZ-2164, I would like to create a module that > is a single point of control of how tez handles protobuf dependency. In the > near future, when tez will upgrade to hadoop 3.3, there will be a couple of > things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf > 3.x and handle the possible mess that can come from the fact that hadoop has > relocated its protobuf. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4252) 期望当基于tez作为计算引擎，对数据倾斜场景的调优
[ https://issues.apache.org/jira/browse/TEZ-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein resolved TEZ-4252. Release Note: [~yang1] Can you please add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? Resolution: Invalid > 期望当基于tez作为计算引擎，对数据倾斜场景的调优 > - > > Key: TEZ-4252 > URL: https://issues.apache.org/jira/browse/TEZ-4252 > Project: Apache Tez > Issue Type: Wish >Reporter: yang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4252) 期望当基于tez作为计算引擎，对数据倾斜场景的调优
[ https://issues.apache.org/jira/browse/TEZ-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated TEZ-4252: --- Release Note: [~yang1] Can you please add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? was: [~yang1] Can you please add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? > 期望当基于tez作为计算引擎，对数据倾斜场景的调优 > - > > Key: TEZ-4252 > URL: https://issues.apache.org/jira/browse/TEZ-4252 > Project: Apache Tez > Issue Type: Wish >Reporter: yang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4252) 期望当基于tez作为计算引擎，对数据倾斜场景的调优
[ https://issues.apache.org/jira/browse/TEZ-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated TEZ-4252: --- Release Note: Please, add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? was: [~yang1] Can you please add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? > 期望当基于tez作为计算引擎，对数据倾斜场景的调优 > - > > Key: TEZ-4252 > URL: https://issues.apache.org/jira/browse/TEZ-4252 > Project: Apache Tez > Issue Type: Wish >Reporter: yang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4251) Acquiring locks for getInputVertices and getOutputVertices is not consistent
[
https://issues.apache.org/jira/browse/TEZ-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237928#comment-17237928
]
Rajesh Balamohan commented on TEZ-4251:
---
Committed to master. Thanks [~kkasa].
> Acquiring locks for getInputVertices and getOutputVertices is not consistent
>
>
> Key: TEZ-4251
> URL: https://issues.apache.org/jira/browse/TEZ-4251
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: TEZ-4251.1.patch, container_jstack.txt
>
>
> *VertexImpl.getInputVertices()* acquires read lock however
> *VertexImpl.getOutputVertices()* doesn't.
> We also faced with deadlock when using Tez from Hive: see
> [^container_jstack.txt]
> 0. Both LlapTaskSchedulerService and VertexImpl defines its own
> ReentrantReadWriteLock instance.
> 1. Thread "LlapScheduler" acquired write lock on
> LlapTaskSchedulerService.lock
> {code:java}
> LlapTaskSchedulerService.java
> protected void schedulePendingTasks() throws InterruptedException {
> Ref downgradedTask = new Ref<>(null);
> writeLock.lock();
> {code}
> 2. Thread "Dispatcher thread \{Central}" acquired write lock on
> VertexImpl.lock
> {code:java}
> VertexImpl.java
> public void handle(VertexEvent event) {
> ...
> try {
> writeLock.lock();
> {code}
> 3. Thread "LlapScheduler" tries acquiring read lock on VertexImpl.lock
> {code:java}
> VertexImpl.java
> @Override
> public Map getInputVertices() {
> readLock.lock();
> {code}
> but it is waiting because Thread "Dispatcher thread \{Central}" holds the
> write lock on VertexImpl.lock
> 4. Thread "Dispatcher thread \{Central}" try acquire read lock on
> LlapTaskSchedulerService.lock
> {code:java}
> LlapTaskSchedulerService.vaja
> @Override
> public Resource getTotalResources() {
> ...
> readLock.lock();
> {code}
> but it is waiting because Thread "LlapScheduler" holds the write lock on
> LlapTaskSchedulerService.lock
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
