[jira] [Created] (TEZ-4252) 期望当基于tez作为计算引擎,对数据倾斜场景的调优
yang created TEZ-4252: - Summary: 期望当基于tez作为计算引擎,对数据倾斜场景的调优 Key: TEZ-4252 URL: https://issues.apache.org/jira/browse/TEZ-4252 Project: Apache Tez Issue Type: Wish Reporter: yang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency
[ https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237342#comment-17237342 ] Emilio Fernandes commented on TEZ-4154: --- +1 for the suggested changes! I hope they will be applied soon! > tez-protobuf: a single module for handling protobuf dependency > -- > > Key: TEZ-4154 > URL: https://issues.apache.org/jira/browse/TEZ-4154 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, > TEZ-4154.04.patch > > > Similarly to what I've seen in TEZ-2164, I would like to create a module that > is a single point of control of how tez handles protobuf dependency. In the > near future, when tez will upgrade to hadoop 3.3, there will be a couple of > things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf > 3.x and handle the possible mess that can come from the fact that hadoop has > relocated its protobuf. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4252) 期望当基于tez作为计算引擎,对数据倾斜场景的调优
[ https://issues.apache.org/jira/browse/TEZ-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein resolved TEZ-4252. Release Note: [~yang1] Can you please add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? Resolution: Invalid > 期望当基于tez作为计算引擎,对数据倾斜场景的调优 > - > > Key: TEZ-4252 > URL: https://issues.apache.org/jira/browse/TEZ-4252 > Project: Apache Tez > Issue Type: Wish >Reporter: yang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4252) 期望当基于tez作为计算引擎,对数据倾斜场景的调优
[ https://issues.apache.org/jira/browse/TEZ-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated TEZ-4252: --- Release Note: [~yang1] Can you please add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? was: [~yang1] Can you please add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? > 期望当基于tez作为计算引擎,对数据倾斜场景的调优 > - > > Key: TEZ-4252 > URL: https://issues.apache.org/jira/browse/TEZ-4252 > Project: Apache Tez > Issue Type: Wish >Reporter: yang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4252) 期望当基于tez作为计算引擎,对数据倾斜场景的调优
[ https://issues.apache.org/jira/browse/TEZ-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated TEZ-4252: --- Release Note: Please, add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? was: [~yang1] Can you please add a clear description of what the issue is about? It is not recommended that Jiras have empty descriptions. Also, Can you please change the title and use a translation so that the lira will be searchable? > 期望当基于tez作为计算引擎,对数据倾斜场景的调优 > - > > Key: TEZ-4252 > URL: https://issues.apache.org/jira/browse/TEZ-4252 > Project: Apache Tez > Issue Type: Wish >Reporter: yang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4251) Acquiring locks for getInputVertices and getOutputVertices is not consistent
[ https://issues.apache.org/jira/browse/TEZ-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237928#comment-17237928 ] Rajesh Balamohan commented on TEZ-4251: --- Committed to master. Thanks [~kkasa]. > Acquiring locks for getInputVertices and getOutputVertices is not consistent > > > Key: TEZ-4251 > URL: https://issues.apache.org/jira/browse/TEZ-4251 > Project: Apache Tez > Issue Type: Bug >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Attachments: TEZ-4251.1.patch, container_jstack.txt > > > *VertexImpl.getInputVertices()* acquires read lock however > *VertexImpl.getOutputVertices()* doesn't. > We also faced with deadlock when using Tez from Hive: see > [^container_jstack.txt] > 0. Both LlapTaskSchedulerService and VertexImpl defines its own > ReentrantReadWriteLock instance. > 1. Thread "LlapScheduler" acquired write lock on > LlapTaskSchedulerService.lock > {code:java} > LlapTaskSchedulerService.java > protected void schedulePendingTasks() throws InterruptedException { > Ref downgradedTask = new Ref<>(null); > writeLock.lock(); > {code} > 2. Thread "Dispatcher thread \{Central}" acquired write lock on > VertexImpl.lock > {code:java} > VertexImpl.java > public void handle(VertexEvent event) { > ... > try { > writeLock.lock(); > {code} > 3. Thread "LlapScheduler" tries acquiring read lock on VertexImpl.lock > {code:java} > VertexImpl.java > @Override > public Map getInputVertices() { > readLock.lock(); > {code} > but it is waiting because Thread "Dispatcher thread \{Central}" holds the > write lock on VertexImpl.lock > 4. Thread "Dispatcher thread \{Central}" try acquire read lock on > LlapTaskSchedulerService.lock > {code:java} > LlapTaskSchedulerService.vaja > @Override > public Resource getTotalResources() { > ... > readLock.lock(); > {code} > but it is waiting because Thread "LlapScheduler" holds the write lock on > LlapTaskSchedulerService.lock -- This message was sent by Atlassian Jira (v8.3.4#803005)