[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180363#comment-15180363 ] Hitesh Shah commented on TEZ-1210: -- Also, please use dev/u...@tez.apache.org for such questions in the future. > TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode > - > > Key: TEZ-1210 > URL: https://issues.apache.org/jira/browse/TEZ-1210 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Alexander Pivovarov > Labels: newbie > Fix For: 0.5.2 > > Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch > > > It writes the dagPlan in text form to the same location. Either it should not > be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180359#comment-15180359 ] Hitesh Shah commented on TEZ-1210: -- You can submit the dag plan via rpc in non-session mode too but from a performance view point it is likely to be a bit slower. The client would need to submit the app, wait for the AM to come up and then submit the DAG. For non-session mode, it is simpler to use local resources as the dag is available to be executed as soon as the AM comes up as it is available in its local container dir. There are other fault tolerance scenario questions too - what if the first AM attempt crashes before the dag could be submitted to it or if the cluster is busy and the AM does not come up for quite some time - this would require the client process to remain around for a long time to submit the dag over rpc. Using local resource in this case allows the client to be a fire and forget which is useful for ETL scripts that submit only one DAG. > TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode > - > > Key: TEZ-1210 > URL: https://issues.apache.org/jira/browse/TEZ-1210 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Alexander Pivovarov > Labels: newbie > Fix For: 0.5.2 > > Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch > > > It writes the dagPlan in text form to the same location. Either it should not > be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15180180#comment-15180180 ] Bikas Saha commented on TEZ-1210: - The DAGPlan is downloaded as a local resource to be used to run the DAG. In session mode the AM is already running and can accept the DAGPlan over RPC. In non-session mode is going to be launched (and there is no connection between it and the client) and thus the DAGPlan needs to be provides indirectly as a YARN LocalResource via HDFS. > TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode > - > > Key: TEZ-1210 > URL: https://issues.apache.org/jira/browse/TEZ-1210 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Alexander Pivovarov > Labels: newbie > Fix For: 0.5.2 > > Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch > > > It writes the dagPlan in text form to the same location. Either it should not > be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179837#comment-15179837 ] Mai H. Dinh commented on TEZ-1210: -- I am not sure this is the best place for this question but I cannot find a better place. In non-session mode, what is the advantage of writing DAGPlan to hadoop file system instead of sending via ProtobufRPC as in session mode? > TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode > - > > Key: TEZ-1210 > URL: https://issues.apache.org/jira/browse/TEZ-1210 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Alexander Pivovarov > Labels: newbie > Fix For: 0.5.2 > > Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch > > > It writes the dagPlan in text form to the same location. Either it should not > be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179454#comment-14179454 ] Bikas Saha commented on TEZ-1210: - Committing this since it addresses the naming issue. We should consider doing it on the client or at least have the DAGAppMaster do it (just like it does for the graphviz). Uploading to HDFS from the client and downloading it to the AM via localization is avoidable overhead. TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Alexander Pivovarov Labels: newbie Attachments: TEZ-1210.1.patch, TEZ-1210.2.patch It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111419#comment-14111419 ] Hitesh Shah commented on TEZ-1210: -- There are a few things to consider: - how does one control whether this text file generated? - where is the text file generated to? On the client, this is tricky as there is no notion of a default location that works across both linux and windows. Furthermore, if accidentally configured to true, this will leave behind a bunch of non-useful files. On the AM, it is simpler, as we can generated this file into the container log dir. - if the question is about whether to upload this to HDFS, I guess another approach would be to just write the binary serialized plan to HDFS but read the plan in the AM and generate a text version into the log dir if the appropriate config/flag has been set. TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Alexander Pivovarov Labels: newbie Attachments: TEZ-1210.1.patch It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111515#comment-14111515 ] Hitesh Shah commented on TEZ-1210: -- In that case, would it be fine for this jira to just address the naming issue and have all the other aspects be addressed in TEZ-974? TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Alexander Pivovarov Labels: newbie Attachments: TEZ-1210.1.patch It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090799#comment-14090799 ] Gavin Mead commented on TEZ-1210: - Thanks [~zjffdu], go ahead and assign TEZ-974 to me. TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Gavin Mead Labels: newbie It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14090013#comment-14090013 ] Jeff Zhang commented on TEZ-1210: - [~gavinmead] I am working on other jiras, may not have bandwidth on this in short time. If you feel OK, you can also take over [Tez-974 | https://issues.apache.org/jira/browse/TEZ-974], I can provide some help if you have any quesiton on [Tez-974 | https://issues.apache.org/jira/browse/TEZ-974] TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Gavin Mead Labels: newbie It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079905#comment-14079905 ] Bikas Saha commented on TEZ-1210: - TEZ-974 should take care of making dot plot and PB text work in the same way. TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Gavin Mead Labels: newbie It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079957#comment-14079957 ] Gavin Mead commented on TEZ-1210: - Is it okay for me to link this issue with a dependency on TEZ-974? Is there anything I can help with for TEZ-974? TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Gavin Mead Labels: newbie It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079982#comment-14079982 ] Bikas Saha commented on TEZ-1210: - In Apache, you can comment on the jira expressing interest to work on it. If the current assignee is not working on it then they usually re-assign it. For this jira, we can resolve it independent of TEZ-974 by simply including a differentiating prefix to the file - eg. appId+Integer of the dag. If this goes in before TEZ-974 then it will have to refresh. TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Gavin Mead Labels: newbie It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (TEZ-1210) TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode
[ https://issues.apache.org/jira/browse/TEZ-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078821#comment-14078821 ] Gavin Mead commented on TEZ-1210: - Does the text version of the DAG Plan have to be stored as a LocalResource? Looking at the DAGAppMaster it writes the dot file to Environment.LOG_DIRS/TezDAGID.toString().dot. Could the PB Text file be written to a location like Environment.LOG_DIRS/TezDAGID.toString()-PB.txt? TezClientUtils.localizeDagPlanAsText() needs to be fixed for session mode - Key: TEZ-1210 URL: https://issues.apache.org/jira/browse/TEZ-1210 Project: Apache Tez Issue Type: Bug Reporter: Bikas Saha Assignee: Gavin Mead Labels: newbie It writes the dagPlan in text form to the same location. Either it should not be invoked in session mode or it should written with a differentiating prefix. -- This message was sent by Atlassian JIRA (v6.2#6252)
