[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193822#comment-15193822 ] Hitesh Shah commented on TEZ-3155: -- Thanks [~aplusplus]. Committed to master. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch, TEZ-3155.additional.2.patch, > TEZ-3155.additional.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193786#comment-15193786 ] Zhiyuan Yang commented on TEZ-3155: --- Additional patch updated. Make log line more user-friendly. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch, TEZ-3155.additional.2.patch, > TEZ-3155.additional.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189966#comment-15189966 ] Hitesh Shah commented on TEZ-3155: -- [~sseth] Missed that. [~aplusplus] mind providing an additional patch for the logging? > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189950#comment-15189950 ] Siddharth Seth commented on TEZ-3155: - A log line indicating that the dag plan is being serialized to disk would be useful. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189800#comment-15189800 ] Zhiyuan Yang commented on TEZ-3155: --- Thank you! Will contribute more patches :-) > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Fix For: 0.8.3 > > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch, TEZ-3155.9.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188463#comment-15188463 ] Hitesh Shah commented on TEZ-3155: -- Thanks for verifying the flaky tests. +1 for the patch. Minor nit on TestDAGClientServer for the non-needed throws IOException which I will fix before commit. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188457#comment-15188457 ] Zhiyuan Yang commented on TEZ-3155: --- On my local machine, TestFaultTolerance never failed. TestCommit failed sometimes, but from the trace I can see it has nothing to do with my patch. Will file another JIRA for this later. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch, > TEZ-3155.8.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188350#comment-15188350
]
TezQA commented on TEZ-3155:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12792349/TEZ-3155.8.patch
against master revision e8269c2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
org.apache.tez.dag.app.dag.impl.TestCommit
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1552//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1552//console
This message is automatically generated.
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch,
> TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch,
> TEZ-3155.8.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187607#comment-15187607
]
TezQA commented on TEZ-3155:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12792292/TEZ-3155.7.patch
against master revision e8269c2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 34 javac
compiler warnings (more than the master's current 33 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestFaultTolerance
org.apache.tez.test.TestTezJobs
org.apache.tez.mapreduce.TestMRRJobsDAGApi
org.apache.tez.test.TestDAGRecovery2
org.apache.tez.test.TestExceptionPropagation
org.apache.tez.test.TestAMRecovery
org.apache.tez.test.TestPipelinedShuffle
org.apache.tez.test.TestSecureShuffle
org.apache.tez.test.TestRecovery
org.apache.tez.test.TestDAGRecovery
org.apache.tez.mapreduce.TestMRRJobs
org.apache.tez.test.TestLocalMode
org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs
org.apache.tez.dag.history.logging.ats.TestATSHistoryWithMiniCluster
org.apache.tez.history.TestHistoryParser
org.apache.tez.dag.app.TestDAGAppMaster
org.apache.tez.dag.app.TestMockDAGAppMaster
org.apache.tez.dag.app.TestPreemption
org.apache.tez.dag.app.rm.TestContainerReuse
org.apache.tez.dag.app.TestSpeculation
org.apache.tez.tests.TestExternalTezServicesErrors
org.apache.tez.tests.TestExternalTezServices
org.apache.tez.analyzer.TestAnalyzer
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1551//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1551//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1551//console
This message is automatically generated.
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch,
> TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187462#comment-15187462 ] Zhiyuan Yang commented on TEZ-3155: --- Thanks for your patience in reviewing my patch! TEZ-3155.7.patch is uploaded to fix these problems. About "request = SubmitDAGRequestProto.parseFrom(fsDataInputStream);" , this is safe because it's creating a new object instead of merging contents into existing object. About recoveryFS, sorry I didn't notice it isn't initialized when DAGClientServer is created..already fix it. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch, TEZ-3155.7.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187408#comment-15187408
]
Hitesh Shah commented on TEZ-3155:
--
Nearly there - thanks for the patience in addressing review comments.
Some minor comments:
{code}
// amResources exceeds max IPC size
176 _testTezClientSessionLargeDAGPlan(10*1024*1024, 3*1024*1024,
3*1024*1024, true);
177 // DAGPlan and amResources together exceed threshold but less than
IPC size
178 _testTezClientSessionLargeDAGPlan(10*1024*1024, 10*1024*1024,
5*1024*1024, true);
{code}
- comments dont seem to match the test params
{code}
8 assertTrue(request.hasSerializedRequestPath());
219 assertFalse(request.hasDAGPlan());
220 } else {
221 assertFalse(request.hasSerializedRequestPath());
222 assertTrue(request.hasDAGPlan());
{code}
- test should also check the additional resources field to see if it is set
or unset similar to dagPlan
DAGClientAMProtocolBlockingPBServerImpl:
- stagingFs should be a final var?
- " request = SubmitDAGRequestProto.parseFrom(fsDataInputStream);" - is this
safe? i.e. merging contents from parseFrom() into an already existing object
with other data set from the rpc call?
DAGAppMaster:
"clientRpcServer = new DAGClientServer(clientHandler, appAttemptID,
recoveryFS);" - are you sure that recoveryFS is initialized correctly by this
point?
TestDAGClientServer:
" clientServer = new DAGClientServer(mockDAGClientHander, mockAppAttempId,
FileSystem.get(conf));" - why not just use a mock of the FS?
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch,
> TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1518#comment-1518 ] Zhiyuan Yang commented on TEZ-3155: --- I've uploaded another patch to address your comments. The big change is instead of only serializing DAGPlan, now we serialize the whole request if request size exceeds the threshold, so that we can also handle the case that local resources become too large. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch, TEZ-3155.6.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186382#comment-15186382
]
TezQA commented on TEZ-3155:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12792087/TEZ-3155.5.patch
against master revision e8269c2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1549//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1549//console
This message is automatically generated.
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch,
> TEZ-3155.4.patch, TEZ-3155.5.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15185961#comment-15185961
]
Hitesh Shah commented on TEZ-3155:
--
Comments:
Maybe
s/TEZ_CLIENT_REQUEST_GAP_TO_MAX_IPC_SIZE_BYTE/TEZ_IPC_PAYLOAD_RESERVED_BYTES
- tez.ipc.payload.reserved.bytes
( as the value is indicate how of the much of the ipc payload is being
reserved for overheads )
- also removed CLIENT as this could be leveraged in other ipc calls later if
needed
{code}
this.stagingFs =
TezCommonUtils.getTezSystemStagingPath(amConfig.getTezConfiguration(),
sessionAppId.toString()).getFileSystem(amConfig.getTezConfiguration());
{code}
- this can be simplified. Look at Filesystem.get(Config)
{code}
// TODO: if additionalAMResources itself exceeds the limit, we still have
problems. This can be solved by serializing
...
try {
546 fsDataOutputStream = stagingFs.create(dagPlanPath, false);
547 dagPlan.writeTo(fsDataOutputStream);
548
requestBuilder.setSerializedDagPlanPath(stagingFs.resolvePath(dagPlanPath).toString());
549 }
{code}
- Sorry - missed this in the earlier review. Any reason why this is not
addressed as part of this patch? As part of the max len check, both dag plan
and local resources could be written to the same file using writeDelimitedTo.
{code}
testTezClientSessionLargeDAGPlan(10*1024*1024, 10, 10, false);
164 testTezClientSessionLargeDAGPlan(10*1024*1024, 5*1024*1024, 10,
true);
165 testTezClientSessionLargeDAGPlan(10*1024*1024, 10, 5*1024*1024,
true);
166 testTezClientSessionLargeDAGPlan(10*1024*1024, 1*1024*1024,
5*1024*1024, true);
167 testTezClientSessionLargeDAGPlan(10*1024*1024, 10*1024*1024,
5*1024*1024, true);
168 // The case that amResourceSize exeeds the maxIPCMsgSize is not
tested because our code won't work in this case
{code}
- the tests belong in a separate @Test function say
testTezClientSessionLargeDAGPlan and rename the current
testTezClientSessionLargeDAGPlan to _testTezClientSessionLargeDAGPlan to
indicate a private function. Also, a comment for what permutation of the test
represents would be good.
{code}
conf.set(TezConfiguration.TEZ_AM_STAGING_DIR, "target");
{code}
- dir should target/TestTezClient/ to ensure that it does not clash with
other test code.
"new DAGClientAMProtocolBlockingPBServerImpl(realInstance, conf);" - should be
changed to "new DAGClientAMProtocolBlockingPBServerImpl(realInstance, fs);" by
appropriate code changes in DAGAppMaster to re-use recoveryFs
{code}
assertTrue(dagPlan.getName().equals(dagPlanName));
92 assertTrue(dagPlan.getVertexCount() == 1);
...
for (int i = 0; i < randomBytes.length; i++) {
98assertTrue(randomBytes[i] == randomBytesDeserialized[i]);
99 }
{code}
- I think Assert supports array equals which can be leveraged here. Also,
instead of assertTrue it is better to use assertEquals
Minor nit:
{code}
(dagPlan.getSerializedSize()+requestBuilder.getAdditionalAmResources().getSerializedSize()
>
540 maxSubmitDAGRequestSizeThroughIPC)
{code}
- please add a parenthesis around the + op for more code clarity
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch,
> TEZ-3155.4.patch, TEZ-3155.5.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15185795#comment-15185795 ] Zhiyuan Yang commented on TEZ-3155: --- I've uploaded another patch which address your comment. Now all FileSystem objects are initialized in constructor or initialization code so it can fail fast if there is anything wrong. The 5MB buffer size become a TezConfiguration property. Also, I've enhanced both client and server side tests: client side tests cover more cases including those you mentioned; server side test verifies the content of UserPayload after deserialization. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch, TEZ-3155.5.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183558#comment-15183558 ] Hitesh Shah commented on TEZ-3155: -- bq. this addition seems to have no relation to the proto being modified - why was this needed? Please ignore. Surprising that findbugs has not reported this earlier for other patches. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183553#comment-15183553
]
Hitesh Shah commented on TEZ-3155:
--
Thanks for addressing the previous comments. Some more comments based on patch
4:
{code}
47
48
49
50
{code}
- this addition seems to have no relation to the proto being modified - why
was this needed?
TezClient:
{code} private FileSystem fs = null; {code}
- rename this to something like stagingFs. Also, this should be initialized
once in init() and re-used.
{code}
137 private static final int gapToMaxIPCSize = 5 * 1024 * 1024;
138 private AtomicInteger serializedDAGPlanCounter = new AtomicInteger(0);
{code}
- above need code comments to describe that the vars are.
- might be good to make gapToMaxIPCSize configurable with default as 5 MB ).
Mark the new config property as Private though
{code}
dagClientConf.getInt(CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH,
530 CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH_DEFAULT)) {
{code}
- this should be a class member var and initialized once. Also it should use
the main tezconf and not dagclientconf
{code} TezConfiguration tezConf = amConfig.getTezConfiguration(); {code}
- no need to create an extra local var. Just use
"amConfig.getTezConfiguration()" directly
{code}
/* we need manually delete the serialized dagplan since staging path here
won't be destroyed */
190 Path dagPlanPath = new Path(request.getSerializedDagPlanPath());
191 FileSystem fs = dagPlanPath.getFileSystem(conf);
192 fs.delete(dagPlanPath, false);
{code}
- this is not reliable if there is a test failure or an exception is thrown
- staging dir should be set to target and also use the local fs
- Using local fs could be done by having a package private method to override
the stagingFs in TezClient with the value of FileSystem::getLocal
- For the dag plan file, use deleteOnExit()
TestTezClient:
{code}
int maxIPCMsgSize = 1024;
173 conf.setInt(CommonConfigurationKeys.IPC_MAXIMUM_DATA_LENGTH,
maxIPCMsgSize);
174
processorDescriptor.setUserPayload(UserPayload.create(ByteBuffer.allocate(2*maxIPCMsgSize)));
{code}
- processorDescriptor.setUserPayload() is not being invoked for the
largeDagPlan false case? - shouldnt it always be set to say 2 MB in both
scenarios and the max limit changed to 1 MB in one scenario and say 8 ( +5 for
the overhead check ) MB in the other scenario? This can played around with to
address my following comments on the buffer and additional resources checks.
- how is the 5 MB buffer check being tested?
- Also, there is no test if additionalResources ( or a combination of dag
plan + additional rsrcs ) exceeds ipc limits?
DAGClientAMProtocolBlockingPBServerImpl:
- fs can be initialized in the ctor itself
{code}
try (FSDataInputStream fsDataInputStream = fs.open(requestPath)) {
173 dagPlan = DAGPlan.parseFrom(fsDataInputStream);
174 } catch (IOException e) {
175 throw wrapException(e);
176 }
{code}
- wont the exception thrown in line 173 be caught be the catch in line 186 ?
testSubmitDagInSessionWithLargeDagPlan
- test could be enhanced to verify the payload contents after deserialization
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch,
> TEZ-3155.4.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182077#comment-15182077
]
TezQA commented on TEZ-3155:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12791660/TEZ-3155.4.patch
against master revision 91e24d7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1547//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1547//console
This message is automatically generated.
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch,
> TEZ-3155.4.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182061#comment-15182061 ] Zhiyuan Yang commented on TEZ-3155: --- It seems in this test the staging dir isn't destroyed after test so I manually delete the serialized DAGPlan file. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182059#comment-15182059 ] Zhiyuan Yang commented on TEZ-3155: --- Thanks for reviewing my code! I've uploaded another patch which fixes the problems you mentioned. Sorry for the mistakes and coding style problems. About TezClient, ratio 0.8 is not necessary especially with large max IPC size, so I changed it to a fixed gap to the max IPC size. About TestTezClient, I've added a test about the negative case. About DAGClientServer, the configuration instance is necessary to get a FileSystem object so I have to pass the configuration object to the constructor. About DAGClientServer, I didn't move the deserialization to DAGAppMaster because that requires change of DAGClientHandler's submitDAG signature which is bigger change than that I did. Also, now both client and server side hold one FileSystem instance without getting one each time which is costly. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch, > TEZ-3155.4.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181516#comment-15181516 ] Hitesh Shah commented on TEZ-3155: -- Test failed due to "org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/tmp/jenkins/tez/staging/.tez/application_0_0001/DAGPlan-1 " The staging dir should be set to use target and not the default value for the test. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Zhiyuan Yang > Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch > > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181400#comment-15181400
]
TezQA commented on TEZ-3155:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12791549/TEZ-3155.3.patch
against master revision 91e24d7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.client.TestTezClient
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1546//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1546//console
This message is automatically generated.
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180725#comment-15180725
]
Hitesh Shah commented on TEZ-3155:
--
Some general overal comments:
- patch looks decent for the most part
- please do not change code lines if they are not needed ( minor edits, new
lines, etc). Small changes are sometimes ok but numerous modifications make it
hard for a reviewer.
Some comments on the patch:
TezClient
- why 0.8 for largeDAGPlanRatio ? Why does it need to be so high for a 64 MB
max limit? Shoudnt it be more like a fixed number to account for the overhead?
- please consider using DAGPlan.writeTo* to write the output stream instead
of the objectoutputstream
- output stream needs to be closed?
{code}
FileSystem fs = dagPlanPath.getFileSystem(dagClientConf);
{code}
- the above should use amCOnfig.getTezConf() and should be done once per
TezClient and the fs object re-used.
{code}
SubmitDAGRequestProto request = requestBuilder.build();
SubmitDAGResponseProto response = proxy.submitDAG(null, request);
{code}
- any reason why this code change was needed?
- requestBuilder.build() is getting called twice - should be fixed to be
called once.
- SERIALIZED_DAGPLAN_NAME_PREFIX can be replaced by TEZ_PB_PLAN_BINARY_NAME
{code}
requestBuilder.setSerializedDagPlanPath(dagPlanPath.toString());
{code}
- this should be a fully resolved path ( check fs.resolve...)
TestTezClient:
- there should be a test for the negative case i.e. one where the dag plan is
within the defined threshold
DAGClientServer:
{code}
DAGClientAMProtocolBlockingPBServerImpl service =
DAGClientAMProtocolBlockingPBServerImpl(realInstance, conf);
{code}
- why was the above needed?
DAGClientAMProtocolBlockingPBServerImpl:
- it might be better to move this aspect of the code into DAGAppMaster where
the staging fs object can be reused.
- also the stream needs to be closed in a finally block
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch, TEZ-3155.3.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180530#comment-15180530
]
TezQA commented on TEZ-3155:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12791516/TEZ-3155.2.patch
against master revision 91e24d7.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 33 javac
compiler warnings (more than the master's current 31 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1545//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1545//console
This message is automatically generated.
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch, TEZ-3155.2.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[
https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179114#comment-15179114
]
TezQA commented on TEZ-3155:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12791361/TEZ-3155.1.patch
against master revision 2af886b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 33 javac
compiler warnings (more than the master's current 31 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 3.0.1) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in :
org.apache.tez.test.TestRecovery
org.apache.tez.test.TestTezJobs
org.apache.tez.test.TestFaultTolerance
Test results:
https://builds.apache.org/job/PreCommit-TEZ-Build/1541//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1541//artifact/patchprocess/newPatchFindbugsWarningstez-dag.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1541//artifact/patchprocess/newPatchFindbugsWarningstez-api.html
Javac warnings:
https://builds.apache.org/job/PreCommit-TEZ-Build/1541//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/1541//console
This message is automatically generated.
> Support a way to submit DAGs to a session where the DAG plan exceeds hadoop
> ipc limits
> ---
>
> Key: TEZ-3155
> URL: https://issues.apache.org/jira/browse/TEZ-3155
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Hitesh Shah
>Assignee: Zhiyuan Yang
> Attachments: TEZ-3155.1.patch
>
>
> Currently, dag submissions fail if the dag plan exceeds the hadoop ipc
> limits. One option would be to fall back to local resources if the dag plan
> is too large.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176314#comment-15176314 ] Hitesh Shah commented on TEZ-3155: -- Thanks [~aplusplus]. Added you to the contributor list - you should now be able to pick up jiras > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-3155) Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits
[ https://issues.apache.org/jira/browse/TEZ-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15176309#comment-15176309 ] Zhiyuan Yang commented on TEZ-3155: --- Would you mind assigning this task to me? I'm willing to take this task. > Support a way to submit DAGs to a session where the DAG plan exceeds hadoop > ipc limits > --- > > Key: TEZ-3155 > URL: https://issues.apache.org/jira/browse/TEZ-3155 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah > > Currently, dag submissions fail if the dag plan exceeds the hadoop ipc > limits. One option would be to fall back to local resources if the dag plan > is too large. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
