[ https://issues.apache.org/jira/browse/TEZ-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17622755#comment-17622755 ]
László Bodor commented on TEZ-4448: ----------------------------------- merged to master, thanks [~karel.kolman] for the patch! > Cannot submit Tez job when dag size is exceeds `ipc.maximum.data.length` and > S3A is the filesystem > -------------------------------------------------------------------------------------------------- > > Key: TEZ-4448 > URL: https://issues.apache.org/jira/browse/TEZ-4448 > Project: Apache Tez > Issue Type: Improvement > Affects Versions: 0.9.2 > Reporter: Karel Kolman > Assignee: Karel Kolman > Priority: Major > Labels: patch > Attachments: TEZ-4448.20021007.1.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Submitting a Tez DAG with serialized size exceeding *ipc.maximum.data.length* > in EMR/S3 environment results in a *FileNotFoundException* > h5. Stacktrace > {noformat} > Caused by: java.io.FileNotFoundException: No such file or directory: > s3://my-s3-bucket/my-job/.staging-a223d0fa-d315-4d84-9594-529c6ba34066/.tez/application_1655350046147_595990/tez-dag.pb1 > at > org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2310) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2204) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2143) > at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888) > at > org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:676) > at org.apache.tez.client.TezClient.submitDAG(TezClient.java:588) > ... 22 more > {noformat} > h5. Reproducing > - set *ipc.maximum.data.length* to *64*1024* instead of the default value of > *64*1024*1024* > - attempt to submit a Tez job (EMR/S3) -- This message was sent by Atlassian Jira (v8.20.10#820010)