Karel Kolman created TEZ-4448:
---------------------------------
Summary: Cannot submit Tez job when dag size is exceeds
`ipc.maximum.data.length` and S3A is the filesystem
Key: TEZ-4448
URL: https://issues.apache.org/jira/browse/TEZ-4448
Project: Apache Tez
Issue Type: Improvement
Affects Versions: 0.9.2
Reporter: Karel Kolman
Submitting a Tez DAG with serialized size exceeding *ipc.maximum.data.length*
in EMR/S3 environment results in a *FileNotFoundException*
h5. Stacktrace
{noformat}
Caused by: java.io.FileNotFoundException: No such file or directory:
s3://my-s3-bucket/my-job/.staging-a223d0fa-d315-4d84-9594-529c6ba34066/.tez/application_1655350046147_595990/tez-dag.pb1
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2310)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2204)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2143)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:676)
at org.apache.tez.client.TezClient.submitDAG(TezClient.java:588)
... 22 more
{noformat}
h5. Reproducing
- set *ipc.maximum.data.length* to *64*1024* instead of the default value of
*64*1024*1024*
- attempt to submit a Tez job (EMR/S3)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
