prashanthvg89 commented on pull request #2069:
URL: https://github.com/apache/hudi/pull/2069#issuecomment-756969267
I saw this failure in Spark streaming job writing to Hudi. Is this still
being worked on and is there an ETA for this?
Hudi version: 0.6.0
Spark version: 2.4.0
EMR version: 5.29.0
Hudi storage type: Copy-On-Write
Number of partitions specified using partition path field: 100
Number of Hudi records currently: 463 million
Caused by: org.apache.hudi.exception.HoodieIOException: Cannot instantiate
an ExternalSpillableMap
at
org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:155)
at
org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:75)
at
org.apache.hudi.table.action.commit.CommitActionExecutor.getUpdateHandle(CommitActionExecutor.java:101)
at
org.apache.hudi.table.action.commit.CommitActionExecutor.handleUpdate(CommitActionExecutor.java:72)
at
org.apache.hudi.table.action.commit.BaseCommitActionExecutor.handleUpsertPartition(BaseCommitActionExecutor.java:257)
... 30 more
Caused by: java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1014)
at
org.apache.hudi.common.util.collection.DiskBasedMap.initFile(DiskBasedMap.java:115)
at
org.apache.hudi.common.util.collection.DiskBasedMap.<init>(DiskBasedMap.java:81)
at
org.apache.hudi.common.util.collection.ExternalSpillableMap.<init>(ExternalSpillableMap.java:86)
at
org.apache.hudi.io.HoodieMergeHandle.init(HoodieMergeHandle.java:152)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
