Reuven Lax created BEAM-2302:
--------------------------------
Summary: WriteFiles with runner-determined sharding and large
numbers of windows causes OOM errors
Key: BEAM-2302
URL: https://issues.apache.org/jira/browse/BEAM-2302
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Reuven Lax
Assignee: Davor Bonaci
This is because the WriteWindowedBundles transform will create many file
writers, and the sheer number of file buffers (which defaults to 64mb per
writer) uses up all memory. The fix is the same as was done in BigQueryIO - if
too many writers are opened, spill into a shuffle, and write the files after
the shuffle
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
