[
https://issues.apache.org/jira/browse/CRUNCH-463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Wills updated CRUNCH-463:
------------------------------
Attachment: CRUNCH-463.patch
Yeah, you're right-- we don't need to copy the Configuration object during
initialization, it will already be configured correctly using the FormatBundle
for the split.
> Copying the Configuration object in every CrunchInputSplit causes OOM errors
> for jobs with lots of splits
> ---------------------------------------------------------------------------------------------------------
>
> Key: CRUNCH-463
> URL: https://issues.apache.org/jira/browse/CRUNCH-463
> Project: Crunch
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.10.0
> Reporter: Hector Izquierdo Seliva
> Assignee: Josh Wills
> Attachments: CRUNCH-463.patch
>
>
> Trying to run a job with 11k input files and that yields about 25k splits
> results in OOM errors due to too many copies of the Configuration object
> being created when the CrunchInputSplit is initialised. I know that that's
> the result of CRUNCH-313, but perhaps a better way to deal with that problem
> should be found.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
