[
https://issues.apache.org/jira/browse/NIFI-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581044#comment-16581044
]
Mark Payne commented on NIFI-3126:
----------------------------------
I agree that SplitRecord, configured with a JsonTreeReader, is the right
approach here. It will load only a single element of the JSON Array at a time.
SplitJSON attempts to load the entire FlowFile into a JSON-oriented object
model, so it is not very memory efficient. SplitRecord handles this much
better. SplitJSON, though, does allow you to split, I believe, at a 'lower
level' so if you need to do that, you'd want to use SplitRecord followed by
UpdateRecord to extract out the field(s) you want.
> A large JSON file consisting of an array of many json elements can cause an
> out of memory error if passed to SplitJSON
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-3126
> URL: https://issues.apache.org/jira/browse/NIFI-3126
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.0.0
> Environment: RHEL 6.5 - single instance of Nifi
> Reporter: Olav Jordens
> Assignee: Otto Fowler
> Priority: Major
>
> Create a flowfile containing a large JSON array (large=many elements in the
> array) with many attributes. Do this in such a way that a set of flowfiles
> corresponding to the individual json elements in the array (with all the
> attributes) would require more RAM than available in your nifi JVM. Now feed
> this flowfile into a SplitJSON processor. I did not see an OOM error in the
> logs and saw very strange behavior as the whole Nifi instance hangs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
