James
You may be able to use alternative JSON components such as those with
record readers/writes.
You could certainly write a nifi processor in either Java or Python that
would do this and be super efficient.
The processor you've chosen just isn't very flexible in regards to larger
objects and
Thanks Eric. So then this in the error message - java.lang.OutOfMemoryError
- isn't really to be taken at face value. FlattenJson tried to index an
array that exceeded the maximum value of an integer, and it choked.
An 8 GB file really isn't that large. I'm hoping someone has encountered
this
Hi James,
I don't have a solution for you off the top of my head. But I can tell you
the failure is because you've got an array longer than the maximum value of
an Int. So, memory is not the limiting factor.
-Eric
On Fri, Jun 14, 2024, 10:59 AM James McMahon wrote:
> I have a json file,
I have a json file, incoming.json. It is 9 GB in size.
I want to flatten the json so that I can tabulate the number of times each
key appears. Am using a FlattenJson 2.0.0-M2 processor, with
this configuration:
Separator .
Flatten Mode
