Re: FlattenJSON fails on large json file

2024-06-14 Thread Joe Witt
James You may be able to use alternative JSON components such as those with record readers/writes. You could certainly write a nifi processor in either Java or Python that would do this and be super efficient. The processor you've chosen just isn't very flexible in regards to larger objects and

Re: FlattenJSON fails on large json file

2024-06-14 Thread James McMahon
Thanks Eric. So then this in the error message - java.lang.OutOfMemoryError - isn't really to be taken at face value. FlattenJson tried to index an array that exceeded the maximum value of an integer, and it choked. An 8 GB file really isn't that large. I'm hoping someone has encountered this

Re: FlattenJSON fails on large json file

2024-06-14 Thread Eric Secules
Hi James, I don't have a solution for you off the top of my head. But I can tell you the failure is because you've got an array longer than the maximum value of an Int. So, memory is not the limiting factor. -Eric On Fri, Jun 14, 2024, 10:59 AM James McMahon wrote: > I have a json file,

FlattenJSON fails on large json file

2024-06-14 Thread James McMahon
I have a json file, incoming.json. It is 9 GB in size. I want to flatten the json so that I can tabulate the number of times each key appears. Am using a FlattenJson 2.0.0-M2 processor, with this configuration: Separator . Flatten Mode