Hi all, Sorry for the late reply. It seems like the issue is with the csv files. I generated csv files using postgreSQL 'COPY' command. According to the postgresql documentation, double quote is set as the default value for both quote character and the escape character for this command. It seems like, when both the quote and escape chars are the same, NIFI CSV reader gets confused and splits the record into many columns causing this issue.
To overcome the issue, I changed the escape character to back-slash when generating CSVs using COPY command. But postgresql inserts the escape character only if quote character is available in data. It doesn't insert escape char if any other special character(delimiter, same escape char etc.) is there in data. Hence, if a back-slash is there at the end of a string data, the delimiter gets escaped when reading from NIFI. Has anyone tried to read postgre generated CSV from NIFI.? Any workaround for this issue. postgreSQL COPY command documentation : https://www.postgresql.org/docs/13/sql-copy.html Thanks & Regards *Vibhath Ileperuma* On Thu, Apr 15, 2021 at 4:59 AM Mike Thomsen <[email protected]> wrote: > I could be totally barking up the wrong tree, but I think this is our > clue: Requested array size exceeds VM limit > > That means that something is causing the reader to try to allocate an > array with a number of entries greater than the VM allows. > > Without seeing the schema, a sample of the CSV and a stacktrace it's > pretty hard to guess what's going on. For what it's worth, I've split > 55GB JSON sets using a custom streaming JSON reader without a hiccup > on a NiFi instance with only 4-8GB of RAM allocated, so I'm fairly > confident we've got some quirky edge case here. > > If you want to sanitize some inputs and share along with a schema that > might help. > > On Wed, Apr 14, 2021 at 1:07 PM Vibhath Ileperuma > <[email protected]> wrote: > > > > Hi Chris, > > > > As you have mentioned, I am trying to split the large csv file in > multiple stages. But this error is thrown at the first stage even without > creating a single flow file. > > It seems like the issue is not with the processor, but with the CSV > record reader. This error is thrown while reading the csv file. I tried to > write the data in the large csv file into a kudu table using a putKudu > processor with the same CSV reader. Then also I got the same error message. > > > > Hi Otto, > > > > Only following information is available in log file related to the > exception > > > > 2021-04-14 17:48:28,628 ERROR [Timer-Driven Process Thread-1] > o.a.nifi.processors.standard.SplitRecord > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] failed to process > session due to java.lang.OutOfMemoryError: Requested array size exceeds VM > limit; Processor Administratively Yielded for 1 sec: > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > > > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > > > 2021-04-14 17:48:28,628 WARN [Timer-Driven Process Thread-1] > o.a.n.controller.tasks.ConnectableTask Administratively Yielding > SplitRecord[id=c9a981db-0178-1000-363d-c767653a6f34] due to uncaught > Exception: java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > > > java.lang.OutOfMemoryError: Requested array size exceeds VM limit > > > > Thanks & Regards > > > > Vibhath Ileperuma > > > > > > > > > > On Wed, Apr 14, 2021 at 7:47 PM Otto Fowler <[email protected]> > wrote: > >> > >> What is the complete stack trace of that exception? > >> > >> On Apr 14, 2021, at 02:36, Vibhath Ileperuma < > [email protected]> wrote: > >> > >> Requested array size exceeds VM limit > >> > >> >
