Re: Data extraction for 100 columns is possible in NiFi?

2017-01-31 Thread prabhu Mahendran
@Mark,@Matt,@Nick Thanks for your valuable information. It really helpful for me. Many thanks, prabhu On Mon, Jan 30, 2017 at 11:38 PM, Nick Carenza < nick.care...@thecontrolgroup.com> wrote: > Hey Prabhu, I just finished up a csv processing flow myself and it looks > like this: > > CSV Flowfi

Re: Data extraction for 100 columns is possible in NiFi?

2017-01-30 Thread Nick Carenza
Hey Prabhu, I just finished up a csv processing flow myself and it looks like this: CSV Flowfile -> InferAvroSchema -> ConvertCSVToAvro -> ConvertAvroToJson You can then use the ConvertJSONToSQL processor to finish things up. I would have liked to be able to go directly from CSV to JSON but I don

Re: Data extraction for 100 columns is possible in NiFi?

2017-01-30 Thread Matt Burgess
Prabhu, I agree with Mark; if you want to use ExecuteScript for this, I have an example of splitting fields (using a bar | delimiter, but you can change to comma) [1]. If you have quoted values that can contain commas, then like Mark said you may want to look at writing a custom processor, or usi

Re: Data extraction for 100 columns is possible in NiFi?

2017-01-30 Thread Mark Payne
Prabhu, My guess is that you probably could find some way to do this with the standard out-of-the-box processors that come with NiFi. Perhaps by using Extract Text to extract the header columns, and then using ReplaceText and perhaps a few other processors. Going down this route though is likely

Data extraction for 100 columns is possible in NiFi?

2017-01-29 Thread prabhu Mahendran
I have a CSV data with 100 columns like below.., ,A,B,C,D,E,F,..[upto 100 Header columns] Date,A1,B1,C1,D1,E1,F1..[upto 100 Header columns] 30/01/2017 23:23:22,Majestic,32,2100.12[upto 100 data columns] In data having first 2 header lines and 3 rd line is data in which i inser