N outputs, you may be able to do this in one go or chain
> a few Processors (say “state”, then “city” then “zipcode”).
>
>
>
> Regards,
>
>
>
> Isha
>
>
>
> *Van:* James McMahon
> *Verzonden:* woensdag 12 april 2023 14:56
> *Aan:* users@nifi.apache.org
:56
Aan: users@nifi.apache.org
Onderwerp: Re: Handling CSVs dynamically with NiFi
Thank you very much Isha. This is helpful. Assuming I wanted to route to N
different output paths, does it follow that I need to use N different Query
Record processors tailored to filter for just one subset?
I&#x
eds of different
> json files with a handful of records each, then splitting per row might be
> quicker than copying the entire json file that many times.
>
>
>
> Regards,
>
>
>
> Isha
>
>
>
> *Van:* James McMahon
> *Verzonden:* vrijdag 7 april 2023 17
: James McMahon
Verzonden: vrijdag 7 april 2023 17:14
Aan: users@nifi.apache.org
Onderwerp: Re: Handling CSVs dynamically with NiFi
Hello Bryan. Thank you for your question.
A downstream consumer requires the complete set in json. So that's part of why
I convert.
Other downstream tools require
werful functions but they run surprisingly fast given the amount of
> string data they are traversing.
> >>
> >>
> >>
> >> Mike Sofen
> >>
> >>
> >>
> >> From: James McMahon
> >> Sent: Thursday, April 06, 2023 2:03
a wide range of validations on
>> individual keys, etc. I use the word amazing because they are not just
>> powerful functions but they run surprisingly fast given the amount of string
>> data they are traversing.
>>
>>
>>
>> Mike Sofen
>>
>>
of
> string data they are traversing.
>
>
>
> Mike Sofen
>
>
>
> *From:* James McMahon
> *Sent:* Thursday, April 06, 2023 2:03 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: Handling CSVs dynamically with NiFi
>
>
>
> Can I ask you one follow-up?
@nifi.apache.org
Subject: Re: Handling CSVs dynamically with NiFi
Can I ask you one follow-up? I've gotten my ConvertRecord to work. I created a
CsvReader service with Schema Access Strategy of Use String Fields From Header.
I created a JsonRecordSetWriter service with Schema Write Strategy of D
use that to dynamically create the DDL sql needed to
> build the staging table in Postgres. In my solution, there are 2 separate
> pipelines running – this pre step and the normal file processing.
>
>
>
> I used the pre step to ensure that all incoming files were from a known
>
: Handling CSVs dynamically with NiFi
Thank you both very much, Bryan and Mike. Mike, had you considered the approach
mentioned by Bryan - a Reader processor to infer schema - and found it wasn't
suitable for your use case, for some reason? For instance, perhaps you were
employing a versi
RY fast and efficient in this, as was Postgres.
>>
>>
>>
>> Mike Sofen
>>
>>
>>
>> From: James McMahon
>> Sent: Thursday, April 06, 2023 4:35 AM
>> To: users
>> Subject: Handling CSVs dynamically with NiFi
>>
>>
>>
&
;
>
>
> Mike Sofen
>
>
>
> *From:* James McMahon
> *Sent:* Thursday, April 06, 2023 4:35 AM
> *To:* users
> *Subject:* Handling CSVs dynamically with NiFi
>
>
>
> We have a task requiring that we transform incoming CSV files to JSON. The
> CSVs vary
a file when processing the actual files
> prior to storing into the destination table.
>
>
>
> Nifi was VERY fast and efficient in this, as was Postgres.
>
>
>
> Mike Sofen
>
>
>
> *From:* James McMahon
> *Sent:* Thursday, April 06, 2023 4:35 AM
>
: James McMahon
Sent: Thursday, April 06, 2023 4:35 AM
To: users
Subject: Handling CSVs dynamically with NiFi
We have a task requiring that we transform incoming CSV files to JSON. The CSVs
vary in schema.
There are a number of interesting flow examples out there illustrating how one
can
For any record reader, including CsvReader, you can choose the "Schema
Access Strategy" of "Infer Schema" and NiFi will read in all the
records and infer the schema from them.
On Thu, Apr 6, 2023 at 7:36 AM James McMahon wrote:
>
> We have a task requiring that we transform incoming CSV files to
We have a task requiring that we transform incoming CSV files to JSON. The
CSVs vary in schema.
There are a number of interesting flow examples out there illustrating how
one can set up a flow to handle the case where the CSV schema is well known
and fixed, but none for the generalized case.
The
16 matches
Mail list logo
