Tim and Matt,
Did the changes that facilitate these settings ever make it into somewhere
public to be pulled into NiFi? I’m needing to set these BasicDataSource
settings for my own needs [1] and would love to reuse someone else’s code
instead of doing it up myself if possible.
[1]:
https://st
Inline.
On Thu, Nov 1, 2018 at 1:40 PM Bryan Bende wrote:
> How big are the initial CSV files?
>
> If they are large, like millions of lines, or even hundreds of
> thousands, then it will be ideal if you can avoid the line-by-line
> split, and instead process the lines in place.
>
> Not million.
Thank you Bryan. I did need to use the variable registry support. That fix
the problem.
From: Bryan Bende
Sent: Wednesday, October 31, 2018 4:52 PM
To: users@nifi.apache.org
Subject: Re: Expression Language
You haven't said which processor/service you are using, but you may want to
check the
How big are the initial CSV files?
If they are large, like millions of lines, or even hundreds of
thousands, then it will be ideal if you can avoid the line-by-line
split, and instead process the lines in place.
This is one of the benefits of the record processors. For example,
with UpdateRecord
Thanks for the input folks.
I had this impression that for actual processing of the data :
- we may have to put in place a custom processor which will have the
transformation framework logic in it.
- Or we can use ExcecuteProcess processor to trigger an external
process(which will be
Ameer,
Depending on how you implemented the custom framework, you may be able to
easily drop it in place into a custom NiFi processor. Without knowing much
about your implementation details, if you can act on Java streams, Strings,
byte arrays and things like that it will probably be very straight
Hi Ameer,
This blog by Mark Payne describes how to manipulate record based data like CSV
using schemas:
https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi. This would
probably be the most efficient method. And another here:
https://bryanbende.com/development/2017/06/20/apache-n
Shawn,
That did the trick. I ended up removing all properties with reference to
hook and adding hcatalog.hive.client.cache.disabled=true in the hive-site.xml
file referenced by PutHiveStreaming in Hive Configuration Resources.
Thank you
Noe
On Wed, Oct 31, 2018 at 10:47 AM Shawn Weeks
wrote:
>
