Re: NiFi copying windows .part files

Imagine a filename construct where you wanted to pick up any file that begins with the phrase 'start' but does NOT end in the phrase 'part'. The name is of a form 'begin.middle.end'. This filename start.middle.ok would get picked up. This filename start.middle.part would not. The pattern for th

Re: NiFi copying windows .part files

Yeah..that is good idea, but we are already using this option to copy file with certain prefix. Not sure how I can use this field to meet both exclusion and inclusion criterion. Any thoughts. Thanks, Ravi Papisetti On 06/12/17, 1:26 AM, "Joe Witt" wrote: Ravi Please use the 'Fil

Re: NiFi copying windows .part files

Ravi Please use the 'File Filter' property of ListFile to control ignoring filenames until they no longer end in 'part'. Thanks On Wed, Dec 6, 2017 at 2:14 AM, Ravi Papisetti (rpapiset) wrote: > Hi, > > > > We are using Apache NiFi 1.3.0 > > > > We have a process flow to copy files from NFS to

NiFi copying windows .part files

Hi, We are using Apache NiFi 1.3.0 We have a process flow to copy files from NFS to HDFS (with processors ListFile, FetchFile and PutHDFS) In the NiFi process flow, ListFile is configured to listen to a directory on NFS. When a file (ex: x.csv) is being copied from a windows machine to NFS (w

Re: unable to start InvokeHTTP processor in secure Nifi 1.4.0 cluster....

Hi Dan/Joe, I have encountered the same issue and after a bit of digging it appears as if during the update to OkHttp3 a bug was introduced in the setSslFactoryMethod. The issue is that the method attempts to prepare a keystore even if properties for the keystore are not defined in the SSLContext

Re: PutParquet with S3

Take a look at the MergeRecord processor, you can use that before PutParquet to create the appropriately sized files. On Tue, Dec 5, 2017 at 10:36 PM Madhukar Thota wrote: > Thanks Joey, > > It worked. Do you know how to control the parquet file size when it writes > to S3. I see lot of small fi

Re: PutParquet with S3

Thanks Joey, It worked. Do you know how to control the parquet file size when it writes to S3. I see lot of small files to s3. Is it possible to right either 512mb or 1GB size file? On Tue, Dec 5, 2017 at 8:57 PM, Joey Frazee wrote: > PutParquet doesn't have the AWS S3 SDK included in it itsel

Re: PutParquet with S3

PutParquet doesn't have the AWS S3 SDK included in it itself but it provides an "Additional Classpath Resources" property that you need to point at a directory with all the S3 dependencies. I just tested this the other day with the following jars: aws-java-sdk-1.7.4.jar hadoop-aws-2.7.3.jar had

RE: [EXT] CDC like updates on Nifi

Alberto, Since it sounds like you have control over the structure of the tables, this should be doable. If you have a changelog table for each table this will probably be easier, and in your changelog table you’ll need to make sure you have a good transaction timestamp column and a change type

PutParquet with S3

Hi Is it possible to use PutParquet processor to write files into S3? I tried by setting s3 bucket in core-site.xml file but i am getting *No FileSystem for scheme: s3a* *core-site.xml* fs.defaultFS s3a://testing fs.s3a.access.key fs.s3a.secret.key xx

CDC like updates on Nifi

Hey folks, I read about Nifi CDC processor for MySQL and other CDC "solutions" with Nifi found on Google, like these: https://community.hortonworks.com/idea/53420/apache-nifi- processor-to-address-cdc-use-cases-for.html https://community.hortonworks.com/questions/88686/change- data-capture-using-

NiFi - replace custom script by available processors

Hi, My task is to analyze some log files to count the number of certain string occurrences and record the number in a SQL table. If the table does not exist, NiFi should create a new SQL table and put the data in. I have the process set up as the following picture:

Re: unable to start InvokeHTTP processor in secure Nifi 1.4.0 cluster....

Hello Joe, Here's the JIRA. LMK if you need additional details. https://issues.apache.org/jira/browse/NIFI-4655 Regards, Dano On Mon, Dec 4, 2017 at 10:46 AM Joe Witt wrote: > Dan > > Please share as much of your config for the processor as you can. > Also, please file a JIRA for this. Ther

RE: NIfi User Details

You can use Apache Ranger, with the NiFi plugin. https://cwiki.apache.org/confluence/display/RANGER/NiFi+Plugin Ranger performs authentication/authorisation of NiFi users, and keeps an audit of user actions. The is written to a SOLR instance for interactive browsing/searching, and can also be w