RE: Help on setting up authentication between NiFi and NiFi Registry

2021-03-22 Thread Sim, Yoosuk
Thanks everyone, @Kevn I wasn't sure how to link the NiFi certificate to the NiFi Registry account as that video wasn't so clear on where that happened. He sorted just went and created some user without fully explaining where that came from or how he went about the certificate. He just said he

Re: Help on setting up authentication between NiFi and NiFi Registry

2021-03-22 Thread Bryan Bende
In addition to what Kevin said... since it is always 2-way TLS between nifi and nifi-registry, if you are seeing "No client certificate found in request" then you may want to double check that NiFI's server cert specified in the keystore jks actually has clientAuth usage. On Mon, Mar 22, 2021 at 4

Re: Help on setting up authentication between NiFi and NiFi Registry

2021-03-22 Thread Kevin Doran
Hi Tony, The typical approach I have seen is to allow NiFi to authenticate to NiFi Registry using a client certificate. This can be achieved alongside LDAP authentication by configuring NiFi Registry using a ConfiguableCompositeUserGroupProvider, with a FileUserGroupProvider to manage certifi

Help on setting up authentication between NiFi and NiFi Registry

2021-03-22 Thread Sim, Yoosuk
Hello everyone, I am trying to setup NiFi and NiFi Registry to talk to each other. They are both using LDAP to authenticate and authorize the users and they seem to talk to each other over SSL. However, I can't seem to figure out how to let NiFi-Registry to authenticate NiFi. After some debuggi

Re: Creating recursive missing folders with PutSmbFile

2021-03-22 Thread Jens M. Kofoed
Dear Mark I would love to help, testing the PR to check that it is working. My only problem is I'm not able to build the nar files my self. So if someone can build the file, I have no problems testing the file. About SMB: I don't like to mount networks drives locally to my NiFi servers which is r

Re: Creating recursive missing folders with PutSmbFile

2021-03-22 Thread Joe Witt
pretty sure SMB is super popular - it is just that for the cases we typically engage in SMB isn't used as the protocol to access data :) Agree with the rest of that Thanks On Mon, Mar 22, 2021 at 11:13 AM Mark Payne wrote: > > Jens, > > In order to review & merge a PR, there are two important t

Re: Creating recursive missing folders with PutSmbFile

2021-03-22 Thread Mark Payne
Jens, In order to review & merge a PR, there are two important things that need to happen: 1. A NiFi committer must review the code to make sure that the changes are safe, make sense, conducive with the architecture, is adhering to best practices, doesn’t break automated tests, etc. 2. The cod

RecordPath...With INNER records

2021-03-22 Thread Greene (US), Geoffrey N
Im making good progress leaning to use using LookupRecord Processor. I have it working so that I know how to turn [{"key": "value1"}, {"key":"value2"}, {"key":"value3"}] Into [{"key": "value1","enhanced":1}, {"key":"value2","enhanced":2}, {"key":"value3","enhanced":3}] BUT What if you have inner

Re: Writing parquet files to S3

2021-03-22 Thread Joe Witt
Not responding to the real question in the thread but "I'm using NIFI 1.13.1.". Please switch to 1.13.2 right away due to a regression in 1.13.1 On Mon, Mar 22, 2021 at 12:24 AM Vibhath Ileperuma wrote: > > Hi Bryan, > > I'm planning to add these generated parquet files to an impala S3 table. >

Re: speeding up ListFile

2021-03-22 Thread James McMahon
Thank you. Excluding those other attributes seems like a good choice to improve performance unless we absolutely need those other attributes. On Mon, Mar 22, 2021 at 9:00 AM Mike Sofen wrote: > I just retested, to be sure, and there is no impact from setting “include > file attributes” to False

RE: speeding up ListFile

2021-03-22 Thread Mike Sofen
I just retested, to be sure, and there is no impact from setting “include file attributes” to False – stopping a flow pointed at a folder tree that had already processed the files, adding one file, then restarting it, the flow only picked up the new file. And it still includes the critical attr

Creating recursive missing folders with PutSmbFile

2021-03-22 Thread Jens M. Kofoed
Hi The following JIRA: https://issues.apache.org/jira/browse/NIFI-7863, was created October 1, 2020 and the user Jaya has created a PR October 9, 2020. but nothing have happens since. Are there someone in the community which is able to help implement a fix? We had looked forward to see the fix in

Re: Writing parquet files to S3

2021-03-22 Thread Vibhath Ileperuma
Hi Bryan, I'm planning to add these generated parquet files to an impala S3 table. I noticed that impala written parquet files contain only one row group. That's why I'm trying to write one row group per file. However, I tried to create small parquet files (Snappy compressed) first and use a Merg