Re: zookeeper error message - nifi 1.11.1/zookeeper 3.5.6
Thank you for your email. Looking at the zookeeper docs, with 3.5.0 it
looks like the format may have changed to support the dynamic
configuration. It may seem that zookeeper is sending back a format that
NiFi isn't expecting??? I.e the :participant
https://zookeeper.apache.org/doc/r3.5.6/zookeeperReconfig.html
On Wed, Feb 12, 2020, 5:53 PM 노대호Daeho Ro wrote:
> In my memory,
>
> zookeepr 3.5.6 needs the new form of zookeeper string such as
>
> server.1=0.0.0.0:2888:3888;2181
>
>
> where the ip is yours.
>
> Hope this help you.
>
>
> 2020년 2월 13일 (목) 오전 1:55, dan young 님이 작성:
>
>> Sorry Joe,
>>
>> Yes, I'll file a JIRA...here's the email again
>>
>> We're seeing the following messages in nifi logs on our cluster nodes.
>> Using
>> Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
>>
>> Functionality seems not to be impacted, but wondering if there's
>> something else
>> going on or the version of zookeeper we're using is causing this.
>>
>> 2020-02-12 15:36:43,959 ERROR [main-EventThread]
>> o.a.c.framework.imps.EnsembleTracker Invalid config event received:
>> {server.1=10.190.3.170:2888:3888:participant, version=0,
>> server.3=10.190.3.91:2888:3888:participant, server.2=10.190.3.172:2888
>> :3888:participant}
>>
>> Regards,
>>
>> Dano
>>
>> On Wed, Feb 12, 2020 at 9:49 AM Joe Witt wrote:
>>
>>> Dan,
>>>
>>> Not sure what others see but for me your email cuts off in the middle of
>>> a line.
>>>
>>> You might want to file a JIRA with your observation/logs.
>>>
>>> Thanks
>>>
>>> On Wed, Feb 12, 2020 at 11:46 AM dan young wrote:
>>>
Hello,
We're seeing the following messages in nifi logs on our cluster nodes.
Using
Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
Functionality seems not to be impacted, but wondering if there's something
else
going on or the version of zookeeper we're using is causing this.
2020-02-12 15:36:43,959 ERROR [main-EventThread]
o.a.c.framework.imps.EnsembleTracker Invalid config event received:
{server.1=10.190.3.170:2888:3888:participant, version=0,
server.3=10.190.3.91:2888:3888:participant,
server.2=10.190.3.172:2888:3888:participant}
Regards,
Dano
Re: zookeeper error message - nifi 1.11.1/zookeeper 3.5.6
In my memory,
zookeepr 3.5.6 needs the new form of zookeeper string such as
server.1=0.0.0.0:2888:3888;2181
where the ip is yours.
Hope this help you.
2020년 2월 13일 (목) 오전 1:55, dan young 님이 작성:
> Sorry Joe,
>
> Yes, I'll file a JIRA...here's the email again
>
> We're seeing the following messages in nifi logs on our cluster nodes.
> Using
> Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
>
> Functionality seems not to be impacted, but wondering if there's something
> else
> going on or the version of zookeeper we're using is causing this.
>
> 2020-02-12 15:36:43,959 ERROR [main-EventThread]
> o.a.c.framework.imps.EnsembleTracker Invalid config event received:
> {server.1=10.190.3.170:2888:3888:participant, version=0,
> server.3=10.190.3.91:2888:3888:participant, server.2=10.190.3.172:2888
> :3888:participant}
>
> Regards,
>
> Dano
>
> On Wed, Feb 12, 2020 at 9:49 AM Joe Witt wrote:
>
>> Dan,
>>
>> Not sure what others see but for me your email cuts off in the middle of
>> a line.
>>
>> You might want to file a JIRA with your observation/logs.
>>
>> Thanks
>>
>> On Wed, Feb 12, 2020 at 11:46 AM dan young wrote:
>>
>>> Hello,
>>>
>>> We're seeing the following messages in nifi logs on our cluster nodes.
>>> Using
>>> Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
>>>
>>> Functionality seems not to be impacted, but wondering if there's something
>>> else
>>> going on or the version of zookeeper we're using is causing this.
>>>
>>> 2020-02-12 15:36:43,959 ERROR [main-EventThread]
>>> o.a.c.framework.imps.EnsembleTracker Invalid config event received:
>>> {server.1=10.190.3.170:2888:3888:participant, version=0,
>>> server.3=10.190.3.91:2888:3888:participant,
>>> server.2=10.190.3.172:2888:3888:participant}
>>>
>>> Regards,
>>>
>>> Dano
>>>
>>>
>>>
Re: NiFi user and access rights
You could use MiNiFi agents on each external resource to consume data in a siloed manner and transmit it to a central NiFi instance over Site-to-site protocol. This would allow each producer of data to remain isolated (either physically disconnected or each using a distinct OS user for ACL with the respective MiNiFi agents running as that user) and communicate the necessary data back to a central processing instance. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 12, 2020, at 6:54 AM, Tomislav Novosel wrote: > > Hi guys, > > I'm having this situation inside my company projects. We are using NiFi as > DataFlow platform and there are multiple projects. > Every project has files on shared disk/folder from which one Nifi > instance(standalone instance) is reading data. > NiFi instance service is running under one generic user which has read rights > for every shared folder/project and that is fine. > > As there will be more and more projects and only one generic user will need > to have read rights on all shared disks/folders of all projects. So which is > better solution: > > To have one NiFi instance running with one generic user which has read rights > on all shared disks/folders. From security standpoint it is not ok. Shared > folders are from various customers. Data volume and load is not too big for > only one standalone NiFi instance. > To have Multiple NiFi instances on one server each running under different > generic user and every generic user belongs to one customer shared folder > regarding read rights, 1:1 relationship. > In the future there will be need to scure NiFi instances with SSL, maybe to > add more nodes and to establish multi-tenancy. > > Is there maybe some other third solution for this situation? How to setup > that kind of data flow where are multiple data sources and security is > important? > > Thanks in advance and best regards. > > Tom
Re: zookeeper error message - nifi 1.11.1/zookeeper 3.5.6
Sorry Joe,
Yes, I'll file a JIRA...here's the email again
We're seeing the following messages in nifi logs on our cluster nodes.
Using
Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
Functionality seems not to be impacted, but wondering if there's something
else
going on or the version of zookeeper we're using is causing this.
2020-02-12 15:36:43,959 ERROR [main-EventThread]
o.a.c.framework.imps.EnsembleTracker Invalid config event received:
{server.1=10.190.3.170:2888:3888:participant, version=0,
server.3=10.190.3.91:2888:3888:participant, server.2=10.190.3.172:2888
:3888:participant}
Regards,
Dano
On Wed, Feb 12, 2020 at 9:49 AM Joe Witt wrote:
> Dan,
>
> Not sure what others see but for me your email cuts off in the middle of a
> line.
>
> You might want to file a JIRA with your observation/logs.
>
> Thanks
>
> On Wed, Feb 12, 2020 at 11:46 AM dan young wrote:
>
>> Hello,
>>
>> We're seeing the following messages in nifi logs on our cluster nodes. Using
>> Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
>>
>> Functionality seems not to be impacted, but wondering if there's something
>> else
>> going on or the version of zookeeper we're using is causing this.
>>
>> 2020-02-12 15:36:43,959 ERROR [main-EventThread]
>> o.a.c.framework.imps.EnsembleTracker Invalid config event received:
>> {server.1=10.190.3.170:2888:3888:participant, version=0,
>> server.3=10.190.3.91:2888:3888:participant,
>> server.2=10.190.3.172:2888:3888:participant}
>>
>> Regards,
>>
>> Dano
>>
>>
>>
Re: zookeeper error message - nifi 1.11.1/zookeeper 3.5.6
Dan,
Not sure what others see but for me your email cuts off in the middle of a
line.
You might want to file a JIRA with your observation/logs.
Thanks
On Wed, Feb 12, 2020 at 11:46 AM dan young wrote:
> Hello,
>
> We're seeing the following messages in nifi logs on our cluster nodes. Using
> Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
>
> Functionality seems not to be impacted, but wondering if there's something
> else
> going on or the version of zookeeper we're using is causing this.
>
> 2020-02-12 15:36:43,959 ERROR [main-EventThread]
> o.a.c.framework.imps.EnsembleTracker Invalid config event received:
> {server.1=10.190.3.170:2888:3888:participant, version=0,
> server.3=10.190.3.91:2888:3888:participant,
> server.2=10.190.3.172:2888:3888:participant}
>
> Regards,
>
> Dano
>
>
>
zookeeper error message - nifi 1.11.1/zookeeper 3.5.6
Hello,
We're seeing the following messages in nifi logs on our cluster nodes. Using
Nifi 1.11.1 and zookeeper (not embedded) version 3.5.6
Functionality seems not to be impacted, but wondering if there's something else
going on or the version of zookeeper we're using is causing this.
2020-02-12 15:36:43,959 ERROR [main-EventThread]
o.a.c.framework.imps.EnsembleTracker Invalid config event received:
{server.1=10.190.3.170:2888:3888:participant, version=0,
server.3=10.190.3.91:2888:3888:participant,
server.2=10.190.3.172:2888:3888:participant}
Regards,
Dano
NiFi user and access rights
Hi guys, I'm having this situation inside my company projects. We are using NiFi as DataFlow platform and there are multiple projects. Every project has files on shared disk/folder from which one Nifi instance(standalone instance) is reading data. NiFi instance service is running under one generic user which has read rights for every shared folder/project and that is fine. As there will be more and more projects and only one generic user will need to have read rights on all shared disks/folders of all projects. So which is better solution: 1. To have one NiFi instance running with one generic user which has read rights on all shared disks/folders. From security standpoint it is not ok. Shared folders are from various customers. Data volume and load is not too big for only one standalone NiFi instance. 2. To have Multiple NiFi instances on one server each running under different generic user and every generic user belongs to one customer shared folder regarding read rights, 1:1 relationship. In the future there will be need to scure NiFi instances with SSL, maybe to add more nodes and to establish multi-tenancy. Is there maybe some other third solution for this situation? How to setup that kind of data flow where are multiple data sources and security is important? Thanks in advance and best regards. Tom
Re: Unable to load the authorizer configuration file
For the sake of testing I have just been running everything as root. Permissions look good to me. [root@wvllab-lnx-nifi conf]# ls -la total 116 drwxr-xr-x 3 root root 4096 Feb 12 00:13 . drwxr-xr-x 16 root root 328 Feb 11 23:36 .. drwxr-xr-x 2 root root86 Feb 12 00:02 archive -rw-r--r-- 1 root root 1813 Feb 11 23:36 authorizations.xml -rw-rw-r-- 1 root root 22076 Feb 12 00:01 authorizers.xml -rw-rw-r-- 1 root root 4026 Jan 5 11:24 bootstrap.conf -rw-rw-r-- 1 root root 2326 Jan 5 11:24 bootstrap-notification-services.xml -rw-r--r-- 1 root root 282 Feb 12 00:02 flow.xml.gz -rw--- 1 root root 3105 Feb 11 23:25 keystore.jks -rw-rw-r-- 1 root root 8666 Jan 5 11:24 logback.xml -rw-rw-r-- 1 root root 6506 Feb 11 23:32 login-identity-providers.xml -rw--- 1 root root 1200 Feb 11 23:25 nifi-cert.pem -rw--- 1 root root 1675 Feb 11 23:25 nifi-key.key -rw-rw-r-- 1 root root 12373 Feb 11 23:27 nifi.properties -rw-rw-r-- 1 root root 8669 Jan 5 11:24 state-management.xml -rw--- 1 root root 911 Feb 11 23:25 truststore.jks -rw-r--r-- 1 root root 382 Feb 11 23:36 users.xml -rw-rw-r-- 1 root root 1718 Jan 5 11:24 zookeeper.properties [root@wvllab-lnx-nifi conf]# pwd /apps/nifi-test/conf On Wed, Feb 12, 2020 at 7:55 AM Bryan Bende wrote: > Hello, > > It looks like basically the same error you had for authorizers.xml, > but now for login-identity-providers.xml. > > It is saying the file does not exist at > /apps/nifi-test/./conf/login-identity-providers.xml > > Can you verify the file exists and it is readable by whatever OS user > NiFi is running as. > > Thanks, > > Bryan > > On Wed, Feb 12, 2020 at 12:55 AM Josh Friberg-Wyckoff > wrote: > > > > I restarted from scratch and tried again. This time it is complaining > about the Login Identity Provider xml > > https://gist.github.com/SquashBuckler/96d23ebd1652908c0b00db0a06198a13 > > > > On Tue, Feb 11, 2020 at 10:26 PM Pierre Villard < > pierre.villard...@gmail.com> wrote: > >> > >> There should be more lines below the ERROR log you gave in your initial > email. The full stack trace here will definitely help. If too long and not > sensitive, you can put in a public gist [1] and give us the link. > >> > >> [1] https://gist.github.com/ > >> > >> Le mar. 11 févr. 2020 à 19:30, Josh Friberg-Wyckoff < > j...@thefribergs.com> a écrit : > >>> > >>> So is that the entire file or just part of it. Over 2300 lines in it. > >>> > >>> On Tue, Feb 11, 2020 at 8:42 PM Pierre Villard < > pierre.villard...@gmail.com> wrote: > > Hi Josh, > > The full stack trace from nifi-app.log would probably provide more > information. This is likely a configuration issue. > > Thanks, > Pierre > > Le mar. 11 févr. 2020 à 15:44, Josh Friberg-Wyckoff < > j...@thefribergs.com> a écrit : > > > > I followed a tutorial from the Mint Ops Blog on how to setup NiFi > with LDAP. > > > > I am getting the following error when trying to start Nifi. Would > be helpful if anyone could point me in the right direction. > > > > 2020-02-11 16:04:59,398 ERROR [NiFi logging handler] > org.apache.nifi.StdErr Failed to start web server: Error creating bean with > name > 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': > Unsatisfied dependency expressed through method > 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is > org.springframework.beans.factory.BeanExpressionException: Expression > parsing failed; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied > dependency expressed through method 'setJwtAuthenticationProvider' > parameter 0; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'jwtAuthenticationProvider' defined in class path resource > [nifi-web-security-context.xml]: Cannot resolve reference to bean > 'authorizer' while setting constructor argument; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'authorizer': FactoryBean threw exception on object > creation; nested exception is java.lang.Exception: Unable to load the > authorizer configuration file at: /apps/nifi/./conf/authorizers.xml > > > > >
Re: Machines for NiFi: general purpose or memory intensive
Most of your pain points with performance will be either CPU or disk I/O unless you happen to have a particularly memory-intensive flow (most aren't). I'd start with a heap of about 4GB of RAM/node and scale up from there as needed while testing. Odds are pretty good that you'll find that adding cores, not a lot of memory, will be the source of most improvements. Start smallish like that and figure out how performant the process must be. My main client, for example, prefers to spend less on cloud resources and accept the longer wait as a way of saving money since there's no business rush to get the data in really fast. On Wed, Feb 12, 2020 at 3:09 AM Anurag Sharma wrote: > Hi All, > > We use Digital Ocean, just wish to check with the community if we should > go for general purpose machines or memory-intensive machines for our NiFi > cluster? > > Regards > Anurag >
Re: Unable to load the authorizer configuration file
Hello, It looks like basically the same error you had for authorizers.xml, but now for login-identity-providers.xml. It is saying the file does not exist at /apps/nifi-test/./conf/login-identity-providers.xml Can you verify the file exists and it is readable by whatever OS user NiFi is running as. Thanks, Bryan On Wed, Feb 12, 2020 at 12:55 AM Josh Friberg-Wyckoff wrote: > > I restarted from scratch and tried again. This time it is complaining about > the Login Identity Provider xml > https://gist.github.com/SquashBuckler/96d23ebd1652908c0b00db0a06198a13 > > On Tue, Feb 11, 2020 at 10:26 PM Pierre Villard > wrote: >> >> There should be more lines below the ERROR log you gave in your initial >> email. The full stack trace here will definitely help. If too long and not >> sensitive, you can put in a public gist [1] and give us the link. >> >> [1] https://gist.github.com/ >> >> Le mar. 11 févr. 2020 à 19:30, Josh Friberg-Wyckoff a >> écrit : >>> >>> So is that the entire file or just part of it. Over 2300 lines in it. >>> >>> On Tue, Feb 11, 2020 at 8:42 PM Pierre Villard >>> wrote: Hi Josh, The full stack trace from nifi-app.log would probably provide more information. This is likely a configuration issue. Thanks, Pierre Le mar. 11 févr. 2020 à 15:44, Josh Friberg-Wyckoff a écrit : > > I followed a tutorial from the Mint Ops Blog on how to setup NiFi with > LDAP. > > I am getting the following error when trying to start Nifi. Would be > helpful if anyone could point me in the right direction. > > 2020-02-11 16:04:59,398 ERROR [NiFi logging handler] > org.apache.nifi.StdErr Failed to start web server: Error creating bean > with name > 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': > Unsatisfied dependency expressed through method > 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is > org.springframework.beans.factory.BeanExpressionException: Expression > parsing failed; nested exception is > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name > 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied > dependency expressed through method 'setJwtAuthenticationProvider' > parameter 0; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'jwtAuthenticationProvider' defined in class path resource > [nifi-web-security-context.xml]: Cannot resolve reference to bean > 'authorizer' while setting constructor argument; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'authorizer': FactoryBean threw exception on object > creation; nested exception is java.lang.Exception: Unable to load the > authorizer configuration file at: /apps/nifi/./conf/authorizers.xml > >
Machines for NiFi: general purpose or memory intensive
Hi All, We use Digital Ocean, just wish to check with the community if we should go for general purpose machines or memory-intensive machines for our NiFi cluster? Regards Anurag
