Hi James - I too would like more information on this. I've been using
NiFi in a kubernetes cluster for over a year now using cetic
(https://github.com/cetic/helm-nifi) and have it working with NiFi
1.25.0. This works well with a 3 node cluster, and can successfully
scale up. Scaling down
Hi all - I'm running a NiFi cluster in kubernetes - version 1.25.0, and
when it's under load, I get these errors:
2024-03-06 22:22:58,081 ERROR [Load-Balanced Client Thread-4]
o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Unable to connect to
0x20 is a space. Maybe that's somewhere in your header?
On 1/12/2024 4:33 PM, James McMahon wrote:
I have a text flowfile that I am trying to send to a translation
service on a remote EC2 instance from my nifi insurance on my EC2. I
am failing with only this somewhat-cryptic error:
You can also set the processors scheduling -> run duration to something
other than 0ms.
I've found NiFi will do heavy disk IO when things have been running for
a while / queue sizes are large. Been using tools like atop to watch
disk IO. Check settings for flow, content, and provenance repos.
We currently have a 3 node NiFi cluster of version 1.23.2 deployed to
our bare-metal kubernetes cluster using https://github.com/cetic/helm-nifi.
It is working very nicely. I'd love to upgrade to 2.0.0M1 and utilize
the configmaps instead of zookeeper. Is there a helm chart available?
Or
Agree! Looks great. Nice.
On 1/9/2024 11:14 AM, Mike Thomsen wrote:
That's a very impressive update!
On Mon, Jan 8, 2024 at 2:38 PM Jörg Hammerbacher
wrote:
Looks Great! Well Done!
Jörg
Am 08.01.2024 um 19:17 schrieb David Handermann:
> Team,
>
> Thanks to a
(might be perfectly
fine for some cases - just curious)
Thanks
On Wed, Jul 12, 2023 at 8:18 AM Joe Obernberger
wrote:
Thank you Joe -
The content repo doesn't seem to be the issue - it's the flowfile
repo.
Here is the section from one of the nodes
FlowFiles per hour then 10 KB is
fine, but if you’re processing a bunch of FlowFiles it adds up very
quickly.
Thanks
-Mark
On Jul 12, 2023, at 11:16 AM, Joe Obernberger
wrote:
Thank you Joe -
The content repo doesn't seem to be the issue - it's the flowfile repo.
Here is the section from one
recommend reducing it to 50KB and
restarting.
Can you show your full 'nifi.content' section from the nifi.properties?
Thanks
On Wed, Jul 12, 2023 at 7:54 AM Joe Obernberger
wrote:
Raising this thread from the dead...
Having issues with IO to the flowfile repository. NiFi will show
. If it stays large like 31 GB while the other stay small,
that would be interesting to know.
Thanks
-Mark
On Mar 22, 2023, at 12:45 PM, Joe Obernberger
wrote:
Thanks for this Mark. I'm not seeing any large attributes at the
moment but will go through this and verify - but I did have one queue
of content into attributes.
And in this case, setting Backpressure Threshold above 10,000 is even
more concerning, as it means even greater disk I/O.
Thanks
-Mark
On Mar 22, 2023, at 11:01 AM, Joe Obernberger
wrote:
Thank you Mark. These are SATA drives - but there's no way for the
flowfile
running NiFi with ssd/nvme drives. Absent
that, if you have multiple disk drives, you could also configure the
content repository to span multiple disks, in order to spread that load.
Thanks
-Mark
On Mar 22, 2023, at 10:41 AM, Joe Obernberger
wrote:
Thank you. Was able to get in.
Currently
secs in
nifi.properties:
nifi.cluster.node.read.timeout : 5 sec
That might be worth increasing.
> On Mar 22, 2023, at 10:24 AM, Joe Obernberger
wrote:
>
> Hi Mark - thank you so much for helping me.
> Any thoughts on the attached?
>
> -Joe
>
> On 3/22/2023 10:21 AM,
, try setting the autoresume setting
in nifi.properties to false and restart node(s).
This will bring up every component/controllerService up
stopped/disabled and may provide some breathing room for the UI to
become available again.
Phil
On Mar 22, 2023 at 10:20 AM -0400, Joe Obernberger
, wrote
more likely the
culprit than disk I/O.
Thanks
-Mark
On Mar 22, 2023, at 10:12 AM, Joe Obernberger
wrote:
I'm getting "java.net.SocketTimeoutException: timeout" from the user interface
of NiFi when load is heavy. This is 1.18.0 running on a 3 node cluster. Disk IO is high
I'm getting "java.net.SocketTimeoutException: timeout" from the user
interface of NiFi when load is heavy. This is 1.18.0 running on a 3
node cluster. Disk IO is high and when that happens, I can't get into
the UI to stop any of the processors.
Any ideas?
I have put the flowfile repository
viewpoint.
*Steve Hindmarch*
*From:*Joe Obernberger
*Sent:* 19 November 2022 18:33
*To:* users@nifi.apache.org; Aian Cantabrana ;
Joe Witt
*Subject:* Re: Exacly once from NiFi to Kafka
Are you by chance using a clustered NiFi? I'm seeing duplicate
messages if I run the consumer on multiple
Are you by chance using a clustered NiFi? I'm seeing duplicate messages
if I run the consumer on multiple NiFi nodes, so I've started running
the consumer only on the parent. This seems to correct the issue, but
leads to other problems. I'd love a solution.
-Joe
On 11/16/2022 3:50 AM,
Hi - I'm using NiFi 1.18.0 in a three node cluster using internal
zookeeper. Occasionally, I will see a queue showing queued messages,
but when I list the queue, the UI says that there are no flow files.
Also the consumer of this queue seems to be idle. I can then empty the
queue and it
Hi - these are three Amazon EC2 instances, 16 cores and 124G of RAM.
-Joe
On 10/5/2022 9:31 AM, Mark Payne wrote:
Joe,
What kind of hardware are you running on?
Thanks
Mark
Sent from my iPhone
On Oct 5, 2022, at 9:23 AM, Joe Obernberger
wrote:
Hi all - I'm using NiFi 1.16.3 in a 3
Hi all - I'm using NiFi 1.16.3 in a 3 node cluster with content
repository settings:
# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=50 KB
Handermann
On Tue, Jul 20, 2021 at 1:59 PM Joe Obernberger
wrote:
Hmm - with:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
I get the following when running "./nifi.sh
set-sensitive-properties-key 12characterpassword"
NiFi Properties Processed
low.xml.gz.
With the existing flow.xml.gz in the conf directory, did you try
running the set-sensitive-properties-key command after setting the PBE
value for the sensitive properties algorithm in nifi.properties?
Regards,
David Handermann
On Tue, Jul 20, 2021 at 12:32 PM Joe Obernberger
wrote:
.
Please pass along any stack traces if NiFi does not startup after
making those changes.
Regards,
David Handermann
On Tue, Jul 20, 2021 at 11:54 AM Joe Obernberger
wrote:
Thank you David.
If my prior flow was not encrypted, what do I set the
nifi.sensitive.props.algorithm to? I've
deployments. As a self-signed certificate, it is not
intended to be used for clustered configurations, so existing
documentation on configuring a secure cluster provides the recommended
approach.
Regards,
David Handermann
On Tue, Jul 20, 2021 at 9:31 AM Joe Obernberger
wrote:
Thank you
rds,
David Handermann
On Mon, Jul 19, 2021 at 6:50 PM Joe Obernberger
wrote:
Trying to go from 1.13.2 to 1.14.0, but am getting this error:
2021-07-19 19:47:36,953 WARN [main]
org.apache.nifi.web.server.JettyServer Failed to start web server...
shu
Trying to go from 1.13.2 to 1.14.0, but am getting this error:
2021-07-19 19:47:36,953 WARN [main]
org.apache.nifi.web.server.JettyServer Failed to start web server...
shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with
Algorithm [AES/GCM/NoPadding]
at
Impressive cluster size! I do not have an answer for you, but could you
change your architecture so that instead of one large NiFi cluster you
have 2 or 3 smaller clusters? Very curious on the answer here as I have
also noticed UI slow-downs as the number of nodes increases.
-Joe
On
Hi All - when I set the Max Poll Records to anything less than 1000, the
processor still returns 1000 records. I can set it higher than 1000,
but it appears 1000 is the smallest that I can set it to.
Is this an issue with the processor, or could something be wrong with
the Kafka cluster?
and avoid data loss, you can just route the
failure relationship from InvokeHTTP back to InvokeHTTP with nothing
in between. This is the typical pattern used in nifi.
Thanks
-Mark
Sent from my iPhone
On Jun 18, 2021, at 1:37 PM, Joe Obernberger
wrote:
Thank you Arpad. We are using NiFi
without it as well.
Regards,
Arpad
On Thu, Jun 17, 2021 at 6:59 PM Joe Obernberger
wrote:
Hi All - I'm wondering if there is an approach to using the
RetryFlowFile that doesn't get 'stuck' if there are lots of failures.
I'm using InvokeHTTP and if it fails, the failure goes
process fails for a long time, the
failure queue fills, the retry queue fills, and when InvokeHTTP is
brought back up, it won't start since both queues are full.
Any ideas?
Thank you!
-Joe Obernberger
32 matches
Mail list logo
