Hi all,
Are there any metrics available in NiFi that could help me track down the
processors/flows causing a lot of data to be written in the flowfile repository?
The production cluster in question typically sees 50-200 MB used in the
flowfile repository under daily load, but we've been seeing p
n Thu, Jun 27, 2024 at 1:29 AM Isha Lamboo
mailto:isha.lam...@virtualsciences.nl>> wrote:
Hi David,
This typically means one of two things:
1. that the DN of the (client) certificate does not match the user in the
receiving NiFi instance *exactly*.
Inspect the logs (I think it’s nifi-u
Hi David,
This typically means one of two things:
1. that the DN of the (client) certificate does not match the user in the
receiving NiFi instance *exactly*.
Inspect the logs (I think it’s nifi-user.log) to find the DN that the sending
NiFi is providing and edit the username in the receivin
Hi Emmanuel,
It looks like the server certificate was signed with a CA certificate that
doesn't have the right usage options set to act as a CA by signing
certificates. Most likely this is a self-signed certificate, in which case both
server and CA/certsign key usage options need to be specifie
Hi Shamsudeen,
We are also using AD for our users and they appear automatically. You say you
are not using groups, but you need to specify at least one group whose members
will be added as NiFi users. If you don’t have a suitable group, there is
probably a Developers group or even the generic U
Hi James,
Have you changed these settings in authorizers.xml since you first started
NiFi? If so, you may need to delete users.xml and authorizations.xml.
A new admin user will not be created if those files already exist.
Otherwise, the trickiest part is usually that the user DN needs to match
Hi James,
I would suggest you try to debug this using the openssl s_client command,
something like this:
openssl s_client -connect : -debug -cert client.pem -key
clientkey.pem -CAfile rootcert.pem
This should give you a lot of details, including information from the server
that specifies whic
We've had this occur when executing complex queries and/or queries on large
tables in ExecuteSQL.
We typically try out some values of Max Rows Per Flow File and Fetch Size (both
to the same value) in the range of 1000, 10k, 50k, 100k to make it work without
memory issues. Changing the Output Ba
Hi Quentin,
I've encountered similar errors in the past when trying to change the
encryption algorithm.
Here are two things that may help:
1. The password/key needs to be at least 12 characters long before you
migrate to NIFI_PBKDF2_AES_GCM_256. If it is not, you have to first change the
Hi Ben,
The cleanup should happen on startup if the usage exceeds 50% and run
regularly. You may even see warnings that NiFi is applying backpressure while
waiting for archiving to free up space.
If archiving doesn’t free up space, you may have the situation that content
files are being kept b
Hi Minh,
You should have messages in the nifi-app.log about the node failing to respond
within the configured cluster comms timeout in nifi.properties. You may want to
increase that and see if it reduces the number of disconnect events.
In my experience, a disconnected node typically happens wh
Hi all,
Hoping for some pointers here as I seem to have run into a gap in the NiFi
featureset: serializing binary files into a data format like Avro that will
allow me to insert them to a database.
This has never been a problem with textbased files, but my usual tricks with
extracting to attri
Hi Urmila,
I see in your screenshot that the listed scope is “Controller”. That means this
controller exists not on the Main NiFi Flow canvas, but in some Process Group,
where it’s only visible to any processors in that group and its child groups.
If you create the controller while on the main
Hi Tiago,
The content is not messed up but converted to scientific notation. This is
valid in json and various other data formats.
Apparently either the database or NiFi considers decimal(x,7) the point at
which scientific notation is more suitable.
Since you’re going to CSV format, you can con
27;ll have to experiment with it to develop more of a feel for how it can be
used.
Thanks again for taking a moment to reply with the suggestion.
Jim
On Wed, Apr 12, 2023 at 6:54 AM Isha Lamboo
mailto:isha.lam...@virtualsciences.nl>> wrote:
Hi James,
One option you can use is the QueryRecor
Hi James,
One option you can use is the QueryRecord processor. It allows you to filter
records with a SQL-like query for any combination of fields that your
downstream tools require. You can add one for each different output required
and send a copy of the main json file to each.
This approach
Hi Richard,
Does the service take more than 5 minutes before it responds? Then your socket
idle timeout might cause the connection to be closed. Try setting that to
higher than the read timeout.
Regards,
Isha
Van: Richard Beare
Verzonden: woensdag 15 maart 2023 03:54
Aan: users@nifi.apache.o
rocessing time
over throughput or not, but it is also a possible optimization opportunity to
distribute the batch in time resulting smaller peaks.
Best regards,
Bence
> On 2023. Feb 21., at 21:45, Isha Lamboo
> wrote:
>
> Hi Simon,
>
> The Hazelcast cache is being used b
way of
usage, it could help to find a more specific answer.
Best regards,
Bence Simon
On 2023. Feb 20., at 15:19, Isha Lamboo
mailto:isha.lam...@virtualsciences.nl>> wrote:
Hi all,
This morning I had to fix up a cluster of NiFi 1.18.0 servers where the primary
was constantly crashi
Hazelcast components
within NiFi?
Met vriendelijke groet,
Isha Lamboo
Data Engineer
[cid:image001.png@01D9453E.4C7ADEB0]
solution for containerized or dynamic deployments.
Hope this gives you some thoughts. There are definitely a lot of recipes and
approaches to containerized NiFi, so do some searching to find one that matches
what you're after. Almost any configuration can be done, based on your needs.
/
Hi all,
I’m looking for some perspectives from people using NiFi deployed in containers
(Docker or otherwise).
It seems to me that the NiFi architecture benefits from having a lot of compute
resources to share for all flows, especially with large batches arriving
periodically. On the other han
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
Van: David Snyder
Verzonden: maandag 16 januari 2023 18:56
Aan: users@nifi.apache.org; Isha Lamboo
Onderwerp: Re: Basic Cluster Install of Nifi
Hello Isha and all -
Isha, thanks for your guidance/feedback!
I did first separately install nifi on each of the
Hi Dave,
There is no separate installation procedure for NiFi as a cluster, it only
takes some configuration changes as described in the basic cluster setup.
You can install and configure the servers separately, then apply the same
configuration changes to each to join them to the cluster, or y
Hi Maxime,
I would try to first use a Recordwriter that doesn’t depend on newlines for
record separation, json being the easiest. Then you can use something like
ReplaceText or JoltTransformJSON (safer but more complex to configure) to
remove the newlines from the field values. Then you use Con
ta mensagem e respetivos anexos.
Caixa Geral de Depósitos, S.A. | Sede Social: Av. João XXI, 63, 1000-300 LISBOA
| Capital Social 3.844.143.735,00 € | CRCL e Contribuinte 500 960 046
From: Isha Lamboo [mailto:isha.lam...@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi
Hi Tiago,
I’ve had a similar experience with migrating the flow encryption algorithm and
in fact, some of them are still on the old one. The nifi.sh commands to update
the sensitive properties key and algorithm are very tricky to use, because they
update the nifi.properties file even if the mig
Hi Jeremy,
Have you tried the JDBC client? NiFi, being a Java application works with JDBC
drivers, not ODBC.
https://code.kx.com/q/interfaces/jdbc-client-for-kdb/
After building the driver jar or obtaining a binary somehow, you can upload it
to the nifi server in an accessible directory and cr
Hi all,
I’m preparing to upgrade some NiFi clusters from 1.12 to 1.18 (after making a
pit stop at 1.16 for sensitive dynamic properties conversion).
After upgrading a test server, all HDFS processors show as invalid with an
error that the path ‘${ADDITIONAL_CLASSPATH}’ could not be found for th
I second Tom's sentiment. It would be very much appreciated that we can go
ahead and upgrade to stop the endless high prio Excel sheets with vulnerability
scanner results showing our NiFi servers having the vulnerable library present
on disk.
The Github pull requests mentions this: "The upgrad
Hi Tiemen,
Parameter contexts and their interaction with the versioning systems are
sometimes a bit counterintuitive. The context a PG links to is included in the
versioned flow definition and it will be created on the destination NiFi when
imported if it doesn’t exist yet. If the destination c
For manual checks, NiFi provides the provenance data. You can check the last
processor in the flow to see recent flowfiles and their lineage.
For a more systematic approach you can use any number of external systems to
help you track this, depending on what’s available.
The simplest is to use Lo
Hi all,
Having recently migrated a flow.xml.gz with hundreds of RPGs from a Nifi 1.12.1
cluster to a 1.15.3 cluster, I just noticed that the status bar displays only a
count of 1 for transmitting RPGs and 0 for non-transmitting on the production
cluster, and a count of 0 / 4 o the acceptance cl
Hi Sanjeeth,
Are you performing the toolkit encryption using the same java installation
that’s running the NiFi server?
If not, you may be running into problems because of encryption limitations on
the java version on your NiFi server.
I think AES256 needs the “Unlimited Strength Encryption” po
for
that scenario.
On Wed, May 4, 2022 at 5:14 AM Isha Lamboo
mailto:isha.lam...@virtualsciences.nl>> wrote:
Hi all,
Is there a way to stop disabled Remote Process Groups from continually
contacting the remote to update the contents?
I’m migrating a cluster with hundreds of Remote P
Hi all,
Is there a way to stop disabled Remote Process Groups from continually
contacting the remote to update the contents?
I’m migrating a cluster with hundreds of Remote Process Groups and the moment I
start up the new cluster with all flows stopped/disabled, the RPGs all start
contacting t
Hi all,
I’m investigating an issue where the content disk of one of our nifi 1.12
clusters is filling up while the active usage is very low (10-12 GB active data
per node while the cluster view shows 98% full of 600 GB per node).
My first idea was that archiving wasn’t being run often enough or
Hi James,
Have you tried the FlattenJson processor? It covers exactly your scenario,
perhaps with some tweaking to field names needed.
Regards,
Isha
Van: James McMahon
Verzonden: woensdag 9 maart 2022 20:47
Aan: users@nifi.apache.org
Onderwerp: Transforming json
Hello. I am not that familiar
Hi Jens,
The behaviour you describe doesn’t seem abnormal if the round-trip time for a
request is around 20ms in you test setup. A single InvokeHTTP would process
requests one by one and that gives 1000 ms /20 ms = 50 requests per second.
Increasing concurrent tasks to send requests in parallel
eter Turcsanyi
On Fri, Feb 25, 2022 at 12:18 PM Isha Lamboo
mailto:isha.lam...@virtualsciences.nl>> wrote:
Hi all,
We have an Azure VM setup with both system-assigned and user-assigned
identities, and when we set “Use Azure Managed Identity” to true, the
Controller appears to get one o
ClientID for this purpose, only for Service Principal
authentication.
Does the ADLSCredentialsControllerService support user-assigned managed
identities at all?
Kind regards,
Isha Lamboo
n from DoS Filter
The last issue is not yet part of a released version, but the other two are
resolved in NiFi 1.15.3.
Although upgrading and migrating to connection load balancing will take some
work, it is the best path forward to address the issues you observed.
Regards,
David Handermann
On Wed,
?
Kind regards,
Isha Lamboo
that I
was able to easily replicate by creating a new process group, just adding the
distributed map cache server, and then copy & paste. Not sure why it would do
that, but we can certainly look into it. You mind filing a Jira for it?
Thanks
-Mark
On Jan 19, 2022, at 9:52 AM, Isha La
mean? Did you copy & paste
the Process Group? Did you create a template and then instantiate it? Did you
download the flow definition and then upload it again? There are a few
different mechanism that you can use for duplicating process groups.
Thanks
-Mark
On Jan 19, 2022, at 7:25 AM, Ish
for the validation/enabling
of the copy to fail.
Is this by design and are the DistributedMapCacheServer services supposed to be
created at the root level, or can I create a bug/improvement ticket for this?
Kind regards,
Isha Lamboo
Hi Emmanuel,
The filename pattern needs to match with the rolling policy. You omitted that
line in your snippet, but by default, the nifi-app looks like this:
SizeAndTimeBased means you have to use both %d (time) and %i (size). If you
only want to use one of those, change the policy to TimeBas
Hi Hendrik,
The behaviour you describe sounds like the first GenerateTableFetch processor
doesn’t find any records with a higher max_value than the initial value.
In that case, it will not emit any flowfile (unless you have Output Empty
FlowFile on Zero Results set to true). Since it hasn’t foun
Hi Stefan,
I see a number of things that are different from our working Kerberos solution,
but one thing in particular: we used a certificate-based user for our initial
admin identity, with the client certificate in the truststore and the DN of
that user in the authorizers file, something like
vriendelijke groet,
Isha Lamboo
Data Engineer
+31 (0)6 20 50 15 91
[cid:image001.png@01D7CB63.3DCBB480]
isha.lam...@virtualsciences.nl<mailto:isha.lam...@virtualsciences.nl>
Edisonbaan 15
3439 MN Nieuwegein
www.virtualsciences.nl<http://www.virtualsciences.nl/>
www.conclu
hanks,
Bryan
On Tue, Sep 28, 2021 at 4:27 AM Marel, J. van der (Jasper)
wrote:
>
> Hi Isha,
>
> We are using nipyapi here to automate the work. If you are interested you can
> contact me and we can have a chat.
>
> With kind regards,
>
> Jasper van der Marel
>
Hi all,
I’m looking into migrating a template-based devops pipeline to NiFi
Registry-based.
The current setup has finished flows being exported to templates and checked
into a git repository branch where they can be merged into a release branch (in
line with changes in the source or target app
Hi Bill,
In a cluster all nodes should indeed apply the changes to the users.xml and
authorizations.xml.
In fact, the nodes should refuse to reconnect if they detect a difference in
these files.
First thing to verify is of course whether the two nodes successfully form a
cluster.
Do you see “2
,
Pierre
Le mar. 20 juil. 2021 à 13:32, Isha Lamboo
mailto:isha.lam...@virtualsciences.nl>> a écrit
:
Hi all,
After working with NiFi (1.9 and 1.12) clusters for some time I’m setting up a
single node and I’m completely stumped on where to find the system stats that
are so nicely presented
.
I am overlooking something obvious or is the single node lacking JVM, storage,
etc info in the GUI?
Regards
Isha Lamboo
to be both
of them at once?
As a side note, I haven’t found a compatibility matrix of NiFi and Zookeeper
versions, is this documented somewhere or do I just go with the embedded
Zookeeper version number of the NiFi release?
Regards,
Isha Lamboo
56 matches
Mail list logo
