Thanks, Mark! Appreciate you retaking another look at this!
From: Mark Payne [mailto:marka...@hotmail.com]
Sent: 26 January 2022 00:21
To: users
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON
Thanks Nathan, the template is helpful. I put up a PR that I believe addresses
the issues.
Tha
Hey Mark,
Apologies if this is more of a dev mailing list issue. Happy to go across to it
if needs be.
I've been testing the fix you've implemented for this and noticed a slight
issue when the default value should be null. It took a little while to work out
what was going on, but I think I've
Hey Mark,
You beat me to the fix, Thanks for taking your time to look at it! I appreciate
that one.
Cheers!
Nathan
From: Mark Payne
Sent: Wednesday, January 19, 2022 6:54:23 PM
To: users@nifi.apache.org
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON
T
Hi There,
We've recently upgraded to NiFi V1.15.2 from V1.12.1 and have noticed a
difference between the JSON Tree Reader Controller Service, which is a slight
annoyance.
When we transform our records, we have some fields that will exist or won't
after the transformation, depending on whether
Hi Joe, Russ,
Many thanks for that information.
Kind Regards,
Nathan
-Original Message-
From: Joe Witt [mailto:joe.w...@gmail.com]
Sent: 13 April 2021 18:16
To: users@nifi.apache.org
Subject: Re: NiFi 1.11.4 Custom Processor Development
Nathan
Nothing has occurred that should result
Hi Joe,
Thanks for that information.
Will there be any issue in running a Custom NAR file built against 1.9.2 on a
different versioned cluster. We haven’t experienced anything yet, but would
hate to spend ages in debugging an issue that is the result of this!
Kind Regards,
Nathan
Hi There,
I am in the process of upgrading the version of the nifi-nar-bundle from 1.9.2
to 1.12.1 however I’ve hit a unexpected issue.
We develop on an offline platform, with Maven Central Mirrored using a Nexus
Repository Manager, with all dependencies delivered through this mechanism.
Today
Hey Mark,
The Validate Record Processor was what I was looking for!
As we are using the _0_10 version of the produce record processor, we don’t
seem to have any of the transactional properties listed? I’m guessing this is
on the newer Kafka API processors?
Kind regards,
Nathan
From: Mark Pay
Hi All,
I've had a look at various processors and the documentation and can't seem to
find any information. So I'm hoping someone may have an idea or point me in the
right direction.
We consume from various data sources, normalise the data and then produce the
records to Kafka in Avro format,
Hi Asmath,
In terms of performance tuning, I’ve found a lot that needs to be considered.
I’ve done tuning it’s mainly been caused large queues building due to the
read-write queues on the content repo disks being very high. I’ve also had
previous threads on the JOLT default transformation, whic
Hi All,
I've managed to get this working as a proof of concept using the Lookup Record
Processor, Putt Distributed Cache Map Processor and the Distributed Cache Map
Services.
One question I have on the Lookup Record processor is when the routing strategy
is set to: 'Route to 'matched' or 'unma
Hi All,
I've got a case where I need to join two or more separate flow files, based on
one or more 's ID within the flow (Multi Record flow file, each record with a
unique ID). The content of the second flow file could also contain one or more
records each again with a matching ID to the record
Hi Joe,
Thanks for getting back to me so quickly.
Our disk setup is as follows:
Path
Storage Type
Format
Capacity
Content
/
100GB OS SSD
ext4
89.9GB
OS, NiFi install, Logs
/data/1/
2 x 4TB SAS Hard Drives in RAID 1
ext4
3.7TB
Database and Flowfile Repos
/data/2/
8 x 4TB SAS Hard Drives in RAID 10
Hi All,
We've got a NiFi 3 Node Cluster running on 3 x 40 CPU, 256GB RAM (32G Java
Heap) servers. However, we have only been able to achieve a consumption of
~9.48GB Consumption Compressed (38.53GB Uncompressed) over 5 minutes, with a
production rate of ~16.84GB out of the cluster over 5 mins.
Hi Mark,
It makes complete sense now, thanks for clearing it up!
Nathan
From: Mark Payne [mailto:marka...@hotmail.com]
Sent: 21 May 2020 15:35
To: users@nifi.apache.org
Subject: Re: State Management in a Cluster
Nathan,
Yes, both are needed. Some processors will store local state while others
Hi All,
Apologies if this an obvious question again, but I've had a search through the
administration guide that's made me more confused!
At the moment in our NiFi cluster deployment, we are configuring the Zookpeer
provider in the state-management.xml and leaving the local provider as it is
(
Hi Bryan,
Would the same also work by using two separate git persistence providers
pointed at two different git repositories? We have a remote production instance
with no remote access (effectively airgapped).
Our plan is when we deploy to production, provide a full exported copy of the
git re
Hi Andy,
Thanks for getting back to me.
I have raised NIFI-7376 on Jira as an improvement. Hopefully that’s enough
detail, if not please let me know as I can add more.
I do agree with the versioning of Avro, it wasn’t something I was expecting to
find in a patch release.
Nathan
From: Andy Lo
Hi There,
Apologies if this has been asked before!
One of our Inputs into our NIFI cluster is looking to start using Avro Single
Object Encoding on their messages. This was added V1.8.2 of the Avro Schema,
but from what I can see NIFI is using V1.8.1 (looking at the pom.xml on GitHub,
https://
19 matches
Mail list logo
