RE: Penalty feature of Processor (Disable)

Hi Edward, Thank you for helping. You can find the information which you want: * LookupAttribute processor uses SimpleDatabaseLookupService, bundle of which is “org.apache.nifi - nifi-lookup-services-nar” * SimpleDatabaseLookupService uses DBCPConnectionPool, bundle of which is “org.apache.nif

Re: Penalty feature of Processor (Disable)

Hi Bilal, Thanks for confirming, it looks like my hunch was correct, and there is a discrepancy in the lookup service code itself. In DatabaseRecordLookupService we catch an IOException and return a lookup failure like is done for an SQLExeception. This isn't done in the SimpleDatabaseLookupServi

Re: Penalty feature of Processor (Disable)

I'd consider changing DatabaseRecordLookupService.. the way it is currently implemented there is no way to tell the difference between a retryable exception like IOException vs a non-retryable exception like SQLException because the columns are wrong and will never work. We could introduce a new Re

Nifi secured cluster can't send heartbeat between nodes

Hi list, I'm facing a weird problem I can't resolve or even understand with my secured nifi cluster. Below is the situation. We have setup a secured nifi cluster with 3 nodes, say node1,node2 and node3. For each of theses nodes, we've manually created a SSL certificate signing request (CSR) (usi

Re: Nifi secured cluster can't send heartbeat between nodes

Hi I don’t know if this will help. But the certificate used by nifi needs be both a server auth and client auth. Normally certificates are only one of them. The nifi certificate use the server auth for the web ui and when other servers connect to it. It is using the client auth when nifi talks

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

Jens, For a bit of background here, the reason that Joe and I have expressed interest in NFS file systems is that the way the protocol works, it is allowed to receive packets/chunks of the file out-of-order. So, what happens is let’s say a 1 MB file is being written. The first 500 KB are receiv

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

And the actual script: import org.apache.nifi.flowfile.FlowFile import java.util.stream.Collectors Map getPreviousHistogram(final FlowFile flowFile) { final Map histogram = flowFile.getAttributes().entrySet().stream() .filter({ entry -> entry.getKey().startsWith("histogram.") })

RE: Nifi secured cluster can't send heartbeat between nodes

Hi Jens, Thanks for your quick reply. After double checking certificate extended key usage, it looks they are configured as required: openssl x509 –in full-nifi-node1.pem –noout –text … X509v3 Extended Key Usage: TLS Web Client Authentication, TLS Web Server Authentication … Emm

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

Hi Mark Thanks for the clarification. I will implement the script when I return to the office at Monday next week ( November 1st). I don’t use NFS, but ext4. But I will implement the script so we can check if it’s the case here. But I think the issue might be after the processors writing conte

Re: Penalty feature of Processor (Disable)

Seems like a nice improvement. I would lean towards introducing the RetryableLookupFailureException. At the same time, I think we should add catch blocks for more specific subclasses of SQLException, such as SQLTransientException, which would also trigger a RetryableLookupFailureException inste

Re: CryptographicHashContent calculates 2 differents sha256 hashes on the same content

Jens, I don't quite follow the EXT4 usage on top of VMFS but the point here is you'll ultimately need to truly understand your underlying storage system and what sorts of guarantees it is giving you. If linux/the jvm/nifi think it has a typical EXT4 type block storage system to work with it can o

Large amounts of data in cluster state

Hi all, I have a question that some of you must have tackled already. On a NiFi cluster (still 1.9 at the moment) that is normally very stable, the users sometimes trigger peak loads that cause disconnections or other issues either in NiFi itself or the external zookeeper cluster. The clear exa

Re: Large amounts of data in cluster state

Hi Isha, ListHDFS does not store the listing of files that it’s found in its state. Doing so would cause a lot of problems. Instead, it only stores the timestamps of the latest file that it has found and the timestamp of the latest file that it has listed/sent out. If peak loads are triggering

Re: Unable to evolve Schema with HWXRegistry

Thanks Christian. Understood, I'll try to have a look into the code and get back to you. Le mar. 26 oct. 2021 à 19:16, Weiss, Christian a écrit : > Hi Pierre, > > the Avro-Files, which we write to a Kafka Topic in the next step, needs to > be compatible with the Confluent Avro-Format, because we