Re: Caused by: java.lang.OutOfMemoryError: Java heap space

It means the processor always running ? How does the impact one on the other ? Thanks a lot Minh Envoyé: mercredi 6 novembre 2024 à 16:13 De: "Mark Payne" À: "users@nifi.apache.org" Objet: Re: Caused by: java.lang.OutOfMemoryError: Java heap space OK so the decompress should

Re: Caused by: java.lang.OutOfMemoryError: Java heap space

à 15:24 De: "Mark Payne" À: "users@nifi.apache.org" Objet: Re: Caused by: java.lang.OutOfMemoryError: Java heap space Hi Minh, It is possible that the heap is being exhausted by EvaluateJsonPath if you are using it to add large JSON chunks as attributes. For example, if you’

Re: Caused by: java.lang.OutOfMemoryError: Java heap space

Hi Minh, It is possible that the heap is being exhausted by EvaluateJsonPath if you are using it to add large JSON chunks as attributes. For example, if you’re creating an attribute from `$.` to put the entire JSON contents into attributes. Generally, attributes should be kept pretty small. Ot

Re: Flowfile transfer without relationship

Hey ML, There’s not really a straight-forward way to do what you’re asking. You’d essentially need to create Output Ports to allow the FlowFile to move up the stack of Process Groups and then into the neighboring Process Group (and potentially back down that other stack). The alternative, as yo

Re: how to retrieve a record reader from a controller service using python processor API?

Hey Chris, If you haven’t already, I’d recommend taking a look at the Python Developer’s Guide [1], particularly the section on RecordTransforms. In short, you should not need a Record Reader in Python. Instead of FlowFileTransform, you would extend from RecordTransform. Then, your transform met

Re: Received SHUTDOWN request from Bootstrap

Hey Minh, Try looking in the nifi-app.log, not the nifi-bootstrap.log Thanks -Mark On Jul 8, 2024, at 10:41 AM, e-soci...@gmx.fr wrote: The only except INFO, there are only this line in bootstrap.log [root@ope-nifi-dfp-vm-19 nifi]# grep -v INFO nifi-bootstrap.log 2024-07-08 07:13:39,883 WARN

Re: Error on first launch (probable newbie misconfiguration)

xpressed through method 'securityFilterChain' >> parameter 2: Error creating bean with name 'org.apache.nifi.web.security.con >> figuration.JwtAuthenticationSecurityConfiguration': Unsatisfied dependency >> expressed through constructor parameter 2: Error creating bean wi

Re: Why are my journal files so large on node 2 of cluster?

o the how long the attribute lives and will monitor over the next couple of days, but likely we will need to cache the contents somewhere and retrieve them later unless someone knows of a better solution here. Thanks for the guidance Dave On Fri, Mar 8, 2024 at 7:05 AM Mark Payne mailto:

Re: Why are my journal files so large on node 2 of cluster?

Dave, When you say that the journal files are huge, I presume you mean the FlowFile repository? There are generally 4 things that can cause this: - OutOfMemoryError causing the FlowFile repo not to properly checkpoint - Out of Disk Space causing the FlowFile repo not to properly checkpoint - Out

Re: Error on first launch (probable newbie misconfiguration)

Hey there Franck, Can you provide the full error message with stack trace that gets printed? Thanks -Mark On Mar 6, 2024, at 1:26 PM, Franck Routier via users wrote: Hi, I'm a first time wanabe user of Nifi, and I followed the installation guide, until the point when I try to launch Nifi:

Re: InvokeHttp - Provenance Events

Greg, Yes, the Relationship is generally only populated for ROUTE events, such as RouteOnAttribute. Thanks -Mark > On Feb 26, 2024, at 11:34 AM, Gregory Foreman > wrote: > > Hello: > > I am having an issue with the InvokeHttp processor provenance events. No > Relationship field is popula

Re: expression language/groovy date handling

n that I'm using Date functions in the expression language I hoped that the generic avro writer would correctly infer the schema so that I didn't have to explicitly provide one. Is this approach the right one? Is there a way I can isolate the expectation of a date component inside

Re: Can we access Queued Duration as an attribute?

Jim, You can actually reference “lastQueueDate” in Expression Language. It is formatted as number of milliseconds since epoch. So you might have a RouteOnAttribute that has a property named “old” with a value of: ${lastQueueDate:lt( ${now():minus(1)} )} So any FlowFile that has been queued

Re: expression language/groovy date handling

Hey Richard, I think you’d need to explain more about what you’re doing in your groovy script. What processor are you using? What’s the script doing? Is it parsing Avro data? On Jan 29, 2024, at 12:26 AM, Richard Beare wrote: Anyone able to offer assistance with this? I think my problem rela

Re: Finding slow down in processing

tprint and any variability regarding data volumes... 20 timer-driven threads could be exhausted pretty quickly. This might cause not only your "stress-test" to slow down but your other flows as well as components might be waiting for available threads to do their jobs. Thanks, Phi

Re: Finding slow down in processing

Aaron, Interestingly, up to version 1.21 of NiFi, if you increase the size of the thread pool, it increased immediately. But if you decreased the size of the thread pool, the decrease didn’t take effect until you restart NiFi. So that’s probably why you’re seeing the behavior you are. Even thou

Re: Finding slow down in processing

Aaron, What version of NiFi are you running? One thing that you can look into if you’re running a pretty recent version, (though the user friendliness is not great) is to update nifi.properties and set the “nifi.performance.tracking.percentage” property from 0 to something like 5 or 10. Restar

Re: Hardware requirement for NIFI instance

Thanks for following up. That actually makes sense. I don’t think Output Batch Size will play a very big role here. But Fetch Size, if I understand correctly, is essentially telling the JDBC Driver “Here’s how many rows you should pull back at once.” And so it’s going to buffer all of those rows

Re: Nifi - Content-repo on AWS-EBS volumes

e benefit to having multiple content repos on multiple EBS volumes? > > Thanks, > Greg > > > >> On Dec 11, 2023, at 8:50 PM, Mark Payne wrote: >> >> Hey Phil, >> >> NiFi will not spread the content of a single file over multiple partitions. >>

Re: ConsumeKafka to PublishKafka doesn't keep the order of the messages in the destination topic

the combination of FIFO and setting the back pressure to 10k, but it didn't preserve the order. Thanks Edi On Wed, Dec 13, 2023 at 3:47 PM Mark Payne mailto:marka...@hotmail.com>> wrote: Hey Edi, By default, nifi doesn’t preserve ordering but you can have it do so by updating the c

Re: Nifi - Content-repo on AWS-EBS volumes

Hey Phil, NiFi will not spread the content of a single file over multiple partitions. It will write the content of FlowFile 1 to content repo 1, then write the next FlowFile to repo 2, etc. so it does round-robin but does not spread a single FlowFile across multiple repos. Thanks -Mark Sent f

Re: Configuring ExecuteStreamCommand on jar flowfiles

Jim, UnpackContent does just that. It does not write to an external directory. Thanks Mark Sent from my iPhone On Dec 3, 2023, at 5:18 PM, James McMahon wrote:  UnpackContent examples seem to require that I output the results of the unpack to a directory outside of the nifi flow. Is it poss

Re: ListSFTP Processor CRON doesn't start

2023 à 16:11 De: "Mark Payne" mailto:marka...@hotmail.com>> À: "users@nifi.apache.org<mailto:users@nifi.apache.org>" mailto:users@nifi.apache.org>> Objet: Re: ListSFTP Processor CRON doesn't start Hi Minh, No - you can configure logging for any Java cla

Re: ListSFTP Processor CRON doesn't start

ot the documentation about exhaustive list about logger we got have in NIFI ? Regards Minh Envoyé: mardi 14 novembre 2023 à 15:25 De: "Mark Payne" mailto:marka...@hotmail.com>> À: "users" mailto:users@nifi.apache.org>> Objet: Re: ListSFTP Processor CRON doesn't start Hi Q

Re: ListSFTP Processor CRON doesn't start

Hi Quentin, What is the CRON schedule that you configured? What version of NiFi are you running? You’ll not see any debug related logs for that Processor by changing its log level, as the Processor is not responsible for scheduling itself. But you can enable DEBUG level logs for org.apache.ni

Re: Expression Language does not work within QueryNifiReportingTask

You’re right, Dogukan. It looks like the “SQL Query” property is documented as supporting Expression Language, but the EL is never evaluated. I filed a JIRA [1] for the issue. Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-12314 On Nov 3, 2023, at 5:09 AM, Doğukan Levendoğlu | O

Re: NiFi hanging during large sql query

Thanks for sharing the solution Mike. Is there something we need to update in nifi to prevent this from biting others? Thanks Mark Sent from my iPhone On Sep 2, 2023, at 9:48 AM, Joe Witt wrote:  Nice. Gald you found it. On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen mailto:mikerthom...@gmai

Re: SOLVED - Re: Putemail - subject contains utf-8 chars - the result is lot of question marks

o not think it does matter. Thank you! István [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.1/org.apache.nifi.processors.standard.PutEmail/index.html 2023. 08. 22, kedd keltezéssel 13.29-kor Mark Payne ezt írta: Hey István, The PutEmail processo

Re: Putemail - subject contains utf-8 chars - the result is lot of question marks

Hey István, The PutEmail processor is using Jakarta Mail as the underlying library. Based on some googling, I found a Stack Overflow issue [1] with similar concerns. The recommendation there is to set a system property to specify mail.mime.charset. Can you try doing that? To do so, you’d update

Re: UI SocketTimeoutException - heavy IO

oe On 7/12/2023 11:27 AM, Mark Payne wrote: Joe, How many FlowFiles are you processing here? Let’s say, per second? How many processors are in those flows? Is the FlowFile Repo a spinning disk, SSD, or NAS? You said you’re using ExtractText to pull 10 KB into an attribute. I presume you’re the

Re: UI SocketTimeoutException - heavy IO

STORAGE, one of the nodes shows 3.74TBytes of usage, but it's actually ~150G on disk. The other nodes are correct. Ideas on what to debug? Thank you! -Joe (NiFi 1.18) On 3/22/2023 12:49 PM, Mark Payne wrote: OK. So changing the checkpoint internal to 300 seconds might help reduce IO a bit.

Re: BufferedReader best option to search through large flowfiles?

Jim, Take a look at RouteText. Thanks -Mark > On Jun 5, 2023, at 8:09 AM, James McMahon wrote: > > Hello. I have a requirement to scan for multiple regex patterns in very large > flowfiles. Given that my flowfiles can be very large, I think my best > approach is to employ an ExecuteGroovySc

Re: [EMBEDDED ZOOKEEPER] Encrypt password

Quentin, I don’t believe there’s any way to encrypt the properties in zookeeper.properties. I will note, though, that the embedded zookeeper should not be used for production use. It’s convenient for testing, proofs of concept, etc. But for any production deployment an external ZooKeeper should

Re: UI SocketTimeoutException - heavy IO

add more nifi nodes. -Joe On 3/22/2023 11:08 AM, Mark Payne wrote: Joe, The errors noted are indicating that NiFi cannot communicate with registry. Either the registry is offline, NiFi’s Registry Client is not configured properly, there’s a firewall in the way, etc. A FlowFile repo of 35 GB is rat

Re: UI SocketTimeoutException - heavy IO

t retrieve version 1 of flow with identifier 7a8c82be-1707-4e7d-a5e7-bb3825e0a38f in bucket 736a8f4b-19be-4c01-b2c3-901d9538c5ef due to: Connection refused (Connection refused) A clue? -joe On 3/22/2023 10:49 AM, Mark Payne wrote: Joe, 1.8 million FlowFiles is not a concern. But when you say

Re: UI SocketTimeoutException - heavy IO

10:20 AM -0400, Joe Obernberger <mailto:joseph.obernber...@gmail.com>, wrote: atop shows the disk as being all red with IO - 100% utilization. There are a lot of flowfiles currently trying to run through, but I can't monitor it becauseUI wont' load. -Joe On 3/22/2023 10:1

Re: UI SocketTimeoutException - heavy IO

Joe, I’d recommend taking a look at garbage collection. It is far more likely the culprit than disk I/O. Thanks -Mark > On Mar 22, 2023, at 10:12 AM, Joe Obernberger > wrote: > > I'm getting "java.net.SocketTimeoutException: timeout" from the user > interface of NiFi when load is heavy. Th

Re: Processor with cron scheduling in middle of flow

gards Jens M. Kofoed Den 22. feb. 2023 kl. 15.24 skrev Mark Payne mailto:marka...@hotmail.com>>: Interesting. Thanks for that feedback Harald. It might make sense to be more surgical about this, disabling it for MergeContent, for example, instead of all interflow processors. Thanks -Mark

Re: Processor with cron scheduling in middle of flow

ing able to > tell the processor in the middle of the flow when not to run. > > -Ursprüngliche Nachricht- > Von: Mark Payne > Gesendet: Dienstag, 21. Februar 2023 21:37 > An: users@nifi.apache.org; John McGinn > Betreff: Re: Processor with cron scheduling in middle of f

Re: Processor with cron scheduling in middle of flow

John, You should not be using CRON driven for any processors in the middle of a flow. In fact, we really should probably just disable that all together. In fact, it’s exceedingly rare that you’d want anything other than Timer-Driven with a Run Schedule of 0 sec. MergeContent will not create any

Re: How to cherry pick a specific line from a flowfile?

James, Have a look at the RouteText processor. I wrote a blog post recently on using it: https://medium.com/cloudera-inc/building-an-effective-nifi-flow-routetext-5068a3b4efb3 Thanks Mark Sent from my iPhone On Feb 9, 2023, at 8:06 PM, James McMahon wrote:  My version of nifi does not have

Re: DistributedMapCacheClient corrupting dataflow

Hey Paul, Looks like you’re running into https://issues.apache.org/jira/browse/NIFI-10246, which was addressed in 1.17.0 When NiFi gets to that point, the service is either in an ENABLING state (even though it says it’s enabled, it may just be enabling) or it’s decided that it needs to disable

Re: Issue with removal and re-add of a cluster node

David, I think you’re running into https://issues.apache.org/jira/browse/NIFI-10453, which was fixed in 1.19. It results in the "Cannot set AnnotationData while processor is running” error. Recommend upgrading to 1.19. In the meantime, though, you should be okay to shutdown node 3, delete the

Re: Expected mergerecord performance

rease the maximum bin age to 30 seconds the size and record count is higher - 12 to 15. Nothing like the behaviour with synthetic data, where the 100 record flowfiles are created almost instantly. Joins are always due to bin age. Can the problem relate to the structure of the avro files? Any w

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

I’ve not followed everything goin on in this thread. But just to offer a bit of clarification: If the flow.json.gz exists then the flow.xml.gz is ignored. But if the flow.json.gz file is not found, it will automatically fall back to the flow.xml.gz So if you were to remove/rename the flow.json.

Re: Expected mergerecord performance

tells you about the reason for merging the data, and we can go from there. Thanks! -Mark On Dec 3, 2022, at 4:38 PM, Mark Payne wrote: Richard, I think just the flow structure shoudl be sufficient. Thanks -Mark On Dec 3, 2022, at 4:32 PM, Richard Beare wrote: Thanks for responding, I re-t

Re: Expected mergerecord performance

py to demonstrate via video conference too. Thanks On Sun, Dec 4, 2022 at 1:37 AM Mark Payne mailto:marka...@hotmail.com>> wrote: Hi Richard, Can you try increasing the Maximum Number of Bins? I think there was an issue that was recently addressed in which the merge processors had an iss

Re: Expected mergerecord performance

Hi Richard, Can you try increasing the Maximum Number of Bins? I think there was an issue that was recently addressed in which the merge processors had an issue when Max Number of Bins = 1. If you still see the same issue, please provide a copy of the flow that can be used to replicate the iss

Re: Empty Queue - but UI shows messages

Joe, The problem is that NiFi is not able to load balance the data. What errors do you see in the logs around load balancing? Thanks -Mark On Nov 22, 2022, at 9:15 AM, Joe Obernberger mailto:joseph.obernber...@gmail.com>> wrote: Hi Joe - this is happening only in queues that do round robin l

Re: Nifi unable to list/empty queue.

Hello, For access to data, because it’s considered more sensitive than the flow definition, both the user and the nifi node accessing the data must be granted permissions to view and modify data. Did you give the nodes permissions to view and modify data? Also moving this from security@ to use

Re: Empty Queue - but UI shows messages

Joe, Given that the icon shows the data is still being load balanced, I am guessing that the system is having trouble load balancing the data. Can you check the logs to see if there are errors related to load balancing? Thanks -Mark On Nov 9, 2022, at 12:04 PM, Joe Obernberger mailto:joseph.

Re: Empty Queue - but UI shows messages

Just to add to what Joe said, in the screenshot the queue that you’re listing shows that blue load balancing icon, meaning that it is actively load balancing. The listing will not include any FlowFiles that are queued up to move to another node. So on a connection like that where it’s actively l

Re: DistributedMapCacheServer persistent directory - Cluster wide same values after primary node changes

I wouldn’t say that it should never be used in production. In fact, it’s used quite heavily in production. But it does have some limitations. Where those limitations are acceptable, it’s still very reasonable to use. Redis offers a lot on top, but it comes with complexity also, having to manage

Re: NiFi 1.18.0 Sensitive Property broken after Upgrade

Hey Josef, I’m sorry about the trouble. It looks like this issue was reported here [1]. We are looking into a fix for it. Fortunately, if you don’t want to wait for the fix there is a workaround available. The work around is to follow these steps: 1. Instead of jumping straight to 1.18, update

Re: Content Repository Performance

Joe, What kind of hardware are you running on? Thanks Mark Sent from my iPhone > On Oct 5, 2022, at 9:23 AM, Joe Obernberger > wrote: > > Hi all - I'm using NiFi 1.16.3 in a 3 node cluster with content repository > settings: > > # Content Repository > nifi.content.repository.implementatio

Re: Can ExecuteStreamCommand do this?

It’s been a while. But if I remember correctly, the reason that NiFi does not natively support 7-zip format is that with 7-zip, the dictionary is written at the end of the file. So when data is compressed, the dictionary is built up during compression and written at the end. This makes sense fro

Re: Trouble configuring logging

This is because of how NiFi is run. When you startup nifi (bin/nifi.sh start) it doesn’t directly start the NiFi process. Instead, it starts a different processor, which is called RunNiFi. This RunNiFi process is responsible for doing a lot of things, including monitoring the nifi process and if

Re: Need help to merge all records in cluster into one flowfile

erge. But when we create more flows the records and filesize will increase and we will be back to the problem that not all files will be merged into one. kind regards Jens Den tir. 30. aug. 2022 kl. 15.40 skrev Mark Payne mailto:marka...@hotmail.com>>: Hey Jens, My recommendation is to t

Re: Need help to merge all records in cluster into one flowfile

Hey Jens, My recommendation is to take a look at the data provenance for MergeRecord (i.e., right-click on the Processor and go to Data Provenance.) Click the little ‘i’ icon on the left for one of the JOIN events. There, it will show a “Details” field, which will tell you why it merged the dat

Re: Crash on startup due to Output port issue

uptive. Thanks On Mon, Aug 1, 2022 at 5:41 AM Mark Payne mailto:marka...@hotmail.com>> wrote: Benji, I would recommend you try to remove (or rename) the flow.json.gz - but not the flow.xml.gz. See if that makes any difference. Thanks -Mark Sent from my iPhone On Jul 31, 2022, at 11

Re: Crash on startup due to Output port issue

Benji, I would recommend you try to remove (or rename) the flow.json.gz - but not the flow.xml.gz. See if that makes any difference. Thanks -Mark Sent from my iPhone On Jul 31, 2022, at 11:35 PM, BeNJ wrote:  Also please see the attached nifi.properties Thanks, Benji On Sun, Jul 31, 2022

Re: retry issue

he original to a routeonattribute > processor that inspects the execution.status code and traps original files > that are errors is an option. Are there any options besides using the second > processor? > > Thanks, > Greg > >> On Jul 20, 2022, at 9:22 AM, Mark Payn

Re: retry issue

Greg, You wouldn’t want to retry the “original” relationship. The processor has 3 relationships: original, output stream, and nonzero status. It should always send the incoming FlowFile to original. So if you retry that relationship you’ll always retry the flowfile, regardless of whether it was

Re: Search only within current Process Group

Jim, Take a look at https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#filters Thanks -Mark On Jul 13, 2022, at 8:55 AM, James McMahon mailto:jsmcmah...@gmail.com>> wrote: Within the UI there is a Search frame in the upper right, and this seems to result in a search globally agains

Re: flow.xml.gz > flow.json.gz migration

Hi Greg, WIth 1.16, NiFi expects to write out both the xml version AND the json version. While we are in the process of making the json version the canonical version of representing a full flow and a flow snippet, the change in 1.16 was very large and critical to get right. So, just in case the

Re: How to configure wait process along with ListSFTP process

Ben, The Wait processor is not intended for this purpose. It’s intended to be used along with the Notify processor. So the processor will block FlowFiles going forward until the Notify processor indicates that the FlowFile should be released. Take a look at ListSFTP’s “Minimum File Age” proper

Re: Fork-Join Enrichment, Input Dependent Behaviour

Hey Steve, Thanks for reporting this. I filed a Jira [1] for the issue. I have a Pull Request up for it, so it should be fixed in the next release. Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-10169 On Jun 24, 2022, at 1:23 PM, stephen.hindma...@bt.com

Re: Requesting Apache NiFi Help/Guidelines

:42 PM Mark Payne mailto:marka...@hotmail.com>> wrote: Ben, So in the message there, you can see that it found 11 files in the /tmp directory, but none of those files matched the filter. So you’ll get no output. What do you have set for the Minimum/Maximum File Age and for the Minimum/Maxim

Re: Nifi custom processor to consume two flow files at once.

Vibhath, That’s correct, all of the data is received as if through a single connection. There’s no notion of named inputs. Unfortunately, that makes this a pattern that’s a bit more difficult to implement than I’d like. Generally, the way this is handled would be to add an attribute to the Flow

Re: Requesting Apache NiFi Help/Guidelines

Ben, ListSFTP -> FetchSFTP is how you’d want to get started. You’d then want to connect that to a PutSFTP to send to Server B and another PutSFTP to send to Server C. You said that it is not working as expected. What does that mean? Are you seeing errors? Not seeing the data show up? Thanks -

Re: Issue with registry and Port error

Agreed. Would definitely recommend 1.16 if possible. Otherwise, you might try stopping the process group, then change version, and then restart the process group. Thanks -Mark On Jun 16, 2022, at 12:50 PM, Joe Witt mailto:joe.w...@gmail.com>> wrote: Dave The issue you flag might be related.

Re: Setting an attribute in EvaluateXPath

James, XPath is namespace-specific. I.e., /root is not the same as /{urn:exo:/root}root or whatever syntax is appropriate (i haven’t used XPath in about a decade so i can’t remember the syntax off the top of my head). But you should be able to use * to wildcard the namespace, I believe. So some

Re: Load balancing just stopped working in NIFI 1.16.1

Jens, So that would tell us that the hostname or the port is wrong, or that NiFi is not running/listening for load balanced connections. I would recommend the following: - Check nifi.properties on all nodes to make sure that the nifi.cluster.load.balance.host property is set to the node’s host

Re: Null Record in ConsumeKafkaRecord

Hi Prasanth, Take a look at the Record Writer that you’re using with ConsumeKafkaRecord. There’s a property name “Suppress Null Values.” You’ll want to set that to “Suppress Missing Values.” That should give you what you’re looking for. Thanks -Mark On May 19, 2022, at 7:53 AM, Prasanth M Sas

Re: Referencing an attribute as the final ifelse/ifelse conditional value

James, If I’m reading it right, it looks like you’re using a comma between the )} and the ${currentAttribute} ? Thanks -Mark On May 6, 2022, at 12:47 PM, James McMahon mailto:jsmcmah...@gmail.com>> wrote: UpdateAttribute is flagging the final } in my expression. How can I make the final v

Re: Issue with version controlled inner PG and parameter contexts

Hei Reinhard, Thanks for reaching out. I believe you’re running into NIFI-9874 [1]. This has been addressed, but hasn’t been released yet. I think a 1.16.1 release will be coming soon, so you should be in good shape once that gets released. Thanks -Mark [1] https://issues.apache.org/jira/brow

Re: NiFi web UI not responsive under load

thread":"main","message":"o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 2858730232217 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size"} {"level":"INFO&quo

Re: NiFi web UI not responsive under load

Eric, I certainly agree with what Joe said. I would also recommend checking in nifi.properties if you have a value set for: nifi.monitor.long.running.task.schedule I recommend setting that to “ hours” In 1.14.0, we introduced the notion of a Long-Running Task Monitor. It’s generally very

Re: Unexpected Behaviour In LookupRecord With "Route To success" Strategy

Yeah, I just created one [1]. Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-9903 On Apr 11, 2022, at 10:16 AM, stephen.hindma...@bt.com<mailto:stephen.hindma...@bt.com> wrote: Thanks Mark, Is there a JIRA open for this? Regards Steve From: Mark Payne mailto

Re: Unexpected Behaviour In LookupRecord With "Route To success" Strategy

Steve, Thanks for the note. Ironically, I ran into this issue just yesterday. Unfortunately, it’s a bug that will have to be addressed. In the meantime, if you define the schema for your Record Writer explicitly, it should work as expected. The issue comes down to the fact that the first record

Re: Cannot Delete Process Group Because Source of Connection is "Running"

processors within a process group? Would it be the one that provides the data for this portion of the UI? Thanks, Eric On Tue, Apr 5, 2022 at 3:29 PM Mark Payne mailto:marka...@hotmail.com>> wrote: Eric, Once a processor state transitions to stopped, it may still have active threads that

Re: Cannot Delete Process Group Because Source of Connection is "Running"

, 2022 at 3:29 PM Mark Payne mailto:marka...@hotmail.com>> wrote: Eric, Once a processor state transitions to stopped, it may still have active threads that haven’t completed yet. You’ll need to wait until the processor is stopped and the active threads on the processor reach 0. Thanks -Ma

Re: Cannot Delete Process Group Because Source of Connection is "Running"

Eric, Once a processor state transitions to stopped, it may still have active threads that haven’t completed yet. You’ll need to wait until the processor is stopped and the active threads on the processor reach 0. Thanks -Mark On Apr 5, 2022, at 6:25 PM, Eric Secules mailto:esecu...@gmail.com

Re: VolatileContentRepository removal

Hey Matthieu, If using a RAM disk, I would recommend trying 1.16 and also setting “nifi.content.claim.max.appendable.size” in nifi.properties to “1 byte”. This will help to ensure that you’re eliminating data from the repository as quickly as possible. Additionally, I would recommend that you c

Re: Insufficient Permissions for Expression Language

Hi Stanley, That error message is not coming from NiFi. I would guess that you have some sort of load balancer, proxy, etc. between you and the NiFi instance? WOuld recommend looking at that to see if you can determine what’s happening there. Thanks -Mark On Mar 30, 2022, at 3:32 PM, Martin,

Re: Where are my processes?

Jean-Sebastian, I’d recommend grabbing a thread dump (bin/nifi.sh dump dump1.txt) and check the thread dump to see what the processors are doing. Would also grep logs for “waiting for” to see if it shows anything. Thanks -Mark Sent from my iPhone On Mar 28, 2022, at 7:27 PM, Jean-Sebastien

Re: QueryRecord with Union type

ate my redesign so I take your point about minimising the number of processes and will look again at combining steps in the query process, although I am also a fan of the JOLT transform as I have used that often in previous projects. Regards Steve Hindmarch From: Mark Payne mailto:marka...@hotma

Re: QueryRecord with Union type

Steve, Because your schema has a union, the SQL engine doesn’t really know how to interpret the data. So it interprets it as a “Java Object.” Essentially, it could be anything. But you can’t compare just anything to true - you need to compare a boolean to true. So you need to tell the SQL engine

Re: Getting java.util.concurrent.RejectedExecutionException upon nifi restart

Hi Shweta, The exception that you see here is rather harmless - it’s indicating that there was an attempt to enable a Controller Service while NiFi was in the middle of shutting down. Recommend you look higher up in the logs to understand why NiFi was shutting down, if this was not intentional.

Re: Records - Best Approach to Enrich Record From Cache

Mike, Steven, Nick, The DistributedMapCache Client does already have that. The method is called subMap. But I don’t think it performs an MGET at present, it just loops over the keys calling GET. But I think the more appropriate approach here would be to provide a RedisRecordLookupService so tha

Re: ListSFTP doesn't follow symlinks

Guille, Thanks for the extra details. I just tried again. In my case, all worked as expected when I had a symlink to a directory. But when I had a symlink to a file, I got the same error and stack trace as you. So looks like we are handling the case properly for symlinked directories but not s

Re: ListSFTP doesn't follow symlinks

Guille, I did a quick test on my MacBook and things worked as expected following symlinks. If I set the property to “false” it didn’t get the files. If I set it to “true” it did retrieve the files. Either way it didn’t error, though - just didn’t follow the symlink. Of course, that’s not to sa

Re: NiFi V1.15.2 Conversion to Avro from JSON

N Hey Mark, You beat me to the fix, Thanks for taking your time to look at it! I appreciate that one. Cheers! Nathan From: Mark Payne mailto:marka...@hotmail.com>> Sent: Wednesday, January 19, 2022 6:54:23 PM To: users@nifi.apache.org<mailto:users@nifi.

Re: NiFi V1.15.2 Conversion to Avro from JSON

Thanks Nathan. I created a Jira [1] for this. I was able to easily replicate with your template. Thanks for including that. Just put up a Pull Request for it, as well. Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-9594 On Jan 18, 2022, at 4:49 AM, nathan.engl...@bt.com

Re: DistributedMapCacheServer not included when making copy of ProcessGroup

server and the DistributedMapCacheServer is created successfully when changing version on the production server, so the component must be present in the flowdefinition. Thanks, Isha Van: Mark Payne mailto:marka...@hotmail.com>> Verzonden: woensdag 19 januari 2022 15:46 Aan: users@nifi.apa

Re: DistributedMapCacheServer not included when making copy of ProcessGroup

Hi Isha, When you say it was “copied” what precisely do you mean? Did you copy & paste the Process Group? Did you create a template and then instantiate it? Did you download the flow definition and then upload it again? There are a few different mechanism that you can use for duplicating proces

Re: PutSQL in combination with ConvertJSONToSQL gives java.sql.SQLException: Invalid column type for Orcale DataType BINARY_DOUBLE

Hi Sven, That is interesting. Looking at the JDBC Types class, there is no BINARY_DOUBLE type - and no constant with a value of 101. So it appears to be a non-standard type. So the script that you have in place is certainly one alternative. It is worth noting, though, that ConvertJsonToSql and

Re: Problem using DistributedMapCache

Christian, It looks like you are attempting to use a DistributedMapCacheClientService with a DistributedSetCacheServer. I.e., you’re using a MAP-based client with a SET-based server. You need to use a DistributedMapCacheServer, not a DistributedSetCacheServer. Thanks -Mark On Jan 10, 2022, a

Re: javax.net.ssl.SSLPeerUnverifiedException

Lior, What do you have set for the “nifi.zookeeper.connect.string” property in nifi.properties? Thanks -Mark On Dec 16, 2021, at 4:26 AM, Lior Halperin mailto:lior.halpe...@outseer.com>> wrote: Hi, We are using nifi 1.15 secured cluster with external zk 3.7.0 defined in the zk conf: ssl.hos

Re: NiFi flow.xml.gz corruption

Hi Ravi, Not sure why you would be seeing the flow get corrupted. When you say “corrupted” - do you mean truly corrupted? As in, the file cannot be read/parsed? Or do you mean that it’s out of sync, meaning that NiFi can read it but won’t join the cluster because its flow is different from the

  1   2   3   4   5   6   7   >