Re: FlattenJSON fails on large json file

Hi James, I don't have a solution for you off the top of my head. But I can tell you the failure is because you've got an array longer than the maximum value of an Int. So, memory is not the limiting factor. -Eric On Fri, Jun 14, 2024, 10:59 AM James McMahon wrote: > I have a json file, incomi

Re: Recommended resource limits

; are specific pod settings youll want to use to avoid it getting nuked by >> k8s. >> >> This blog may give you great things to consider >> https://home.robusta.dev/blog/kubernetes-memory-limit >> >> Thanks >> >> On Wed, Aug 16, 2023 at 3:29 PM Eric Secu

Re: NiFi OpenTelemetry JavaAgent

ciate it if they shared their experience. Thanks, Eric On Fri, Sep 8, 2023 at 4:22 PM Joe Witt wrote: > Eric > > Any sign anything in the nifi app log? Did it perhaps run out of mem and > hang? > > In general such agents should be fine. > > Thanks > > On Fri, Sep 8,

NiFi OpenTelemetry JavaAgent

Hello, I have tried adding opentelemetry to NiFi by running the OpenTelemetry javaagent ( https://opentelemetry.io/docs/instrumentation/java/getting-started/#instrumentation) with NiFi and it mostly works. I am able to see spans created by jetty, reddison, etc. However NiFi also dies and gets rest

Re: Recommended resource limits

framework creating many processors and connections but deleting a processor also makes its component status unavailable, so I would assume that also releases associated memory. On Wed, Aug 16, 2023 at 2:35 PM Eric Secules wrote: > Hi, > > I was wondering what the recommendation is for how m

Recommended resource limits

Hi, I was wondering what the recommendation is for how much memory to start the nifi jvm with and then how much to set the docker container memory limit as a function of the jvm memory setting. I have nifi started with -xms and -xmx set to 4gb and docker memory resource limits of 8GB on the conta

Re: how SCP full directory to remote location

Hi Ben, To clarify, do you want to move files from the same server nifi is running on to an sftp server? -Eric On Thu., May 11, 2023, 1:58 a.m. Ben .T.George, wrote: > Hello, > > How can we scp directory with many sub folders and files to remote sftp > server? > > Regards, > Ben >

Re: Need Help in migrating Giant CSV from S3 to SFTP

ingle I have one single > file as 400GB all the load goes to any one of the node that picks up the > transfer. I wanted to know if we can do this any other way of configuring > the flow. If not please tell me the metrics for nifi to tweak. > > > > *From:* Eric Secules &g

Re: Maximum number of processors?

gt; would be no flowfile sitting there. So you'd not really burn > resources meaningfully in that model. This is part of why it scales > so well as there are so many flows all on the same nodes all the time. > But you might want to lower the scheduled run frequency of processors

Re: Maximum number of processors?

nd up with 100s of thousands of processors and would want > node slicing of flows in the cluster. > > Thanks > > On Wed, Mar 8, 2023 at 9:31 AM Eric Secules wrote: > > > > Hello, > > > > Is there any upper limit on the number of processors that I can have in >

Maximum number of processors?

Hello, Is there any upper limit on the number of processors that I can have in my nifi canvas? Would 10 still be okay? As I understand it, each processor takes up space on the heap as an instance of a class. If this is a problem my idea would be to use multiple unclustered nifi nodes and spre

Which repositories would benefit from a higher performance disk?

Hello, I am working on configuring nifi to improve throughput. I already have each of the repositories attached to their own Standard SSD disk. I am wondering whether I am correct in thinking that I'd see the most improvement by putting the content repository on a faster SSD SKU, or is it worth it

Re: Merge Content De-fragment Max Bins

On Tue, Sep 27, 2022 at 4:55 PM Eric Secules wrote: > >> Hello, >> >> I am using MergeContent in defragment mode. I am having a problem when >> bins complete too early because they were evicted due to reaching the max >> bin count. However, this behaviour doesn&#x

Merge Content De-fragment Max Bins

Hello, I am using MergeContent in defragment mode. I am having a problem when bins complete too early because they were evicted due to reaching the max bin count. However, this behaviour doesn't make sense for defragment mode. What I expect it would do is penalize and requeue flowfiles that didn't

Re: NiFi web UI not responsive under load

rious if maybe oom_killer > or some process like that is killing nifi. Might want to check system logs > like dmesg to see if there’s anything there. > > Thanks > -Mark > > On Apr 19, 2022, at 4:28 PM, Eric Secules wrote: > > Hi Joe & Mark, > > I'll work on

Re: NiFi web UI not responsive under load

hours” > > In 1.14.0, we introduced the notion of a Long-Running Task Monitor. It’s > generally very fast. Typically runs in 10s of milliseconds on my macbook. > But it relies on JVM-specific code, and we’ve seen in some environments > that can cause the UI responsiveness to be ver

Re: NiFi web UI not responsive under load

2168 nanoseconds (4737 seconds)."} > On Tue, Apr 19, 2022 at 12:34 PM Eric Secules wrote: > Hello, > > When my nifi system goes under high load the web UI becomes unresponsive > until load comes down. Is there a way I can see what's going on (processor > status summa

NiFi web UI not responsive under load

Hello, When my nifi system goes under high load the web UI becomes unresponsive until load comes down. Is there a way I can see what's going on (processor status summary, queued count, active thread count) when the UI is unresponsive? The logs are not showing any errors and the various repositori

Re: Cannot Delete Process Group Because Source of Connection is "Running"

Side note, that's the behavior I'd expect from a "stopping" processor but not one that's in the "stopped" state. On Tue., Apr. 5, 2022, 4:07 p.m. Eric Secules, wrote: > Hi Mark, > > Is there an API for this that can filter by processors within

Re: Cannot Delete Process Group Because Source of Connection is "Running"

itions to stopped, it may still have active > threads that haven’t completed yet. You’ll need to wait until the processor > is stopped and the active threads on the processor reach 0. > > Thanks > -Mark > > On Apr 5, 2022, at 6:25 PM, Eric Secules wrote: > > Hello, > > I

Cannot Delete Process Group Because Source of Connection is "Running"

Hello, I have this program which stops and deletes flows from NiFi, when we're done with them and once in a long while we fail this operation because of this: 2022-04-05 13:03:18,569 WARN [NiFi Web Server-281] > o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: > Destinatio

Re: NIFI - Performance issues

Another thing is you want to check the actual disk usage. If it's above a threshold limit defined by your NiFi Properties the system will stall seemingly randomly. See this thread for more details: https://lists.apache.org/thread.html/r70dcc6387dccd062cbae09cfa6470c8d7443db63a495be35450e2b00%40%3Cu

Dynamic Attribute Naming

Hello, I was wondering if it's possible to name an attribute based on an EL statement like MyAttr.${fragment.index} Then when flow files are merged back together all the attributes are preserved. Thanks, Eric

Re: ExtractText Improvement

20, 2:06 p.m. Eric Secules, wrote: > Hello everyone, > > I was wondering if there could be an improvement to ExtractText so that > the entire content of the flowfile is scanned for matches in chunks of > MAX_BUFFER_SIZE which overlap by MAX_CAPTURE_GROUP_LENGTH. That way we can >

Re: NiFi Missing Provenance Data

rty to > something longer than that, such as 10 days. > > Thanks > -Mark > > [1] https://issues.apache.org/jira/browse/NIFI-7856 > > > On Dec 17, 2020, at 3:50 PM, Eric Secules wrote: > > Leon and I did some more digging and found this in both of our logs when >

Re: NiFi Missing Provenance Data

ory and I was > able to find the component ids and the filenames I was setting in some of > the files there. But nothing showed up in the UI. > > I am not sure of the conditions needed to reproduce this, but I do know it > happens often enough to be a problem. > > Thanks, >

Re: NiFi Missing Provenance Data

enough to be a problem. Thanks, Eric On Wed, Dec 9, 2020 at 10:41 AM Eric Secules wrote: > Hi Leon, > > I want to try to rule out something. Are there any logs we can search for > that would help us find when provenance events are cleared out? I want to > rule out the CREATE eve

ExtractText Improvement

Hello everyone, I was wondering if there could be an improvement to ExtractText so that the entire content of the flowfile is scanned for matches in chunks of MAX_BUFFER_SIZE which overlap by MAX_CAPTURE_GROUP_LENGTH. That way we can do pattern extraction over arbitrary size files while keeping me

Re: [Bug] Duplicate Flow Import From Registry

) talking to a Registry > running on some IaaS infrastructure? There is no load balancer/proxy/etc.. > in between ? > > Thanks > > On Thu, Dec 10, 2020 at 4:17 PM Eric Secules wrote: > >> Hello everyone, >> >> My team is encountering a bug where we import a f

[Bug] Duplicate Flow Import From Registry

Hello everyone, My team is encountering a bug where we import a flow from our registry residing in the cloud to our laptops. The import takes a long time and we end up with multiple copies (about 6) of the same flow one on top of each other on the canvas. The canvas becomes unresponsive and we are

Re: NiFi Missing Provenance Data

Hi Leon, I want to try to rule out something. Are there any logs we can search for that would help us find when provenance events are cleared out? I want to rule out the CREATE event getting created and getting immediately deleted due to some race condition. Some more detail is that the initial p

Re: Tuning for flow with lots of processors

Hi Mark, It was because the main disk was filling up! We increased the disk size to 128GB and speed improved! Thanks, Eric On Wed., Nov. 25, 2020, 12:34 p.m. Eric Secules, wrote: > Hi Mark, > > Thanks for the quick response, I grepped the logs and did find several > hits

Re: Tuning for flow with lots of processors

space that you have. > > Thanks > -Mark > > > > On Nov 25, 2020, at 3:21 PM, Eric Secules wrote: > > > > Thanks for the tips Mark! > > > > I looked at the summary and there are a fair number of processors at the > top of the list which create flo

Re: Tuning for flow with lots of processors

sing a second thread > will improve the performance. > > Thanks > -Mark > > > On Nov 24, 2020, at 5:40 PM, Eric Secules wrote: > > Hi Mark, > > Watching the video now, and will plan to watch more of the series. Thanks! > As for questions, > > I have NiFi on

Re: Tuning for flow with lots of processors

used? There are a > series of videos on YouTube where I’ve discussed nifi anti-patterns. One of > those [1] discusses how to tune the Timer-Driven Thread Pool, which may be > helpful to you. > > Thanks > -Mark > > [1] https://www.youtube.com/watch?v=pZq0EbfDBy4 > > >

Tuning for flow with lots of processors

Hello everyone, I was wondering if there was a metric for the amount of time tImer-driven processors spend in a queue ready and waiting to be run. I use NiFi in an atypical way and my flow has over 2000 processors running on a single node, but there are usually less than 10 connections that have o

Flow Hotspots

Hello everyone, I was wondering if the nifi summary view could have a summary of the connections where flowfiles spend the most time in waiting, that would help identify slow points in a complicated flow. Alternatively, does anyone know of some tool which might be able to provide this analysis al

Re: Provenance queries effect on processing

ready at the output port, this would include the original flow > file and any other child flow files. > > On Wed, Nov 4, 2020 at 12:58 PM Joe Witt wrote: > >> Eric >> >> Nope Im not aware of anything specific. >> >> Thanks >> Joe >> >> O

Re: Provenance queries effect on processing

anges and I'm trying to figure out what's the cause for many of our tests suddenly timing out. Thanks, Eric On Tue., Nov. 3, 2020, 6:18 p.m. Eric Secules, wrote: > Hi Joe, > > Thanks for the explanation, is there a Jira ticket for for a job > management mechanism? Is t

Re: Provenance queries effect on processing

st any > blending of the flow execution with provenance queries. What you're trying > to do though is a great idea and aligns to what Mark Payne has talked about > previous as a job management mechanism. This can/should be done without > provenance itself. > > > Thanks &

Provenance queries effect on processing

Hello everyone, I was wondering if it were possible for excessive use of the provenance api would cause flowfile processing to slow down and even come to a halt? My test setup queries the provenance API to see if all flowfiles that descended from a given input file have completed processing. This

Re: PutAzureBlobStorage OutOfMemoryError

much detail as you can. > > Thanks > > On Wed, Oct 28, 2020 at 12:40 PM Eric Secules wrote: > >> Hello everyone, >> >> I am trying to upload a 300 MB file to azure blob storage using >> PutAzureBlobStorage and the processor is failing due to an >> O

PutAzureBlobStorage OutOfMemoryError

Hello everyone, I am trying to upload a 300 MB file to azure blob storage using PutAzureBlobStorage and the processor is failing due to an OutOfMemoryError. My JVM heap size is set to 512 MB, but I wouldn't expect this to be an issue because the PutAzureBlobStorage processor should be using stream

Re: Possible Regression in PutAzureBlobStorage 1.12.0

A/B/test.json >> >> New object: A/B/${filename}.json >> >> New object with new pseudo-dirs: >> ${random():mod(10):plus(1)}/${random():mod(10):plus(1)}/${filename}.json >> >> -joey >> >> On Sep 16, 2020, 11:02 AM -0700, Eric Secules , >> w

Re: Possible Regression in PutAzureBlobStorage 1.12.0

Hello everyone, I was able to see why this is an issue. It's an issue that the blob is stored several layers deep at "my-container/A/B/my_test_blob.json" -Eric On Wed, Sep 16, 2020 at 10:49 AM Eric Secules wrote: > Hello everyone, > > I tried upgrading to 1.12.0 and

Possible Regression in PutAzureBlobStorage 1.12.0

Hello everyone, I tried upgrading to 1.12.0 and right away noticed that PutAzureBlobStorage is failing due to the following error. I don't think it's an issue with access because I can use a ListAzureBlobStorage on the same container and I haven't changed the permissions of the container. And I di

Re: NiFi Merge Content Processor Use Case

I have created a JIRA issue for this: https://issues.apache.org/jira/browse/NIFI-7699 On Fri, Jul 31, 2020 at 10:20 AM Eric Secules wrote: > Hi Joe, > > Thanks for your suggestions and analysis! > > I was also thinking that the MergeContent processor could also have a wait &g

Re: NiFi Merge Content Processor Use Case

w and utilize a FIFO queue for instance to overcome that. Just needs a > JIRA/analysis to get over the real hump here. > > Thanks > > > On Fri, Jul 31, 2020 at 8:16 AM Eric Secules wrote: > >> Is it possible to surround the merge content processor with a wait/notify >

NiFi Merge Content Processor Use Case

27;d rather the incoming flowfiles wait and then only expire existing buckets after a timeout. -- Forwarded message ----- From: Eric Secules Date: Thu., Jul. 30, 2020, 6:13 p.m. Subject: NiFi Merge Content Processor Use Case To: Cc: Hello, I have a use case for the merge content pr

Validating JSON Values are neither null not empty

Hello, I am using the ValidateRecord processor and an avro schema to validate that a json documents of roughly 50 fields contains 20 required fields. For those required fields I am also supposed to validate that they are neither null not empty-string. I don't think an avro schema will help me with

NullPointerException when downloading process group

Does anyone have a suggestion for a workaround? I'm at a loss right now. On Wed., Jul. 8, 2020, 11:11 p.m. Eric Secules, wrote: > Hello, > > I believe I found a bug regarding downloading versioned process groups > from a registry. I have logged a bug. > > https://issues

NullPointerException when downloading process group

Hello, I believe I found a bug regarding downloading versioned process groups from a registry. I have logged a bug. https://issues.apache.org/jira/browse/NIFI-7622 Thanks, Eric

Processor Extensibility

Hello, I was wondering if there was a recommendation on how to extend the functionality of nifi processors without forking the nifi repository. I'm looking for a way to include a processor's nar in my project and extend from it. I'd also like to be able to extend that processor's test suite so I c

Provenance Query Improvements

Hello everyone, I am working on a tool to tell whether all processing has completed for a given input filename. Since it's possible for the flow to change the "filename" attribute, that is not a reliable way to get all the events related to an input file. My current solution involves recursively c

NiFi NARs Included in Docker

Hello everyone, I noticed that not all the NARs from the NiFi project are included in the Docker image on Dockerhub. For example, the nifi-atlas-nar is not included in the Docker image. Is this on purpose or an oversight? Thanks, Eric

Re: Connecting Controller Services Automatically

I have put together a bug on NiFi and added steps to reproduce with a simple template. https://jira.apache.org/jira/browse/NIFI-7489 -Eric On Mon, May 25, 2020 at 10:28 AM Eric Secules wrote: > Thanks for the link Mark, > > I have found out that my problems are caused by this `continu

Re: Connecting Controller Services Automatically

t, May 23, 2020 at 8:00 PM Andrew Grande wrote: > >> Maybe something is going on with specific types or hierarchies. I've >> noticed DefaultSslContext didn't get assigned, even though it was the only >> one available. Does autowiring logic apply to this one? >>

Re: Connecting Controller Services Automatically

deployment would need a manual intervention to reference the >>> specific controller service in the proper component. >>> >>> Andy LoPresto >>> alopre...@apache.org >>> *alopresto.apa...@gmail.com * >>> He/Him >>> PGP Fingerprint: 70EC B3E5

Re: Connecting Controller Services Automatically

a...@gmail.com * > He/Him > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On May 22, 2020, at 3:01 PM, Eric Secules wrote: > > Hello everyone, > > I am running into an issue with automated deployment using nipyapi > <https://nipyapi.readthedoc

Connecting Controller Services Automatically

Hello everyone, I am running into an issue with automated deployment using nipyapi . We would like to be able to pull down flows from a registry and have them ready to go once all their controller services have been turned on. But there are a few issues.

Re: Nifi - how to achieve a concurrent development and CI/CD

do you write/run for NiFi? We were > just researching this topic. > > On Thu, May 14, 2020 at 6:38 PM Eric Secules wrote: > >> Hi Michal, >> >> I'm also using a single registry for development and production. It >> doesn't help with collaborating on the

Re: Nifi - how to achieve a concurrent development and CI/CD

Hi Michal, I'm also using a single registry for development and production. It doesn't help with collaborating on the same process group as there is way for it to reconcile merge conflicts. Instead, the registry will earn you that you're about to overwrite someone else's changes. Another pain of c

Re: Is provenance data preserved when processors are deleted?

omsen wrote: > It copies all of the provenance data, and no, there's no way yet to back > the provenance repository with one of those nosql databases yet > unfortunately. > > On Mon, May 4, 2020 at 6:40 PM Eric Secules wrote: > >> What information is transmitted by Si

Re: Is provenance data preserved when processors are deleted?

> On Mon, May 4, 2020 at 5:47 PM Eric Secules wrote: > >> Hello everyone, >> >> If I am upgrading a process group to the latest version, do you know >> whether provenance is preserved for processors that may get deleted in the >> upgrade? >> I have noticed

Is provenance data preserved when processors are deleted?

Hello everyone, If I am upgrading a process group to the latest version, do you know whether provenance is preserved for processors that may get deleted in the upgrade? I have noticed that if I delete my process group and redownload it from the registry, I am no longer able to see the provenance d

Re: Deleting/Modifying Parameters with Nifi Registry

nks > > > > On Tue, Apr 14, 2020 at 7:41 PM Eric Secules wrote: > >> Hello everyone, >> >> I am noticing that when I delete a parameter from a parameter context in >> NiFi a number of strange things result. >> >>- It doesn't register as a ch

Deleting/Modifying Parameters with Nifi Registry

Hello everyone, I am noticing that when I delete a parameter from a parameter context in NiFi a number of strange things result. - It doesn't register as a change that I can commit to the registry - When I do make a change and commit to the registry, the deleted parameter remains there a

Automatic Grid Placement

Hello everyone, Is there a way to tell the NiFi rest api to place my component in an empty space on the grid? If I do not specify the position, everything gets put down on top of each other at (0,0) right? Additionally it would be great if there was an auto-arrange button that organized your flow

Re: Suggestions for Flow Development Lifestyle

ionality, I > don't know that I would expect users to approve changes by examining diffs > in the XML files. > > However, I did have one thing to add that I hope might help: > On Tue, Feb 25, 2020 at 12:32 AM Eric Secules wrote: > >> I've also tried backing up my lo

Re: Suggestions for Flow Development Lifestyle

flows of other versioned flows, and limit > authorship of a versioned flow to a single author at any one time unless > the other author is going to fork from a particular version to make a new > versioned flow. > > Again great e-mail. Your expectation of what it should do is very

Suggestions for Flow Development Lifestyle

Hello everyone, Im starting to use nifi and nifi registry on my development team and we're running into issues working together on the same versioned process groups. The nifi registry doesn't support branching, merging and code review nativly so we all have ended up developing on the same branch o