[ANNOUNCE] Apache NiFi 1.15.2 release

2021-12-22 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.15.2. This is a bug fix and security focused release. The primary intent is upgrading to the latest log4j 2.x libraries 2.17, eliminating usage of log4j 2.x core, and updating to the latest logback. Apache NiFi is an

Re: ConsumeJMS causing multiple BIND/UNBIND request

2022-01-13 Thread Joe Witt
Solace is a commonly used JMS provider with NIFi and it is very likely such an issue has been addressed. Please test/attempt this on a newer version of NiFi. Fixed related to JMS since 1.11 https://issues.apache.org/jira/browse/NIFI-9239?jql=project%20%3D%20NIFI%20AND%20status%20in%20(Resolved%2C

Re: Running NiFi with uncompressed flow.xml?

2022-01-13 Thread Joe Witt
Troels, At this time there is not. The flow.xml.gz file isn't meant for external usage generally and it might as well be called 'nifivoodoofiledontmesswiththis.custom' The better answer from a version management point of view is usage of the Flow Registry and versioned flows. NiFi isn't 'design

Re: Question bigqueryStreaming processor

2022-01-13 Thread Joe Witt
Franco I've pinged someone in the community who I suspect will know or be able to quickly find out and reply back. But he is in France and it is already later in the day so hang tight for a reply - hopefully tomorrow. Thanks On Thu, Jan 13, 2022 at 12:40 PM walter franco wrote: > > hello, > >

Re: Question bigqueryStreaming processor

2022-01-13 Thread Joe Witt
this new API (not sure the existing processor can be > updated to use the new API, but I'll check). > > Hope this helps, > Thanks, > Pierre > > > Le jeu. 13 janv. 2022 à 20:55, walter franco a écrit : >> >> Thanks a lot Joe >> >> Nifi version 1.1

Re: Inconsistent behaviour in GenerateTableFetch 1.15.0

2022-01-14 Thread Joe Witt
Hello That processor has changed only once since March 2021 (when 1.13.2 came out). That was this commit [1] for this JIRA [2]. >From a quick look it isn't obvious to me how that could be related. Mattyb/MarkP any thoughts? [1] https://github.com/apache/nifi/commit/ecacfdaa4c663ff2831dd13a840a

[ANNOUNCE] Apache NiFi 1.15.3 release

2022-01-18 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.15.3. This is a bug fix and security focused release. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable d

Re: Nifi 1.15.2 and 1.15.3 compilation errors

2022-01-19 Thread Joe Witt
We should discontinue Kite processors from the build at all in 1.16.0. Ill file a Jira Thanks On Wed, Jan 19, 2022 at 7:47 AM David Handermann < exceptionfact...@apache.org> wrote: > Mike, > > Thanks for the additional details. According to the original output, the > problem appears to be in th

Re: Nifi 1.15.2 and 1.15.3 compilation errors

2022-01-20 Thread Joe Witt
too? >> >> Or you discontinue only a part of org.apache.nifi.processors.kite schema and >> the rest will be somehow moved to other schemas? >> >> >> >> Regards, >> >> Mike >> >> >> >> >> >> From: Joe Witt >>

Re: nifi 15.3 - puthive3streaming

2022-01-21 Thread Joe Witt
Hello We need to see the full stack trace. Most likely involves log4j Thanks On Fri, Jan 21, 2022 at 7:14 AM Hengesbach, Martin < martin.hengesb...@fiz-karlsruhe.de> wrote: > Hi Mike, > > > > I have downloaded the compiled version of nifi-hive3-nar-1.15.*3*.nar and > have the same problem as y

Re: Kerberos Issue with OpenJDK 1.8.292

2022-01-27 Thread Joe Witt
Guille We are trying to be broadly compatible with every Java 8 and Java 11 release we can but of course the older they get certain ones become unusable and the newer they get sometimes new behavior are introduced. We test a lot of combinations directly plus we hear a lot from threads like this.

Re: Kerberos Issue with OpenJDK 1.8.292

2022-01-27 Thread Joe Witt
va:680) > at java.security.AccessController.doPrivileged(Native Method) > at > javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) > at javax.security.auth.login.LoginContext.login(LoginContext.java:587) > at > org.apache.nifi.security.krb.AbstractKerbe

Re: IPDR-SP ETL using Apache Nifi

2022-02-08 Thread Joe Witt
Breno I'm not aware of any specific IPDR usage/components built for NiFi but we often don't hear about all the custom components users/companies write. If you do build these and find they can be Apache license compatible and are interested in contributing please let us know. This is a perfectly f

Re: ListenTCP not generating flowfiles

2022-02-09 Thread Joe Witt
Correct. With both ListenTCP and ListenTCPRecord it listens for data and needs some way of framing them to generate records/bundles of data as flowfiles. What kind of data do you want to listen for? Thanks On Wed, Feb 9, 2022 at 6:26 PM Michael Loftis wrote: > ListenTCP isn’t fit for ingestin

Re: ListenTCP not generating flowfiles

2022-02-09 Thread Joe Witt
ow nifi >> is interpreting the data when It its received ? >> >> Em qua., 9 de fev. de 2022 22:31, Joe Witt escreveu: >> >>> Correct. With both ListenTCP and ListenTCPRecord it listens for data >>> and needs some way of framing them to generate records/bund

Re: Use case requires deletion of counters

2022-02-13 Thread Joe Witt
James Counters offer no such facilities as they have q very narrow purpose/usage as built which is purely for basic monitoring of component behaviors useful mostly for a processor developer. They have no persistence/etc… For your case you likely want to interact with a database. Thanks On Sun,

[ANNOUNCE] Apache NiFi 1.16.0 release

2022-04-04 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.16.0. This is a substantial feature, improvement, bug fix and security focused release with more than 400 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data.

Re: Possible to convert .msg to .eml?

2022-04-07 Thread Joe Witt
James, >From a tool/purpose point of view yes NiFi is an ideal tool for the job assuming you're talking about needing to do this continuously in an automated flow fashion. But, we of course need some transform written for this or you could certainly build your own custom one as a full on processo

Re: NiFi web UI not responsive under load

2022-04-19 Thread Joe Witt
Eric When the UI isn't responsive it would be great to have a snapshot of: - CPU usage at that time - GC behavior/logging at/around that time. - IO Utilization around that time - NiFi Thread dump precisely during it and ideally also one after it responds again NiFi Restarting itself is very inter

[ANNOUNCE] Apache NiFi 1.16.1 release

2022-04-29 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.16.1. This is an improvement, bug fix and security focused release with more than 80 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made

Re: Rest API to query for nifi.properties settings

2022-05-11 Thread Joe Witt
James, There is no API to do so that I'm aware of - we'd intentionally not want that available. The nifi.properties file contains inherently node centric values and some sensitive values. Joe On Wed, May 11, 2022 at 10:03 AM James McMahon wrote: > Are there any nifi REST API calls that return

Re: nifi-content-viewer with mime.type of image/webp

2022-05-12 Thread Joe Witt
Ryan. Nope - just means we dont have a viewer for that content type. Would need to be added. Thanks On Thu, May 12, 2022 at 9:19 AM Ryan Hendrickson < ryan.andrew.hendrick...@gmail.com> wrote: > Hi all, > I'm curious if anyone has used the nifi-content-viewer to view images with > a mime.type

Re: Co-habitating log files and flow_repo - bad idea?

2022-05-17 Thread Joe Witt
Hello Performance wise not as concerning really unless this is a truly performance intense high volume of flowfile situation. But must ensure logs only take up limited space and plenty remains for the flowfile repo particularly when there are backlogs/etc. Joe On Tue, May 17, 2022 at 12:21 PM

[ANNOUNCE] Apache NiFi 1.16.2 release

2022-05-25 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.16.2. This is an improvement, bug fix and security focused release with more than 30 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made

[ANNOUNCE] Apache NiFi 1.16.3 release

2022-06-14 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.16.3. This is a pure bug fix and security focused release with 13 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It s

Re: Issue with registry and Port error

2022-06-16 Thread Joe Witt
Dave The issue you flag might be related. In the 1.15/1.16 lines we've improved a good bit of funky lifecycle state handling. If you're able I recommend trying this on the 1.16.3 release for both NiFi and Registry. If that still occurs we would like to see the nifi-app.log entries of nifi. Tha

Re: Encrypted smb v3

2022-06-17 Thread Joe Witt
Robert Would you be comfortable trying to make a pull request for it? Thanks On Fri, Jun 17, 2022 at 3:12 AM Richter, Robert wrote: > Hello, > > > > We’ve tried to put some files on a smb v3 with PutSmbFile processor. This > works fine if the share is not encrypted and throws access denied fai

Re: Changes not persisted to flow.xml.gz

2022-06-30 Thread Joe Witt
Jim, If I think back a very very long time I can think of when this might have occurred/been possible but it certainly isn't something I've heard of in a very long time. We'd need to understand the nifi version, the os/version, java version as a starter. But also please look in the nifi-app.log

Re: Issue with SiteToSiteProvenanceReportingTask 1.15.3

2022-07-15 Thread Joe Witt
Hello Is that the totality of what the logs contain? Ideally we're going to need to see the full logs. The other case you link is rooted in having exhausted the open file limits. Your case appears different. Would be good to file a JIRA with the logs, details about the JVM version at play, det

Re: Crash on startup due to Output port issue

2022-07-31 Thread Joe Witt
Can you share the flow or does it contain sensitve info? On Sun, Jul 31, 2022 at 10:01 AM BeNJ wrote: > Hi, > I'm a little at a loss so I figured maybe someone here would have an idea > of how to get out of this situation. > My NiFI is crashing on startup with the following: > --

Re: Crash on startup due to Output port issue

2022-07-31 Thread Joe Witt
my iPhone > > On Jul 31, 2022, at 12:25, Joe Witt wrote: > >  > > Can you share the flow or does it contain sensitve info? > > On Sun, Jul 31, 2022 at 10:01 AM BeNJ wrote: > >> Hi, >> I'm a little at a loss so I figured maybe someone here would have an id

Re: Crash on startup due to Output port issue

2022-07-31 Thread Joe Witt
ng on > Linux)? > > Dave > > > On Sun, Jul 31, 2022, 2:21 PM Benji Benning wrote: > >> Not really around the below error. >> Anything I can move to debug or trace in logback.xml to add some color >> around this? >> >> Thanks, >> Benji >> &g

Re: Crash on startup due to Output port issue

2022-08-01 Thread Joe Witt
zer.java:423) >> at >> org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) >> at >> org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) >&

[ANNOUNCE] Apache NiFi 1.17.0 release

2022-08-01 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.17.0. This is a feature, improvement, bug fix and security release with 310 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataf

Re: Crash on startup due to Output port issue

2022-08-01 Thread Joe Witt
ew?usp=sharing >> [2] https://issues.apache.org/jira/browse/NIFI-10306 >> >> On Mon, Aug 1, 2022 at 12:46 PM BeNJ wrote: >> > >> > Updates: >> > Mark: Deleting the json allows nifi to start up(!!) - note that restarting >> > (with the newly g

Re: MiniFi codebase question

2022-09-07 Thread Joe Witt
Marc Please only post to one list. Moved security and dev to bcc as this is just a user question. Apache NiFi's MiNiFi has two implementations one is Java based and the other is CPP based. Given the version you reference I suspect you're asking about the MiNiFI Java implementation. We used to

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
Lars What version are you using? Thanks On Tue, Sep 13, 2022 at 3:11 AM Lars Winderling wrote: > Dear community, > > sometimes our content repository grows out of bounds. Since it has been > separated on disk from the rest of NiFi, we can still use the NiFi UI and > empty the respective queues

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
nifi.content.repository.always.sync=false Thanks On Tue, Sep 13, 2022 at 7:04 AM Lars Winderling wrote: > > I'm using 1.16.3 from upstream (no custom build) on java 11 temurin, debian > 10, virtualized, no docker setup. > > On 13 September 2022 13:37:15 CEST, Joe Witt wrote: >> >>

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
ck if the issue persists. Your help is really > appreciated, thanks! > > On 13 September 2022 16:33:53 CEST, Joe Witt wrote: >> >> Lars >> >> The issue that came to mind is >> https://issues.apache.org/jira/browse/NIFI-10023 but that is fixed in >> 1.16.2

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
; On 13 September 2022 16:45:30 CEST, Joe Witt wrote: >> >> Lars >> >> You should not have to update to 1.17. While I'm always fond of >> peoople being on the latest the issue i mentioned is fixed in 1.16.3. >> >> HOWEVER, please do confirm your values.

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
read that again and hopefully it was obvious I was joking. But I am looking forward to hearing what you learn. Thanks On Tue, Sep 13, 2022 at 10:10 AM Joe Witt wrote: > > Lars > > I need you to drive back to work because now I am very vested in the outcome > :) > >

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
ntent-repository On Tue, Sep 13, 2022 at 10:15 AM Joe Witt wrote: > > read that again and hopefully it was obvious I was joking. But I am > looking forward to hearing what you learn. > > Thanks > > On Tue, Sep 13, 2022 at 10:10 AM Joe Witt wrote: > > > > Lars &g

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
hours in Germany, but you have NiFi brothers and > >> sisters around the world that are counting on you ... please don't > >> let us down. > >> > >> :) <- international smiley/joking symbol > >> > >> > >> On 9/13/2022 10:15 AM, J

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
). And thanks > Joe (and Pat) for both you technical and emotional support :-) > > Enjoy your day, see you on the list and may the flow be with you. Always. > > Best, Lars > > On 22-09-13 20:17, Joe Witt wrote: > > Hahah Pat! > > > > Lars > > > > O

Re: Merge Content De-fragment Max Bins

2022-09-27 Thread Joe Witt
Eric Yeah I think I agree. Can you file a JIRA? Thanks On Tue, Sep 27, 2022 at 4:55 PM Eric Secules wrote: > Hello, > > I am using MergeContent in defragment mode. I am having a problem when > bins complete too early because they were evicted due to reaching the max > bin count. However, thi

Re: Merge Content De-fragment Max Bins

2022-09-27 Thread Joe Witt
Cool. Thanks for both On Tue, Sep 27, 2022 at 6:23 PM Eric Secules wrote: > Hi Joe, > > It is done. https://issues.apache.org/jira/browse/NIFI-10553 I have a > unit test that I added on my fork and it is failing as expected. > > -Eric > > On Tue, Sep 27, 2022 at

Re: Thank You community

2022-10-04 Thread Joe Witt
Carlos Thanks for taking the time to write a nice email like this and glad you found nifi valuable. Best wishes. Joe On Tue, Oct 4, 2022 at 2:39 AM Dobbernack, Harald (Key-Work) wrote: > > | Perhaps I will explain this better in a medium Post. > > oh please do! Am looking forward to reading ab

[ANNOUNCE] Apache NiFi 1.18.0 release

2022-10-06 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.18.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: CVE-2022-42889 and impact on Apache Nifi

2022-10-21 Thread Joe Witt
Tom In the future if you're concerned or have questions about a vulnerability/potential vulnerability please follow the guidance here. https://nifi.apache.org/security.html Here you can see what we've done for this already on main https://issues.apache.org/jira/browse/NIFI-10648 with more info

Re: Empty Queue - but UI shows messages

2022-11-09 Thread Joe Witt
Hello This likely means the processor consuming from this queue has the flowfiles held being processed. MergeContent is a common processor that would do this but others certainly can. What processor do you have there? If you stop the target processor then delete it should always work though you

Re: CVE-2022-42889 and impact on Apache Nifi

2022-11-10 Thread Joe Witt
hat has been done! >> >> 1.18.1 could be a good idea I think as you say it would let people more >> comfortable as there is much noise around this CVE and the immediate >> conclusion that a patch is needed while it may be a product is not >> vulnerable to it althou

Re: Deprecated Processors

2022-11-12 Thread Joe Witt
Please post to a single list. Can you show the log output you see? Thanks On Sat, Nov 12, 2022 at 5:28 PM Mohnish Vaid wrote: > Greetings! > > Not sure which email to use, but I wanted to reach out and ask if it's > possible to find a list of deprecated processors (or future deprecated > ones).

Re: Exacly once from NiFi to Kafka

2022-11-15 Thread Joe Witt
Aian, How can you tell there are duplicates in Kafka and are you certain that no duplicates exist in the source topic? Given NiFi's data provenance capabilities you should be able to pin point a given duplicate and figure out whether it happened at the source, in nifi, or otherwise. Note much ha

Re: Nifi with Nginx(ssl) reverse proxy

2022-11-15 Thread Joe Witt
Thanks! On Tue, Nov 15, 2022 at 10:06 AM Ben .T.George wrote: > Hello, > > I have achieved this with nginx - reverse proxy method. > > Regards, > Ben > > On Sun, Nov 13, 2022 at 7:22 PM Ben .T.George > wrote: > >> Hello, >> >> i have a nifi instance running fine on port 8443, i would like to >>

Re: Exacly once from NiFi to Kafka

2022-11-19 Thread Joe Witt
; In any case, messages have json format (one json per flowfile) but since I > am sending and storing them in kafka in plain text I am using > *no-record-oriented* Kafka publisher. Is PublishKafkaRecord more > reliable? Would it be better to use it? > > Thanks, > > Aian > &g

[ANNOUNCE] Apache NiFi 1.19.0 release

2022-11-28 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.19.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: Expected mergerecord performance

2022-12-02 Thread Joe Witt
Hello Run schedule should be 0. 50 should be the min number of records 5 seconds is the max bin age it sounds like you want. Start with these changes and let us know what youre seeing. Thanks On Fri, Dec 2, 2022 at 10:12 PM Richard Beare wrote: > Hi, > I'm having a great deal of trouble con

[ANNOUNCE] Apache NiFi 1.19.1 release.

2022-12-07 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.19.1. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: Error on nifi start

2022-12-13 Thread Joe Witt
We support 8, 11, and 17 now. As far as NiFi 1.14 which James mentioned he was using we didn't support it then. Support started in NiFi 1.16. Thanks On Tue, Dec 13, 2022 at 8:43 AM James Srinivasan wrote: > I think nifi currently supports java 8 or 11, not 17: > > > https://nifi.apache.org/do

Re: CSV with header - but always, even for 0 record flowfiles

2022-12-19 Thread Joe Witt
Josef Please file a JIRA with this information above. Makes sense what you're looking for. Just not sure where this 'concern' would live whether it is in the processors themselves or the controller services for the writers. Thanks On Mon, Dec 19, 2022 at 9:21 AM wrote: > Hi guys > > > > I’ve

Re: Which repositories would benefit from a higher performance disk?

2023-01-16 Thread Joe Witt
Eric So assuming each ff is 50KB you currently see 1.6MB/sec for 120,000 ff/hr and want to achieve at least 26MB/sec for 2,000,000 ff/hr. These should be quite achievable rates for single nodes with modest resources on a wide range of flow requirements. Can you share a bit more about the cpu, me

[ANNOUNCE] Apache NiFi 1.20.0 release.

2023-02-09 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.20.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

[ANNOUNCE] Apache NiFi 1.20.0 release.

2023-02-09 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.20.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: Web Based Nifi for Python

2023-02-21 Thread Joe Witt
great. Go have fun. Thanks On Tue, Feb 21, 2023 at 7:27 AM Patrick Timmins wrote: > > I guess Joe already responded on this front ... more than a year and a half > ago! > > > > Forwarded Message > From:Joe Witt > Date:Sat, 10 Jul 2021 09:4

Re: Processor with cron scheduling in middle of flow

2023-02-21 Thread Joe Witt
John MergeContent cannot reliably work well with cron scheduling. That component is designed to get threads consistently so it can perform its bin packing function and time and size based kick out functions. If it ever worked with cron scheduling that was mostly on accident I'd say. Thanks Joe

Re: Processor with cron scheduling in middle of flow

2023-02-22 Thread Joe Witt
Yep. Though I'd still say we should first look at simply making it targeted that certain processors do not allow cron scheduling such as MergeContent which is not designed to work well unless it is consistently getting a chance to crunch numbers/data. @CronSchedulingDisabled or something... I do

Re: Maximum number of processors?

2023-03-09 Thread Joe Witt
Eric There is a practical limit in terms of memory, browser performance, etc... But there isn't otherwise any real hard limit set. We've seen flows with many 10s of thousands of processors that are part of what can be many dozens or hundreds of process groups. But the challenge that comes up i

Re: Maximum number of processors?

2023-03-10 Thread Joe Witt
essors that have flowfiles in the input queue and processors that have no > input? > > Thanks, > Eric > > On Thu, Mar 9, 2023 at 10:32 AM Joe Witt wrote: >> >> Eric >> >> There is a practical limit in terms of memory, browser performance, >> etc... Bu

Re: Cluster modification

2023-03-14 Thread Joe Witt
Phil Yes it should happen then too. Thanks On Tue, Mar 14, 2023 at 7:05 AM Phillip Lord wrote: > I see... Thanks Pierre. This all makes sense... I'm going to assume the > reconnecting node uses the Cluster Coordinator to retrieve the latest > flow-version. In Mark's video he demonstrated rec

Re: Broken pipe: InvokeHTTP

2023-03-14 Thread Joe Witt
Hello I believe this is the remote service killing the socket. Do you have logs for that service to check? Thanks On Tue, Mar 14, 2023 at 7:54 PM Richard Beare wrote: > Hi Everyone, > I have an InvokeHTTP processor experiencing the error below on a small > proportion of flowfiles. The service

[ANNOUNCE] Apache NiFi 1.21.0 release.

2023-04-07 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.21.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: PutSFTP TransportException (Timeout expired: 30000 MILLISECONDS) for sending data to Synology NAS - Data Timeout?

2023-04-24 Thread Joe Witt
Hello If going from 8 nodes with many errors to 1 node with few errors then you likely hit max connection limits on the sftp server. You can change that value on fhe sftp server. How many concurrent tasks do you allow the processor. If Y tasks you will want Y times 8 connections allowed. Than

Re: How to debug why a node isn't rejoining a cluster

2023-05-02 Thread Joe Witt
Mike If nothing is in the logs on the impacted node or anything else in the cluster you will want to grab a thread dump on the affected node and see what is actually happening. There has to be logs though If you have a large back log then you could be waiting for the initial repository healt

Re: Need Help in migrating Giant CSV from S3 to SFTP

2023-05-09 Thread Joe Witt
Nilesh, These processors generally are not memory sensitive as they should only ever have small amounts in memory at a time so it is likely this should work well up to 100s of GB objects and so on. We of course dont really test that much but it is technically reasonable and designed as such. So

Re: how SCP full directory to remote location

2023-05-11 Thread Joe Witt
Hello NiFi doesn't offer an SCP specific processor. Instead you would use List/FetchSFTP to pull from the target directory and remote source server (and recurse to subdirs as desired) and then use PutSFTP to write to the target directory (including subdirs as desired) of the remote destination se

Re: Processor ID/Processor Name as a default attribute

2023-05-24 Thread Joe Witt
Hello Can you describe how you would use this information? These kinds of details and more are present in provenance data now. Thanks On Wed, May 24, 2023 at 7:45 AM Chirthani, Deepak Reddy < c-deepakreddy.chirth...@charter.com> wrote: > Is there any chance where Processor_ID or Processor_Nam

Fwd: [ANNOUNCE] Apache NiFi 1.22.0 release.

2023-06-11 Thread Joe Witt
-- Forwarded message - From: Joe Witt Date: Sun, Jun 11, 2023 at 9:27 PM Subject: [ANNOUNCE] Apache NiFi 1.22.0 release. To: Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.22.0. Apache NiFi is an easy to use, powerful, and reliable system to

Re: NiFi not rolling logs

2023-07-07 Thread Joe Witt
H. Interesting. Can you capture these bits of fun in a jira? Thanks On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen wrote: > After doing some research, it appears that is a wonky > setting WRT how well it's honored by logback. I let a GenerateFlowFile > > LogAttribute flow run for a long tim

Re: UI SocketTimeoutException - heavy IO

2023-07-12 Thread Joe Witt
Joe I dont recall the specific version in which we got it truly sorted but there was an issue with our default settings for an important content repo property and how we handled mixture of large/small flowfiles written within the same underlying slab/claim in the content repository. Please check

Re: UI SocketTimeoutException - heavy IO

2023-07-12 Thread Joe Witt
e.repository.checkpoint.interval=300 secs > nifi.flowfile.repository.always.sync=false > nifi.flowfile.repository.retain.orphaned.flowfiles=true > > -Joe > On 7/12/2023 11:07 AM, Joe Witt wrote: > > Joe > > I dont recall the specific version in which we got it truly sorted but > there was an issue wi

Re: Schema Registry

2023-07-19 Thread Joe Witt
Hello You will want to check with Cloudera for the status of the Schema Registry in question here. From a NiFi point of view we will continue to integrate with schema registries we see in common usage and it should be set up well to plug in new ones and phase older ones as we go. Support for the

Re: Schema Registry

2023-07-19 Thread Joe Witt
And looks like with NiFi 2.0 though we will drop the Hortonworks Schema Registry as shown here [1]. If maintenance becomes more active for it we could revisit as needed. [1] https://issues.apache.org/jira/browse/NIFI-11095 On Wed, Jul 19, 2023 at 12:32 PM Joe Witt wrote: > Hello >

Re: source build issue - Nifi 1.23.0

2023-08-07 Thread Joe Witt
Greg Yeah seems likely there is a build issue with the RPM profile. I think we just dont have enough people exercising that thing often enough anymore and so when it breaks it is silent. The only thing we meant to have an RPM for as far as I recall was the nifi assembly itself but perhaps we had

Re: Node/processor/queue status in grafana

2023-08-13 Thread Joe Witt
This list is perfectly fine. All of our metrics are available via push, pull, and prometheus so it can def be done. It would be great to see what you end up with. Thanks On Sun, Aug 13, 2023 at 3:56 PM Aaron Rich wrote: > Hi, > > Not sure if this belongs on the dev or user mailing list but f

Re: Recommended resource limits

2023-08-16 Thread Joe Witt
Eric Try using 2GB for the heap and seeing that helps. I also believe there are specific pod settings youll want to use to avoid it getting nuked by k8s. This blog may give you great things to consider https://home.robusta.dev/blog/kubernetes-memory-limit Thanks On Wed, Aug 16, 2023 at 3:29 PM

Re: Re: Files in /tmp folder

2023-09-01 Thread Joe Witt
Van You might be able to run 'lsof -p ' periodically and figure out who/what is placing that file there. Thanks Joe On Fri, Sep 1, 2023 at 12:53 PM Williams, Van wrote: > Here are the uncommented parts of that file: > > > > > > local-provider > > > org.apache.nifi.controller.sta

Re: NiFi hanging during large sql query

2023-09-02 Thread Joe Witt
Nice. Gald you found it. On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen wrote: > It was the PostgreSQL JDBC driver. If you don't paginate the query > aggressively, it will try to load a significant chunk of the table into > memory rather than just pulling chunks, even with fetchSize set low. > > O

Re: [NIFI] ExecuteScript

2023-09-08 Thread Joe Witt
NiFi 2.0 should get you much further with its Python processor support. Are you able to build main or would you need help with that? We could put a dev build somewhere if needed. Thanks On Fri, Sep 8, 2023 at 4:28 PM Jeremy Dyer wrote: > Rafael is right. This isn’t reasonably possible today. Y

Re: NiFi OpenTelemetry JavaAgent

2023-09-08 Thread Joe Witt
Eric Any sign anything in the nifi app log? Did it perhaps run out of mem and hang? In general such agents should be fine. Thanks On Fri, Sep 8, 2023 at 6:15 PM Eric Secules wrote: > Hello, > > I have tried adding opentelemetry to NiFi by running the OpenTelemetry > javaagent ( > https://ope

Re: Large json string field errors

2023-09-18 Thread Joe Witt
Greg Can you share details on how you're seeing this limit? Is there a resulting stack trace or other output for it in the nifi-app.log? What version of NiFi are you on? Thanks On Mon, Sep 18, 2023 at 11:19 AM Gregory M. Foreman < gfore...@spinnerconsulting.com> wrote: > Hello: > > It appears

Re: Large json string field errors

2023-09-18 Thread Joe Witt
Greg When you mention a flow file with a string field are you saying the 'content' of the flowfile OR are you saying there is a flowfile attribute? The reference to 'field' is throwing me off a bit I think. You might want to share a picture of the relevant portions of the flow and/or the actual

Re: How can I View my flowfile records?

2023-09-24 Thread Joe Witt
Jim, If a content type attribute exists and is not a type NiFi understands it will not be able to render it. Can you show what flowfile attributes are present at the point you attempt to view it? Thanks On Sun, Sep 24, 2023 at 3:03 PM James McMahon wrote: > Hello. I have converted incoming Ex

Re: How can I View my flowfile records?

2023-09-24 Thread Joe Witt
> sourcing.sourceMD5 > b48840c161b645a0169e622dcb8f5083 > sourcing.sourceSHA256 > 4847ac157fd30d6f2e53cb3c4e879ae063d498709da2686c6f61ba6019456afa > triage.datatype > excel > uuid > d72ec2e9-cfbd-435e-9954-4f7fae55c550 > > Thanks for any help. Perhaps my data is there but I

Re: How can I View my flowfile records?

2023-09-25 Thread Joe Witt
pected downstream content viewer >> behaviour. >> >> On Mon, 25 Sept 2023, 06:54 Matt Burgess, wrote: >> >>> I added MIME Type properties to ExecuteProcess and ExecuteStream command >>> so you can set it explicitly if you want [1]. They will be in the 1.24

Re: Analyze : "Diagnostics on Stop" logs

2023-10-28 Thread Joe Witt
Hello We need to better understand what it means to say the node stopped. Did the nifi process die? The nifi-app.log should almost certainly expose something interesting. Is this on a Linux box, or in a container, etc. How is it run and what precisely stopped? Thanks On Sat, Oct 28, 2023 at 2

Re: unsubscribe me

2023-11-01 Thread Joe Witt
Hello Please follow the guidance on https://nifi.apache.org/mailing_lists.html. Primarily for this mailing list that means sending an email to users-unsubscr...@nifi.apache.org Thanks On Wed, Nov 1, 2023 at 1:49 PM McIntyre, Christina < c-christina.mcint...@charter.com> wrote: > I no longer wis

Re: Bug : PutGCSObject : Version NIFI 1.23.2

2023-11-08 Thread Joe Witt
Minh Do you need to override that value? Did Google specify a different URL for you or you have a private connection? If you dont set that value what happens? Thanks On Wed, Nov 8, 2023 at 4:57 AM wrote: > Hello, > > I'm trying to user the overwriting storage url parameter in the nifi > vers

Re: Bug : PutGCSObject : Version NIFI 1.23.2

2023-11-08 Thread Joe Witt
; with the private url and leave empty > "proxy configuration service" > > Regards > > > *Envoyé:* mercredi 8 novembre 2023 à 14:31 > *De:* "Joe Witt" > *À:* users@nifi.apache.org > *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2 > Minh > &g

Re: Bug : PutGCSObject : Version NIFI 1.23.2

2023-11-08 Thread Joe Witt
oxy configuration service" because we are behind the corporate network. >> >> Since some month ago we have configured "private connection" with google. >> So I have setup "storage api url" with the private url and leave empty >> "proxy configu

Re: NiFi custom extension and deploy phase on Maven

2023-12-09 Thread Joe Witt
Etienne, Are you using our nifi poms as your parent pom of your extensions? I do wonder if perhaps we should offer one that doesn't tie to all our ASF-isms Thanks On Sat, Dec 9, 2023 at 10:54 AM Etienne Jouvin wrote: > Hello all; > > Since now, I never push my extensions to any Nexus or Artif

<    6   7   8   9   10   11   12   >