Re: Apache Nifi Professional?

Sylvain Cloudera provides professional support, https://www.cloudera.com/products/dataflow.html There may be other companies providing support, but have not used their services so can’t say anything about their offerings. Hope this helps, Vijay Chhipa > On Oct 1, 2024, at 09:20, Sylv

Re: Datatype Transformation

Hi Deepanshu, Can you please post the configuration of your UpdateRecord Processor, This is a common use case and I believe this will benefit the community. Thanks Vijay > On Sep 10, 2024, at 04:44, Deepanshu Lodhi wrote: > > Hii Dennis, > Thanks for the reply > > I'll practice your sugge

RedisConnectionPoolService integration with Redis in AWS

Hey NiFi users, Has anyone successfully set up a RedisConnectionPoolService to be used by Put/FetchDistributedMapCache to talk to Redis in AWS ? What does your Redis configuration look like? AWS has a few options for Redis and would like to know how you made it work. Upon searching found a co

Re: ECMAScript support missing from ExecuteScript

versions of > Java don’t include the Nashorn (ECMAScript) library, but I believe we added > it explicitly, perhaps for the 1.20 release (not at computer right now) > > Sent from my iPhone > >> On Jan 16, 2023, at 6:28 PM, Vijay Chhipa wrote: >> >> Hi All, >

Re: Expected mergerecord performance

Hi Richard Have you tried JoltTransformJSON or JoltTransformRecord I believe you should be able to do this Quick start here: https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350 > On Dec 20, 2022, at 4:13 AM, Richard Beare wrote: > >

Re: Migrate NiFi users

Hi HORNEMAN, What I have done is the first option you mentioned, "move the users.xml and authorizations.xml files to the new instances and then add the instances manually” The reason of this is that the authorizations for instances are just a few while authorization for users could be a lot ba

Flowfile disk space is not released from the content-repository until the entire dataflow is completed

Hi all, We have a use case where we list out the contents of a website and then download each item in the list and process it. What I expected is that when each item (a file) is downloaded, after processing is completed, and the flowfile is not in any of the queues the disk storage will be re

Re: Order of upgrading Nifi and Zookeeper clusters

Isha, When we migrated off of 1.9.1 we faced the same issue. You need to upgrade zookeeper to 3.5.x, better yet you should upgrade to 3.6+ as it allows the zookeeper nodes to be started in any order, otherwise they need to be started in the order of their id. Vijay > On Jul 8, 2021,

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

Avinash, Did you try to tune Max Timer Driven Thread : 100 Set this value to (number of cores per node) * (number of nodes) i.e. 48 *13 = 624 https://docs.cloudera.com/cfm/2.1.1/nifi-tuning/topics/cfm-tuning-recommendations.html

Re: Issue when extracting a large dataset into CSV files

Hi Vibhath, There is this property on the processor Max Rows Per Flow File Per docs: If the value specified is zero, then all rows are returned in a single FlowFile. Which seems to be what is happening in your case. > On May 9, 2021, at 12:11 PM, Vibhath Ileperuma > wrote: > > execu

Re: InvokeHTTP hangs after several successful calls

Jeanne, This is not a JDK issue. It is specific to HTTP. You can try to apply the fix for https://issues.apache.org/jira/browse/NIFI-8181 as a patch to your current version. HTH, Vijay > On Apr 14, 2021, at 8:20 AM, jeanne-herndon > wrot

Re: InvokeHTTP hangs after several successful calls

/NIFI-8181> implemented. Regards, Vijay > On Jan 29, 2021, at 1:34 PM, Vijay Chhipa wrote: > > Joe, > > behavior is the same with the 1.12.1 version. > > Mark, > I will make the change to the OkHttp client and try. I do know that this flow > was working

Re: InvokeHTTP hangs after several successful calls

unWorker(java.base@11.0.5/ThreadPoolExecutor.java:1128) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628) >> at java.lang.Thread.run(java.base@11.0.5/Thread.java:834) >> >> This implies that it is either ge

Re: PrometheusReportingTask Metrics

Pavel What problem did you run into? If it is pulling an empty result set, you just need to add the trailing slash. See the solution here: https://stackoverflow.com/questions/63549400/prometheusreportingtask-not-emitting-metrics-in-nifi > On Oct 28, 2020, at 7:18 PM, Pavel S wrote: > > I do

Re: Acks from Kafka not received (PutKafka, PublishKafka)

James, Port 9092 is the only port you need. See this post: https://stackoverflow.com/questions/38531054/kafka-and-firewall-rules Check that the firewall rules allow bidirectional traffic on that port. This is my setup:

Re: 2 Node Cluster High CPU usage

Cenk, Do you see anything interesting in the nifi-app.log or the nifi-bootstrap.log on the 2nd node? You can also set the logging level to DEBUG on the second node to get a better idea. Vijay > On Sep 11, 2020, at 11:58 AM, Cenk Aktas wrote: > > Hello all, > > We have a 2 node cluster s

Re: Securing NiFI behind a proxy (NGINX).

Scotty, When you turn an unsecured cluster to secured one, authentication is turned on. You have to configure one of the authentication mechanism (Kerberos, LDAP, OIDC, ClientCertificates) in your nifi.properties. > On Sep 7, 2020, at 3:28 AM, scotty wrote: > > Hi, > > 1. I had an usecure

PrometheusReportingTask not publishing any metrics

Hello NiFi users, The inclusion of PrometheusReportingTask in NiFi-1.10 + is very exciting. I created this reporting task using the hamburger menu and then chose "All Components" strategy. However when I cURL the publishing endpoint, I don't find any metrics. There is no error, just no outp

Re: In memoriam of Jeff Storck

I waited on pins and needles for the Java 11 support to come out. Little did I know that Jeff was the man behind it. Thanks for all of your efforts Jeff, because of you we were able to meet critical deadlines. RIP Jeff. Vijay > On Jun 16, 2020, at 10:03 AM, Kevin Doran wrote: > > Jeff, y

NIFI-3065 Fine-grained logging in multi-tenant context

Hi Team, This is an old issue from 2019: NIFI-3065 Fine-grained logging in multi-tenant context : http://mail-archives.apache.org/mod_mbox/nifi-issues/201612.mbox/%3cjira.13021713.1479459766000.583688.1482431698...@atlassian.jira%3E This is very useful feature. Do we know if there is any

Re: invokehttp timeout

Michael, Just to rule out timeouts set along the network hops, can you use cURL or Postman to ensure that you can make this call successfully from the host that's running NiFi? Vijay > On Oct 30, 2019, at 1:52 PM, Michael Di Domenico > wrote: > > i'm trying to use invokehttp to post a fil

Re: InvokeHTTP with dynamic query parameters based on previous request response

Devraj, You will need to parse the response JSON produced by the first InvokeHTTP. You can do extract the timestamp you want using the EL syntax. If you only care when you made the previous request and want to store that timestamp somewhere, you can do so by adding a new attribute in the flowf

Re: Getting the value of a variable defined in a processor group

rocessor/StandardProcessContext.java#L87 >> [2] >> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/registry/variable/MutableVariableRegistry.java#L27 >> >> >> Andy LoPr

Getting the value of a variable defined in a processor group

Hello I am writing a new NiFi processor. In the source code I would like to use a variable defined in the NiFi processor group using the UI. Is there a way to get the map of all the variables available to this processor? I am on NiFi 1.9.1 Thanks smime.p7s Description: S/MIME cryptogra

Re: InvokeHTTP POST

Martjin, Can you show what the HTTP error code and the message is? Also, please list all the properties you have set on the InvokeHTTP processor. A screenshot works too, Vijay > On Apr 26, 2019, at 6:27 AM, Martijn Dekkers wrote: > > Hi, > > I am working to implement a flow where we need t

Re: When do I download the older releases of Apache NiFi

Joe > > On Wed, Feb 27, 2019 at 10:11 AM Vijay Chhipa <mailto:vchh...@apple.com>> wrote: > Thanks Bryan and Tomislav. > > Is it possible to put a link to that on the current download page: > https://nifi.apache.org/download.html <https://nifi.apache.org/downlo

Re: When do I download the older releases of Apache NiFi

n find older releases: > https://archive.apache.org/dist/nifi/ <https://archive.apache.org/dist/nifi/> > > BR, > Tom > > On Wed, 27 Feb 2019, 18:52 Vijay Chhipa, <mailto:vchh...@apple.com>> wrote: > Hi, > > Noticed that with the release of NiFi 1.

When do I download the older releases of Apache NiFi

Hi, Noticed that with the release of NiFi 1.9.0 there is not a link to download version 1.7.1 and older, Is there a link to the archived versions? Thanks smime.p7s Description: S/MIME cryptographic signature

Re: running multiple commands from a single ExecuteStreamCommand processor

you to pipe the output of the first command > to the input of the second. Does that make sense? > > Thanks > -Mark > > > >> On Feb 13, 2019, at 3:26 PM, Vijay Chhipa wrote: >> >> Mark, >> >> Thanks for your quick response, >> When ca

Re: running multiple commands from a single ExecuteStreamCommand processor

script that executes the desired > commands and invoke > that from the processor. Or, of course, you can chain together multiple > processors. > > Thanks > -Mark > > >> On Feb 13, 2019, at 1:48 PM, Vijay Chhipa wrote: >> >> Hi, >>

running multiple commands from a single ExecuteStreamCommand processor

Hi, I have a ExecuteStreamCommand processor running a single command, (executing a -jar ) and it runs fine, I need to run the same command but with different arguments. My question is: Can I put multiple lines as command arguments and still have a single instance of the ExecuteStreamComm

Re: How to remove lines from a flow file that don't start with a certain prefix?

cient. > > Thanks > -Mark > > Sent from my iPhone > > On Dec 29, 2018, at 7:56 AM, Vijay Chhipa <mailto:vchh...@apple.com>> wrote: > >> Hi All, >> >> I have an output from a processor that contains lines that are JSON >> structured, i.e. e

Re: Unable to List Queue on a connection

...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Dec 17, 2018, at 2:45 PM, Vijay Chhipa > <mailto:vchh...@apple.com>> wrote: >> >> Hi >> >&g

Unable to List Queue on a connection

Hi I have a secure NiFi instance setup. I put my self in the Administrator group and created policies for the administrator. When I right click on a connection and click on the List Queue, I get the following error: Insufficient Permissions: No Applicable policies could be foun

Re: Can a nar bundle host a servlet outside of the NiFi web framework?

, Vijay > On Nov 6, 2018, at 4:35 PM, Vijay Chhipa wrote: > > Bryan > > Thats perfect, I was planning to launch my own Jetty instance from within the > nar, but ListenHttp or HandleHttpRequest is even better > > Thank you, > > >> On Nov 6, 2018, at 4:2

Re: Can a nar bundle host a servlet outside of the NiFi web framework?

uest gets turned into a flow file and be processed by > the rest of the flow, and a response can be sent with > HandleHttpResponse. > On Tue, Nov 6, 2018 at 4:52 PM Vijay Chhipa wrote: >> >> Hello >> >> I would like to create a nar bundle that can serve up some co

Can a nar bundle host a servlet outside of the NiFi web framework?

Hello I would like to create a nar bundle that can serve up some content, I would like this to be outside of the secure NiFI web context. Essentially it has nothing to do with the "https://:/nifi" and the nifi.properties This would be a unsecured endpoint accessible from the host thats

Cluster setup in multiple data centers

Hello We are putting NiFi in production. To ensure that there is no single point of failure we have two options. a) Setup a single NiFi cluster with nodes distributed across data centers. b) Setup a cluster in one data center and for failover have a hot stand-by cluster in another data cente

How do I logout of NiFi UI

Hello I can't find the 'logout' link on the canvas or under my profile name or in the hamburger menu. Whats the recommended way to logout of Apache NiFi Can you please point me to this link on a screenshot or if its somewhere in the docs a pointer to that is highly appreciated. Thanks

SAML based identity provider

Hello, I am setting up NiFi in the company, but the out-of-the-box authentication modules are not an option for me. I would like to write a SAML based login identity provider, Is there one out there already ? I am on NiFi 1.7.1, with Java 8, SAML 2.0, What do I need to get started with writin

Re: Send data to ElasticSearch

For value of the date field, ensure that you have 'Z' at the end to indicate the timezone, even if your timezone is UTC. I ran into a similar issue working with ES HTTP API from Python > On Jul 16, 2018, at 1:59 PM, Mike Thomsen wrote: > > ISO8601 smime.p7s Description: S/MIME cryptograp