Re: [DISCUSS] Apache NiFi distribution has grown too large

Just an idea, can we also manger Nar, custom Nars in NiFi Repository and let fresh NiFi installers to add required nars and their dependencies on the fly vi NiFi UI ? -Sumanth > On Jan 17, 2018, at 8:05 AM, Matt Burgess wrote: > > I'd like to echo many of the comments / discussion points her

VolatileProvenanceRepository for MiNiFi

Hi, In NiFi , I can set following properties in 'nifi.property'. nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository nifi.provenance.repository.buffer.size=1000 What is the equivalent process for MiNiFi configuration ? -- Thanks, Sumanth

Re: MiNiFi logs not removed as per l maxHistory in ogback.xml

resolved since including newer releases of the logback lib. If you >> would be able to evaluate please let us know if there continue to be issues. >> >>> On May 15, 2017, at 13:43, Sumanth Chinthagunta >> wrote: >>> >>> We have an issue with MiNi

MiNiFi logs not removed as per l maxHistory in ogback.xml

We have an issue with MiNiFi (minifi-0.1.0 Java). It is not rotating its logs as per conf/logback.xml maxHistory value. Over the time, it is filling the drive and leaving many log files on desk. many "minifi-app.log15558502285619089.tmp" files are left on desk. please advice. Thank

How to disable MiNiFi's provenance repository.

How to disable MiNiFi's (java version) provenance repository. Thanks Sent from my iPhone

TailFile Multiple files and Rolling Filename Pattern not working.

Hi I am trying to use TailFile Processor to aggregate logs. my app produce three log files : 1. app-sumo-audit.log 2. app-sumo-api.log 3. app-sumo-logging.log and corresponding rolling files app-sumo-audit.log.2017-03-11.0 app-sumo-audit.log.2017-03-11.1 app-sumo-api.log.2017-03-11.0 app-sumo-api

Re: Odd inter-operation behavior when using clustered NiFi against secure MapR cluster

Andre, Sorry for late replay. Yes we are able to successfully integrate nifi with secure MapR cluster with JAAS method Sumo Sent from my iPhone > On Oct 27, 2016, at 1:38 AM, Andre wrote: > > Sumo, > > Have you tried the workaround? Did it work? > > Cheers > >> On Thu, Oct 27, 2016 at 3:

Re: DB2 to MS SQL data transfer

ile, then > transfer it with NiFi to the target system, then use PutSQL to do a > BULK INSERT [2]. > > Regards, > Matt > > [1] > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.QueryDatabaseTable/index.html > [2] https://msdn.micros

Re: DB2 to MS SQL data transfer

ello Sumo, > > Are you looking at transferring data from a specific table between two > databases, or all tables? > > On Wed, Aug 10, 2016 at 3:17 PM Sumanth Chinthagunta > wrote: > >> I have same scheme in DB2 and SQL server. >> Is there a simple solution to transfer

DB2 to MS SQL data transfer

I have same scheme in DB2 and SQL server. Is there a simple solution to transfer data with NiFi? ( one time ) Hoping for somebody have better idea to build the flow :) Thanks Sumo Sent from my iPhone

Re: DistributedMapCacheClient Groovy example

u have shown. Mind writing an improvement Jira to capture > this? > > Thanks, > Matt > > >> On Jul 17, 2016, at 4:51 AM, Sumanth Chinthagunta wrote: >> >> I had to custom build NiFi to add distributed-cache support for scripting >> processors as des

Re: ConsumeKafka vs getKafka

Done. https://issues.apache.org/jira/browse/NIFI-2298 <https://issues.apache.org/jira/browse/NIFI-2298> Thanks -Sumo > On Jul 17, 2016, at 1:37 PM, Sumanth Chinthagunta wrote: > > Env: Linux , Kafka 9.0.x, NiFi: 0.7 > > We like to use new ConsumeKafka processo

ConsumeKafka vs getKafka

Env: Linux , Kafka 9.0.x, NiFi: 0.7 We like to use new ConsumeKafka processor but found critical limitations compared to old getKafka processor. New ConsumeKafka is not writing critical Kafka attributes i.e., kafka.key, kafka.offset, kafka.partition etc into flowFile attributes. Old getKa

Re: DistributedMapCacheClient Groovy example

apr-build.md#add-optional> Please consider adding this support in next release. -Sumo > On Jul 16, 2016, at 10:17 PM, Sumanth Chinthagunta wrote: > > Hi Matt, > > Did you find any solution to use DistributedMapCacheClientService from > Scripting processors ? > > I

Re: DistributedMapCacheClient Groovy example

t; > I'll give it a try myself to see if I can find a way (using closures > -- but not interface coercion since I can't reference the class) to > achieve this, but otherwise you can use the first blog post from my > last email to write a simple client of your o

Re: DistributedMapCacheClient Groovy example

processor ! Thanks Sumo > On Jul 13, 2016, at 9:12 PM, Sumanth Chinthagunta wrote: > > Matt, > I setup DistributedMapCacheServer controller service and trying to run > following script. Am I doing correct? > I have couple of issues: > 1. How do I make org.apache.nifi.distri

Re: DistributedMapCacheClient Groovy example

post) you can call its methods to > get and put values. > > Regards, > Matt > > On Wed, Jul 13, 2016 at 8:26 PM, Sumanth Chinthagunta > wrote: >> looking for example script ( for ExecuteScript processor) that uses >> DistributedMapCacheClient to put/get key/value pair. >> >> Thanks >> -Sumo

Re: DistributedMapCacheClient Groovy example

call its methods to > get and put values. > > Regards, > Matt > >> On Wed, Jul 13, 2016 at 8:26 PM, Sumanth Chinthagunta >> wrote: >> looking for example script ( for ExecuteScript processor) that uses >> DistributedMapCacheClient to put/get key/value pair. >> >> Thanks >> -Sumo

DistributedMapCacheClient Groovy example

looking for example script ( for ExecuteScript processor) that uses DistributedMapCacheClient to put/get key/value pair. Thanks -Sumo

Re: State sharing

n backed by a ZK is not meant for high speed > concurrent use. Distributed caches are, however. > > Andrew > > On Sun, Jul 10, 2016, 10:58 PM Sumanth Chinthagunta > wrote: > >> Thanks Bryan. >> Would be nice if we get support for state sharing across diffrent &g

Re: State sharing

t >> local to this processor node. >> >> Andrew >> >> On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta > > >> wrote: >> >>> If I set state from one ExecuteScript processor via stateManager , can I >>> access same state from other processor ? >>> Thanks >>> Sumo >>> >>> Sent from my iPhone > > > -- > Sent from Gmail Mobile

Re: State sharing

> > IIRC there's a node one selects when setting state. If you invoke with a > cluster mode, the state will be set to a ZK by default. Otherwise just > local to this processor node. > > Andrew > > On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta > wrote: > >&g

State sharing

If I set state from one ExecuteScript processor via stateManager , can I access same state from other processor ? Thanks Sumo Sent from my iPhone

Re: MergeContent to group related files

I just noticed you set 'tar' so presumably you are using > tar merging strategy and most likely this is to address how to keep these > objects separate and avoid the need for header/foot/demarcator/etc.. Good > choice as well. > > There are a lot of ways to sl

Feature Request: NodeID of NiFi cluster nodes as FlowFile common attribute.

In my NiFi cluster environment, for a given flow, each node in the cluster is producing afinal FlowFile that is written to HDFS. I needed to append NodeID to filename attribute (e.g., node1_filename1.tar) so that files written to HDFS has unique name and I can easily identify which node produce

Re: NiFi error with embedded ZooKeeper for State Management

arameter from bootstrap.conf > - Set the nifi.state.management.embedded.zookeeper.start property to false > - Change the conf/state-management.xml to point to the external ZooKeeper via > the Connect String property. > > Does all of this make sense? > > Thanks > -Mark

NiFi error with embedded ZooKeeper for State Management

I am getting following error when NiFi Cluster started with embedded ZooKeeper enabled. I would like to configure external ZooKeeper for NiFi State Management without interfering with java.security.auth.login.config setting for MapR HDFS. Can somebody provide me instructions how to configur

Scripted controllers

Since we have scripted processors, any plans to support scripted controllers in the roadmap? -sumo

Re: Git NiFi processor

Sure. Sent from my iPhone > On Feb 18, 2016, at 4:12 AM, Joe Witt wrote: > > Sumo, > > Agreed. If interested in helping with the discussion/idea formation > for this can you add to this [1]. > > Looks like [2] gives us a potentially strong option for a Java-based > Git client and [3] shows

Re: [DISCUSS] Proposal for an Apache NiFi sub project - MiNiFi

Good idea. There will be many possibilities if we can make MiNIFi run on android / iOS or other embedded devices. Wonder how back-pressure works in this kind of distributed setup. I was reading about reactivesocket project, This project is trying to solve reactive / back-pressure problem over n

Error: is not the most recent version of this flow file within this session

I am creating a new flowFile in my processor’s onTrigger method and getting following error! Please advice if I am doing anything wrong ….. My requirement is: I need to create a new outgoing flowFile with a set of derived attributes and route to success path. ```groovy public void onTrigger

Re: Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

ed concurrently > (like [S]FTP for instance). > > Matt > > On Fri, Dec 18, 2015 at 7:26 PM, Sumanth Chinthagunta <mailto:xmlk...@gmail.com>> > wrote: > >> Issue 1: >> in my NiFi installation I granted only read access to NIFI_HOME/lib and I >> got thi

Re: Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

f7d-45d7-a1f4-f9e9269b489b/processors/bceff62d-435d-4c47-a907-bb2a26cd0e56]> while in safe mode please let me know if you find anything wrong with my setup. Thanks Sumo > On Dec 18, 2015, at 11:37 AM, Sumanth Chinthagunta wrote: > > Thanks Corey and Mark for quick response. >

Re: Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

that mean that you made a copy of the first directory, or > that you untarred the .tar.gz again? > > It looks like the same nifi.pid file is in the bin/ directory of both > instances. You should be able to delete the nifi.pid file from the node > and then start it up. > > Tha

Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

Hi I am following clustering instructions as per the link below: NCM, Node1 in Server1 and Node2 on Server2. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering I created two copie

Re: [GitHub] nifi pull request: NIFI-1218 upgraded Kafka to 0.9.0.0 client API

Hi Oleg, For my case NiFi (consumer) has to talk to Kafka cluster behind firewall and it was difficult got Kafka brokers and ZooKeeper ports opened. I noticed that 0.9.0.0 support consumers directly communicate to Kafka brokers with `bootstrap.servers` setting. it might be good idea if you con

Re: remote command execution via SSH?

record of it on Jira than just on the > Dev list. > > Joe > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com > > > > > On Wednesday, November 25, 2015 2:12 PM, Sumanth Chinthagunta > wrote: > I have first-cut

Re: remote command execution via SSH?

' } } 3. User supplied Arguments which will be available in Run DSL anything that is assigned to ‘result’ in RunDSL is passed as flowfile to success relationship. Any suggestions for improvements are welcome. -Sumo > On Nov 24, 2015, at 8:19 PM, Adam Taft wrote: > > Sumo, >

bundling shared jars

Sorry if this question may have been answered previously: I am implementing two custom processors (bundled separately as NAR files), both depends on Groovy Jar. instead of bundling Groovy Jars in each NAR, is there a way to share jars that don’t have any processors. Should we consider NiFi su

Re: remote command execution via SSH?

>> Cheers >> Oleg >> >>> On Nov 24, 2015, at 11:51 AM, Joe Witt wrote: >>> >>> Hello Sumo, >>> >>> At present there are no such processors to do this. I know it has >>> been done in the past but that was not in an o

remote command execution via SSH?

Are there any NiFi processors to execute remote commands via SSH? I need to SSH to a remoter server and run a shell script on schedule basses. thinking of using NiFi’s scheduling and argument passing capability. I find this lib can be used, if no such processor exist. https://github.com/int12

Re: EvaluateJsonPath error: Unable to return a scalar value for the expression

expression. In this case, if you change the property to json, it should > work as anticipated. The property itself is more of a validation of the > data that is being extracted (in that it is an object/array or a simple > value). > > On Mon, Nov 16, 2015 at 3:20 PM, Sumanth Chin

EvaluateJsonPath error: Unable to return a scalar value for the expression

I am trying to extract data into attribute using EvaluateJsonPath. when what JsonPath return complex type, I am getting error: Unable to return a scalar value for the expression $['data'] for FlowFile 152. Evaluated value was {id=1…..}. Transferring to failure data - $.data <— Error id

Re: NiFi won't start with my Processor

nd what is happening here. > > Thanks > Joe > > On Wed, Oct 21, 2015 at 8:56 PM, Sumanth Chinthagunta > wrote: > > I noticed similar error when I have my flow using older version of a > custom > > nar and when I deployed new version of nar with package refactor

Re: NiFi won't start with my Processor

I noticed similar error when I have my flow using older version of a custom nar and when I deployed new version of nar with package refactoring. My workaround to bring the server back is to redeploy old nar, start nifi and delete the flow then recreate the flow after deploying refactored nar. Nifi

MonitorActivity bug?

Noticed "copy attributes" is implemented only for REL_ACTIVITY_RESTORED case. I needed original attributes for REL_INACTIVE case as well. wonder why it is limited to activity restored relationship only. https://github.com/apache/nifi/blob/31fba6b3332978ca2f6a1d693f6053d719fb9daa/nifi-nar-bundles

Re: NiFi Authentication Mechanisms

JSON Web Tokens (JWT) can be an option. It will provide claims required for authorization without needing verification with issuer. Auth0.com has more info on this method. JWT can be use to propagate identity along the flow so that it can be used later by processors On Mon, Oct 5, 2015, 7:41 PM

Re: attributes to json

cument from flow file > attributes. Something like... > > Replacement Text = { "myField" : "${attribute1}" } > > -Bryan > > On Saturday, October 3, 2015, Sumanth Chinthagunta > wrote: > >> do we have attributes to json processor ? &g

attributes to json

do we have attributes to json processor ? I am thinking to use it along with ExtractText where the matching data is stored in attributes. now I need to convert those attributes into json flowFile. if we don’t have such processor, any ideas to use, existing processors to compose what I needed? th

Re: need help with Async code

andard/JmsConsumer.java > > <https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java> > > On Sun, Sep 27, 2015 at 3:24 PM, Sumanth Chinthagunta <mailto:xmlk...@gmai

Re: nifi question

d = false; > > @OnScheduled > public void doExpensiveSetup(final ProcessContext context) { >if (!expensiveActionPerformed) { > >// do expensive action > >expensiveActionPerformed = true; > } > } > > > Thanks > -Mark > > >>

Re: nifi question

o have an example to look at. > > Let us know if you have any more questions, or if anything is still not clear! > > Thanks > -Mark > > > [1] > http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#interacting-with-controller-service > > > >

nifi question

Hi All, I am new to NiFi and I'm stuck with couple issues: 1. Unable to get hold of ControllerService from Processor’s init method. I wanted to pre-set some dependencies in my Processor during init phase instead of querying them repeatedly in onTrigger method. I am gettin

need help with Async code

Hi All, I am new to NiFi and I'm stuck with couple issues: 1. Unable to get hold of ControllerService from Processor’s init method. I wanted to pre-set some dependencies during init phase instead of querying them repeatedly in onTrigger method. I am getting null for servi