Re: Providing Config and Attributes to NiFi

James, In NiFi 1.x line we introduced the variable registry concept for exactly this purpose. Right now it reads from a properties file but we intend to also grow this to support other more dynamic/central options. https://issues.apache.org/jira/browse/NIFI-2208

Re: Providing Config and Attributes to NiFi

pecification document in the wiki is written in future tense, but is it > in fact what got implemented in the two jiras (all linked below)? > Thanks, > --Matt > > On 3/24/17, 5:50 AM, "Joe Witt" <joe.w...@gmail.com> wrote: > > James, > > In Ni

Re: Cannot Restart Nifi

you disagree. -Jim > > On Tue, Mar 28, 2017 at 9:37 AM, Joe Witt <joe.w...@gmail.com> wrote: >> >> jim - definitely take the time to walk through the best practices >> guide. Some are more like "if you dont do this it will probably kill >> the process - pr

Re: Cannot Restart Nifi

false in your nifi.properties. >>> Depending on how your flow and scripts are constructed, this may allow you >>> to piecewise alleviate any large queues/processing of files that could be >>> causing the issue at hand. You could additionally bypass the possible >

Re: Access Denied after initial setup

thanks for following up with the resolution! On Tue, Mar 28, 2017 at 8:43 AM, Bram vd Klinkenberg wrote: > Solved it! > > > I used CN=admin with the toolkit and cn=admin in the users.xml Changed > this to CN=admin and works fine now :). > > > > >

Re: Providing Config and Attributes to NiFi

> Hope this helps, > > Russ > > On 03/24/2017 11:07 AM, Joe Witt wrote: > > We made our initial capability available but we still have more work > to go to reach the full vision of that feature proposal/vision as well > as related visions for extensions and flow versions.

Re: NiFi 1.2?

Dan No but we really do need to push it as many folks have asked. I'll try my best to start swirling the release management process soon. Thanks Joe On Mar 22, 2017 6:34 PM, "dan young" wrote: Any update on when 1.2 might be released into the wild? Regards, Dano

Re: NIFI 1.1.1 fails to start

o > monitor it's upper bound so that stuff doesn't get corrupted when the disk > fills up. > > Thanks, > Mike. > > On Tue, Apr 4, 2017 at 3:20 PM, Joe Witt <joe.w...@gmail.com> wrote: > >> Mike >> >> It looks like the flow file repo has partial w

Re: NIFI 1.1.1 fails to start

Mike It looks like the flow file repo has partial written data at the end and is problematic on restart. Strongly recommend configuring per best practices and making sure the flow file repo in particular is in its own logical partition. Also, we can make that more resilient on startup and there

Re: Logging changes to workflow

t;. > Coincidentally we are having to meet similar requirement. > > Our solution was to get that through restAPI but if there is a cleaner way > though reporting tasks, would like to experiment with that too. > > On Mon, Apr 10, 2017 at 9:09 AM Joe Witt <joe.w...@gmail.com> wro

Re: Logging changes to workflow

; What is meant by the following statement: "From that one can access the list > of changes from a given offset onward." > ReportingTask.java is an interface class. That does mean there exists a class > specifically for reporting configuration changes? > > Regards John > &g

Re: Help with SFTP processor

definitely agree with Juan's suggestion to get more details on what the actual authentication process is when trying ssh -vvv. Also, be sure to check what order of authorization occurs. It is possible it is trying keyboard-interactive before the certs and this could create problems so ordering

Re: Avoiding Forced Refreshes that Result in Lost Changes

Hello This has been addressed in NiFi 1.x. You can have multiple writers to various things in parallel and it handles it nicely. Thanks Joe On Wed, Apr 12, 2017 at 12:03 PM, James McMahon wrote: > We have multiple people working is separate distinct processor groups

Re: Help with SFTP processor

pdate the documentation to > make it a little clearer? Let me know and I will take a crack at expandign > the information to help other users. > > Thanks. > > Jim K. > > On Tue, Apr 4, 2017 at 12:31 PM Joe Witt <joe.w...@gmail.com> wrote: >> >> defi

Re: Help with SFTP processor

at 5:08 PM, James Keeney <nextves...@gmail.com> wrote: > Thanks again. When you refer to live queue listing and data viewing what are > you referring to? The dashboard or something else. > > Jim K. > > On Thu, Apr 6, 2017 at 4:49 PM Joe Witt <joe.w...@gmail.com> wrote: &

Re: Options for increasing performance?

Much of this goodness can be found in the help->Users Guide. Adjusting run durection/scheduling factors: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#scheduling-tab These are the latest docs but I'm sure there is coverage in the older stuff. Thanks On Wed, Apr 5, 2017 at 2:23

Re: Options for increasing performance?

There are a number of things you can do for a given flow to improve its performance: - Add tasks to the component(s) - Adjust scheduling period - Add threads to the controller overall - Adjust the run duration to tell nifi you are willing to trade small latency for higher throughput Be sure your

Re: Logging changes to workflow

John To build on Andrew's comments there is already a very significant amount of audit data made available for flow changes. We store this in an audit database so we can drive some user experiences off it and we make it available via reporting tasks so one could report this information to an

Re: NiFi failed to restart after power outage on server

Hello Did the partition nifi use get full? We are making the error handling in such case more resilient as at time it can result in partial writes. Joe Thanks Joe On Apr 14, 2017 5:06 PM, "Ravi Papisetti (rpapiset)" wrote: > Hi, > > > > NiFi version: 1.1.2, Linux > > > >

Re: NiFi failed to restart after power outage on server

s, > > Ravi Papisetti > > > > From: Joe Witt <joe.w...@gmail.com> > Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> > Date: Friday, April 14, 2017 at 5:48 PM > To: "users@nifi.apache.org" <users@nifi.apache.org> > Subject: Re: NiFi fa

Re: Data flow best practise

Brian, This is a tough question to respond to meaningfully as it is pretty open ended. You're right that often different flows created by different people will varying levels of optimization, efficiency, simplicity, and so on much like written code. One of the aspects that helps here is the

Re: Netflow parser

he logstash processing work to nifi, we want to move the >>> netflow parsing to nifi too. >>> >>> https://github.com/logstash-plugins/logstash-codec-netflow >>> >>> http://blogs.cisco.com/security/step-by-step-setup-of-elk-for-netflow-analytics >>> >

Re: ERRORs when trying to make new RPGs

Hello It appears to be throwing an NPE when establishing the position of the component you're trying to add. While clearly an unfriendly error message as-is did you specify the position in your request? Thanks JOe On Wed, Apr 19, 2017 at 10:36 AM, Pushkara R wrote: >

Re: Decompressing content consumed from Kafka topics?

Hello UnpackContent is for archive formats. gzip/snappy/lz4 aren't archive formats but are compression types. You want CompressContent in decompress mode. I think the first question is why they didn't decompress. Did you verify the were indeed still compressed? You can use IdentifyMimeType

Re: Provenance Event performance

You're right that the generation and indexing of provenance data creates overhead. We've put considerable effort in minimizing that overhead to a point where you should not have to think about it and still get all the powerful user experience/auditing gains it provides. However, when you're

Re: Provenance Event Use Cases?

We also use it for troubleshooting as we can see the nifi attributes ( > metadata ) and its content ( if configured ) > > You can also use provenance to "replay" your data at specific points during > its dataflow life cycle. > > Please reference similar answer given on s

Re: tag processors with deprecated

I think this is an important concept. We should have something meaning deprecated but with redirectors to better options. We have 'SeeAlso' for docs so something similar. On Thu, Mar 9, 2017 at 4:28 PM, Juan Sequeiros wrote: > > Hi all, > > Lately I've been discussing

Re: How to recover from a full provenance repository?

in I understood clearly: I should go ahead and rm -rf > ./provenance_repository? Won't the problem just reappear again as soon as it > recreates provenance_repository at start up? > > On Mon, Mar 13, 2017 at 5:04 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >> Jim >>

Re: How to recover from a full provenance repository?

Actually in following the referenced PR to the JIRA it references it appears it went out in Apache NiFi 0.7.1 as https://issues.apache.org/jira/browse/NIFI-2429. So if you can get onto that version it should address that condition. Thanks Joe On Mon, Mar 13, 2017 at 3:20 PM, Joe Witt <jo

Re: How to recover from a full provenance repository?

Jim, Processors create provenance events which get written to the provenenance repo but that is the end of their relationship. The provenance repository is then self managing. Can you share your nifi.properties provenance configuration? Thanks Joe On Mon, Mar 13, 2017 at 1:05 PM, James

Re: Merge join processor for nifi

Purbon The MergeContent processor supports merging data using a correlation key. The processor does not support SQL style inner/outer joins as it is not joining data based on per record/tuple type logic but rather at a per flowfile level. We can correlated based on any flowfile attribute or

Re: RouteText / Replcace Text to remove first line of file

Juan, I think RouteText is the right answer. It would indeed need to check all lines to determine whether the condition is satisfied and to remove the first line it will need to write out all the remaining lines. If a majority of the input files do not have this problematic header I'd use

Re: Re: Re: Re: new Nifi Processors

Uwe To progress toward a pull request for inclusion in the nifi community we will need a LICENSE/NOTICE to end up in the nar bundle itself. You can see many examples of this in other nars such as [1] If you don't need to edit the LICENSE you can not provide it and a default one will be

Re: HandleHttpRequest API Port Exposure

Ryan, We had a very similar question just a bit ago. https://lists.apache.org/thread.html/1289510c6d69176907273be08f1d1548182af6bdec0c4f51dee1c38b@%3Cdev.nifi.apache.org%3E Thanks Joe On Thu, Mar 2, 2017 at 3:32 PM, Ryan H wrote: > Hello All, > > I am

Re: Help with SFTP processor

t is getting created is /www/files/newfile.txt and not > /www/files/test/newfile.txt > > Any suggestions? > > Jim K. > > > On Thu, Apr 6, 2017 at 12:19 PM Joe Witt <joe.w...@gmail.com> wrote: >> >> Jim, >> >> Glad you've made prog

Re: Latency v Throughput

hink about it might be a bit off base. :) > > On Fri, Apr 7, 2017 at 2:31 PM Joe Witt <joe.w...@gmail.com> wrote: >> >> The concept of run duration there is one of the ways we allow users to >> hint to the framework what their preference is. In general all users >&

Re: Latency v Throughput

The concept of run duration there is one of the ways we allow users to hint to the framework what their preference is. In general all users want the thing to 'go fast'. But what 'fast' means for you is throughput and what fast means for someone else is low latency. What this really means under

Re: NIFI 1.1.1 fails to start

Mike Just to follow up it looks like the issue you hit and another related one was just addressed. https://issues.apache.org/jira/browse/NIFI-3678 Thanks Joe On Apr 4, 2017 8:30 PM, "Joe Witt" <joe.w...@gmail.com> wrote: > It agree it should. For content it does. > &g

Re: Options for increasing performance?

If you're seeing 'transfer relationship not specified' it means one or more of the flowfiles you've pulled have not been acted upon for transfer meaning you have a logic/code flow path whereby a flowfile can be pulled from the queue but never transferred. Thanks On Fri, Apr 7, 2017 at 12:57 PM,

Re: FW: Content repository filling up

Olav, How large is your content repository? How large is a large file? How many transformation steps exist in your flow from receipt through delivery of that large file? Thanks Joe On Mon, Apr 24, 2017 at 9:31 PM, Olav Jordens < olav.jord...@2degreesmobile.co.nz> wrote: > Apologies – forgot

Re: Content repository filling up

Olav Your understanding sounds spot on to me and your settings seem quite fine for what you're trying to do. What version of NiFi are you working with? Thanks Joe On Mon, Apr 24, 2017 at 9:26 PM, Olav Jordens < olav.jord...@2degreesmobile.co.nz> wrote: > Hi Users, > > > > I have had this

Re: sftp put processor

Brett I think originally it was because the sftp client we used didn't support jks. One could probably bridge that to something it can support but not sure. Please feel free to file a jira. Using a CS for such things could be useful too. Thanks On Jul 28, 2017 7:53 AM, "Brett Tiplitz"

Re: Moving nifi dataflows through environments

Brian, There have been issues in the past with regard to template importing and it causing duplicate controller service entries. But these should be much improved. What version are you using? Also, the registry component that is now being worked on should help this quite a bit. The

Re: Problem with hanging processor in wifi

Hello Please do a stack dump by running 'bin/nifi.sh dump' then share the contents of the nifi-bootstrap.log file. Thanks Joe On Thu, Jul 13, 2017 at 12:36 PM, Jin Park wrote: > I have GetAzureEventHub I am working to setup. For testing purposes, I > have ran the job then

Re: Safeguarding against List/Fetch of partial files

Jim, Ultimately this comes down to whether any consuming process (not just NiFi) can reliably know that a given file is 'ready to be consumed'. If the writer of those files offers no 'protocol' by which you can know then unfortunately it is about 'having a pretty good guess' that they're done.

Re: ElasticSearch and record API use case

Mike, The record reader/writers do support hierarchical data models and the RecordPath expressions allow you to refer to nested elements similar to XPath for XML. There is a PutElasticsearchHttpRecord [1] [1]

Re: Processor using Kerberos keytab auth - can't renew TGT

James, I know Jeff Storck has recently been doing some work around Kerberos/TGT renewal. Hopefully he can share some of his observations/work back on this thread soon. Thanks On Fri, Jul 14, 2017 at 8:48 AM, James Srinivasan wrote: > Hi all, > > I have a NiFi

Re: Problem with hanging processor in wifi

something is still running when in the problematic condition? Also, is this thread dump definitely from when there was a hung thread/hung process? Thanks Joe On Mon, Jul 17, 2017 at 1:57 PM, Jin Park <jinp...@payoff.com> wrote: > Here is the dump for wifi… > > > > On Jul 13,

Re: Need to view all the flow files in the queue via list queue option

Manojkumar The support is for viewing the top 100 items. It isn't meant to provide exhaustive access or pagination but rather to see the top elements so that one could look at the items, understand how they're ordered, look at their attributes, click to content, etc.. It offers a live look into

Re: MergeContent Inquiry

Yep. Very common. Set the desired size or number of object targets and set the 'Max Bin Age' so that it will kick out whatever you've got by that time. On Thu, Jul 20, 2017 at 7:38 AM, James McMahon wrote: > Good morning. I have a situation where I have a staging

Re: Application is no longer processing files

Good point Pierre. Could also be the api to list directories that have so many files in them. I recommend trying on a directory with less things to list and see if that helps too as that means we need to evaluate other things. Also, we clearly could make it easier for users to see what is going

Re: Jetty failure to start: org.jasypt.exceptions.EncryptionOperationNotPossibleException

Has the version of java being used changed by chance on the system? And if so, or perhaps even if not, were the JCE extensions installed/configured previously and now it is not? Other than that the only other thing that comes to mind is if the sensitive properties key was changed On Wed, Jul 26,

Re: NiFi UI Not Starting

Dan - are you able to share the nifi-app and nifi-bootstrap logs? Thanks On Wed, Jul 26, 2017 at 9:21 AM, Dan Morris wrote: > Hello, > > > > I’m having an issue getting NiFi to start correctly. Here’s my situation: > > I’m currently running v0.7.0. > I have a production

Re: Application is no longer processing files

p. > > Thanks all for your input and quick responses to assist me on this matter. > > > On 26 July 2017 at 16:27, Joe Witt <joe.w...@gmail.com> wrote: >> >> Good point Pierre. Could also be the api to list directories that >> have so many files in them. I recommend

Re: NiFi UI Not Starting

5B 2F7D EF69 > > On Jul 26, 2017, at 10:44 AM, Dan Morris <dgmorri...@gmail.com> wrote: > > Hi Joe, > > I’ve attached relevant files… tried to redact sensitive info… hope I didn’t > cut too much from the logs… > > Thanks, > Dan > > > > On 7/26/17, 9:30

user question on nifi support for large data set handling

Paul Nahay on the wiki asked the following. I've directed him here so we can discuss as appropriate on the mailing lists. What is the current state, and future plans, for NiFi transferring large data sets? How large can data attached to FlowFiles be, currently? Whatever the answer to #2 above

Re: Does PutFile block subsequent files on failure for a flowfile?

...but in the meantime it can process others... On Jun 29, 2017 8:40 AM, "Pierre Villard" wrote: > Hi Jim, > > That's correct. When going in the failure relationship, the flow files are > penalized with a duration of 30s (by default, this can be changed in the >

Re: Jython in 1.3 ExecuteScript config?

To understand jython/python relationship [1] To understand example usage of the processor a bit more [2, 3, 4]. [1] http://www.jython.org/archive/21/docs/whatis.html [2] https://community.hortonworks.com/articles/75032/executescript-cookbook-part-1.html [3]

Re: Issues with debugging

no worries tim. this is a good place to ask and I guarantee this thread will help someone else later. On Thu, Apr 27, 2017 at 10:30 AM, Tim Zimmerman wrote: > Bryan, > > Thanks for the quick response. Yes I thought it quite strange myself. I > was trying to debug a

Re: FW: Content repository filling up

while the larger files are processed. > > The surprising thing for me is that even with nifi.content.claim.max.flow. > files=1 > > I still see multiple pieces of content within a single file in the content > repository. Some are small bits os sql / json, and others are huge 1GB text >

Re: FW: Content repository filling up

ther than a rule, does it make sense to > reduce the max.append.size from 10MB down to say 1 KB? > > > > Thanks, > > Olav > > > > > > *From:* Joe Witt [mailto:joe.w...@gmail.com] > *Sent:* Tuesday, 25 April 2017 2:16 p.m. > > *To:* users@nifi.apache.org &

RE: [EXT] NiFi 1.3: Simplest way possible of creating CSV files from SQL queries

There are some great points here. Am on a phone right now so will be very brief. The current 1.4.0 snapshot on master has some nice improvements for ease of use with record handling. This includes writers inheriting schema from the reader and others. Thanks On Aug 1, 2017 1:20 AM, "Peter

Re: *Kafka processors

Lauren, The PublishKafka processor series is where we really dug into the details of the Kafka API and build the processors in a manner far more aligned to the Kafka style. They work well. The PublishKafkaRecord processor though is *really* nice in terms of performance and it means it is part

Re: provenance content

Frank There are three repositories. - FlowFile (tracks metadata about things actively in the flow) - Content (holds the actual underlying content for things in the flow or still in archive) - Provenance (holds metadata about the events that went through the flow) Provenance and content

Re: Archive selected "feeds"

The concept of being able to provide hints to the underlying content repository about which content is more important to keep longer than others is a good one. I'd recommend filing a JIRA for the ask and providing your proposed concept as one to consider. Definitely see something like this being

Re: sFTP processor not working well when fetching big file

Hello Can you please clarify the processor(s) you're using. It is the ListSftp and FetchSftp processors? Or is it GetSFTP? For the processor to send the data to the success relationship it should mean that the transfer session reports a successful conclusion. Can you show us the logs for the

Re: Storage buffer separation in a multi-tenant cluster.

Kris, Now that five months have passed on this thread I suppose it is time to reply :-) Sorry about the delay To the basis of the question we do not provide in Apache NiFi today any built-in or integrated quota management for the various tenants of a given system. The multi-tenancy today is

Re: how to set version for NIFI customized Processor?

Hello They will remain unversioned until they are built with the latest nar plugin. This is described briefly in the migration guidance [1]. For anything built with the older nifi nar plugin the resulting nar will not have suffiicient bundle data for the framework to have it support component

Re: Extract data from Microsoft Access with NIFI Processors

Ben If you can get a hold of a jdbc/odbc driver then you might be able to use existing processors and just point at the jdbc jar. Otherwise, you might need to write one. Please let us know which path you go. Thanks On Wed, Jun 21, 2017 at 11:01 PM, 尹文才 wrote: > Hi guys,

Re: issues with Kafka ConsumeKafka_0_10 (1.0.0)

e > new version will likely help us out a lot. We do not plan on getting this > far behind again. > > [image: Payoff, Inc.] > *Jeremy Farbota* > Software Engineer, Data > Payoff, Inc. > > jfarb...@payoff.com > (217) 898-8110 <+2178988110> > > On Fri, Jun

Re: issues with Kafka ConsumeKafka_0_10 (1.0.0)

Jeremy It is possible that backpressure was being engaged in NiFi and causing our consumer code to handle it poorly. We did fix that a while ago and I think it ended up in NiFi 1.2.0 (off top of mind anyway). Between your current release and the latest 1.3.0 release a few bugs with those

Re: Nifi clusters : duplicate nodes shown in cluster overview

That is correct. On May 19, 2017 9:47 AM, "ddewaele" wrote: > We're using docker, and in our failover scenario the machine is rebooted > and/or the docker system is restarted. > > We're currently volume mapping the following : > > -

Re: Nifi Cluster fails to disconnect node when node was killed

Thanks, > Neil > > On Thu, May 18, 2017 at 9:56 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >> Neil, >> >> Want to make sure I understand what you're saying. What are stating >> is a single point of failure? >> >> Thanks >> Joe >

Re: Nifi clusters : duplicate nodes shown in cluster overview

When a node joins a cluster it writes its node identifier into its local state. This is in a state directory. Is that directory being removed? If so it will get a new identifier. Otherwise when the node is started it will reuse that identifier. On May 19, 2017 4:13 AM, "ddewaele"

Re: use DistributedMapCacheClientService in NIFI cluster

down on that node(or the node > itself goes down), does it support HA? > > > 2017-05-25 20:46 GMT+08:00 Joe Witt <joe.w...@gmail.com>: >> >> Hello >> >> You put the DistributedMapCacheServer controller service on as well >> and then point at it from t

Re: NiFi will not restart, missing status file message in bootstrap

_STOPPED >>> [date] [time],### INFO [main] org.apache.nifi.bootstrap.RunNiFi >>> Registered no Notification Services for Notification Type NIFI_DIED >>> [date] [time],### INFO [main] org.apache.nifi.bootstrap.Command Apache >>> NiFi is not running >>> >

Re: NiFi will not restart, missing status file message in bootstrap

vement with 1.2.0 sounds like it will make a big difference. Sadly > as you recall it may be some time before 1.2.x is available to me. > > On Thu, May 25, 2017 at 11:05 AM, Joe Witt <joe.w...@gmail.com> wrote: >> >> jim >> >> that provenance warning is not

Re: NiFi will not restart, missing status file message in bootstrap

Jim, The code relevant to that log output is here [1]. Can you share the bootstrap output before/after that output? [1] https://github.com/apache/nifi/blob/rel/nifi-0.7.1/nifi-bootstrap/src/main/java/org/apache/nifi/bootstrap/RunNiFi.java Thanks Joe On Thu, May 25, 2017 at 9:11 AM, James

Re: use DistributedMapCacheClientService in NIFI cluster

Hello You put the DistributedMapCacheServer controller service on as well and then point at it from the client services. So in your three nodes all three will have the server service but on all the clients they'll only point to the server service on one of the nodes. Thanks Joe On Thu, May 25,

Re: NiFi will not restart, missing status file message in bootstrap

a situation, and whether there is a more refined way to solve > it than full system reboot. There are other things running on the resource > and I should try to minimize impact to them by fully rebooting. > > Let me see about that log content. Thank you again. > > On Thu, May 25, 2017 at 9:

Re: Nifi Cluster fails to disconnect node when node was killed

odified and/or gracefully shutdown (afaik) is > what I was referring to. > > On Fri, May 19, 2017 at 12:40 PM, Joe Witt <joe.w...@gmail.com> wrote: > >> I believe at the state you describe that down node is now considered >> disconnected. The cluster behavior pr

Re: Nifi Cluster fails to disconnect node when node was killed

t can't be reported on either, i.e. some GETs fail, like > nifi-api/flow/process-groups/root. > > Are we describing the same behavior? > > On Fri, May 19, 2017 at 11:12 AM, Joe Witt <joe.w...@gmail.com> wrote: > >> If there is no longer a quorum then we cannot drive thing

Re: Nifi Cluster fails to disconnect node when node was killed

Neil, Want to make sure I understand what you're saying. What are stating is a single point of failure? Thanks Joe On Thu, May 18, 2017 at 5:27 PM, Neil Derraugh wrote: > Thanks for the insight Matt. > > It's a disaster recovery issue. It's not something

Re: some processors runs only once in NiFi

prabhu - can you please share screenshots and or logs showing that it is running only once? Thanks On Wed, May 24, 2017 at 12:42 AM, prabhu Mahendran wrote: > Aldrin, > > Thanks for your response. > > For GetHTTP ,I have checked to download different files even it could

Re: difficulty killing NiFi w/ nifi.sh stop under 1.2.0 now vs. 1.1.2 when run as systemd service

I'm not an expert in this arena but there a couple of JIRAs that may be relevant which are targeted for the NiFi 1.3.0 release. https://issues.apache.org/jira/browse/NIFI-3112 https://issues.apache.org/jira/browse/NIFI-2199 On Tue, May 30, 2017 at 10:17 PM, Jeremy Taylor

Re: Questions about record processors

Mika And yes to question one. What you're doing us fine. There will also soon be an option for the writer to just infer its schema from the reader which may be useful in simplifying the config more. Glad you are finding these useful. If you'd like to contribute a sample flow that does this we

Re: NiFi relaunching every few minutes

Is there anything interesting in the nifi-bootstrap.log? On May 5, 2017 6:52 AM, "Janaki Joshi" wrote: > Hello all, > > I've been using NiFi for a while now. Since last evening I've started > facing an issue that I am unable to get to the root of. My NiFi (Deployed > on an

Re: Is it possible to reference python requests module in ExecuteScript?

It is worth discussing whether there is sufficient value to warrant keeping jython/python support in the processors or whether we should pull it. It is certainly something we can document as being highly limited but we don't really know how limited. Frankly given the performance I've seen with

Re: Quickly find queues with data...

Go here: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Summary_Page It shows you how to get the summary view. Click on connections. Sort on the column of interest. Thanks On Fri, May 5, 2017 at 5:35 PM, Russell Bateman wrote: > Is there some NiFi

Re: Apache nifi 1.0.0 consuming high CPU utilization

Kunal, Solution to what though? Are you trying to minimize CPU usage in general or are you not hitting the latencies you need? You can lengthen the bored yield duration in nifi.properties and still get solid performance. Joe On Tue, May 2, 2017 at 9:15 AM, kunal

Re: how to set version for NIFI customized Processor?

the version in that processor selection dialog? > > 2017-05-18 10:42 GMT+08:00 Joe Witt <joe.w...@gmail.com>: >> >> Hello >> >> They will remain unversioned until they are built with the latest nar >> plugin. This is described briefly in the migration guidance [1]. &

Re: How to lock getfile upto putfile write into same file?

PutFile while writing automatically writes with a dot prepended and once the write is complete it removes the dot. If you have multiple instances of PutFile writing to the same file with the intent of appending to that file it will like fail in strange ways. I would recommend you simply build up

Re: is it possible for a NIFI processor to be both timer driven and event driven?

Hello It is certainly possible to have a process act as a listening daemon and you can see this in various Listen processors. Basically once they are scheduled they open up some port under some protocol and listen for incoming data. The custom processor you're mentioning can spawn a thread to

Re: Create a CovertCSVToJSON Processor

Venkat I think you'll want to take a closer look at the apache nifi 1.2.0 release support for record readers and record writers. It handles schema aware parsing/transformation and more for things like csv, json, avro, can be easily extended, and supports scripted readers and writers written

Re: Encryption and Hashing in Nifi

Raman, There are no out of the box apache nifi processors to encrypt specific fields of a given record. With the record reader/writer capabilities that came out in 1.3.0 we can probably much more easily provide one but in the mean time it is a pretty straight forward extension to build. You

Re: PutEmail with Multiple Attachements

Mika, Could it be ok for your case to use MergeContent to package data together in some archive format like Zip/Tar? If yes then you can just add a tar/zip to your email. Otherwise, the changes necessary to PutEmail for supporting multiple attachments would be quite involved. Thanks On Wed,

Re: PutTCP connector not cleaning up dangling connections

nelSender.java#L45-L78 > > Should be doing something like this where it catches, closes, and rethrows: > > https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/remote/io/socket/ssl/SSLSocketChannel.java#L140-L179 > > > On Mo

Re: PutTCP connector not cleaning up dangling connections

his processor and all of them end up getting closed correctly. > > So the "hanging" connection originated from the built-in PutTCP processor. > > > Joe Witt wrote >> If you stop the processor manually does it clean them up? >> >> When the connections

Re: PutTCP connector not cleaning up dangling connections

Davy If you could give the PR a try and see if it helps I'd be happy to help get it reviewed and in for 1.4 if timing works out. Thanks On Mon, Sep 18, 2017 at 4:39 PM, Bryan Bende wrote: > Davy, > > I just pushed a second commit to the PR that will log the port from > the

Re: New ListFTP (and possibly ListXXX) processors fail to pickup new files

causing my primary node to shift, or why the processors are unable to > recover when the node shifts. If I find any more info I'll raise a ticket. > > On Thu, Sep 14, 2017 at 8:08 AM, Joe Witt <joe.w...@gmail.com> wrote: >> >> Thanks Gino. If you can confirm/replicate

<    1   2   3   4   5   6   7   8   9   10   >