Re: [EXT] [discuss] release apache nifi 1.9.0
Hoping I am not too late to the party, but it would be fantastic if https://github.com/apache/nifi/pull/3253 could have the comments on it addressed and included the 1.9 release -Charlie On Wed, Feb 6, 2019 at 9:08 PM Joe Witt wrote: > MarkB - i dont have any issue with that being include but we need to find > someone familiar enough with that to review it. If that happens in the > next couple days then great. Any takers familiar with JMS and that class? > The changes are few but the implications are unclear to me. > > Thanks > > On Wed, Feb 6, 2019 at 9:55 PM Mark Bean wrote: > > > I have an open PR I'd like included in 1.9. It's a very simple one. > > > > https://github.com/apache/nifi/pull/3053 > > > > Thanks, > > Mark > > > > On Wed, Feb 6, 2019 at 7:43 PM Joe Witt wrote: > > > > > i have that one ready to go andy. ill take care of the dev docs one. > > > > > > thanks > > > > > > On Wed, Feb 6, 2019, 7:24 PM Andy LoPresto wrote: > > > > > > > I also want to merge https://github.com/apache/nifi/pull/3283 < > > > > https://github.com/apache/nifi/pull/3283> so we can get the > Developer > > > > Guide more visibility in the docs. Will do today. > > > > > > > > Andy LoPresto > > > > alopre...@apache.org > > > > alopresto.apa...@gmail.com > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > > > > > On Feb 7, 2019, at 10:12 AM, Ed B wrote: > > > > > > > > > > Haha, just using this opportunity to pull attention and get > reviewer > > :D > > > > > > > > > > But totally agreed, it's time for 1.9 to be finalized. > > > > > > > > > > Thanks, > > > > > Ed. > > > > > > > > > > > > > > > On Wed, Feb 6, 2019 at 5:08 PM Joe Witt > wrote: > > > > > > > > > >> Thanks Andy sounds good. > > > > >> > > > > >> EdB: Sure we just need to find a reviewer. There are tons of good > > prs > > > > out > > > > >> there its just about getting good review cycles/progress. > > > > >> > > > > >> I'd like to shoot for early next week RC beginning if we can pull > it > > > > off. > > > > >> > > > > >> Thanks > > > > >> Joe > > > > >> > > > > >> On Wed, Feb 6, 2019 at 5:07 PM Andy LoPresto < > > > > alopresto.apa...@gmail.com> > > > > >> wrote: > > > > >> > > > > >>> The PR for adding HTTP headers is complete, I just need to merge > > it. > > > > Will > > > > >>> do today. > > > > >>> > > > > >>> Andy LoPresto > > > > >>> alopre...@apache.org > > > > >>> alopresto.apa...@gmail.com > > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D > EF69 > > > > >>> > > > > On Feb 7, 2019, at 08:23, Ed B wrote: > > > > > > > > if not too late, would be nice to have JNDI bug fix included in > > new > > > > >>> release: > > > > > > > > "JMS Connection Fails After JMS servers Change behind JNDI" > > > > https://issues.apache.org/jira/browse/NIFI-5869 > > > > PR is available: > > > > https://github.com/apache/nifi/pull/3261 > > > > > > > > Thank you, > > > > Ed. > > > > > > > > > > > > > On Wed, Feb 6, 2019 at 1:13 PM Joe Witt > > > wrote: > > > > > > > > > > thanks for bumping that peter. Mark P just got it merged. > > > > > > > > > > otherwise I see a few straggling items that look like they need > > to > > > be > > > > > merged and i'll start the process! Somehow this release has > > > sneakily > > > > > become packed with good fixes and improvements. > > > > > > > > > > Thanks > > > > > > > > > >> On Wed, Feb 6, 2019 at 10:34 AM Joe Witt > > > > wrote: > > > > >> > > > > >> agreed. looks important enough to warrant pulling in and I > just > > > > >> tagged > > > > > as > > > > >> 1.9.0 fix version to at least force discussion. > > > > >> > > > > >> On Wed, Feb 6, 2019 at 10:30 AM Peter Wicks (pwicks) < > > > > >>> pwi...@micron.com> > > > > >> wrote: > > > > >> > > > > >>> I'd like to see a review of this bug before we release 1.9.0, > > as > > > it > > > > > helps > > > > >>> mature the Node Offload feature. > > > > >>> > > > > >>> NIFI-5940 Cluster Node Offload Hangs if any RPG on flow is > > > Disabled > > > > >>> https://github.com/apache/nifi/pull/3255 > > > > >>> > > > > >>> Thanks, > > > > >>> Peter > > > > >>> > > > > >>> -Original Message- > > > > >>> From: Joe Witt [mailto:joew...@apache.org] > > > > >>> Sent: Tuesday, February 5, 2019 10:51 AM > > > > >>> To: dev@nifi.apache.org > > > > >>> Subject: [EXT] [discuss] release apache nifi 1.9.0 > > > > >>> > > > > >>> Team, > > > > >>> > > > > >>> > > > > >>> > > > > > > > > > >>> > > > > >> > > > > > > > > > > https://issues.apache.org/jira/browse/NIFI-5995?jql=project%20%3D%20NIFI%20AND%20status%20in%20(Resolved%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%201.9.0%20ORDER%20BY%20key%20DESC > > > > >>> > > > > >>> We have a pretty large list of bugs/features added already to >
Re: Load issues
Also, we've seen flows automatically throttled if provenance (or other repositories) can't keep up. Tuning the configs and switching drives (gp2 -> io1) helped On Tue, Oct 23, 2018, 19:38 Milan Das wrote: > Did you increase the no. of Concurrent Tasks and Run Duration on slow > processors ? > > Thanks, > Milan Das > > On 10/23/18, 8:26 PM, "Phil H" wrote: > > Hi team, > > My NiFi is struggling to process data (lots of back pressure in various > queues) but it is only using a very small amount (2-5%) of CPU > according to > top. > > Any ideas? > > > >
Re: Nifi build failed - 1.7.1
Hi, I had a similar problem just a bit ago and Scott kindly helped me out. The solution he provided might help you as well: http://apache-nifi-developer-list.39713.n7.nabble.com/failed-to-build-nifi-1-8-0-SNAPSHOT-on-OSX-high-sierra-td19426.html On Mon, Sep 3, 2018 at 1:28 PM Arunkumar wrote: > Hello Everyone, > > Any pointers would be helpful. I am trying to build NiFi from source on > Amazon Linux (AL2012) > > - > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.4:npm (npm install) on project > nifi-web-ui: Failed to run task: 'npm --cache-min Infinity install' failed. > (error code 1) -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR]mvn -rf :nifi-web-ui > BUILD FAILED > > I tried downgrading com.github.eirslett:frontend-maven-plugin:1.4:npm to > 1.1 > as well, still the error is the same. Kindly help. > 87530-nifi-build-failure.txt > < > http://apache-nifi-developer-list.39713.n7.nabble.com/file/t1050/87530-nifi-build-failure.txt> > > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ >
Re: failed to build nifi 1.8.0-SNAPSHOT on OSX high sierra
Thanks Scott, that fixed it! On Thu, Aug 16, 2018 at 9:56 AM Scott Aslan wrote: > Hi Charlie, > > Sorry to hear you are having issues building NiFi. The > frontend-maven-plugin has been designed to use a local installation of > node. Using a globally installed version has been requested before > <https://github.com/eirslett/frontend-maven-plugin/issues/277> but the > developer's position is that node does not take up much space and will only > download if missing. Installing node locally allows developers that have > not installed node globally or are using different versions to build the > project without having to do anything more complicated than mvn clean > install. > > I assume you used brew to install node? > > I am wondering if there is a permission issue and if you uninstall node > from your machine are you able to successfully build? Try brew uninstall > --force node and then try mvn clean install. > > -Scott > > On Thu, Aug 16, 2018 at 10:27 AM Charlie Meyer < > charlie.me...@civitaslearning.com> wrote: > > > Hello, > > > > I'm attempting to build 1.8.0-snap on osx high sierra and am running into > > the following nodejs error: > > > > [INFO] Running 'npm --cache-min Infinity install' in > > > > > /Users/cmeyer/opt/nifi/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/target/frontend-working-directory > > [ERROR] module.js:549 > > [ERROR] throw err; > > [ERROR] ^ > > [ERROR] > > [ERROR] Error: Cannot find module '../lib/utils/unsupported.js' > > [ERROR] at Function.Module._resolveFilename (module.js:547:15) > > [ERROR] at Function.Module._load (module.js:474:25) > > [ERROR] at Module.require (module.js:596:17) > > [ERROR] at require (internal/module.js:11:18) > > [ERROR] at > > > > > /Users/cmeyer/opt/nifi/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/target/frontend-working-directory/node/node_modules/npm/bin/npm-cli.js:19:21 > > [ERROR] at Object. > > > > > (/Users/cmeyer/opt/nifi/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/target/frontend-working-directory/node/node_modules/npm/bin/npm-cli.js:92:3) > > [ERROR] at Module._compile (module.js:652:30) > > [ERROR] at Object.Module._extensions..js (module.js:663:10) > > [ERROR] at Module.load (module.js:565:32) > > [ERROR] at tryModuleLoad (module.js:505:12) > > ... > > ... > > ... > > [ERROR] Failed to execute goal > > com.github.eirslett:frontend-maven-plugin:1.4:npm (npm install) on > project > > nifi-jolt-transform-json-ui: Failed to run task: 'npm --cache-min > Infinity > > install' failed. org.apache.commons.exec.ExecuteException: Process exited > > with an error: 1 (Exit value: 1) -> [Help 1] > > [ERROR] > > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e > > switch. > > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > > [ERROR] > > [ERROR] For more information about the errors and possible solutions, > > please read the following articles: > > [ERROR] [Help 1] > > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > > [ERROR] > > [ERROR] After correcting the problems, you can resume the build with the > > command > > [ERROR] mvn -rf :nifi-jolt-transform-json-ui > > > > I tried solutions suggested online such as reinstalling node, etc but > have > > still not had luck. I've been able to replicate the error on 2 different > > machines, but only on osx (was able to build fine on ubuntu). Is this a > > known issue and is there a workaround? > > > > thanks! > > >
failed to build nifi 1.8.0-SNAPSHOT on OSX high sierra
Hello, I'm attempting to build 1.8.0-snap on osx high sierra and am running into the following nodejs error: [INFO] Running 'npm --cache-min Infinity install' in /Users/cmeyer/opt/nifi/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/target/frontend-working-directory [ERROR] module.js:549 [ERROR] throw err; [ERROR] ^ [ERROR] [ERROR] Error: Cannot find module '../lib/utils/unsupported.js' [ERROR] at Function.Module._resolveFilename (module.js:547:15) [ERROR] at Function.Module._load (module.js:474:25) [ERROR] at Module.require (module.js:596:17) [ERROR] at require (internal/module.js:11:18) [ERROR] at /Users/cmeyer/opt/nifi/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/target/frontend-working-directory/node/node_modules/npm/bin/npm-cli.js:19:21 [ERROR] at Object. (/Users/cmeyer/opt/nifi/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/target/frontend-working-directory/node/node_modules/npm/bin/npm-cli.js:92:3) [ERROR] at Module._compile (module.js:652:30) [ERROR] at Object.Module._extensions..js (module.js:663:10) [ERROR] at Module.load (module.js:565:32) [ERROR] at tryModuleLoad (module.js:505:12) ... ... ... [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.4:npm (npm install) on project nifi-jolt-transform-json-ui: Failed to run task: 'npm --cache-min Infinity install' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :nifi-jolt-transform-json-ui I tried solutions suggested online such as reinstalling node, etc but have still not had luck. I've been able to replicate the error on 2 different machines, but only on osx (was able to build fine on ubuntu). Is this a known issue and is there a workaround? thanks!
Re: Should the elastic search client's tllmomorrow and ask service impl get renamed before 1.7?
I'm gonna jump me know a day without women are you going so let a man and I'll nknnock knock it k and I'll fill n never get mm forget kkjinnock it out and let it me I can illoyiintjvhjhmbmhbnnnjkyolokiklu to the of you know know if I meNikki'sklnobn the I don't own the car parked Atkins I will be home at your On Sun, Jun 10, 2018, 06:46 Mike Thomsen wrote: > The current implementation uses the last stable release of the 5.X client > from Elastic, and 6.X is already mature so we'll need to be able to have > room to create a new implementation copy that uses that client if there are > things we have to change between them. So does it make sense to throw in a > new ticket to rename the service to something that indicates that this > implementation is officially for 5.X? As of 1.6 I think only one processor, > JsonQueryElasticSearch, uses it so not many uses would likely be impacted > yet. > > Thanks, > > Mike >
Re: Should the elastic search client service impl get renamed before 1.7? I'm I getting muyyiugjjjjj your lkpmllomnj is like o huglll is m
I'm going out for dinner tonight if we Mom mmomnlokk let other LM sorry to be homlmmmkmmolle with On Sun, Jun 10, 2018, 06:46 Mike Thomsen wrote: > The current implementation uses the last stable release of the 5.X client > from Elastic, and 6.X is already mature so we'll need to be able to have > room to create a new implementation copy that uses that client if there are > things we have to change between them. So does it make sense to throw in a > new ticket to rename the service to something that indicates that this > implementation is officially for 5.X? As of 1.6 I think only one processor, > JsonQueryElasticSearch, uses it so not many uses would likely be impacted > yet. > > Thanks, > > Mike >
Re: NiFi code re-use
We do this often by leveraging the variable registery and the expression language to make components be more dynamic and reusable -Charlie On Sat, May 12, 2018, 20:01 scott wrote: > Hi Devs, > > I've got a question about an observation I've had while working with > NiFi. Is there a better way to re-use process groups similar to how > programming languages reference functions, libraries, classes, or > pointers. I know about remote process groups and templates, but neither > do exactly what I was thinking. RPGs are great, but I think the output > goes to the root canvas level, and you have to have have connectors all > the way back up your flow hierarchy, and that's not practical. > Ultimately, I'm looking for an easy way to re-use process groups that > contain common logic in many of my flows, so that I reduce the amount of > places I have to change. > > Hopefully that made sense. Appreciate your thoughts. > > Scott > >
Re: Custom Controller Service
Hello,
I'm working to see what we can share, but for a start, here is how we
handle the delegating controller service:
package com.civitaslearning.nifi.service.impl;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.controller.AbstractControllerService;
import org.apache.nifi.dbcp.DBCPService;
import java.util.Collections;
import java.util.List;
import java.util.UUID;
public class DelegatingDBCPServiceImpl extends
AbstractControllerService implements DelegatingDBCPService {
private static final List properties =
Collections.emptyList();
@Override
public DBCPService getDBCPService(UUID uuid) {
DBCPService dbcpService =
(DBCPService)
getControllerServiceLookup().getControllerService(uuid.toString());
if (dbcpService == null) {
throw new NullPointerException("Couldn't find DBCP service with
ID " + uuid);
}
return dbcpService;
}
@Override
public List getSupportedPropertyDescriptors() {
return properties;
}
}
>From there, we took existing nifi processor code such as ExecuteSQL and
modified it to require a reference to a delegating service rather than a
DBCP service and added another property to specify which flow file
attribute has the id of the dbcp service that should be used, similar to:
final DelegatingDBCPService delegatingDBCPService =
context.getProperty(DELEGATING_DBCP_SERVICE).asControllerService(DelegatingDBCPService.class);
final UUID dbcpServiceId =
UUID.fromString(context.getProperty(DBCP_SERVICE_ATTRIBUTE).evaluateAttributeExpressions(fileToProcess).getValue());
final DBCPService dbcpService =
delegatingDBCPService.getDBCPService(dbcpServiceId);
thanks
On Thu, Apr 26, 2018 at 4:07 AM, RP wrote:
> Hi Charlie,
>
> Thanks for the reply. It seems that you already have a solution to my
> problem. If possible, can you please share you code for the custom
> controller service and the custom processor?
>
>
>
> -
> RP
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Re: Custom Controller Service
Chiming in a bit late on this, but we faced this same issue and got around it by implementing a custom controller service which acts as a "router" to different dbcp services. It exposes a method which given a uuid, returns back the DBCPservice that corresponds with that uuid if it exists using DBCPService dbcpService = (DBCPService) getControllerServiceLookup().getControllerService(uuid); >From there, we created processors we needed based on the stock ones which relied on our "router" service rather than a single DBCP. Our custom processors read an attribute from incoming flow files, then send that to the router which returns back the connection pool. On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende wrote: > Here is a proposal for how to modify the existing API to support both > scenarios: > > https://issues.apache.org/jira/browse/NIFI-5121 > > The scope of that ticket would be to make the interface change, and > then update all of NiFi's DB processors to pass in the attribute map. > > Then a separate effort to provide a new service implementation that > used the attribute map to somehow manage multiple connection pools, or > create connections on the fly, or whatever the desired behavior is. > > On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende wrote: > > To Otto's question... > > > > For simplicity sake, there is a new implementation of > > DBCPConnectionPool that behind the scenes has two connection pools, > > one for DB A and one for DB B, it doesn't matter how these are > > configured. > > > > Now a flow file comes into the ExecuteSQL and it goes to > > connectionPoolService.getConnection()... > > > > How does it know which DB to return a connection for? > > > > > > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna > wrote: > >> Option 2 and 3 seem to be a probable approach. However creating varying > >> number of connections based on *each* flowfile still sounds to be > >> suboptimal. If the requirement still demands to take that road, then > it’s > >> better to do some prep-work.. as in the list of probable connections > that > >> are required are taken and connection pools are created for them and > then > >> based on the flowfiles (which connection it needs), we use the relevant > >> one. > >> > >> Thanks, > >> Sivaprasanna > >> > >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende wrote: > >> > >>> The issue here is more about the service API and not the > implementations. > >>> > >>> The current API has no way to pass information between the processor > and > >>> service. > >>> > >>> The options boil down to... > >>> > >>> - Make a new API, but then you need all new processors that use the > new API > >>> > >>> - Modify the current API to have a new method, but then we are combing > two > >>> concepts into one API and some impls may not implement both > >>> > >>> - Modify the processors to use two different service APIs, but enforce > that > >>> only one can be used at a time, so it can have either the original > >>> connection pool service or can have some new dynamic connection > factory, > >>> but not both, and then modify all DB processors to have logic to > determine > >>> which service to use. > >>> > >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler > >>> wrote: > >>> > >>> > Or you could just call every time you needed properties more likely. > >>> > This would still be custom unless integrated…. > >>> > > >>> > > >>> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > >>> > wrote: > >>> > > >>> > Can services work with other controller services? > >>> > Maybe a PropertiesControllerService, FilePropertiesControllerService > >>> could > >>> > work with your service? > >>> > the PCS could fire events on property changes etc. > >>> > > >>> > > >>> > > >>> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > >>> > wrote: > >>> > > >>> > Shot in the dark here, but what you try to do is create a custom > >>> connection > >>> > pool service that uses dynamic properties to build a "pool of > connection > >>> > pools." You could then use the property names as hints for where to > send > >>> > the queries. > >>> > > >>> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad < > rishabprasad...@gmail.com > >>> > > >>> > wrote: > >>> > > >>> > > Hi, > >>> > > > >>> > > Basically, there are 'n' number of databases that we are dealing > with. > >>> We > >>> > > need to fetch the data from the source database into HDFS. Now > since we > >>> > are > >>> > > dealing with many databases, the source database is not static and > >>> > changes > >>> > > every now and then. And every time the source database changes we > >>> > manually > >>> > > need to change the value for the connection parameters in > >>> > > DBCPConnectionPool. Now, people suggest that for 'n' databases > create > >>> 'n' > >>> > > connections for each database, but that is not possible because > 'n' is > >>> a > >>> > > big number and creating that many connections in > DBCPConnectionPool is > >>> > not >
Re: ListSFTP incoming relationship
Just a thought, Could a processor that did the scan and stored state be implemented similar to GenerateTableFetch, where there is a minimum value attribute that is specified that could be read from the source (such as created date, updated date, etc)? That way the state could potentially be manageable. On Thu, Mar 29, 2018 at 2:43 PM, Andy LoPresto wrote: > Bryan, > > No, that was exactly what I was referencing regarding the attribute > output. It would have been clearer if I had said it like you did. Thanks. > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Mar 29, 2018, at 10:46 AM, Bryan Bende wrote: > > Scott, > > You are correct that the overall discussion is about allowing incoming > flow files to ListSFTP. > > However, the previous discussion on this thread highlighted that the > main reason ListSFTP currently doesn't allow incoming flow files is > because of challenges when storing state. > > This led to the proposal of a new processor that allowed incoming flow > files, but did not store state, thus avoiding the challenges mentioned > above. If we were going to store state in this new processor, then > we'd be back to the exact same challenges. > > Providing an option to turn on state also doesn't really help, because > if there is an option provided to users,then the option will be used, > and it needs to work when it is used. > > If we can come up with something that stores state and works well for > all scenarios, then we aren't against it, we just need to handle the > challenges highlighted by Joe's original email. > > Regarding some of the other ideas... > > The current output of ListSFTP already includes flow file attributes > for each listing that include the full path, filename, last update > time, owner, group, permissions, and file size were you thinking > of something different than that? > > See the "Writes Attributes" section here: > https://nifi.apache.org/docs/nifi-docs/components/org. > apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi. > processors.standard.ListSFTP/index.html > > Thanks, > > Bryan > > > > On Thu, Mar 29, 2018 at 12:43 PM, Andy LoPresto > wrote: > > Scott, > > I think there are two conversations going on here. You are finding the > requirements for your specific use case, and that’s great. But I echo > Bryan’s point that a community processor for this scenario should not store > state at all. Sivaprasanna’s point that given dynamic directory input, > storing state based on that can cause massive data ingestion problems still > stands. > > For your specific use case, you can prototype (or possibly even get to a > stable and robust-enough point) using ExecuteScript to model the behavior > you need. > > In regards to the desired output format, I would suggest a few items: > > * Avro requires a schema to be defined, and this raises the barrier to use > of the processor. Also, unless being sent to a processor that understands > Avro, the result will need to be converted anyway using Record* processors. > * If the output is individual flowfiles on a 1:1 basis, each should have as > many attributes populated with the parsed information as possible (i.e. > file.name, file.path, file.size, file.owner, file.permissions, etc.). This > allows for easily-consumable and routable flowfiles. > * If the output is a full directory listing, I would suggest `ls -al` type > raw text output, or JSON (arbitrary human-readable and machine-readable > format with many consuming/transforming processors). > > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Mar 29, 2018, at 9:34 AM, scott wrote: > > Sorry Bryan, but I disagree with you. Not storing state is NOT the main > point of this new processor. The main point is to allow an incoming > relationship flowfile to trigger the action, and allow variables to be used > from the attributes therein. > > I agree that if the NiFi community deems it too risky to distribute this > processor with state keeping optionally available, even if the default is > to > disable it, then so be it. If state is not included optionally, then how > about making the output flowfile content include more than just the file > names? Have it include last updated time along with the filename. If it > searches recursively, you'll want to include the path to the file also. > Maybe it would be best to output the results into a structured format, such > as AVRO? Or, maybe it would just be best to output one flowfile per remote > file found, and include updated time and fully qualified path as > attributes? > > Scott > > > On 03/29/2018 04:32 AM, Bryan Bende wrote: > > The main point of the new processor is to NOT store state so that it > becomes more reasonable to allow incoming flow files. > > You could probably implement your own custom processor that does both > be
Re: AWS Processors and Proxy User/Password
For these, I have used the AWS Credentials Provider controller service -Charlie ᐧ Charlie Meyer On Fri, Mar 16, 2018 at 1:25 PM, Otto Fowler wrote: > Does anyone know why the AWS Processors support proxy host and port but not > user and password as InvokeHttp does? > > ottO >
Re: 答复: Re: Is there a REST API to run a dataflow on demand?
Hi, I've found the rest api was designed to support actions that can be done from the UI and too much much beyond. That being said, if you can do it from the UI, you can write up code to mimic that via the rest api. So, to accomplish the task you are looking to do, you might need to write code that makes the same api calls the UI does that let you know your flow needs attention (on a loop or some other mechanism) then makes the same rest calls the UI would when you take corrective action when called for. Hope that helps On Wed, Feb 21, 2018 at 7:57 PM, wrote: > Thanks a lot. > > But I want to know if there is a REST API that triggers a dataflow on > demand? > I don't find the API in the page. > > > > > 发件人: > Charlie Meyer > 收件人: > dev@nifi.apache.org > 日期: > 2018/02/22 09:36 > 主题: > Re: Is there a REST API to run a dataflow on demand? > > > > Yep, when you make the changes in the UI, open developer tools in your > browser and see what calls to the nifi api it is making then mimic those > with code. > > The nifi team also kindly publishes > https://nifi.apache.org/docs/nifi-docs/rest-api/index.html which help a > lot. > > Best of luck! > > -Charlie > > On Wed, Feb 21, 2018 at 7:34 PM, wrote: > > > Hi, team, > > > > We set up several NiFi dataflows for data processing. > > These dataflows are configured to run once per day in the midnight. > > > > But sometimes, some dataflows are failed,I want to run the dataflow > again > > immediately after fixing the issue instead of waiting for running it in > > the midnight to > > make sure that the issue is really fixed. > > > > The only way I know to do this is to change the time of running the > > dataflow to the 5 mintutes from now for example > > and then change it back to midnight. > > > > It's a little inconvenient. > > > > Is there any REST API that I can use to trigger the dataflow on demand > > i.e. without change the time back and forth? > > > > Thanks > > > > Boying > > > > > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对 > 外 > > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发 > 件 > > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。 > > > > > > This email message may contain confidential and/or privileged > information. > > If you are not the intended recipient, please do not read, save, > forward, > > disclose or copy the contents of this email or open any file attached to > > this email. We will be grateful if you could advise the sender > immediately > > by replying this email, and delete this email and any attachment or > links > > to this email completely and immediately from your computer system. > > > > > > > > > > > > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外 > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件 > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。 > > > This email message may contain confidential and/or privileged information. > If you are not the intended recipient, please do not read, save, forward, > disclose or copy the contents of this email or open any file attached to > this email. We will be grateful if you could advise the sender immediately > by replying this email, and delete this email and any attachment or links > to this email completely and immediately from your computer system. > > > >
Re: Is there a REST API to run a dataflow on demand?
Yep, when you make the changes in the UI, open developer tools in your browser and see what calls to the nifi api it is making then mimic those with code. The nifi team also kindly publishes https://nifi.apache.org/docs/nifi-docs/rest-api/index.html which help a lot. Best of luck! -Charlie On Wed, Feb 21, 2018 at 7:34 PM, wrote: > Hi, team, > > We set up several NiFi dataflows for data processing. > These dataflows are configured to run once per day in the midnight. > > But sometimes, some dataflows are failed,I want to run the dataflow again > immediately after fixing the issue instead of waiting for running it in > the midnight to > make sure that the issue is really fixed. > > The only way I know to do this is to change the time of running the > dataflow to the 5 mintutes from now for example > and then change it back to midnight. > > It's a little inconvenient. > > Is there any REST API that I can use to trigger the dataflow on demand > i.e. without change the time back and forth? > > Thanks > > Boying > > > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外 > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件 > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。 > > > This email message may contain confidential and/or privileged information. > If you are not the intended recipient, please do not read, save, forward, > disclose or copy the contents of this email or open any file attached to > this email. We will be grateful if you could advise the sender immediately > by replying this email, and delete this email and any attachment or links > to this email completely and immediately from your computer system. > > > >
Re: NiFi 1.5.0 can't start with AWS public ip is set to nifi.web.https.host
i ran into that earlier today actually, setting the nifi.web.http(s).network.interface* fixed the issue for me -Charlie On Wed, Feb 21, 2018 at 4:48 PM, tanezavm wrote: > Hi Andy, > > As far as I know, there's no other services running in the instance that is > preventing NiFi from binding to it. Previous NiFi instance was shutdown > too. > Does this version of NiFi allows to be configured with an IP that is not > associated to any of the interfaces? Since It worked with using the private > IP of the instance. > > Regards, > Virgil > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ >
Re: [EXT] Re: Embedded Nifi
Hi Vincent, We do this for our test cases using https://github.com/palantir/docker-compose-rule. It works fairly well, but is a bit heavy weight to get tests started up locally. Also, with 1.5.0, the host header issue surfaces here and makes running in docker a pain. Hope that helps a bit! On Mon, Feb 5, 2018 at 6:07 PM, Vincent Russell wrote: > Perhaps Dan, but integrating docker into an automated test might prove a > little challenging. > > I'll check it out though. > > Thanks, > > On Wed, Jan 31, 2018 at 3:00 AM, Daniel Chaffelson > wrote: > > > Vincent, > > I do something along these lines in Python to test NiFi automation work. > > NiPyApi creates the requisite Docker container(s) for the test suite, > > procedurally creates the NiFi/Registry objects (Process Groups, > Processors, > > Buckets, Flows, etc), recursively tests itself, then tears itself down. > > > > I know this is not 'embedded', but could automated and containerised be > > close enough to meet your needs? > > > > Thanks, > > Dan > > > > On Wed, Jan 24, 2018 at 2:22 AM Peter Wicks (pwicks) > > wrote: > > > > > Vincent, > > > > > > Embedded NiFi still has a long ways to go to be really useful, in my > > > opinion; and I don't know if anyone is actively working on those > > > improvements. > > > > > > The PR Andy mentioned simply allows you to startup NiFi inside your > > > process instead of running it directly from a startup script, but that > > > doesn't mean you magically have access to all of NiFi's internals > > (someone > > > can correct me if I'm wrong). If you want to actually interact with > your > > > new NiFi instance you will still need to use the REST API. > > > > > > Thanks, > > > Peter > > > > > > -Original Message- > > > From: Vincent Russell [mailto:vincent.russ...@gmail.com] > > > Sent: Tuesday, January 23, 2018 03:07 > > > To: dev@nifi.apache.org > > > Subject: [EXT] Re: Embedded Nifi > > > > > > Thanks Andy, > > > > > > This looks a great first step. It would be nice to have a builder > > pattern > > > and the ability to download the "executable" from a nexus or the local > > > filesystem like embedded elastic, but perhaps that might be better in > > some > > > third party library. > > > > > > https://github.com/allegro/embedded-elasticsearch > > > > > > -Vincent > > > > > > On Mon, Jan 22, 2018 at 1:37 PM, Andy LoPresto > > > wrote: > > > > > > > Vincent, > > > > > > > > I plan to merge this pull request [1] for NIFI-4424 [2] by Peter > > > > Horvath today. Does this satisfy your requirements? > > > > > > > > [1] https://github.com/apache/nifi/pull/2251 > > > > [2] https://issues.apache.org/jira/browse/NIFI-4424 > > > > > > > > Andy LoPresto > > > > alopre...@apache.org > > > > *alopresto.apa...@gmail.com * PGP > > > > Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > > > > > > > On Jan 21, 2018, at 7:35 AM, Vincent Russell > > > > > > > > wrote: > > > > > > > > Devs, > > > > > > > > Does an embedded nifi exist that would start a nifi with a provided > > > > workflow? I am aware of the Mock framework, but I am looking for > > > > something for integration tests. > > > > > > > > Thanks, > > > > Vincent > > > > > > > > > > > > > > > > > >
Incorrect return classes specified on api annotations
Hi, I opened up https://issues.apache.org/jira/browse/NIFI-4835 earlier today for a few places where the return type class did not reflect the entity that is being returned. For now, I've locally patched my build and was able to regenerate a correct swagger json, but figured id bring it up here as well so it can hopefully make it into the next release. Thanks!
