RE: 1.0 Run Status Colors
I created a ticket and put in a link to a mostly accurate screenshot, at least close enough to give you a feel for what I was thinking: https://issues.apache.org/jira/browse/NIFI-2603 Direct screenshot link: https://goo.gl/photos/RZzz45RGeTAmWwiC7 -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Wednesday, August 17, 2016 3:46 PM To: dev@nifi.apache.org Subject: Re: 1.0 Run Status Colors no big discussion really - just contributions. If you could kick off a JIRA and attach a screenshot of what you have in mind that could help the discussion move along. On Wed, Aug 17, 2016 at 2:33 PM, Peter Wicks (pwicks) wrote: > We've been playing with NiFi 1.0 and of course the new theme/layout is a big > change. > One thing we don't particularly care for is the bland gray > Running/Stopped/Disabled colors. I would be willing to have them bland in the > header, but on the Processor/Process Group level I feel they really need to > be Green/Red/whatever color. > > I would be happy to submit the change, but I wanted to make sure there hadn't > been some kind heated discussion over colors or lack thereof before putting > in a PR.
Re: User-supplied attributes for processors
Kirk, The processors have to explicitly know about dynamic properties (and their intent) in order to use them appropriately. For a processor like ListSFTP, it could be beneficial to have custom attributes (as parameters to the SFTP session perhaps?) but the domain knowledge on how they'd be used would have to be coded into the processor itself. If you have a use case in mind, please feel free to file an improvement Jira for ListSFTP to use dynamic properties. Regards, Matt On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk wrote: > Is there some reason why Processors, like ListSFTP, don’t allow custom > attributes to be added? Relatedly, why do these processors allow users to add > custom attributes & values in the UI even though it always throws the error: > ‘[attribute]’ validated against ‘[value]’ is invalid because ‘[attribute]’ is > not a supported property > > - Kirk Tarou
Re: nifi.sensitive.props.key from external source
Devin, I am working on a PR to support this. I hope to have it completed by tomorrow. NIFI-1831 has a description of the task, PR 834 has my current changes. Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Aug 18, 2016, at 10:33, Devin Fisher > wrote: > > We are looking to deploy nifi as part of an appliance. Normally, we use > etckeeper [1] to maintain, track and backup our configuration across the > board. Etckeeper puts configuration files into a git repo. I would like to > store nifi configuration in the same way but I'm concerned about storing > the sensitive properties key there along with the flow.xml.gz. I would like > to store that key somewhere else and load it in at start up time. Any > thoughts on how that could be done. Ultimately, I just want is to not back > up the key with the flow.xml.gz (which has the encrypted data). That way if > someone gets a hold of the backup it would not trivial to decrypt the > sensitive data in flow.xml.gz. > > I thought I might be able to do this by adding a custom java.arg to the > bootstrap.conf that would point to an environment variable. > > Something like: > java.arg.99=-Dnifi.sensitive.props.key=$NIFI_SENSITIVE_PROPS_KEY > > But I'm not sure if System properties can stand in for nifi.perperies > values and if the boot loader launches nifi in such a way to use > environment variables. > > > [1] https://github.com/joeyh/etckeeper
Re: Apache Yetus annotations : clarifying public vs private api
Team, I've gone ahead and focused more on ensuring that the versioning/scheme and compatibility model we've been following is more clearly articulated on [1] and also more specifically defined what is considered part of the nifi-api. I still think the yetus annotations or something inspired by them is the right way to go but at this point wanted to avoid adding a dependency to the nifi-api. I'm actually inclined to think that if we go further with this it should be done by bringing the yetus annotations in as a source dependency rather than a binary one. In any event, the near term pain was to more clearly articulate the API compatibility model and i believe that is resolved. Also added some comments about it on the JIRA i raised for it [2]. [1] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility [2] https://issues.apache.org/jira/browse/NIFI-1373 On Sun, Jul 31, 2016 at 7:15 PM, Joe Witt wrote: > Team, > > The big ticket back-end changes for NiFi 1.x line seem to be closing > down and Apache Yetus has pushed their artifacts to maven central so I > am proceeding with NIFI-1373. I plan to start with a limited rollout > but specifically will annotate the nifi-api completely. I also > updated our wiki which talks about our version scheme [1]. We can see > how this goes and then if we want to expand it to other parts of the > codebase we can do so later. > > The specific text i added to that was: > " > Also, starting with the Apache NiFi 1.x codebase we have adopted > Apache Yetus 'Audience Annotations' to explicitly mark code for things > like interfaces, classes, and methods to indicate the intended > audience and stability of those APIs. If not otherwise marked please > consider the interface, class, or method to be private/internal and > unstable. The vast majority of 'nifi-api' should be both public use > and stable meaning we will treat any compatibility changes for these > items as things which would likely require a major release. > Otherwise, as a community we should be able to more easily navigate > the necessary evolution and improvement of the codebase for these > interfaces, classes, and methods for which we do not have to be so > strictly adherent to backward compatibility. A further consideration > here though is as mentioned previously which is that even if we want > to change classes, interfaces, and methods which are not public and > stable we must also be mindful of things like configuration, user > experience, REST API, etc... as these are also an important part of > our 'interface' and compatibility which we must honor and account for. > " > > The idea is that unless we have marked something it should be assumed > to be private and unstable as far as implementation goes though this > doesn't eliminate our need to honor the holistic interpretation of our > "interface". > > [1] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme > > Thanks > Joe > > On Tue, Jul 26, 2016 at 7:59 AM, Joe Skora wrote: >> +1 >> >> This should help clarify things when learning NiFi and when diving in new >> areas of the codebase. >> >> Thanks, I'm looking forward to it. >> >> On Mon, Jul 25, 2016 at 7:45 PM, Matt Burgess wrote: >> >>> agreed, big +1 here, will make things much more clear re: stability >>> and extensibility. >>> >>> On Mon, Jul 25, 2016 at 7:43 PM, Tony Kurc wrote: >>> > awesome! >>> > >>> > On Mon, Jul 25, 2016 at 7:32 PM, Joe Witt wrote: >>> > >>> >> Team >>> >> >>> >> For those interested I am going to take a stab at working through the >>> >> codebase [1] to apply Apache Yetus [2] annotations to make it more >>> >> clear and well communicated what are the public vs private aspects of >>> >> the API. >>> >> >>> >> We've had the 'nifi-api' module for some time now but unless you know >>> >> that this was meant to be the public API for supported extension types >>> >> you'd not really know that other parts of NiFi were not meant to be >>> >> altered and could be changed. Therefore, we've not been able to >>> >> evolve the framework as directly as we'd like at times. We had to >>> >> wait until 1.0 to make changes which otherwise should have been ok to >>> >> make. Also, previously 'nifi-api' had a lot of extraneous bits in it >>> >> that have now been refactored out to where they belong such as core >>> >> framework itself or in a 'nifi-framework-api' module. This means >>> >> things in 'nifi-api' should be quite stable and public. >>> >> >>> >> Yetus gives us a chance to clearly articulate which parts of the API >>> >> are meant for public use and which are meant for private use only and >>> >> therefore are subject to change. Furthermore, even for those public >>> >> elements we can more effectively articulate stability in ways that map >>> >> really nicely to our versioning scheme. >>> >> >>> >> My intent is to be very cautious in labeling anything beyond the >>> >> nifi-api as public. We can open things up as they prove to
nifi.sensitive.props.key from external source
We are looking to deploy nifi as part of an appliance. Normally, we use etckeeper [1] to maintain, track and backup our configuration across the board. Etckeeper puts configuration files into a git repo. I would like to store nifi configuration in the same way but I'm concerned about storing the sensitive properties key there along with the flow.xml.gz. I would like to store that key somewhere else and load it in at start up time. Any thoughts on how that could be done. Ultimately, I just want is to not back up the key with the flow.xml.gz (which has the encrypted data). That way if someone gets a hold of the backup it would not trivial to decrypt the sensitive data in flow.xml.gz. I thought I might be able to do this by adding a custom java.arg to the bootstrap.conf that would point to an environment variable. Something like: java.arg.99=-Dnifi.sensitive.props.key=$NIFI_SENSITIVE_PROPS_KEY But I'm not sure if System properties can stand in for nifi.perperies values and if the boot loader launches nifi in such a way to use environment variables. [1] https://github.com/joeyh/etckeeper
Re: Apache NiFi Clarification
Hello, NiFi can gather data from as many sources as needed. When data is brought into NiFi it is written into NiFi's internal repositories and stored there as it moves through the graph of processors, the data can then be delivered to any desired system. To bring data from a database to HDFS you would likely use the ExecuteSQL or QueryDatabase processor to get the data out of the database, then some intermediary processors if you need to convert the data, and then PutHDFS to write the data to HDFS. Thanks, Bryan On Thu, Aug 18, 2016 at 6:17 AM, wrote: > Hi, > Good Day! > I Saw your Blogs and Videos its Pretty Interesting and I am new in Apache > Nifi.So i just want to know if the data will only to fetch into nifi or > else data can export to any other databases or API. I need to your guides I > am working on Nifi in My organization So i just to know how the flow to > connect with Sql Server Or Any other Databases and How can i create a data > flow to fetch the Server into Hadoop .but In my case its pretty different > we need to connect multiple API Data into Nifi and Transfer Via Sqoop To > Hive .If in this case we can choose Nifi . please give some guides about > the Apache Nifi and how the flow it will work . I am new to Apache Nifi So > Please give Guides about Apache Nifi in my case Thanks & > Regards,Manikandan Kolanjinathan > Junior Software Engineer > Pinnacle Seven Technologies 0422-4208736, 4506535 > www.pinnacleseven.comDelivering > your business apps on cloud! > >
Re:Apache NiFi Clarification
Hi, Good Day! I Saw your Blogs and Videos its Pretty Interesting and I am new in Apache Nifi.So i just want to know if the data will only to fetch into nifi or else data can export to any other databases or API. I need to your guides I am working on Nifi in My organization So i just to know how the flow to connect with Sql Server Or Any other Databases and How can i create a data flow to fetch the Server into Hadoop .but In my case its pretty different we need to connect multiple API Data into Nifi and Transfer Via Sqoop To Hive .If in this case we can choose Nifi . please give some guides about the Apache Nifi and how the flow it will work . I am new to Apache Nifi So Please give Guides about Apache Nifi in my case Thanks & Regards,Manikandan Kolanjinathan Junior Software Engineer Pinnacle Seven Technologies 0422-4208736, 4506535 www.pinnacleseven.comDelivering your business apps on cloud!