Re: "External" extensions

[Tony Kurc]

Not very strong opinions on this?
On Oct 30, 2015 10:53 AM, "Joe Witt"  wrote:

> Tony,
>
> I completely agree we should do this.  A quick github search reveals
> there are some nice utilities/processors folks have built for NiFi but
> for which they're not necessarily going to submit them as PRs.  We
> should link to these as much as possible but we should also help folks
> understand these aren't 'apache' things and are not of the Apache NiFi
> community directly but they are good for users and developers to know
> about.
>
> Perhaps a wiki page linking to these is good provided we have the
> above sort of disclaimer and a healthy recognition such references
> will become stale...
>
> Thanks
> Joe
>
> On Fri, Oct 30, 2015 at 10:48 AM, Tony Kurc  wrote:
> > All,
> > I wanted to start a conversation about projects that are good for people
> > using or developing NiFi, but either can't or don't belong in the source
> > tree. This could be due to licensing issues (for example not compatible
> (or
> > not yet determined if it is compatible (GPL [1])) with the Apache
> License),
> > or other thought provoking mild concerns like we're discussing on
> NIFI-1074
> > [2].
> >
> > I'd like to propose either capturing these on the website or on the wiki
> or
> > some other approach I didn't think of. I was hoping to find a good
> > archetype for this type of documentation in another apache project, but
> > didn't find anything I personally liked. If you have seen something you
> > like or don't, I'd be interested to hear.
> >
> >
> > [1] http://www.apache.org/licenses/GPL-compatibility.html
> > [2] https://issues.apache.org/jira/browse/NIFI-1074
>

Re: "External" extensions

[Adam Estrada]

This has been suggested before. It's a great idea!!! I suggest creating a repo 
on github for NiFi-Processors or something like that. There are many more folks 
searching through GitHub than on the Apache wikis, IMO. This will inevitably 
help spread the word...

A

Sent from my iPhone

> On Nov 1, 2015, at 12:55 PM, Tony Kurc  wrote:
> 
> Not very strong opinions on this?
>> On Oct 30, 2015 10:53 AM, "Joe Witt"  wrote:
>> 
>> Tony,
>> 
>> I completely agree we should do this.  A quick github search reveals
>> there are some nice utilities/processors folks have built for NiFi but
>> for which they're not necessarily going to submit them as PRs.  We
>> should link to these as much as possible but we should also help folks
>> understand these aren't 'apache' things and are not of the Apache NiFi
>> community directly but they are good for users and developers to know
>> about.
>> 
>> Perhaps a wiki page linking to these is good provided we have the
>> above sort of disclaimer and a healthy recognition such references
>> will become stale...
>> 
>> Thanks
>> Joe
>> 
>>> On Fri, Oct 30, 2015 at 10:48 AM, Tony Kurc  wrote:
>>> All,
>>> I wanted to start a conversation about projects that are good for people
>>> using or developing NiFi, but either can't or don't belong in the source
>>> tree. This could be due to licensing issues (for example not compatible
>> (or
>>> not yet determined if it is compatible (GPL [1])) with the Apache
>> License),
>>> or other thought provoking mild concerns like we're discussing on
>> NIFI-1074
>>> [2].
>>> 
>>> I'd like to propose either capturing these on the website or on the wiki
>> or
>>> some other approach I didn't think of. I was hoping to find a good
>>> archetype for this type of documentation in another apache project, but
>>> didn't find anything I personally liked. If you have seen something you
>>> like or don't, I'd be interested to hear.
>>> 
>>> 
>>> [1] http://www.apache.org/licenses/GPL-compatibility.html
>>> [2] https://issues.apache.org/jira/browse/NIFI-1074
>> 

Re: LogAttribute - Sending that output to a custom logger?

[Joe Witt]

Mark Petronic,

I share Payne's perspective on this.  But I'd also like to work with
you to better understand the workflow.  For those of us that have used
this tool for a long time there is a lot we take for granted from a
new user perspective.  We believe the provenance feature to provide a
far superior option to understanding how an item went through the
system and the timing and what we knew when and so on.  But, it would
be great to understand it from your perspective as someone learning
NiFi.  Not meaning to take away from your proposed contrib - that
would be great too.  Just want to see if the prov user experience
solves what you're looking for and if not can we make it do that.

Thanks
Joe

On Sun, Nov 1, 2015 at 11:23 AM, Mark Payne  wrote:
> Mark,
>
> To make sure that I understand what you're proposing, you want to add a 
> property to
> LogAttribute that allows users to provide a custom logger name?
>
> If that is indeed what you are suggesting then I think it's a great idea.
>
> That being said, in practice I rarely ever use LogAttribute and we even 
> considered removing
> it from the codebase before we open sourced, because the Data Provenance 
> provides a
> much better view of what's going on to debug your flows.
>
> I know you're pretty new to NiFi, so if you've not yet had a chance to play 
> with the Provenance,
> you can see the section in the User Guide at 
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance 
> 
>
> If you're interested in updating the LogAttribute processor, though, we'd be 
> happy to have
> that contribution added, as it does make the Processor more usable.
>
> Thanks
> -Mark
>
>> On Oct 31, 2015, at 12:35 PM, Mark Petronic  wrote:
>>
>> From the code, it appears it cannot be done as the attribute logging
>> goes the same getLogger() instance as the normal nifi-app traces. Has
>> anyone considered making that configurable, maybe allowing you do
>> define a different logger name for LogAttribute then creating that
>> logger definition in log back conf allowing flexibility? I'm using
>> attribute logging heavily as I try to better learn/debug Nifi (it give
>> you a nice 'under the hood' view of the flow) and build up some flows
>> and feel it would be beneficial to be able to capture the LogAttribte
>> message by themselves for more clarity on what is happening. I would
>> not mind maybe trying to implement this feature as my first crack at
>> contributing to the project. Seems like a fairly easy one that would
>> allow me to "go through the motions" of a full pull request process
>> and iron out the process. Anyone have any thoughts on this?
>

Re: LogAttribute - Sending that output to a custom logger?

[Mark Payne]

Mark,

To make sure that I understand what you're proposing, you want to add a 
property to
LogAttribute that allows users to provide a custom logger name?

If that is indeed what you are suggesting then I think it's a great idea.

That being said, in practice I rarely ever use LogAttribute and we even 
considered removing
it from the codebase before we open sourced, because the Data Provenance 
provides a
much better view of what's going on to debug your flows. 

I know you're pretty new to NiFi, so if you've not yet had a chance to play 
with the Provenance,
you can see the section in the User Guide at 
http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance 


If you're interested in updating the LogAttribute processor, though, we'd be 
happy to have
that contribution added, as it does make the Processor more usable.

Thanks
-Mark

> On Oct 31, 2015, at 12:35 PM, Mark Petronic  wrote:
> 
> From the code, it appears it cannot be done as the attribute logging
> goes the same getLogger() instance as the normal nifi-app traces. Has
> anyone considered making that configurable, maybe allowing you do
> define a different logger name for LogAttribute then creating that
> logger definition in log back conf allowing flexibility? I'm using
> attribute logging heavily as I try to better learn/debug Nifi (it give
> you a nice 'under the hood' view of the flow) and build up some flows
> and feel it would be beneficial to be able to capture the LogAttribte
> message by themselves for more clarity on what is happening. I would
> not mind maybe trying to implement this feature as my first crack at
> contributing to the project. Seems like a fairly easy one that would
> allow me to "go through the motions" of a full pull request process
> and iron out the process. Anyone have any thoughts on this?

Re: "External" extensions

[Oleg Zhurakousky]

Well the question still remains unanswered, what relationship those projects 
have to ASF distribution of NiFi? I seriously doubt that anyone on this list 
suggests that all have to be part of the release. And if they are not then they 
are just individual projects managed in/out of ASF, right?

Sent from my iPhone

> On Nov 1, 2015, at 17:10, Adam Estrada  wrote:
> 
> The elasticsearch project has a really cool plugin utility that automatically 
> downloads and builds plugins from GitHub, BitBucket, etc...
> 
> Has anyone taken a look at that?
> 
> A
> 
> Sent from my iPhone
> 
>> On Nov 1, 2015, at 3:54 PM, Benson Margulies  wrote:
>> 
>> ASF policy; a PMC should not be in the business of creating and
>> maintaining code 'somewhere else' and/or under another license, for
>> fear of confusion.
>> 
>> Gray area -- some PMC members can be in that business, as long as the
>> boundary is clear.
>> 
>> There was a thing called 'apache extras' for this. Unfortunately, it
>> was hosted as part of google code, which is defunct. As far as I know,
>> various plans to replace it have not come to fruition, but I might be
>> behind.
>> 
>> 
>> 
>> 
>> 
>>> On Sun, Nov 1, 2015 at 3:04 PM,   wrote:
>>> How about maintaining a registry like npm https://www.npmjs.com or 
>>> https://github.com/jspm/registry where individuals host their modules on 
>>> github and users can discover them via registry?
>>> 
>>> Sent from my iPad
>>> 
 On Nov 1, 2015, at 10:35 AM, Joe Witt  wrote:
 
 " but raises several questions, all pertaining to the relationship of
 this project with ASF, its ownership and control."
 
 ...that is what I'm struggling to respond to as well.
 
 It feels like the right path within the ASF is to establish child
 projects of Apache NiFi.  I think we knew we needed to do this anyway
 as we've mentioned before.  It just might be time now...
 
 On Sun, Nov 1, 2015 at 1:34 PM, Oleg Zhurakousky
  wrote:
> Tony, plenty of opinion but so are the questions/concerns.
> Managing it on GitHub is perfect, but raises several questions, all 
> pertaining to the relationship of this project with ASF, its ownership 
> and control.
> Perhaps some PMCs on the list can shed some light as to how it could be 
> done?
> 
> Cheers
> Oleg
> 
> Sent from my iPhone
> 
>> On Nov 1, 2015, at 13:08, Adam Estrada  wrote:
>> 
>> This has been suggested before. It's a great idea!!! I suggest creating 
>> a repo on github for NiFi-Processors or something like that. There are 
>> many more folks searching through GitHub than on the Apache wikis, IMO. 
>> This will inevitably help spread the word...
>> 
>> A
>> 
>> Sent from my iPhone
>> 
>>> On Nov 1, 2015, at 12:55 PM, Tony Kurc  wrote:
>>> 
>>> Not very strong opinions on this?
 On Oct 30, 2015 10:53 AM, "Joe Witt"  wrote:
 
 Tony,
 
 I completely agree we should do this.  A quick github search reveals
 there are some nice utilities/processors folks have built for NiFi but
 for which they're not necessarily going to submit them as PRs.  We
 should link to these as much as possible but we should also help folks
 understand these aren't 'apache' things and are not of the Apache NiFi
 community directly but they are good for users and developers to know
 about.
 
 Perhaps a wiki page linking to these is good provided we have the
 above sort of disclaimer and a healthy recognition such references
 will become stale...
 
 Thanks
 Joe
 
> On Fri, Oct 30, 2015 at 10:48 AM, Tony Kurc  wrote:
> All,
> I wanted to start a conversation about projects that are good for 
> people
> using or developing NiFi, but either can't or don't belong in the 
> source
> tree. This could be due to licensing issues (for example not 
> compatible
 (or
> not yet determined if it is compatible (GPL [1])) with the Apache
 License),
> or other thought provoking mild concerns like we're discussing on
 NIFI-1074
> [2].
> 
> I'd like to propose either capturing these on the website or on the 
> wiki
 or
> some other approach I didn't think of. I was hoping to find a good
> archetype for this type of documentation in another apache project, 
> but
> didn't find anything I personally liked. If you have seen something 
> you
> like or don't, I'd be interested to hear.
> 
> 
> [1] 

Re: "External" extensions

[Benson Margulies]

ASF policy; a PMC should not be in the business of creating and
maintaining code 'somewhere else' and/or under another license, for
fear of confusion.

Gray area -- some PMC members can be in that business, as long as the
boundary is clear.

There was a thing called 'apache extras' for this. Unfortunately, it
was hosted as part of google code, which is defunct. As far as I know,
various plans to replace it have not come to fruition, but I might be
behind.





On Sun, Nov 1, 2015 at 3:04 PM,   wrote:
> How about maintaining a registry like npm https://www.npmjs.com or 
> https://github.com/jspm/registry where individuals host their modules on 
> github and users can discover them via registry?
>
> Sent from my iPad
>
>> On Nov 1, 2015, at 10:35 AM, Joe Witt  wrote:
>>
>> " but raises several questions, all pertaining to the relationship of
>> this project with ASF, its ownership and control."
>>
>> ...that is what I'm struggling to respond to as well.
>>
>> It feels like the right path within the ASF is to establish child
>> projects of Apache NiFi.  I think we knew we needed to do this anyway
>> as we've mentioned before.  It just might be time now...
>>
>> On Sun, Nov 1, 2015 at 1:34 PM, Oleg Zhurakousky
>>  wrote:
>>> Tony, plenty of opinion but so are the questions/concerns.
>>> Managing it on GitHub is perfect, but raises several questions, all 
>>> pertaining to the relationship of this project with ASF, its ownership and 
>>> control.
>>> Perhaps some PMCs on the list can shed some light as to how it could be 
>>> done?
>>>
>>> Cheers
>>> Oleg
>>>
>>> Sent from my iPhone
>>>
 On Nov 1, 2015, at 13:08, Adam Estrada  wrote:

 This has been suggested before. It's a great idea!!! I suggest creating a 
 repo on github for NiFi-Processors or something like that. There are many 
 more folks searching through GitHub than on the Apache wikis, IMO. This 
 will inevitably help spread the word...

 A

 Sent from my iPhone

> On Nov 1, 2015, at 12:55 PM, Tony Kurc  wrote:
>
> Not very strong opinions on this?
>> On Oct 30, 2015 10:53 AM, "Joe Witt"  wrote:
>>
>> Tony,
>>
>> I completely agree we should do this.  A quick github search reveals
>> there are some nice utilities/processors folks have built for NiFi but
>> for which they're not necessarily going to submit them as PRs.  We
>> should link to these as much as possible but we should also help folks
>> understand these aren't 'apache' things and are not of the Apache NiFi
>> community directly but they are good for users and developers to know
>> about.
>>
>> Perhaps a wiki page linking to these is good provided we have the
>> above sort of disclaimer and a healthy recognition such references
>> will become stale...
>>
>> Thanks
>> Joe
>>
>>> On Fri, Oct 30, 2015 at 10:48 AM, Tony Kurc  wrote:
>>> All,
>>> I wanted to start a conversation about projects that are good for people
>>> using or developing NiFi, but either can't or don't belong in the source
>>> tree. This could be due to licensing issues (for example not compatible
>> (or
>>> not yet determined if it is compatible (GPL [1])) with the Apache
>> License),
>>> or other thought provoking mild concerns like we're discussing on
>> NIFI-1074
>>> [2].
>>>
>>> I'd like to propose either capturing these on the website or on the wiki
>> or
>>> some other approach I didn't think of. I was hoping to find a good
>>> archetype for this type of documentation in another apache project, but
>>> didn't find anything I personally liked. If you have seen something you
>>> like or don't, I'd be interested to hear.
>>>
>>>
>>> [1] http://www.apache.org/licenses/GPL-compatibility.html
>>> [2] https://issues.apache.org/jira/browse/NIFI-1074