Re: Scripted controllers

[Matt Burgess]

Sumo et al,

I thought about scripted Controller Services when I added NIFI-1458 for
scripting reporting tasks. The thing I ran into was that Processors and
Reporting Tasks are technically "consumers", at least in the sense that no
other NiFi "object" knows about their interfaces, innards, etc. Controller
Services are "producers", in the sense that the consumers need to know
about them (and their dependencies) beforehand, or at least the APIs.

For example, the InvokeHttp processor is built into a NAR while declaring a
dependency on the ssl-context-service-api, so they have a compile-time
dependency on the SSLContextService interface. For scripted controller
services, the objects that would use them (presumably scripted processors)
would need to have runtime access to the scripted controller services'
classloaders as they wouldn't have a priori knowledge of the controller
service's interface, and with the NAR setup it doesn't seem available at
runtime.  However ControllerService implementations are loaded via
ServiceLoader, which also makes it a bit harder for scripted CS impls. I
would think some classloader black-magic could be done for such a thing,
but haven't tried it yet.

Having said all that, you can use an existing ControllerService from a
scripting processor. The trick/rule is whether you have access to the
Objects referred to in the ControllerService's interface. For example, with
ExecuteScript (using Groovy) you can get access to a DBCPConnectionPool by
name and do SQL things with it, since the API is part of the normal Java
runtime. I will put together a blog post about this to illustrate, and will
include the above notes.

If you have ideas about how to do a scripted Controller Service, I am
definitely all ears :) There are probably two opportunities: 1) a scripted
impl of an existing interface, and 2) a new scripted ControllerService impl
that could be used in other (presumably scripted) processors.  I welcome
all thoughts, comments, questions, and suggestions!

Regards,
Matt

On Sat, Feb 20, 2016 at 1:40 PM, Joe Witt  wrote:

> Hello Sumo
>
> https://issues.apache.org/jira/browse/NIFI-1458
>
> That is to allow for scripted reporting tasks.
>
> No plans for scripted controller services yet.  What do you have in mind?
>
> Thanks
> Joe
> On Feb 20, 2016 1:26 PM, "Sumanth Chinthagunta"  wrote:
>
> > Since we have scripted processors, any plans to support scripted
> > controllers in the roadmap?
> >
> > -sumo
>

Re: Scripted controllers

[xmlking]

Was planing to upgrade my websocket processor/controller  to 0.5.1.
Just checking, so that I don't have to build if controller scripting is in 
roadmap, Nothing important 
Thanks 
Sumo

> On Feb 20, 2016, at 10:40 AM, Joe Witt  wrote:
> 
> Hello Sumo
> 
> https://issues.apache.org/jira/browse/NIFI-1458
> 
> That is to allow for scripted reporting tasks.
> 
> No plans for scripted controller services yet.  What do you have in mind?
> 
> Thanks
> Joe
>> On Feb 20, 2016 1:26 PM, "Sumanth Chinthagunta"  wrote:
>> 
>> Since we have scripted processors, any plans to support scripted
>> controllers in the roadmap?
>> 
>> -sumo

Re: Accessing the name of a Processor programatically

[Matt Burgess]

I think Aldrin's points about being able to use properties for the UI are
pretty cool, but I also think there's room for the inverse (which is what I
think the OP meant). The name of the processor is tied to its instance
(though not guaranteed unique between instances, and definitely not unique
between tasks within an instance). I would think any
property/attribute/value associated with an instance of a processor should
be available to it (so if name has been a "UI field", it would be no
longer). The tradeoff, then, is when you have to name your processor a
certain way to get it function correctly. I think this should not be the
normal NiFi way, but should be available with the caveat that the
documentation should reflect such a dependency. Sometimes when you're given
enough rope to hang yourself, you build a bridge with it. Other times you
just get dead :-P

Regards,
Matt

On Sun, Feb 21, 2016 at 3:25 PM, Aldrin Piri  wrote:

> I think the functionality is certainly useful and like the deduplication,
> certainly a great idea to be a bit more efficient.  However, I find myself
> a bit hesitant to drive processor configuration via the processor UI
> fields.  At first thought, I think something along the lines of EL that
> could make use of processor properties for UI components like name and
> potentially notes could be helpful.  In this case, the processor UI could
> have its name be defined as something akin to "Retrieve data from ${url}
> every ${run.time}"
>
> Obviously some caveats would be needed such as handling the case of a
> property that is also using EL likely wouldn't work, but personally find
> driving presentation from processor logic to be more intuitive than the
> inverse.
>
> On Sun, Feb 21, 2016 at 7:45 AM, Joe Witt  wrote:
>
> > Richard,
> >
> > I don't believe the processor name (as you see in the UI or through
> > the REST API) is made available to the processor instance itself at
> > this time.  There is no strong/obvious case I can think of quickly to
> > restrict access to the logical name of a ConfigurableComponent like
> > this.
> >
> > Other opinions?
> >
> > Thanks
> > Joe
> >
> > On Sun, Feb 21, 2016 at 8:32 AM, Richard Miskin 
> > wrote:
> > > Hi,
> > >
> > > I’ve a use case where each instance of a custom Processor on my flow
> > represents a different instance of a remote system. The message produced
> by
> > my Processor needs to include the remote system name, and currently I’m
> > using a PropertyDescriptor to allow this to be configured. This means
> that
> > the name is effectively set twice, once to be shown in the UI and once to
> > be used by my Processor internally. This leads to the possibility of the
> > two values being inconsistent which is likely to confuse people in
> future.
> > >
> > > Is there any way I can access the name of a Processor from within an
> > AbstractProcessor?
> > >
> > > Cheers,
> > > Richard
> > >
> > >
> >
>

[GitHub] nifi pull request: NIFI-1542 PutS3Object Processor Logs Errors Wit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/240


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request: NIFI-1542 PutS3Object Processor Logs Errors Wit...

[trkurc]

Github user trkurc commented on the pull request:

https://github.com/apache/nifi/pull/240#issuecomment-186926703
  
Will start reviewing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] NiFi 0.5.1 release

[Oleg Zhurakousky]

1536 will be in shortly 

Sent from my iPhone

> On Feb 21, 2016, at 14:17, Joe Witt  wrote:
> 
> Tony,
> 
> Just chatted with Oleg.  Sounds like he has a potential solution for
> NIFI-1536.  I did some testing for NIFI-1527 that I've reopened so we
> can validate what was observed.  NIFI-1542 is open.
> 
> NIFI-1536 and NIFI-1542 could be moved off of 0.5.1 but NIFI-1527 should not 
> be.
> 
> So looks like we're not quite ready yet.
> 
> Thanks
> Joe
> 
>> On Fri, Feb 19, 2016 at 5:05 PM, Tony Kurc  wrote:
>> sunday works for me.
>> 
>>> On Fri, Feb 19, 2016 at 4:18 PM, Joe Witt  wrote:
>>> 
>>> Ok so looks like the major players for the 0.5.1 release are in.
>>> Today during some testing it was discovered that there can be issues
>>> with multiple HDFS processors with differing configurations and
>>> yesterday on StackOverflow it was reported that there are some issues
>>> talking to Kafka but there is a viable workaround for now.  I
>>> personally feel like what we have for an 0.5.1 is good right now
>>> without those two JIRAs and that we could do an 0.5.2 if those get
>>> sorted soon or they can go in 0.6.0 which if we stick to the
>>> previously discussed idea would be mid-March.
>>> 
>>> What do ya'll think?
>>> 
>>> If we did a kick-off of RC say late Sunday (assuming RM is on board
>>> too of course) we'd be looking at a midweek release for 0.5.1
>>> 
>>> Thanks
>>> Joe
>>> 
 On Wed, Feb 17, 2016 at 9:13 PM, Joe Witt  wrote:
 +1
 
> On Wed, Feb 17, 2016 at 10:12 PM, Tony Kurc  wrote:
> All,
> Just so we're clear, my expectations is that the fixes we need above
>>> will
> go into master (0.6.0-SNAPSHOT). I (or the contributor) will attempt to
> also apply those fixes to a branch I made off of nifi-0.5.0, called
> "support/nifi-0.5.x", and when done we can test and when satisfied,
>>> start
> the 0.5.1 release process.
> 
> 
>> On Wed, Feb 17, 2016 at 8:28 PM, Tony Kurc  wrote:
>> 
>> I lieu of a decision on a git branching strategy, does anyone object
>>> to me
>> naming a branch "support/0.5.x" so I can start pulling together 0.5.1?
>> 
>>> On Wed, Feb 17, 2016 at 8:17 PM, Joe Witt  wrote:
>>> 
>>> Team,
>>> 
>>> First, congrats on getting 0.5.0 out the door.  Very nice release with
>>> excellent new features, enhancements, and bug fixes with involvement
>>> of many new contributors.
>>> 
>>> The effort toward 0.6.0 is well underway and if we are shooting for
>>> our original target would still arrive around mid march-ish.  Seems
>>> reasonable given progress so far.
>>> 
>>> That said, I'd like to propose we go ahead and put out an 0.5.1 as
>>> soon as the patches are addressed.  The finding that Lars helped
>>> expose and that Mark Payne is now working  [1] in my view warrants a
>>> release by itself as it is data loss related.  Also, we can address
>>> the work Matt Gilman is doing to ensure properly authorized users can
>>> access resources they're being blocked by [2] and the license/notice
>>> findings Sean discovered during the last release [3].  There are a
>>> couple odd/ends in the list as well [4].  Purely a bug fix release as
>>> spec'd out in release notes [5].
>>> 
>>> I checked with Tony and he is happy to continue performing RM tasks on
>>> the 0.5.x line.
>>> 
>>> [1] https://issues.apache.org/jira/browse/NIFI-1527
>>> [2] https://issues.apache.org/jira/browse/NIFI-1497
>>> [3] https://issues.apache.org/jira/browse/NIFI-1520
>>> [4] https://s.apache.org/nifi-0.5.1-candidate
>>> [5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
>>> 
>>> Thanks
>>> Joe
> 

Re: [DISCUSS] NiFi 0.5.1 release

[Joe Skora]

I hope to have a patch for NIFI-1542 this afternoon, unless I run into
problems.

On Sun, Feb 21, 2016 at 2:17 PM, Joe Witt  wrote:

> Tony,
>
> Just chatted with Oleg.  Sounds like he has a potential solution for
> NIFI-1536.  I did some testing for NIFI-1527 that I've reopened so we
> can validate what was observed.  NIFI-1542 is open.
>
> NIFI-1536 and NIFI-1542 could be moved off of 0.5.1 but NIFI-1527 should
> not be.
>
> So looks like we're not quite ready yet.
>
> Thanks
> Joe
>
> On Fri, Feb 19, 2016 at 5:05 PM, Tony Kurc  wrote:
> > sunday works for me.
> >
> > On Fri, Feb 19, 2016 at 4:18 PM, Joe Witt  wrote:
> >
> >> Ok so looks like the major players for the 0.5.1 release are in.
> >> Today during some testing it was discovered that there can be issues
> >> with multiple HDFS processors with differing configurations and
> >> yesterday on StackOverflow it was reported that there are some issues
> >> talking to Kafka but there is a viable workaround for now.  I
> >> personally feel like what we have for an 0.5.1 is good right now
> >> without those two JIRAs and that we could do an 0.5.2 if those get
> >> sorted soon or they can go in 0.6.0 which if we stick to the
> >> previously discussed idea would be mid-March.
> >>
> >> What do ya'll think?
> >>
> >> If we did a kick-off of RC say late Sunday (assuming RM is on board
> >> too of course) we'd be looking at a midweek release for 0.5.1
> >>
> >> Thanks
> >> Joe
> >>
> >> On Wed, Feb 17, 2016 at 9:13 PM, Joe Witt  wrote:
> >> > +1
> >> >
> >> > On Wed, Feb 17, 2016 at 10:12 PM, Tony Kurc  wrote:
> >> >> All,
> >> >> Just so we're clear, my expectations is that the fixes we need above
> >> will
> >> >> go into master (0.6.0-SNAPSHOT). I (or the contributor) will attempt
> to
> >> >> also apply those fixes to a branch I made off of nifi-0.5.0, called
> >> >> "support/nifi-0.5.x", and when done we can test and when satisfied,
> >> start
> >> >> the 0.5.1 release process.
> >> >>
> >> >>
> >> >> On Wed, Feb 17, 2016 at 8:28 PM, Tony Kurc  wrote:
> >> >>
> >> >>> I lieu of a decision on a git branching strategy, does anyone object
> >> to me
> >> >>> naming a branch "support/0.5.x" so I can start pulling together
> 0.5.1?
> >> >>>
> >> >>> On Wed, Feb 17, 2016 at 8:17 PM, Joe Witt 
> wrote:
> >> >>>
> >>  Team,
> >> 
> >>  First, congrats on getting 0.5.0 out the door.  Very nice release
> with
> >>  excellent new features, enhancements, and bug fixes with
> involvement
> >>  of many new contributors.
> >> 
> >>  The effort toward 0.6.0 is well underway and if we are shooting for
> >>  our original target would still arrive around mid march-ish.  Seems
> >>  reasonable given progress so far.
> >> 
> >>  That said, I'd like to propose we go ahead and put out an 0.5.1 as
> >>  soon as the patches are addressed.  The finding that Lars helped
> >>  expose and that Mark Payne is now working  [1] in my view warrants
> a
> >>  release by itself as it is data loss related.  Also, we can address
> >>  the work Matt Gilman is doing to ensure properly authorized users
> can
> >>  access resources they're being blocked by [2] and the
> license/notice
> >>  findings Sean discovered during the last release [3].  There are a
> >>  couple odd/ends in the list as well [4].  Purely a bug fix release
> as
> >>  spec'd out in release notes [5].
> >> 
> >>  I checked with Tony and he is happy to continue performing RM
> tasks on
> >>  the 0.5.x line.
> >> 
> >>  [1] https://issues.apache.org/jira/browse/NIFI-1527
> >>  [2] https://issues.apache.org/jira/browse/NIFI-1497
> >>  [3] https://issues.apache.org/jira/browse/NIFI-1520
> >>  [4] https://s.apache.org/nifi-0.5.1-candidate
> >>  [5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
> >> 
> >>  Thanks
> >>  Joe
> >> 
> >> >>>
> >> >>>
> >>
>

Re: [DISCUSS] NiFi 0.5.1 release

[Joe Witt]

Tony,

Just chatted with Oleg.  Sounds like he has a potential solution for
NIFI-1536.  I did some testing for NIFI-1527 that I've reopened so we
can validate what was observed.  NIFI-1542 is open.

NIFI-1536 and NIFI-1542 could be moved off of 0.5.1 but NIFI-1527 should not be.

So looks like we're not quite ready yet.

Thanks
Joe

On Fri, Feb 19, 2016 at 5:05 PM, Tony Kurc  wrote:
> sunday works for me.
>
> On Fri, Feb 19, 2016 at 4:18 PM, Joe Witt  wrote:
>
>> Ok so looks like the major players for the 0.5.1 release are in.
>> Today during some testing it was discovered that there can be issues
>> with multiple HDFS processors with differing configurations and
>> yesterday on StackOverflow it was reported that there are some issues
>> talking to Kafka but there is a viable workaround for now.  I
>> personally feel like what we have for an 0.5.1 is good right now
>> without those two JIRAs and that we could do an 0.5.2 if those get
>> sorted soon or they can go in 0.6.0 which if we stick to the
>> previously discussed idea would be mid-March.
>>
>> What do ya'll think?
>>
>> If we did a kick-off of RC say late Sunday (assuming RM is on board
>> too of course) we'd be looking at a midweek release for 0.5.1
>>
>> Thanks
>> Joe
>>
>> On Wed, Feb 17, 2016 at 9:13 PM, Joe Witt  wrote:
>> > +1
>> >
>> > On Wed, Feb 17, 2016 at 10:12 PM, Tony Kurc  wrote:
>> >> All,
>> >> Just so we're clear, my expectations is that the fixes we need above
>> will
>> >> go into master (0.6.0-SNAPSHOT). I (or the contributor) will attempt to
>> >> also apply those fixes to a branch I made off of nifi-0.5.0, called
>> >> "support/nifi-0.5.x", and when done we can test and when satisfied,
>> start
>> >> the 0.5.1 release process.
>> >>
>> >>
>> >> On Wed, Feb 17, 2016 at 8:28 PM, Tony Kurc  wrote:
>> >>
>> >>> I lieu of a decision on a git branching strategy, does anyone object
>> to me
>> >>> naming a branch "support/0.5.x" so I can start pulling together 0.5.1?
>> >>>
>> >>> On Wed, Feb 17, 2016 at 8:17 PM, Joe Witt  wrote:
>> >>>
>>  Team,
>> 
>>  First, congrats on getting 0.5.0 out the door.  Very nice release with
>>  excellent new features, enhancements, and bug fixes with involvement
>>  of many new contributors.
>> 
>>  The effort toward 0.6.0 is well underway and if we are shooting for
>>  our original target would still arrive around mid march-ish.  Seems
>>  reasonable given progress so far.
>> 
>>  That said, I'd like to propose we go ahead and put out an 0.5.1 as
>>  soon as the patches are addressed.  The finding that Lars helped
>>  expose and that Mark Payne is now working  [1] in my view warrants a
>>  release by itself as it is data loss related.  Also, we can address
>>  the work Matt Gilman is doing to ensure properly authorized users can
>>  access resources they're being blocked by [2] and the license/notice
>>  findings Sean discovered during the last release [3].  There are a
>>  couple odd/ends in the list as well [4].  Purely a bug fix release as
>>  spec'd out in release notes [5].
>> 
>>  I checked with Tony and he is happy to continue performing RM tasks on
>>  the 0.5.x line.
>> 
>>  [1] https://issues.apache.org/jira/browse/NIFI-1527
>>  [2] https://issues.apache.org/jira/browse/NIFI-1497
>>  [3] https://issues.apache.org/jira/browse/NIFI-1520
>>  [4] https://s.apache.org/nifi-0.5.1-candidate
>>  [5] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes
>> 
>>  Thanks
>>  Joe
>> 
>> >>>
>> >>>
>>

[GitHub] nifi pull request: Nifi 1540 - AWS Kinesis Get and Put Processors

[mans2singh]

GitHub user mans2singh opened a pull request:

https://github.com/apache/nifi/pull/239

Nifi 1540 - AWS Kinesis Get and Put Processors

Hi:  I have implemented kinesis consumer and producer.  I've created a new 
base class for aws processors which do not depend on aws client directly.  
Please let me know if you have any comments/recommendations.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mans2singh/nifi Nifi-1540

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/239.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #239


commit d4e3dd7b8e6969c1801fbfe9e4485a4fed3d334b
Author: mans2singh 
Date:   2016-02-12T03:39:08Z

created base class for aws processors

commit 2376bdeb95e1ff83fc81f6e7b76c7d6945d936d2
Author: mans2singh 
Date:   2016-02-12T03:39:32Z

updated pom file for client libs

commit 10f9d84f277096167057bc696e5224999a0e5a18
Author: mans2singh 
Date:   2016-02-14T16:47:22Z

kinesis consumer test

commit 27db6adc466b93325e8412c2d756df9c78e13b6b
Author: mans2singh 
Date:   2016-02-17T04:03:35Z

renamed test class

commit 07cfce2c7dfe13c6db005a9369c273a86d5fcef5
Author: mans2singh 
Date:   2016-02-17T15:35:13Z

support for consumer and refactoring

updated with record handling always allowed

updated consumer properties and added comments

added exception handling

corrected description

refactoring and tests

removed record processor factory from member varible

integration test

added integration tests

added integration and unit tests

commit bf3bda16a03166329f5a235b8a6d31c15d23f8d5
Author: mans2singh 
Date:   2016-02-21T04:29:42Z

updated comment

commit 14cd73f9ff7d3ab1cce5671de0af29f4d5184692
Author: mans2singh 
Date:   2016-02-21T04:35:55Z

comments cleanup and ignoreintegration tests

commit d2cc2a03f0744df1951e9914e5d0d33d403fe9bf
Author: mans2singh 
Date:   2016-02-21T07:13:57Z

synched with mastr and resolved conflicts

commit 34278599d48b141c1059fb3ef7a0b6abc7c6caef
Author: mans2singh 
Date:   2016-02-21T07:20:21Z

uncommented integration tests

commit 4f5eb90f446324e1aec1b1d737c5371ce6006f81
Author: mans2singh 
Date:   2016-02-21T07:22:41Z

removed unused imports

commit 4020abdf277df1d5e252e7cbc13d947852050532
Author: mans2singh 
Date:   2016-02-21T15:32:05Z

commented out integration tests

commit 7d85f655d483bb19f41bb74d9a7650f9d1f4ce04
Author: mans2singh 
Date:   2016-02-21T16:21:23Z

checkstyle errors resolution

commit fcd3eaf6ed27fe272ccb0245f82035154bc290b3
Author: mans2singh 
Date:   2016-02-21T16:27:19Z

minor comment updates




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Accessing the name of a Processor programatically

[Joe Witt]

Richard,

I don't believe the processor name (as you see in the UI or through
the REST API) is made available to the processor instance itself at
this time.  There is no strong/obvious case I can think of quickly to
restrict access to the logical name of a ConfigurableComponent like
this.

Other opinions?

Thanks
Joe

On Sun, Feb 21, 2016 at 8:32 AM, Richard Miskin  wrote:
> Hi,
>
> I’ve a use case where each instance of a custom Processor on my flow 
> represents a different instance of a remote system. The message produced by 
> my Processor needs to include the remote system name, and currently I’m using 
> a PropertyDescriptor to allow this to be configured. This means that the name 
> is effectively set twice, once to be shown in the UI and once to be used by 
> my Processor internally. This leads to the possibility of the two values 
> being inconsistent which is likely to confuse people in future.
>
> Is there any way I can access the name of a Processor from within an 
> AbstractProcessor?
>
> Cheers,
> Richard
>
>

Accessing the name of a Processor programatically

[Richard Miskin]

Hi,

I’ve a use case where each instance of a custom Processor on my flow represents 
a different instance of a remote system. The message produced by my Processor 
needs to include the remote system name, and currently I’m using a 
PropertyDescriptor to allow this to be configured. This means that the name is 
effectively set twice, once to be shown in the UI and once to be used by my 
Processor internally. This leads to the possibility of the two values being 
inconsistent which is likely to confuse people in future.

Is there any way I can access the name of a Processor from within an 
AbstractProcessor?

Cheers,
Richard