Re: Re: a question about nifi's source code

[YuNing]

I fround the problem, recently. there is no existance check of tenants 
before doUpdateGroup() at the most recent version on github. but it checks at  
nifi-1.3.0 release version. you can see this at 
https://github.com/apache/nifi/blob/rel/nifi-1.3.0/nifi-framework-api/src/main/java/org/apache/nifi/authorization/AbstractPolicyBasedAuthorizer.java
 
and it doesn't seams to be a bug ,because everything just works fine.



Best Regards
YuNing
 
From: YuNing
Date: 2017-08-02 14:43
To: dev
Subject: Re: Re: a question about nifi's source code
Hi，Andy 
It seems I found the problems, I compare the sourcecode which i am viewing 
to the sourcecode on github, it appears to be different, and i am confused, 
because i didn't change it myslf, i will keep this update as long as i found 
the reason. 
thanks you very much.



Best Regards
YuNing
 
From: Andy LoPresto
Date: 2017-08-02 14:01
To: dev
Subject: Re: a question about nifi's source code
Hi YuNing,
 
The Abstract policy authorizer delegates the updateGroup action to 
doUpdateGroup, which is implemented by the extending concrete class. I'm not 
sure where you are seeing that it checks for the absence of the group, but 
there is an addGroup method for adding a new group. 
 
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
 
> On Aug 1, 2017, at 19:48, YuNing  wrote:
> 
> Hello，everyone
>I am work on reading nifi's source code . it confuse me when i read 
> the class "AbstractPolicyBasedAuthorizer", I found there is no method to add 
> a user to an existing group. I noticed there is an "updateGroup" method 
> ,which i don't think can be used to add users to existing group, because it 
> checks the input group to make sure it didn't exist.
>Am i right?  can anyone help me, thanks alot !
> 
> Best Regards
> YuNing
> 

Re: how to execute code when processor is stopping

[尹文才]

Hi guys, about my case, I have another question, if I implement the retry
logic inside the ontrigger method and I need to retry until the database
connection is back online, in case user needs to stop the processor in NIFI
UI while the database is still offline, according to my understanding
ontrigger will keep executing the retry logic and the processor couldn't be
stopped even if user tries to stop it, is there any way to solve this
problem? Thanks.

Regards,
Ben

2017-08-12 6:30 GMT+08:00 尹文才 :

> Hi Bryan and Matt, thanks for all your suggestions, I was trying to make
> sure that the OnUnscheduled method was not called too frequently when the
> connection is offline.
> You guys were right, these sort of logic should not be placed inside the
> scheduling methods, I need to refactor my code to place them into onTrigger.
>
> Regards,
> Ben
>
> 2017-08-12 0:53 GMT+08:00 Matt Burgess :
>
>> I'm a fan of Bryan's last suggestion. For dynamic/automatic retry
>> (such as database connection retries), I recommend putting the
>> connection logic in the onTrigger() method. If you can check
>> connectivity, then your onTrigger() would know whether it needs to try
>> to reconnect before it does any work. If it tries to reconnect and is
>> unsuccessful, you can yield the processor if you want, so as not to
>> hammer the DB with connection attempts. The CaptureChangeMySQL
>> processor does this, it has a retry loop for trying various nodes in a
>> MySQL cluster, but once it's connected, it goes on about its work, and
>> if a connection fails, it will retry the connection loop before it
>> does any more work. It only uses onTrigger and none of the scheduling
>> stuff.
>>
>> Regards,
>> Matt
>>
>> On Fri, Aug 11, 2017 at 11:06 AM, Bryan Bende  wrote:
>> > Ben,
>> >
>> > I apologize if I am not understanding the situation, but...
>> >
>> > In the case where your OnScheduled code is in a retry loop, if someone
>> > stops the processor it will call your OnUnscheduled code which will
>> > set the flag to bounce out of the loop. This sounds like what you
>> > want, right?
>> >
>> > In the case where OnScheduled times out, the framework is calling
>> > OnUnscheduled which would call your code to set the flag, but wouldn't
>> > that not matter at this point because you aren't looping anymore
>> > anyway?
>> >
>> > If the framework calls OnScheduled again, your code should set the
>> > flag back to whatever it needs to be to start looping again right?
>> >
>> > An alternative that might avoid some of this would be to lazily
>> > initialize the connection in the onTrigger method of the processor.
>> >
>> > -Bryan
>> >
>> >
>> > On Fri, Aug 11, 2017 at 9:16 AM, 尹文才  wrote:
>> >> thanks Pierre, my case is that I need to implement a database
>> connection
>> >> retry logic inside my OnScheduled method, when the database is not
>> >> available I will retry until the connection is back online.
>> >> The problem is when the database is offline it will throw timed out
>> >> execution exception inside OnScheduled and then call OnUnscheduled. But
>> >> when I manually stop the processor the OnUnsheduled
>> >> will also get called. I know my logic sounds a little weird but I need
>> to
>> >> set some flag in the OnUnscheduled method to stop the retry logic
>> inside
>> >> OnScheduled in order to be able to stop the processor,
>> >> otherwise the processor is not able to be stopped unless I restart the
>> >> whole NIFI.
>> >>
>> >> Regards,
>> >> Ben
>> >>
>> >> 2017-08-11 17:18 GMT+08:00 Pierre Villard > >:
>> >>
>> >>> Oh OK, get it now!
>> >>>
>> >>> Not sure what's your use case, but I don't think you can do that
>> unless you
>> >>> set some information when the process actually executes onTrigger for
>> the
>> >>> first time and you then check this value in your OnUnscheduled
>> annotated
>> >>> method.
>> >>>
>> >>> Pierre
>> >>>
>> >>> 2017-08-11 10:11 GMT+02:00 尹文才 :
>> >>>
>> >>> > Hi Pierre, I've checked the developer guide before I sent the email
>> and
>> >>> > according to the developer guide, the method annotated with
>> OnUnScheduled
>> >>> > will be called in 2 cases according to my understanding, please
>> correct
>> >>> me
>> >>> > if I'm wrong:
>> >>> > 1. when user tries to stop the processor in the NIFI UI, thus the
>> >>> processor
>> >>> > is no longer scheduled to run in this case, and the method will be
>> >>> called.
>> >>> > 2. when method annotated with OnScheduled throws exceptions, for
>> example
>> >>> > time out execution exception, the OnUnScheduled method will also be
>> >>> called.
>> >>> >
>> >>> > My question is how to tell the first scenario from the second one?
>> >>> Thanks.
>> >>> >
>> >>> > Regards,
>> >>> > Ben
>> >>> >
>> >>> > 2017-08-11 15:51 GMT+08:00 Pierre Villard <
>> pierre.villard...@gmail.com>:
>> >>> >
>> >>> > > Hi Ben,
>> >>> > >
>> >>> > > You might 

Re: NIFI-4198 - *ElasticsearchHttp processors do not expose Proxy settings

[Arun Manivannan]

Hi,

I notice that the AbstractElasticsearchHttpProcessor already has support
for the proxy host and port.  I suppose all I need to do now is to

1. Add the properties to the PropertyDescriptors list
2. Add support for proxy authentication
3. System test.

Can I get away with just this?

Cheers,
Arun



On Sun, Aug 13, 2017 at 9:06 AM Arun Manivannan  wrote:

> Absolutely, I will.
>
> Is uploading the steps as a word document to the JIRA an appropriate way?
> I understand that these steps are specific to a set of JIRAs and the nifi
> dev guide may or may not be a good home for this. (I can look up how to do
> that if this is the case).
>
> Thanks again, Joe.
> Cheers
> Arun
>
> On Sun, Aug 13, 2017, 03:54 Joe Witt  wrote:
>
>> Thanks Arun.  If you can document the steps you went through the setup
>> a representative testing environment that would *greatly* help
>> whomever reviews to do the same/similar.  The review pipeline for
>> contributions like this can be tough and it is almost always due to
>> helping the reviewer setup the environment.
>>
>> Thanks
>>
>> On Sat, Aug 12, 2017 at 12:48 PM, Arun Manivannan 
>> wrote:
>> > Great.  Thanks a lot, Joe.  Appreciate your help.
>> >
>> > Will get it up and running before I assign the issue to myself.
>> >
>> > Regards,
>> > Arun
>> >
>> > On Sun, Aug 13, 2017 at 3:24 AM Joe Witt  wrote:
>> >
>> >> Arun
>> >>
>> >> Very cool that you are planning to jump in on this.  Your approach
>> >> sounds like a good start.  As far as system testing you're hitting on
>> >> one of the more challenging parts of the equation here.  Your unit
>> >> tests of course won't integrate with a real ES instance but manual
>> >> testing can be done against a system as you mention.  I think a lot of
>> >> folks use things like Docker to do such testing or sometimes these
>> >> systems offer quick start configurations.  You could put a squid proxy
>> >> instance/container in front as well.  Perhaps this one can help
>> >> https://github.com/sameersbn/docker-squid
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Sat, Aug 12, 2017 at 11:49 AM, Arun Manivannan 
>> wrote:
>> >> > Hi,
>> >> >
>> >> > Very Good morning.
>> >> >
>> >> > I would like to make an attempt at resolving NIFI-4198
>> >> > .  Looking at the
>> >> code, I
>> >> > would think by introducing the proxy url and authentication
>> properties
>> >> and
>> >> > delegating them to the OkHttpClient would be a good way to do it.
>> >> >
>> >> >
>> >>
>> https://stackoverflow.com/questions/35554380/okhttpclient-proxy-authentication-how-to
>> >> >
>> >> > The question I have is beside the test cases, I would like to test it
>> >> > against an actual elastic server to be absolutely sure of the fix.
>> >> What's
>> >> > the easiest/good way to run an elastic instance behind a proxy and
>> test
>> >> the
>> >> > modified component?
>> >> >
>> >> > Thanks in advance.
>> >> >
>> >> > Cheers,
>> >> > Arun
>> >>
>>
>