Re: Redshift driver class is not working dbcpconnectionpool for 1.16.1 version
Hi , Any updates on the trailed mail is really helpful. Thanks & Regards Sanjeet On Tue, 7 Jun 2022 at 10:18 PM, sanjeet rath wrote: > Hi , > > I am seeing one different behaviour of dbcpconnectionpool service in > 1.16.1 version of nifi. > > I am passing the Database driver class name : Jdbc.redshift.RedshiftDriver > And DB connection URL. I am passing the required name & pwd and other > things. > Driver path i am not passing anything. > > With these values I am seeing no error in 1.12.1 version of nofi , > however in 1.16.1 i am seeing “Driver class jdbc.redshift.redshift.redshift > not found.” Screenshot is attached for your reference. > > Could you please let me know this behaviour is due to some change in > 1.16.1 as the same controller with the same config 1.12.1 no issue . > > Also please let me know how to solve this issue. > > Thanks & Regards > Sanjeet > > > > -- Sanjeet Kumar Rath, mob- +91 8777577470
Redshift driver class is not working dbcpconnectionpool for 1.16.1 version
Hi , I am seeing one different behaviour of dbcpconnectionpool service in 1.16.1 version of nifi. I am passing the Database driver class name : Jdbc.redshift.RedshiftDriver And DB connection URL. I am passing the required name & pwd and other things. Driver path i am not passing anything. With these values I am seeing no error in 1.12.1 version of nofi , however in 1.16.1 i am seeing “Driver class jdbc.redshift.redshift.redshift not found.” Screenshot is attached for your reference. Could you please let me know this behaviour is due to some change in 1.16.1 as the same controller with the same config 1.12.1 no issue . Also please let me know how to solve this issue. Thanks & Regards Sanjeet
Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments
Thanks David,
Really appreciate it.
It clarified everything.
On Fri, 14 Jan 2022, 12:40 am David Handermann,
wrote:
> Hi Sanjeet,
>
> That's correct. Setting the JVM arguments for proxy server access can
> create the problems you observed, and it is unlikely to work as expected
> when it comes to proxy access. Various NiFi components use different
> methods for creating network connections. For this reason, enabling
> outgoing proxy server access will be more reliable using the
> ProxyConfigurationService. If there specific components that have issues
> with proxy server connectivity, it would be worth creating a Jira issue for
> those components. If proxy server access is limited to your custom
> component, then the best approach is to integrate the
> ProxyConfigurationService and determine how that should be wired to your
> custom component library.
>
> Regards,
> David Handermann
>
> On Thu, Jan 13, 2022 at 1:02 PM sanjeet rath
> wrote:
>
> > Thanks a lot for the quick response.
> >
> > So u r suggesting we should not use proxy in jvm argument(nifi
> > bootstrap.conf file) lable also as it will impact other component.like in
> > my case its impacting nifi-api.
> >
> > Regards,
> > Sanjeet
> >
> > On Fri, 14 Jan 2022, 12:19 am David Handermann, <
> > exceptionfact...@apache.org>
> > wrote:
> >
> > > Hi Sanjeet,
> > >
> > > Thank you for providing the stack trace and details of your
> > configuration.
> > > Setting Java System properties in custom code is not a safe or
> supported
> > > operation in NiFi. As you have observed, setting system properties
> > alters
> > > the behavior of other components, leading to unexpected results.
> > >
> > > If you need to support access through a proxy server, the
> > > ProxyConfigurationService interface and
> StandardProxyConfigurationService
> > > implementation provide a way to specify proxy server properties:
> > >
> > >
> > >
> >
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I encounter one "java.net.NoRouteToHostException: No route to host
> > > (Host
> > > > unreachable)" in *Nifi UI login page *.
> > > > The after debugging i realised when i am setting the proxy & port in
> > > > *System.setProperty("proxy address") & System.setProperty("port
> > address")
> > > > *in
> > > > my custom processor. then this issue is appearing .
> > > >
> > > > The other way i also replicated when i am setting at the* jvm
> > lable(in
> > > > bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application
> > this
> > > > exception is coming in nifi ui login page:, After removal of this
> > > argument
> > > > it works fine.
> > > >
> > > > Could someone help me to understand what could be the issue.
> > > >
> > > > Nifi version: 1.12.1
> > > > cluster : 3 node (amazon EC2 linux cluster)
> > > >
> > > > *Detail exception from nifi-app.log:*
> > > >
> > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> > GET
> > > > /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> > > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > > >
> > > > 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> > > > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> > > >
> > > > java.net.NoRouteToHostException: No route to host (Host unreachable)
> > > >
> > > > at java.base/java.net.PlainSocketImpl.socketConnect(Native
> > > Method)
> > > >
> > > > at
> > > > java.base/java.net
> > > > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
> > > >
> > > > at
> > > > java.base/java.net
> > > >
> > >
> >
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
> > > >
> > > > at
> > > >
Re: NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments
Thanks a lot for the quick response.
So u r suggesting we should not use proxy in jvm argument(nifi
bootstrap.conf file) lable also as it will impact other component.like in
my case its impacting nifi-api.
Regards,
Sanjeet
On Fri, 14 Jan 2022, 12:19 am David Handermann,
wrote:
> Hi Sanjeet,
>
> Thank you for providing the stack trace and details of your configuration.
> Setting Java System properties in custom code is not a safe or supported
> operation in NiFi. As you have observed, setting system properties alters
> the behavior of other components, leading to unexpected results.
>
> If you need to support access through a proxy server, the
> ProxyConfigurationService interface and StandardProxyConfigurationService
> implementation provide a way to specify proxy server properties:
>
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-proxy-configuration-nar/1.15.2/org.apache.nifi.proxy.StandardProxyConfigurationService/index.html
>
> Regards,
> David Handermann
>
> On Thu, Jan 13, 2022 at 12:39 PM sanjeet rath
> wrote:
>
> > Hi,
> >
> > I encounter one "java.net.NoRouteToHostException: No route to host
> (Host
> > unreachable)" in *Nifi UI login page *.
> > The after debugging i realised when i am setting the proxy & port in
> > *System.setProperty("proxy address") & System.setProperty("port address")
> > *in
> > my custom processor. then this issue is appearing .
> >
> > The other way i also replicated when i am setting at the* jvm lable(in
> > bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application this
> > exception is coming in nifi ui login page:, After removal of this
> argument
> > it works fine.
> >
> > Could someone help me to understand what could be the issue.
> >
> > Nifi version: 1.12.1
> > cluster : 3 node (amazon EC2 linux cluster)
> >
> > *Detail exception from nifi-app.log:*
> >
> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> > /nifi-api/flow/current-user to "exampledumyserveraddress" due to
> > java.net.NoRouteToHostException: No route to host (Host unreachable)
> >
> > 2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
> > o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> >
> > java.net.NoRouteToHostException: No route to host (Host unreachable)
> >
> > at java.base/java.net.PlainSocketImpl.socketConnect(Native
> Method)
> >
> > at
> > java.base/java.net
> > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
> >
> > at
> > java.base/java.net
> >
> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
> >
> > at
> > java.base/java.net
> > .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
> >
> > at
> > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
> >
> > at java.base/java.net.Socket.connect(Socket.java:609)
> >
> > at
> > okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
> >
> > at
> >
> >
> okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
> >
> > at
> >
> >
> okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)
> >
> > at
> >
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)
> >
> > at
> >
> >
> okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
> >
> > at
> >
> >
> okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
> >
> > at
> > okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
> >
> > at
> > okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
> >
> > at
> >
> >
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
> >
> > at
> >
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
> >
> > at
> >
> >
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorCh
NoRouteToHostException is coming in NIfi UI login pageafter setting proxy in jvm arguments
Hi,
I encounter one "java.net.NoRouteToHostException: No route to host (Host
unreachable)" in *Nifi UI login page *.
The after debugging i realised when i am setting the proxy & port in
*System.setProperty("proxy address") & System.setProperty("port address") *in
my custom processor. then this issue is appearing .
The other way i also replicated when i am setting at the* jvm lable(in
bootstrap.conf file -Dhttp.proxyHost=address) *for nifi application this
exception is coming in nifi ui login page:, After removal of this argument
it works fine.
Could someone help me to understand what could be the issue.
Nifi version: 1.12.1
cluster : 3 node (amazon EC2 linux cluster)
*Detail exception from nifi-app.log:*
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
/nifi-api/flow/current-user to "exampledumyserveraddress" due to
java.net.NoRouteToHostException: No route to host (Host unreachable)
2022-01-13 12:57:23,722 WARN [Replicate Request Thread-5]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator
java.net.NoRouteToHostException: No route to host (Host unreachable)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at
java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at
java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:609)
at
okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
at
okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
at
okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.java:235)
at
okhttp3.internal.connection.RealConnection.connect(RealConnection.java:177)
at
okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at
okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at
okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at
okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at
okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at
okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at
okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at
okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
at okhttp3.RealCall.execute(RealCall.java:81)
at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:132)
at
org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:126)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647)
at
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Thanks & Regards
--
Sanjeet Kumar Rath,
mob- +91 8777577470
End of Life for mail-1.4.7.jar file
Hi The latest 1.15 Nifi version contain mail-1.4.7.jar file(inside \lib\bootstrap folder). We r using 1.12.1 version of nifi same issue is also there. Actually this mail-1.4.7.jar has reached end of life (latest verion was released on 2013). Could someone let me know r we planing to update this jar in future version. Thanks, Sanjeet
Re: How to restrict custom processor execution time
Thanks Chris & Joe for the wonderfull ideas. Now i am clear how to acheive this functionality. Regards, Sanjeet On Thu, 12 Aug 2021, 11:06 pm Chris Sampson, wrote: > Consider using a static class member variable maybe (i.e. use a java-based > solution instead of trying to find a nifi specific one). Bear in mind that > "last execution" would then be reset if the node on which the processor is > scheduled gets restarted. If you're in a clustered environment then the > processor scheduling would move to another node automatically, but that may > trigger another execution within the same hour. > > You could start using State in the processor (local or cluster) if you > wanted a nifi-based solution, although you'll have to consider what happens > when the state is reset, e.g. by a user in the UI or if the node on which > the processor is scheduled changes within the cluster (and you're using > local state) - would this be OK or cause a problem, etc.? Using > remote/cluster state would allow the same "last execution" to be stored for > all nodes across the cluster, so even if the processor scheduling is moved > between nodes, it should be able to determine whether to run. > > There are @TriggerSerially and @PrimaryNodeOnly annotations for processors > to ensure they can only be scheduled on a single node (current "primary") > within a cluster and to only run one instance at any one time. > > @DefaultSchedule annotation can be used to configure the default processor > schedule (e.g. every "1 hour"), but a user could still change that in the > UI. > > Last bit of advice from me would be to call "context.yield" in "onTrigger" > if the processor detects it should skip the current execution because it's > within the same hour as the "last execution" - this allows nifi to schedule > other processors to run instead of immediately retrying this processor and > taking up one of the available threads in the pool (if someone accidentally > configures it to run continually). > > > Cheers, > > Chris Sampson > > On Thu, 12 Aug 2021, 18:05 sanjeet rath, wrote: > > > Hi Joe, > > > > Keeping track of last execution & comparing with current time looks good > > solution for my use case.There is no additional complication also. > > > > I am thinking of storing the "last execution time" in a processor > > parameter.after every succesfull execution i will populate the value .so > > Unless "current time" of execution > last execution + 60 min . The > > processor won't do it function. > > > > Is there any way can i hide this parameter in UI. > > > > Regards, > > Sanjeet > > > > On Thu, 12 Aug 2021, 10:18 pm Joe Witt, wrote: > > > > > Sanjeet > > > > > > What about keeping track of 'last execution' and 'current time' to > > > give you enough to decide if it has been at least 60 mins? Are there > > > additional complications to consider? > > > > > > Thanks > > > > > > On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath > > > wrote: > > > > > > > > Thanks , Joe for the quick reply. > > > > I was wondering how can i figure out inside the processor's Ontrigger > > > > method when it was run previously. > > > > Basicaly want to implement ur suggestion of restricting the execution > > > > inside the processor's code . > > > > > > > > Regards, > > > > Sanjeet > > > > > > > > On Thu, 12 Aug 2021, 9:43 pm Joe Witt, wrote: > > > > > > > > > Sanjeet > > > > > > > > > > We dont presently allow the processor developer to put such a > > > > > restriction into the code. Your best bet for now would be to > > document > > > > > that the processor even if scheduled more than once in a one hour > > > > > period will not execute its function more than once. Then in your > > > > > code you can protect/ensure it only truly does the execution once > and > > > > > hour. > > > > > > > > > > Thanks > > > > > Joe > > > > > > > > > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath < > rath.sanj...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > Hi , > > > > > > > > > > > > I am building a custom processor and there is restriction i want > to > > > put > > > > > for > > > > > > the processor that it should not schedule to run 2 times in 1 > hour > > > time > > > > > > period. > > > > > > > > > > > > I can acheive this by passing "run schedule" 60 mins. > > > > > > > > > > > > Is there any other way i can do in my custom processor code, So > > that > > > it > > > > > > won't allow the user to select "run schedule " time less than 60 > > > > > > mins.basically similar to we can restrict the procesor to execute > > on > > > > > > prinary node. > > > > > > > > > > > > Any other thought is really helpfull. > > > > > > > > > > > > Thanks, > > > > > > Sanjeet > > > > > > > > > > >
Re: How to restrict custom processor execution time
Hi Joe, Keeping track of last execution & comparing with current time looks good solution for my use case.There is no additional complication also. I am thinking of storing the "last execution time" in a processor parameter.after every succesfull execution i will populate the value .so Unless "current time" of execution > last execution + 60 min . The processor won't do it function. Is there any way can i hide this parameter in UI. Regards, Sanjeet On Thu, 12 Aug 2021, 10:18 pm Joe Witt, wrote: > Sanjeet > > What about keeping track of 'last execution' and 'current time' to > give you enough to decide if it has been at least 60 mins? Are there > additional complications to consider? > > Thanks > > On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath > wrote: > > > > Thanks , Joe for the quick reply. > > I was wondering how can i figure out inside the processor's Ontrigger > > method when it was run previously. > > Basicaly want to implement ur suggestion of restricting the execution > > inside the processor's code . > > > > Regards, > > Sanjeet > > > > On Thu, 12 Aug 2021, 9:43 pm Joe Witt, wrote: > > > > > Sanjeet > > > > > > We dont presently allow the processor developer to put such a > > > restriction into the code. Your best bet for now would be to document > > > that the processor even if scheduled more than once in a one hour > > > period will not execute its function more than once. Then in your > > > code you can protect/ensure it only truly does the execution once and > > > hour. > > > > > > Thanks > > > Joe > > > > > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath > > > wrote: > > > > > > > > Hi , > > > > > > > > I am building a custom processor and there is restriction i want to > put > > > for > > > > the processor that it should not schedule to run 2 times in 1 hour > time > > > > period. > > > > > > > > I can acheive this by passing "run schedule" 60 mins. > > > > > > > > Is there any other way i can do in my custom processor code, So that > it > > > > won't allow the user to select "run schedule " time less than 60 > > > > mins.basically similar to we can restrict the procesor to execute on > > > > prinary node. > > > > > > > > Any other thought is really helpfull. > > > > > > > > Thanks, > > > > Sanjeet > > > >
Re: How to restrict custom processor execution time
Thanks , Joe for the quick reply. I was wondering how can i figure out inside the processor's Ontrigger method when it was run previously. Basicaly want to implement ur suggestion of restricting the execution inside the processor's code . Regards, Sanjeet On Thu, 12 Aug 2021, 9:43 pm Joe Witt, wrote: > Sanjeet > > We dont presently allow the processor developer to put such a > restriction into the code. Your best bet for now would be to document > that the processor even if scheduled more than once in a one hour > period will not execute its function more than once. Then in your > code you can protect/ensure it only truly does the execution once and > hour. > > Thanks > Joe > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath > wrote: > > > > Hi , > > > > I am building a custom processor and there is restriction i want to put > for > > the processor that it should not schedule to run 2 times in 1 hour time > > period. > > > > I can acheive this by passing "run schedule" 60 mins. > > > > Is there any other way i can do in my custom processor code, So that it > > won't allow the user to select "run schedule " time less than 60 > > mins.basically similar to we can restrict the procesor to execute on > > prinary node. > > > > Any other thought is really helpfull. > > > > Thanks, > > Sanjeet >
How to restrict custom processor execution time
Hi , I am building a custom processor and there is restriction i want to put for the processor that it should not schedule to run 2 times in 1 hour time period. I can acheive this by passing "run schedule" 60 mins. Is there any other way i can do in my custom processor code, So that it won't allow the user to select "run schedule " time less than 60 mins.basically similar to we can restrict the procesor to execute on prinary node. Any other thought is really helpfull. Thanks, Sanjeet
Re: Regarding AWScredentialproviderService
Hi , please ignore my previous mail.as i got the answer. On Wed, 30 Oct, 2019, 8:16 AM sanjeet rath, wrote: > Hi, > I have built custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE) > and use this one in processor service(NIFI-AWS-SERVICE).for puts3Object. > In this controller service i have used custom parameters to connect to aws > and getting token , also using this token i am able to to use connect AWS > and able to upload object using puts3object processor. > > But i am not able to understand few things , > 1) How token refresh works in custom controller service. > 2) Does it refresh after time out. > > Regards, > Sanjeet Kumar Rath, > mob- +91 8777577470 > >
Regarding AWScredentialproviderService
Hi, I have built custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE) and use this one in processor service(NIFI-AWS-SERVICE).for puts3Object. In this controller service i have used custom parameters to connect to aws and getting token , also using this token i am able to to use connect AWS and able to upload object using puts3object processor. But i am not able to understand few things , 1) How token refresh works in custom controller service. 2) Does it refresh after time out. Regards, Sanjeet Kumar Rath, mob- +91 8777577470
Clarification Regarding custom controller service & AWS-CREDENTIAL-CONTROLER-SERVICE
Hi Team, I have a requirement of building a custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE) and custom processor service(NIFI-AWS-SERVICE).for puts3Object. Already code changes are done for both of it. and i am building 2 separate nars . The custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE) nar contain dependency of custom processor service(NIFI-AWS-SERVICE). So for puts3Object processor, it should use my custom controller service. My question is, Should i build 2 separate nars , 1 for custom processor service(NIFI-AWS-SERVICE) and 1 for custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE). and putting both the nars , in lib folder to make nifi work. Or should i build only 1 nar file for custom controller service(AWS-CREDENTIAL-CONTROLER-SERVICE) as it has a pom dependency of custom processor service(NIFI-AWS-SERVICE). Please clarify. Regards, Sanjeet -- Sanjeet Kumar Rath, mob- +91 8777577470
Re: Unable to do unit testing Awscredentialprovidercontrolerservice
Hi Team,
please help me with the trailed mail issue.i got stuck here.
Thanks & Regards
sanjeet
On Fri, 18 Oct, 2019, 10:02 PM sanjeet rath, wrote:
> Hi ,
>
> While testing a controller service (Awscredentialprovidercontrolerservice)
> i am getting IlligalStateException,
> in the runner.enablecontrolerService(...) , when i am removing LINE3 code
> i am getting null pointer exception in the useStratergy value(in the custom
> validate).
> What i have annalysed the properties list, i am getting all the
> properties values that are present in the default
> Awscredentialprovidercontrolerservice class.
> It should have my properties list which i have defined in the custom
> Awscredentialprovidercontrolerservice class.
>
>
> code:
> @Test
> public void testAwscredentialprovidercontrolerservice() {
> final TestRunner runner = TestRunners.newTestRunner(puts3Object.class);
> Final Awscredentialprovidercontrolerservice serviceImpl = new
> Awscredentialprovidercontrolerservice()
> Runner.setProperty(..)
>
> #LINE3#runner.setProperty(serviceImpl,AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE,"awsCredsProviderService");
> runner.enablecontrolerService(serviceImpl)
>}
>
> In the custom Awscredentialprovidercontrolerservice I have added ,
>
> public static final PropertyDescriptor AWS_CONTROLER_SERVICE = new
> PropertyDescriptor.Builder()
> .name(“AWS Service")
> .description("Specified the SSL Context Service that can be used to
> create secure connections")
> .required(true)
> .identifiesControllerService(Awscredentialprovidercontrolerservice
> .class)
>
> Thanks,
> Sanjeet
>
>
> On Thu, 17 Oct, 2019, 11:42 PM Bryan Bende, wrote:
>
>> You should be able use mockito (or any other mocking framework),
>> something like...
>>
>> AWSCredentialsProviderService service =
>> Mockito.mock(AWSCredentialsProviderService.class)
>> //setup whatever you want the mock to do
>>
>> runner.setProperty(service,
>> AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE,
>> "awsCredsProviderService");
>> runner.enableControllerService(service);
>>
>> On Thu, Oct 17, 2019 at 1:54 PM sanjeet rath
>> wrote:
>> >
>> > Thanks a lot bryan , for helping me to identify proper UNIT testing.
>> > I will go with ur suggestion, will write unit testing for custom
>> > controlerservice and separate for custom processor service.
>> > If possible Can u give me a example of mock the
>> > AWSCredentialsProviderService interface.
>> >
>> > Regards,
>> > sanjeet
>> >
>> > On Thu, Oct 17, 2019 at 9:41 PM Bryan Bende wrote:
>> >
>> > > Please only email one list, removing users...
>> > >
>> > > The AWS processors depend on an interface called
>> > > AWSCredentialsProviderService. The unit test of the processor should
>> > > not need to depend on real implementations of this interface,
>> > > otherwise it is really an integration test of multiple components.
>> > >
>> > > I would suggest testing your processor and service separately... the
>> > > processor test can mock the AWSCredentialsProviderService interface.
>> > >
>> > >
>> > > On Thu, Oct 17, 2019 at 11:50 AM sanjeet rath > >
>> > > wrote:
>> > > >
>> > > > Hi Team,
>> > > >
>> > > > I am new to this community, today i have joined and this is my first
>> > > query.(already from last 15 days i got stuck here)
>> > > >
>> > > > I have build a nifi custom processor.to put a object in AWS s3
>> > > bucket.(Just Modified Puts3Object processor in NIFI- AWS-PROCESSOR
>> project)
>> > > > In this processor Awscredentialprovidercontrolerservice is the
>> default
>> > > one to connect AWS.
>> > > >
>> > > > But our project structure we have a custom controller service
>> > > ,Awscredentialprovidercontrolerservice controller service to connect
>> AWS
>> > > with our defined attributes.(This is present in a separate project
>> > > NIFI-AWS-CUSTOM_PING_CONTROLER)
>> > > > I want to use this custom controller service , instead of the
>> default
>> > > Awscredentialprovidercontrolerservice one which is present in the
>> NIFI-
>> > > AWS-PROCESSOR project
>> > > >
>> > > > My question is I want to do the unit testing (Using JUnit &
Re: Unable to do unit testing Awscredentialprovidercontrolerservice
Hi ,
While testing a controller service (Awscredentialprovidercontrolerservice)
i am getting IlligalStateException,
in the runner.enablecontrolerService(...) , when i am removing LINE3 code i
am getting null pointer exception in the useStratergy value(in the custom
validate).
What i have annalysed the properties list, i am getting all the properties
values that are present in the default
Awscredentialprovidercontrolerservice class.
It should have my properties list which i have defined in the custom
Awscredentialprovidercontrolerservice class.
code:
@Test
public void testAwscredentialprovidercontrolerservice() {
final TestRunner runner = TestRunners.newTestRunner(puts3Object.class);
Final Awscredentialprovidercontrolerservice serviceImpl = new
Awscredentialprovidercontrolerservice()
Runner.setProperty(..)
#LINE3#runner.setProperty(serviceImpl,AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE,"awsCredsProviderService");
runner.enablecontrolerService(serviceImpl)
}
In the custom Awscredentialprovidercontrolerservice I have added ,
public static final PropertyDescriptor AWS_CONTROLER_SERVICE = new
PropertyDescriptor.Builder()
.name(“AWS Service")
.description("Specified the SSL Context Service that can be used to
create secure connections")
.required(true)
.identifiesControllerService(Awscredentialprovidercontrolerservice
.class)
Thanks,
Sanjeet
On Thu, 17 Oct, 2019, 11:42 PM Bryan Bende, wrote:
> You should be able use mockito (or any other mocking framework),
> something like...
>
> AWSCredentialsProviderService service =
> Mockito.mock(AWSCredentialsProviderService.class)
> //setup whatever you want the mock to do
>
> runner.setProperty(service,
> AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE,
> "awsCredsProviderService");
> runner.enableControllerService(service);
>
> On Thu, Oct 17, 2019 at 1:54 PM sanjeet rath
> wrote:
> >
> > Thanks a lot bryan , for helping me to identify proper UNIT testing.
> > I will go with ur suggestion, will write unit testing for custom
> > controlerservice and separate for custom processor service.
> > If possible Can u give me a example of mock the
> > AWSCredentialsProviderService interface.
> >
> > Regards,
> > sanjeet
> >
> > On Thu, Oct 17, 2019 at 9:41 PM Bryan Bende wrote:
> >
> > > Please only email one list, removing users...
> > >
> > > The AWS processors depend on an interface called
> > > AWSCredentialsProviderService. The unit test of the processor should
> > > not need to depend on real implementations of this interface,
> > > otherwise it is really an integration test of multiple components.
> > >
> > > I would suggest testing your processor and service separately... the
> > > processor test can mock the AWSCredentialsProviderService interface.
> > >
> > >
> > > On Thu, Oct 17, 2019 at 11:50 AM sanjeet rath
> > > wrote:
> > > >
> > > > Hi Team,
> > > >
> > > > I am new to this community, today i have joined and this is my first
> > > query.(already from last 15 days i got stuck here)
> > > >
> > > > I have build a nifi custom processor.to put a object in AWS s3
> > > bucket.(Just Modified Puts3Object processor in NIFI- AWS-PROCESSOR
> project)
> > > > In this processor Awscredentialprovidercontrolerservice is the
> default
> > > one to connect AWS.
> > > >
> > > > But our project structure we have a custom controller service
> > > ,Awscredentialprovidercontrolerservice controller service to connect
> AWS
> > > with our defined attributes.(This is present in a separate project
> > > NIFI-AWS-CUSTOM_PING_CONTROLER)
> > > > I want to use this custom controller service , instead of the default
> > > Awscredentialprovidercontrolerservice one which is present in the NIFI-
> > > AWS-PROCESSOR project
> > > >
> > > > My question is I want to do the unit testing (Using JUnit &
> Mockito) to
> > > validate this custom Awscredentialprovidercontrolerservice is working
> or
> > > not like if I get invalid bucket name then test case should failed.
> > > >
> > > > My code structure:
> > > > @Test
> > > > public void testRetryLogin() {
> > > > final TestRunner runner = TestRunners.newTestRunner(new
> > > puts3Object);
> > > > final Awscredentialprovidercontrolerservice serviceimpl= new
> > > Awscredentialprovidercontrolerservice()
> > > > Runner.setproperty(…)
> >
Re: Unable to do unit testing Awscredentialprovidercontrolerservice
Thanks a lot bryan , for helping me to identify proper UNIT testing.
I will go with ur suggestion, will write unit testing for custom
controlerservice and separate for custom processor service.
If possible Can u give me a example of mock the
AWSCredentialsProviderService interface.
Regards,
sanjeet
On Thu, Oct 17, 2019 at 9:41 PM Bryan Bende wrote:
> Please only email one list, removing users...
>
> The AWS processors depend on an interface called
> AWSCredentialsProviderService. The unit test of the processor should
> not need to depend on real implementations of this interface,
> otherwise it is really an integration test of multiple components.
>
> I would suggest testing your processor and service separately... the
> processor test can mock the AWSCredentialsProviderService interface.
>
>
> On Thu, Oct 17, 2019 at 11:50 AM sanjeet rath
> wrote:
> >
> > Hi Team,
> >
> > I am new to this community, today i have joined and this is my first
> query.(already from last 15 days i got stuck here)
> >
> > I have build a nifi custom processor.to put a object in AWS s3
> bucket.(Just Modified Puts3Object processor in NIFI- AWS-PROCESSOR project)
> > In this processor Awscredentialprovidercontrolerservice is the default
> one to connect AWS.
> >
> > But our project structure we have a custom controller service
> ,Awscredentialprovidercontrolerservice controller service to connect AWS
> with our defined attributes.(This is present in a separate project
> NIFI-AWS-CUSTOM_PING_CONTROLER)
> > I want to use this custom controller service , instead of the default
> Awscredentialprovidercontrolerservice one which is present in the NIFI-
> AWS-PROCESSOR project
> >
> > My question is I want to do the unit testing (Using JUnit & Mockito) to
> validate this custom Awscredentialprovidercontrolerservice is working or
> not like if I get invalid bucket name then test case should failed.
> >
> > My code structure:
> > @Test
> > public void testRetryLogin() {
> > final TestRunner runner = TestRunners.newTestRunner(new
> puts3Object);
> > final Awscredentialprovidercontrolerservice serviceimpl= new
> Awscredentialprovidercontrolerservice()
> > Runner.setproperty(…)
> > Runner.enablecontrolerservice(serviceimpl)
> > runner.run();
> >// will do assert }
> >
> > This gives me compiler error in error in
> Awscredentialprovidercontrolerservice instance creation line as custom
> controller service project is not available to this processor service
> project. If we add dependency in pom file to make it available then it will
> be circular dependency.
> > Because already in custom controlerservice pom file , custom processor
> is already added in dependency..
> >
> > Thanks in advance :)
> >
> > --
> > Sanjeet Kumar Rath,
> > mob- +91 8777577470
> >
>
--
Sanjeet Kumar Rath,
mob- +91 8777577470
Unable to do unit testing Awscredentialprovidercontrolerservice
Hi Team,
I am new to this community, today i have joined and this is my first
query.(already from last 15 days i got stuck here)
I have build a nifi custom processor.to put a object in AWS s3 bucket.(Just
Modified Puts3Object processor in NIFI- AWS-PROCESSOR project)
In this processor Awscredentialprovidercontrolerservice is the default one
to connect AWS.
But our project structure we have a custom controller service
,Awscredentialprovidercontrolerservice controller service to connect AWS
with our defined attributes.(This is present in a separate project
NIFI-AWS-CUSTOM_PING_CONTROLER)
I want to use this custom controller service , instead of the default
Awscredentialprovidercontrolerservice one which is present in the NIFI-
AWS-PROCESSOR project
My question is I want to do the unit testing (Using JUnit & Mockito) to
validate this custom Awscredentialprovidercontrolerservice is working or
not like if I get invalid bucket name then test case should failed.
My code structure:
@Test
public void testRetryLogin() {
final TestRunner runner = TestRunners.newTestRunner(new
puts3Object);
final Awscredentialprovidercontrolerservice serviceimpl= new
Awscredentialprovidercontrolerservice()
Runner.setproperty(…)
Runner.enablecontrolerservice(serviceimpl)
runner.run();
// will do assert }
This gives me compiler error in error in
Awscredentialprovidercontrolerservice instance creation line as custom
controller service project is not available to this processor service
project. If we add dependency in pom file to make it available then it will
be circular dependency.
Because already in custom controlerservice pom file , custom processor is
already added in dependency..
Thanks in advance :)
--
Sanjeet Kumar Rath,
mob- +91 8777577470
