Re: Custom Controller Service
Hello, Others who have worked on the DB related services and processors can correct me if I'm wrong here, but... In general the idea of a connection pool is that creating connections is somewhat expensive, and for a high-volume of operations you don't want to create a connection for each DB operation, so the connection pool creates some number of connections ahead of time and makes them re-usable across operations, thus being more efficient. In your case you want dynamically created connections based on flow file attributes, which means potentially the connection information could be different for each flow file. At that point it starts to feel like it isn't really a connection pool and is just a factory to obtain a one-time use connection because otherwise would end-up needing to obtain multiple connection pools behind the scenes. A controller service has an API and then implementations of the API, and the API just a Java interface. The Java interface (API) is the contract with processors... a processor gets a reference to an object that implements the interface, and the processor can call methods on the interface. So if you want to pass information from flow files to a controller service, then the methods in the interface need to somehow accept that information. The DBCPConnectionPool interface [1] is just a single method "getConnection()" and it is designed to be a re-usable pool of connections against a single DB (as I described in the first paragraph). You could define your own "DynamicDBCPConnectionPool" API with a method like "getConnection(String dbIdentifier)" and have an implementation that loaded connection information from properties file and kept a lookup from dbIdentifier to connection info, but then you also need your own set DB processor because none of the existing DB processors work against your new API. Hope this helps. -Bryan [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-api/src/main/java/org/apache/nifi/dbcp/DBCPService.java#L33 On Wed, Apr 25, 2018 at 3:24 AM, Rishab Prasad wrote: > Hi, > > Basically, there are 'n' number of databases that we are dealing with. We > need to fetch the data from the source database into HDFS. Now since we are > dealing with many databases, the source database is not static and changes > every now and then. And every time the source database changes we manually > need to change the value for the connection parameters in > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > connections for each database, but that is not possible because 'n' is a > big number and creating that many connections in DBCPConnectionPool is not > possible. So we were looking for a way where we can specify all the > connection parameters in a file present in our local system and then make > the DBCPConnectionPool controller service to read the values from the file. > In that way we can simply change the value in the file present in the local > system. No need to alter anything in the dataflow. But it turns out that > FlowFile attributes are not available to the controller services as the > expression language is evaluated at the time of service enable. > > So can you suggest a way where I can achieve my requirement (except > 'variable.registry' ) ? I am looking to develop a custom controller service > that can serve the requirement but how do I make the flowfile attributes > available to the service?
Re: Custom Controller Service
Shot in the dark here, but what you try to do is create a custom connection pool service that uses dynamic properties to build a "pool of connection pools." You could then use the property names as hints for where to send the queries. On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad wrote: > Hi, > > Basically, there are 'n' number of databases that we are dealing with. We > need to fetch the data from the source database into HDFS. Now since we are > dealing with many databases, the source database is not static and changes > every now and then. And every time the source database changes we manually > need to change the value for the connection parameters in > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > connections for each database, but that is not possible because 'n' is a > big number and creating that many connections in DBCPConnectionPool is not > possible. So we were looking for a way where we can specify all the > connection parameters in a file present in our local system and then make > the DBCPConnectionPool controller service to read the values from the file. > In that way we can simply change the value in the file present in the local > system. No need to alter anything in the dataflow. But it turns out that > FlowFile attributes are not available to the controller services as the > expression language is evaluated at the time of service enable. > > So can you suggest a way where I can achieve my requirement (except > 'variable.registry' ) ? I am looking to develop a custom controller service > that can serve the requirement but how do I make the flowfile attributes > available to the service? >
Re: Custom Controller Service
Can services work with other controller services? Maybe a PropertiesControllerService, FilePropertiesControllerService could work with your service? the PCS could fire events on property changes etc. On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) wrote: Shot in the dark here, but what you try to do is create a custom connection pool service that uses dynamic properties to build a "pool of connection pools." You could then use the property names as hints for where to send the queries. On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad wrote: > Hi, > > Basically, there are 'n' number of databases that we are dealing with. We > need to fetch the data from the source database into HDFS. Now since we are > dealing with many databases, the source database is not static and changes > every now and then. And every time the source database changes we manually > need to change the value for the connection parameters in > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > connections for each database, but that is not possible because 'n' is a > big number and creating that many connections in DBCPConnectionPool is not > possible. So we were looking for a way where we can specify all the > connection parameters in a file present in our local system and then make > the DBCPConnectionPool controller service to read the values from the file. > In that way we can simply change the value in the file present in the local > system. No need to alter anything in the dataflow. But it turns out that > FlowFile attributes are not available to the controller services as the > expression language is evaluated at the time of service enable. > > So can you suggest a way where I can achieve my requirement (except > 'variable.registry' ) ? I am looking to develop a custom controller service > that can serve the requirement but how do I make the flowfile attributes > available to the service? >
Re: Custom Controller Service
Or you could just call every time you needed properties more likely. This would still be custom unless integrated…. On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) wrote: Can services work with other controller services? Maybe a PropertiesControllerService, FilePropertiesControllerService could work with your service? the PCS could fire events on property changes etc. On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) wrote: Shot in the dark here, but what you try to do is create a custom connection pool service that uses dynamic properties to build a "pool of connection pools." You could then use the property names as hints for where to send the queries. On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad wrote: > Hi, > > Basically, there are 'n' number of databases that we are dealing with. We > need to fetch the data from the source database into HDFS. Now since we are > dealing with many databases, the source database is not static and changes > every now and then. And every time the source database changes we manually > need to change the value for the connection parameters in > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > connections for each database, but that is not possible because 'n' is a > big number and creating that many connections in DBCPConnectionPool is not > possible. So we were looking for a way where we can specify all the > connection parameters in a file present in our local system and then make > the DBCPConnectionPool controller service to read the values from the file. > In that way we can simply change the value in the file present in the local > system. No need to alter anything in the dataflow. But it turns out that > FlowFile attributes are not available to the controller services as the > expression language is evaluated at the time of service enable. > > So can you suggest a way where I can achieve my requirement (except > 'variable.registry' ) ? I am looking to develop a custom controller service > that can serve the requirement but how do I make the flowfile attributes > available to the service? >
Re: Custom Controller Service
The issue here is more about the service API and not the implementations. The current API has no way to pass information between the processor and service. The options boil down to... - Make a new API, but then you need all new processors that use the new API - Modify the current API to have a new method, but then we are combing two concepts into one API and some impls may not implement both - Modify the processors to use two different service APIs, but enforce that only one can be used at a time, so it can have either the original connection pool service or can have some new dynamic connection factory, but not both, and then modify all DB processors to have logic to determine which service to use. On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler wrote: > Or you could just call every time you needed properties more likely. > This would still be custom unless integrated…. > > > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Can services work with other controller services? > Maybe a PropertiesControllerService, FilePropertiesControllerService could > work with your service? > the PCS could fire events on property changes etc. > > > > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > wrote: > > Shot in the dark here, but what you try to do is create a custom connection > pool service that uses dynamic properties to build a "pool of connection > pools." You could then use the property names as hints for where to send > the queries. > > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad > wrote: > > > Hi, > > > > Basically, there are 'n' number of databases that we are dealing with. We > > need to fetch the data from the source database into HDFS. Now since we > are > > dealing with many databases, the source database is not static and > changes > > every now and then. And every time the source database changes we > manually > > need to change the value for the connection parameters in > > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > > connections for each database, but that is not possible because 'n' is a > > big number and creating that many connections in DBCPConnectionPool is > not > > possible. So we were looking for a way where we can specify all the > > connection parameters in a file present in our local system and then make > > the DBCPConnectionPool controller service to read the values from the > file. > > In that way we can simply change the value in the file present in the > local > > system. No need to alter anything in the dataflow. But it turns out that > > FlowFile attributes are not available to the controller services as the > > expression language is evaluated at the time of service enable. > > > > So can you suggest a way where I can achieve my requirement (except > > 'variable.registry' ) ? I am looking to develop a custom controller > service > > that can serve the requirement but how do I make the flowfile attributes > > available to the service? > > > -- Sent from Gmail Mobile
Re: Custom Controller Service
If any controller service optionally supported this external service ( like the AWS processors optional support the credentials service ) then there is no need for the processor to change though right? On April 25, 2018 at 08:37:50, Bryan Bende (bbe...@gmail.com) wrote: The issue here is more about the service API and not the implementations. The current API has no way to pass information between the processor and service. The options boil down to... - Make a new API, but then you need all new processors that use the new API - Modify the current API to have a new method, but then we are combing two concepts into one API and some impls may not implement both - Modify the processors to use two different service APIs, but enforce that only one can be used at a time, so it can have either the original connection pool service or can have some new dynamic connection factory, but not both, and then modify all DB processors to have logic to determine which service to use. On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler wrote: > Or you could just call every time you needed properties more likely. > This would still be custom unless integrated…. > > > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > Can services work with other controller services? > Maybe a PropertiesControllerService, FilePropertiesControllerService could > work with your service? > the PCS could fire events on property changes etc. > > > > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > wrote: > > Shot in the dark here, but what you try to do is create a custom connection > pool service that uses dynamic properties to build a "pool of connection > pools." You could then use the property names as hints for where to send > the queries. > > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad > wrote: > > > Hi, > > > > Basically, there are 'n' number of databases that we are dealing with. We > > need to fetch the data from the source database into HDFS. Now since we > are > > dealing with many databases, the source database is not static and > changes > > every now and then. And every time the source database changes we > manually > > need to change the value for the connection parameters in > > DBCPConnectionPool. Now, people suggest that for 'n' databases create 'n' > > connections for each database, but that is not possible because 'n' is a > > big number and creating that many connections in DBCPConnectionPool is > not > > possible. So we were looking for a way where we can specify all the > > connection parameters in a file present in our local system and then make > > the DBCPConnectionPool controller service to read the values from the > file. > > In that way we can simply change the value in the file present in the > local > > system. No need to alter anything in the dataflow. But it turns out that > > FlowFile attributes are not available to the controller services as the > > expression language is evaluated at the time of service enable. > > > > So can you suggest a way where I can achieve my requirement (except > > 'variable.registry' ) ? I am looking to develop a custom controller > service > > that can serve the requirement but how do I make the flowfile attributes > > available to the service? > > > -- Sent from Gmail Mobile
Re: Custom Controller Service
Option 2 and 3 seem to be a probable approach. However creating varying number of connections based on *each* flowfile still sounds to be suboptimal. If the requirement still demands to take that road, then it’s better to do some prep-work.. as in the list of probable connections that are required are taken and connection pools are created for them and then based on the flowfiles (which connection it needs), we use the relevant one. Thanks, Sivaprasanna On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende wrote: > The issue here is more about the service API and not the implementations. > > The current API has no way to pass information between the processor and > service. > > The options boil down to... > > - Make a new API, but then you need all new processors that use the new API > > - Modify the current API to have a new method, but then we are combing two > concepts into one API and some impls may not implement both > > - Modify the processors to use two different service APIs, but enforce that > only one can be used at a time, so it can have either the original > connection pool service or can have some new dynamic connection factory, > but not both, and then modify all DB processors to have logic to determine > which service to use. > > On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler > wrote: > > > Or you could just call every time you needed properties more likely. > > This would still be custom unless integrated…. > > > > > > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > > wrote: > > > > Can services work with other controller services? > > Maybe a PropertiesControllerService, FilePropertiesControllerService > could > > work with your service? > > the PCS could fire events on property changes etc. > > > > > > > > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > > wrote: > > > > Shot in the dark here, but what you try to do is create a custom > connection > > pool service that uses dynamic properties to build a "pool of connection > > pools." You could then use the property names as hints for where to send > > the queries. > > > > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad > > > wrote: > > > > > Hi, > > > > > > Basically, there are 'n' number of databases that we are dealing with. > We > > > need to fetch the data from the source database into HDFS. Now since we > > are > > > dealing with many databases, the source database is not static and > > changes > > > every now and then. And every time the source database changes we > > manually > > > need to change the value for the connection parameters in > > > DBCPConnectionPool. Now, people suggest that for 'n' databases create > 'n' > > > connections for each database, but that is not possible because 'n' is > a > > > big number and creating that many connections in DBCPConnectionPool is > > not > > > possible. So we were looking for a way where we can specify all the > > > connection parameters in a file present in our local system and then > make > > > the DBCPConnectionPool controller service to read the values from the > > file. > > > In that way we can simply change the value in the file present in the > > local > > > system. No need to alter anything in the dataflow. But it turns out > that > > > FlowFile attributes are not available to the controller services as the > > > expression language is evaluated at the time of service enable. > > > > > > So can you suggest a way where I can achieve my requirement (except > > > 'variable.registry' ) ? I am looking to develop a custom controller > > service > > > that can serve the requirement but how do I make the flowfile > attributes > > > available to the service? > > > > > > -- > Sent from Gmail Mobile >
Re: Custom Controller Service
To Otto's question... For simplicity sake, there is a new implementation of DBCPConnectionPool that behind the scenes has two connection pools, one for DB A and one for DB B, it doesn't matter how these are configured. Now a flow file comes into the ExecuteSQL and it goes to connectionPoolService.getConnection()... How does it know which DB to return a connection for? On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna wrote: > Option 2 and 3 seem to be a probable approach. However creating varying > number of connections based on *each* flowfile still sounds to be > suboptimal. If the requirement still demands to take that road, then it’s > better to do some prep-work.. as in the list of probable connections that > are required are taken and connection pools are created for them and then > based on the flowfiles (which connection it needs), we use the relevant > one. > > Thanks, > Sivaprasanna > > On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende wrote: > >> The issue here is more about the service API and not the implementations. >> >> The current API has no way to pass information between the processor and >> service. >> >> The options boil down to... >> >> - Make a new API, but then you need all new processors that use the new API >> >> - Modify the current API to have a new method, but then we are combing two >> concepts into one API and some impls may not implement both >> >> - Modify the processors to use two different service APIs, but enforce that >> only one can be used at a time, so it can have either the original >> connection pool service or can have some new dynamic connection factory, >> but not both, and then modify all DB processors to have logic to determine >> which service to use. >> >> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler >> wrote: >> >> > Or you could just call every time you needed properties more likely. >> > This would still be custom unless integrated…. >> > >> > >> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) >> > wrote: >> > >> > Can services work with other controller services? >> > Maybe a PropertiesControllerService, FilePropertiesControllerService >> could >> > work with your service? >> > the PCS could fire events on property changes etc. >> > >> > >> > >> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) >> > wrote: >> > >> > Shot in the dark here, but what you try to do is create a custom >> connection >> > pool service that uses dynamic properties to build a "pool of connection >> > pools." You could then use the property names as hints for where to send >> > the queries. >> > >> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad > > >> > wrote: >> > >> > > Hi, >> > > >> > > Basically, there are 'n' number of databases that we are dealing with. >> We >> > > need to fetch the data from the source database into HDFS. Now since we >> > are >> > > dealing with many databases, the source database is not static and >> > changes >> > > every now and then. And every time the source database changes we >> > manually >> > > need to change the value for the connection parameters in >> > > DBCPConnectionPool. Now, people suggest that for 'n' databases create >> 'n' >> > > connections for each database, but that is not possible because 'n' is >> a >> > > big number and creating that many connections in DBCPConnectionPool is >> > not >> > > possible. So we were looking for a way where we can specify all the >> > > connection parameters in a file present in our local system and then >> make >> > > the DBCPConnectionPool controller service to read the values from the >> > file. >> > > In that way we can simply change the value in the file present in the >> > local >> > > system. No need to alter anything in the dataflow. But it turns out >> that >> > > FlowFile attributes are not available to the controller services as the >> > > expression language is evaluated at the time of service enable. >> > > >> > > So can you suggest a way where I can achieve my requirement (except >> > > 'variable.registry' ) ? I am looking to develop a custom controller >> > service >> > > that can serve the requirement but how do I make the flowfile >> attributes >> > > available to the service? >> > > >> > >> -- >> Sent from Gmail Mobile >>
Re: Custom Controller Service
Here is a proposal for how to modify the existing API to support both scenarios: https://issues.apache.org/jira/browse/NIFI-5121 The scope of that ticket would be to make the interface change, and then update all of NiFi's DB processors to pass in the attribute map. Then a separate effort to provide a new service implementation that used the attribute map to somehow manage multiple connection pools, or create connections on the fly, or whatever the desired behavior is. On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende wrote: > To Otto's question... > > For simplicity sake, there is a new implementation of > DBCPConnectionPool that behind the scenes has two connection pools, > one for DB A and one for DB B, it doesn't matter how these are > configured. > > Now a flow file comes into the ExecuteSQL and it goes to > connectionPoolService.getConnection()... > > How does it know which DB to return a connection for? > > > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna > wrote: >> Option 2 and 3 seem to be a probable approach. However creating varying >> number of connections based on *each* flowfile still sounds to be >> suboptimal. If the requirement still demands to take that road, then it’s >> better to do some prep-work.. as in the list of probable connections that >> are required are taken and connection pools are created for them and then >> based on the flowfiles (which connection it needs), we use the relevant >> one. >> >> Thanks, >> Sivaprasanna >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende wrote: >> >>> The issue here is more about the service API and not the implementations. >>> >>> The current API has no way to pass information between the processor and >>> service. >>> >>> The options boil down to... >>> >>> - Make a new API, but then you need all new processors that use the new API >>> >>> - Modify the current API to have a new method, but then we are combing two >>> concepts into one API and some impls may not implement both >>> >>> - Modify the processors to use two different service APIs, but enforce that >>> only one can be used at a time, so it can have either the original >>> connection pool service or can have some new dynamic connection factory, >>> but not both, and then modify all DB processors to have logic to determine >>> which service to use. >>> >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler >>> wrote: >>> >>> > Or you could just call every time you needed properties more likely. >>> > This would still be custom unless integrated…. >>> > >>> > >>> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) >>> > wrote: >>> > >>> > Can services work with other controller services? >>> > Maybe a PropertiesControllerService, FilePropertiesControllerService >>> could >>> > work with your service? >>> > the PCS could fire events on property changes etc. >>> > >>> > >>> > >>> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) >>> > wrote: >>> > >>> > Shot in the dark here, but what you try to do is create a custom >>> connection >>> > pool service that uses dynamic properties to build a "pool of connection >>> > pools." You could then use the property names as hints for where to send >>> > the queries. >>> > >>> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad >> > >>> > wrote: >>> > >>> > > Hi, >>> > > >>> > > Basically, there are 'n' number of databases that we are dealing with. >>> We >>> > > need to fetch the data from the source database into HDFS. Now since we >>> > are >>> > > dealing with many databases, the source database is not static and >>> > changes >>> > > every now and then. And every time the source database changes we >>> > manually >>> > > need to change the value for the connection parameters in >>> > > DBCPConnectionPool. Now, people suggest that for 'n' databases create >>> 'n' >>> > > connections for each database, but that is not possible because 'n' is >>> a >>> > > big number and creating that many connections in DBCPConnectionPool is >>> > not >>> > > possible. So we were looking for a way where we can specify all the >>> > > connection parameters in a file present in our local system and then >>> make >>> > > the DBCPConnectionPool controller service to read the values from the >>> > file. >>> > > In that way we can simply change the value in the file present in the >>> > local >>> > > system. No need to alter anything in the dataflow. But it turns out >>> that >>> > > FlowFile attributes are not available to the controller services as the >>> > > expression language is evaluated at the time of service enable. >>> > > >>> > > So can you suggest a way where I can achieve my requirement (except >>> > > 'variable.registry' ) ? I am looking to develop a custom controller >>> > service >>> > > that can serve the requirement but how do I make the flowfile >>> attributes >>> > > available to the service? >>> > > >>> > >>> -- >>> Sent from Gmail Mobile >>>
Re: Custom Controller Service
Chiming in a bit late on this, but we faced this same issue and got around it by implementing a custom controller service which acts as a "router" to different dbcp services. It exposes a method which given a uuid, returns back the DBCPservice that corresponds with that uuid if it exists using DBCPService dbcpService = (DBCPService) getControllerServiceLookup().getControllerService(uuid); >From there, we created processors we needed based on the stock ones which relied on our "router" service rather than a single DBCP. Our custom processors read an attribute from incoming flow files, then send that to the router which returns back the connection pool. On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende wrote: > Here is a proposal for how to modify the existing API to support both > scenarios: > > https://issues.apache.org/jira/browse/NIFI-5121 > > The scope of that ticket would be to make the interface change, and > then update all of NiFi's DB processors to pass in the attribute map. > > Then a separate effort to provide a new service implementation that > used the attribute map to somehow manage multiple connection pools, or > create connections on the fly, or whatever the desired behavior is. > > On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende wrote: > > To Otto's question... > > > > For simplicity sake, there is a new implementation of > > DBCPConnectionPool that behind the scenes has two connection pools, > > one for DB A and one for DB B, it doesn't matter how these are > > configured. > > > > Now a flow file comes into the ExecuteSQL and it goes to > > connectionPoolService.getConnection()... > > > > How does it know which DB to return a connection for? > > > > > > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna > wrote: > >> Option 2 and 3 seem to be a probable approach. However creating varying > >> number of connections based on *each* flowfile still sounds to be > >> suboptimal. If the requirement still demands to take that road, then > it’s > >> better to do some prep-work.. as in the list of probable connections > that > >> are required are taken and connection pools are created for them and > then > >> based on the flowfiles (which connection it needs), we use the relevant > >> one. > >> > >> Thanks, > >> Sivaprasanna > >> > >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende wrote: > >> > >>> The issue here is more about the service API and not the > implementations. > >>> > >>> The current API has no way to pass information between the processor > and > >>> service. > >>> > >>> The options boil down to... > >>> > >>> - Make a new API, but then you need all new processors that use the > new API > >>> > >>> - Modify the current API to have a new method, but then we are combing > two > >>> concepts into one API and some impls may not implement both > >>> > >>> - Modify the processors to use two different service APIs, but enforce > that > >>> only one can be used at a time, so it can have either the original > >>> connection pool service or can have some new dynamic connection > factory, > >>> but not both, and then modify all DB processors to have logic to > determine > >>> which service to use. > >>> > >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler > >>> wrote: > >>> > >>> > Or you could just call every time you needed properties more likely. > >>> > This would still be custom unless integrated…. > >>> > > >>> > > >>> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) > >>> > wrote: > >>> > > >>> > Can services work with other controller services? > >>> > Maybe a PropertiesControllerService, FilePropertiesControllerService > >>> could > >>> > work with your service? > >>> > the PCS could fire events on property changes etc. > >>> > > >>> > > >>> > > >>> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) > >>> > wrote: > >>> > > >>> > Shot in the dark here, but what you try to do is create a custom > >>> connection > >>> > pool service that uses dynamic properties to build a "pool of > connection > >>> > pools." You could then use the property names as hints for where to > send > >>> > the queries. > >>> > > >>> > On Wed, Apr 25, 2018 at 6:19 AM Rishab Prasad < > rishabprasad...@gmail.com > >>> > > >>> > wrote: > >>> > > >>> > > Hi, > >>> > > > >>> > > Basically, there are 'n' number of databases that we are dealing > with. > >>> We > >>> > > need to fetch the data from the source database into HDFS. Now > since we > >>> > are > >>> > > dealing with many databases, the source database is not static and > >>> > changes > >>> > > every now and then. And every time the source database changes we > >>> > manually > >>> > > need to change the value for the connection parameters in > >>> > > DBCPConnectionPool. Now, people suggest that for 'n' databases > create > >>> 'n' > >>> > > connections for each database, but that is not possible because > 'n' is > >>> a > >>> > > big number and creating that many connections in > DBCPConnectionPool is > >>> > not >
Re: Custom Controller Service
Charlie, That is a really nice solution, thanks for sharing. If we make the API changes in that JIRA I just sent, I could see having a new implementation of DBCPService that does something very similar. Basically there could be "DelegatingDBCPService" which still implemented the same DBCPService interface but followed a convention where it always looked in the attribute map for an attribute called "db.id", and then it itself has dynamic properties to define many DBCPServices where the property name was the db.id, and it would just return a Connection from the one with the given id. There are definitely other options for how to implement the dynamic connection service, but this would be a good one to have. -Bryan On Wed, Apr 25, 2018 at 10:58 AM, Charlie Meyer wrote: > Chiming in a bit late on this, but we faced this same issue and got around > it by implementing a custom controller service which acts as a "router" to > different dbcp services. It exposes a method which given a uuid, returns > back the DBCPservice that corresponds with that uuid if it exists using > > DBCPService dbcpService = > (DBCPService) > getControllerServiceLookup().getControllerService(uuid); > > From there, we created processors we needed based on the stock ones which > relied on our "router" service rather than a single DBCP. Our custom > processors read an attribute from incoming flow files, then send that to > the router which returns back the connection pool. > > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende wrote: > >> Here is a proposal for how to modify the existing API to support both >> scenarios: >> >> https://issues.apache.org/jira/browse/NIFI-5121 >> >> The scope of that ticket would be to make the interface change, and >> then update all of NiFi's DB processors to pass in the attribute map. >> >> Then a separate effort to provide a new service implementation that >> used the attribute map to somehow manage multiple connection pools, or >> create connections on the fly, or whatever the desired behavior is. >> >> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende wrote: >> > To Otto's question... >> > >> > For simplicity sake, there is a new implementation of >> > DBCPConnectionPool that behind the scenes has two connection pools, >> > one for DB A and one for DB B, it doesn't matter how these are >> > configured. >> > >> > Now a flow file comes into the ExecuteSQL and it goes to >> > connectionPoolService.getConnection()... >> > >> > How does it know which DB to return a connection for? >> > >> > >> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna >> wrote: >> >> Option 2 and 3 seem to be a probable approach. However creating varying >> >> number of connections based on *each* flowfile still sounds to be >> >> suboptimal. If the requirement still demands to take that road, then >> it’s >> >> better to do some prep-work.. as in the list of probable connections >> that >> >> are required are taken and connection pools are created for them and >> then >> >> based on the flowfiles (which connection it needs), we use the relevant >> >> one. >> >> >> >> Thanks, >> >> Sivaprasanna >> >> >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende wrote: >> >> >> >>> The issue here is more about the service API and not the >> implementations. >> >>> >> >>> The current API has no way to pass information between the processor >> and >> >>> service. >> >>> >> >>> The options boil down to... >> >>> >> >>> - Make a new API, but then you need all new processors that use the >> new API >> >>> >> >>> - Modify the current API to have a new method, but then we are combing >> two >> >>> concepts into one API and some impls may not implement both >> >>> >> >>> - Modify the processors to use two different service APIs, but enforce >> that >> >>> only one can be used at a time, so it can have either the original >> >>> connection pool service or can have some new dynamic connection >> factory, >> >>> but not both, and then modify all DB processors to have logic to >> determine >> >>> which service to use. >> >>> >> >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler >> >>> wrote: >> >>> >> >>> > Or you could just call every time you needed properties more likely. >> >>> > This would still be custom unless integrated…. >> >>> > >> >>> > >> >>> > On April 25, 2018 at 08:26:57, Otto Fowler (ottobackwa...@gmail.com) >> >>> > wrote: >> >>> > >> >>> > Can services work with other controller services? >> >>> > Maybe a PropertiesControllerService, FilePropertiesControllerService >> >>> could >> >>> > work with your service? >> >>> > the PCS could fire events on property changes etc. >> >>> > >> >>> > >> >>> > >> >>> > On April 25, 2018 at 08:05:27, Mike Thomsen (mikerthom...@gmail.com) >> >>> > wrote: >> >>> > >> >>> > Shot in the dark here, but what you try to do is create a custom >> >>> connection >> >>> > pool service that uses dynamic properties to build a "pool of >> connection >> >>> > pools." You could then use the property names as hints fo
Re: Custom Controller Service
Okay.. but two questions: 1. We are passing the attribute 'db.id' that means, we'll be using 'UpdateAttribute' to do add that attribute to flowfile? 2. If we are to use 'UpdateAttribute' to set the value for 'db.id', we need to know before hand, right? - Sivaprasanna On Wed, Apr 25, 2018 at 8:38 PM, Bryan Bende wrote: > Charlie, > > That is a really nice solution, thanks for sharing. > > If we make the API changes in that JIRA I just sent, I could see > having a new implementation of DBCPService that does something very > similar. > > Basically there could be "DelegatingDBCPService" which still > implemented the same DBCPService interface but followed a convention > where it always looked in the attribute map for an attribute called > "db.id", and then it itself has dynamic properties to define many > DBCPServices where the property name was the db.id, and it would just > return a Connection from the one with the given id. > > There are definitely other options for how to implement the dynamic > connection service, but this would be a good one to have. > > -Bryan > > > On Wed, Apr 25, 2018 at 10:58 AM, Charlie Meyer > wrote: > > Chiming in a bit late on this, but we faced this same issue and got > around > > it by implementing a custom controller service which acts as a "router" > to > > different dbcp services. It exposes a method which given a uuid, returns > > back the DBCPservice that corresponds with that uuid if it exists using > > > > DBCPService dbcpService = > > (DBCPService) > > getControllerServiceLookup().getControllerService(uuid); > > > > From there, we created processors we needed based on the stock ones which > > relied on our "router" service rather than a single DBCP. Our custom > > processors read an attribute from incoming flow files, then send that to > > the router which returns back the connection pool. > > > > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende wrote: > > > >> Here is a proposal for how to modify the existing API to support both > >> scenarios: > >> > >> https://issues.apache.org/jira/browse/NIFI-5121 > >> > >> The scope of that ticket would be to make the interface change, and > >> then update all of NiFi's DB processors to pass in the attribute map. > >> > >> Then a separate effort to provide a new service implementation that > >> used the attribute map to somehow manage multiple connection pools, or > >> create connections on the fly, or whatever the desired behavior is. > >> > >> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende wrote: > >> > To Otto's question... > >> > > >> > For simplicity sake, there is a new implementation of > >> > DBCPConnectionPool that behind the scenes has two connection pools, > >> > one for DB A and one for DB B, it doesn't matter how these are > >> > configured. > >> > > >> > Now a flow file comes into the ExecuteSQL and it goes to > >> > connectionPoolService.getConnection()... > >> > > >> > How does it know which DB to return a connection for? > >> > > >> > > >> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna < > sivaprasanna...@gmail.com> > >> wrote: > >> >> Option 2 and 3 seem to be a probable approach. However creating > varying > >> >> number of connections based on *each* flowfile still sounds to be > >> >> suboptimal. If the requirement still demands to take that road, then > >> it’s > >> >> better to do some prep-work.. as in the list of probable connections > >> that > >> >> are required are taken and connection pools are created for them and > >> then > >> >> based on the flowfiles (which connection it needs), we use the > relevant > >> >> one. > >> >> > >> >> Thanks, > >> >> Sivaprasanna > >> >> > >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende > wrote: > >> >> > >> >>> The issue here is more about the service API and not the > >> implementations. > >> >>> > >> >>> The current API has no way to pass information between the processor > >> and > >> >>> service. > >> >>> > >> >>> The options boil down to... > >> >>> > >> >>> - Make a new API, but then you need all new processors that use the > >> new API > >> >>> > >> >>> - Modify the current API to have a new method, but then we are > combing > >> two > >> >>> concepts into one API and some impls may not implement both > >> >>> > >> >>> - Modify the processors to use two different service APIs, but > enforce > >> that > >> >>> only one can be used at a time, so it can have either the original > >> >>> connection pool service or can have some new dynamic connection > >> factory, > >> >>> but not both, and then modify all DB processors to have logic to > >> determine > >> >>> which service to use. > >> >>> > >> >>> On Wed, Apr 25, 2018 at 8:28 AM Otto Fowler < > ottobackwa...@gmail.com> > >> >>> wrote: > >> >>> > >> >>> > Or you could just call every time you needed properties more > likely. > >> >>> > This would still be custom unless integrated…. > >> >>> > > >> >>> > > >> >>> > On April 25, 2018 at 08:26:57, Otto Fowler ( > ottobackwa...@gmail.com)
Re: Custom Controller Service
Yes, this was just one idea based on Charlie's solution. I'm not saying that approach solves the original request in this email, I was just saying its another nice idea that could be easily implemented once we make the changes in the JIRA. There can be as many "dynamic" DBCPService implementations as we want, and anyone can implement their own, the key is making the API changes to allow for it. On Wed, Apr 25, 2018 at 12:05 PM, Sivaprasanna wrote: > Okay.. but two questions: > > >1. We are passing the attribute 'db.id' that means, we'll be using >'UpdateAttribute' to do add that attribute to flowfile? >2. If we are to use 'UpdateAttribute' to set the value for 'db.id', we >need to know before hand, right? > > - > > Sivaprasanna > > On Wed, Apr 25, 2018 at 8:38 PM, Bryan Bende wrote: > >> Charlie, >> >> That is a really nice solution, thanks for sharing. >> >> If we make the API changes in that JIRA I just sent, I could see >> having a new implementation of DBCPService that does something very >> similar. >> >> Basically there could be "DelegatingDBCPService" which still >> implemented the same DBCPService interface but followed a convention >> where it always looked in the attribute map for an attribute called >> "db.id", and then it itself has dynamic properties to define many >> DBCPServices where the property name was the db.id, and it would just >> return a Connection from the one with the given id. >> >> There are definitely other options for how to implement the dynamic >> connection service, but this would be a good one to have. >> >> -Bryan >> >> >> On Wed, Apr 25, 2018 at 10:58 AM, Charlie Meyer >> wrote: >> > Chiming in a bit late on this, but we faced this same issue and got >> around >> > it by implementing a custom controller service which acts as a "router" >> to >> > different dbcp services. It exposes a method which given a uuid, returns >> > back the DBCPservice that corresponds with that uuid if it exists using >> > >> > DBCPService dbcpService = >> > (DBCPService) >> > getControllerServiceLookup().getControllerService(uuid); >> > >> > From there, we created processors we needed based on the stock ones which >> > relied on our "router" service rather than a single DBCP. Our custom >> > processors read an attribute from incoming flow files, then send that to >> > the router which returns back the connection pool. >> > >> > On Wed, Apr 25, 2018 at 9:48 AM, Bryan Bende wrote: >> > >> >> Here is a proposal for how to modify the existing API to support both >> >> scenarios: >> >> >> >> https://issues.apache.org/jira/browse/NIFI-5121 >> >> >> >> The scope of that ticket would be to make the interface change, and >> >> then update all of NiFi's DB processors to pass in the attribute map. >> >> >> >> Then a separate effort to provide a new service implementation that >> >> used the attribute map to somehow manage multiple connection pools, or >> >> create connections on the fly, or whatever the desired behavior is. >> >> >> >> On Wed, Apr 25, 2018 at 9:34 AM, Bryan Bende wrote: >> >> > To Otto's question... >> >> > >> >> > For simplicity sake, there is a new implementation of >> >> > DBCPConnectionPool that behind the scenes has two connection pools, >> >> > one for DB A and one for DB B, it doesn't matter how these are >> >> > configured. >> >> > >> >> > Now a flow file comes into the ExecuteSQL and it goes to >> >> > connectionPoolService.getConnection()... >> >> > >> >> > How does it know which DB to return a connection for? >> >> > >> >> > >> >> > On Wed, Apr 25, 2018 at 9:01 AM, Sivaprasanna < >> sivaprasanna...@gmail.com> >> >> wrote: >> >> >> Option 2 and 3 seem to be a probable approach. However creating >> varying >> >> >> number of connections based on *each* flowfile still sounds to be >> >> >> suboptimal. If the requirement still demands to take that road, then >> >> it’s >> >> >> better to do some prep-work.. as in the list of probable connections >> >> that >> >> >> are required are taken and connection pools are created for them and >> >> then >> >> >> based on the flowfiles (which connection it needs), we use the >> relevant >> >> >> one. >> >> >> >> >> >> Thanks, >> >> >> Sivaprasanna >> >> >> >> >> >> On Wed, 25 Apr 2018 at 6:07 PM, Bryan Bende >> wrote: >> >> >> >> >> >>> The issue here is more about the service API and not the >> >> implementations. >> >> >>> >> >> >>> The current API has no way to pass information between the processor >> >> and >> >> >>> service. >> >> >>> >> >> >>> The options boil down to... >> >> >>> >> >> >>> - Make a new API, but then you need all new processors that use the >> >> new API >> >> >>> >> >> >>> - Modify the current API to have a new method, but then we are >> combing >> >> two >> >> >>> concepts into one API and some impls may not implement both >> >> >>> >> >> >>> - Modify the processors to use two different service APIs, but >> enforce >> >> that >> >> >>> only one can be used at a time, so it can have
Re: Custom Controller Service
Hi Charlie, Thanks for the reply. It seems that you already have a solution to my problem. If possible, can you please share you code for the custom controller service and the custom processor? - RP -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Custom Controller Service
Hello,
I'm working to see what we can share, but for a start, here is how we
handle the delegating controller service:
package com.civitaslearning.nifi.service.impl;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.controller.AbstractControllerService;
import org.apache.nifi.dbcp.DBCPService;
import java.util.Collections;
import java.util.List;
import java.util.UUID;
public class DelegatingDBCPServiceImpl extends
AbstractControllerService implements DelegatingDBCPService {
private static final List properties =
Collections.emptyList();
@Override
public DBCPService getDBCPService(UUID uuid) {
DBCPService dbcpService =
(DBCPService)
getControllerServiceLookup().getControllerService(uuid.toString());
if (dbcpService == null) {
throw new NullPointerException("Couldn't find DBCP service with
ID " + uuid);
}
return dbcpService;
}
@Override
public List getSupportedPropertyDescriptors() {
return properties;
}
}
>From there, we took existing nifi processor code such as ExecuteSQL and
modified it to require a reference to a delegating service rather than a
DBCP service and added another property to specify which flow file
attribute has the id of the dbcp service that should be used, similar to:
final DelegatingDBCPService delegatingDBCPService =
context.getProperty(DELEGATING_DBCP_SERVICE).asControllerService(DelegatingDBCPService.class);
final UUID dbcpServiceId =
UUID.fromString(context.getProperty(DBCP_SERVICE_ATTRIBUTE).evaluateAttributeExpressions(fileToProcess).getValue());
final DBCPService dbcpService =
delegatingDBCPService.getDBCPService(dbcpServiceId);
thanks
On Thu, Apr 26, 2018 at 4:07 AM, RP wrote:
> Hi Charlie,
>
> Thanks for the reply. It seems that you already have a solution to my
> problem. If possible, can you please share you code for the custom
> controller service and the custom processor?
>
>
>
> -
> RP
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Re: Custom Controller Service
In case it helps, Matt B was nice enough to quickly put up a PR for
the API change discussed yesterday:
https://github.com/apache/nifi/pull/2658
Once this is tested and merged it make it would make it easier to
implement Charlie's approach without needing a new interface.
On Thu, Apr 26, 2018 at 12:36 PM, Charlie Meyer
wrote:
> Hello,
>
> I'm working to see what we can share, but for a start, here is how we
> handle the delegating controller service:
>
> package com.civitaslearning.nifi.service.impl;
>
>
> import org.apache.nifi.components.PropertyDescriptor;
> import org.apache.nifi.controller.AbstractControllerService;
> import org.apache.nifi.dbcp.DBCPService;
>
> import java.util.Collections;
> import java.util.List;
> import java.util.UUID;
>
> public class DelegatingDBCPServiceImpl extends
> AbstractControllerService implements DelegatingDBCPService {
>
> private static final List properties =
> Collections.emptyList();
>
> @Override
> public DBCPService getDBCPService(UUID uuid) {
> DBCPService dbcpService =
> (DBCPService)
> getControllerServiceLookup().getControllerService(uuid.toString());
>
> if (dbcpService == null) {
> throw new NullPointerException("Couldn't find DBCP service with
> ID " + uuid);
> }
>
> return dbcpService;
> }
>
> @Override
> public List getSupportedPropertyDescriptors() {
> return properties;
> }
> }
>
>
> From there, we took existing nifi processor code such as ExecuteSQL and
> modified it to require a reference to a delegating service rather than a
> DBCP service and added another property to specify which flow file
> attribute has the id of the dbcp service that should be used, similar to:
>
> final DelegatingDBCPService delegatingDBCPService =
> context.getProperty(DELEGATING_DBCP_SERVICE).asControllerService(DelegatingDBCPService.class);
> final UUID dbcpServiceId =
> UUID.fromString(context.getProperty(DBCP_SERVICE_ATTRIBUTE).evaluateAttributeExpressions(fileToProcess).getValue());
> final DBCPService dbcpService =
> delegatingDBCPService.getDBCPService(dbcpServiceId);
>
> thanks
>
> On Thu, Apr 26, 2018 at 4:07 AM, RP wrote:
>
>> Hi Charlie,
>>
>> Thanks for the reply. It seems that you already have a solution to my
>> problem. If possible, can you please share you code for the custom
>> controller service and the custom processor?
>>
>>
>>
>> -
>> RP
>> --
>> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>>
Re: Custom controller service
Hey Varun, It would br helpful if you could share the error stacktrace. By the way, you can use DBCP Connection Pool for connecting to databases. On Thu, 8 Mar 2018 at 12:14 PM, varun yadav wrote: > Hii, > > I am trying to create a custom controller service for connection to a local > database. > I am getting so many issues in that. If you can help with a few documents > or videos, it will be very helpful for me to proceed. > > Thanks Varun >
Re: Custom controller service
Varun, Bryan Bende has a github repo with examples[1] as well as nifi.rocks[2] has an example. Thanks, Chad 1. https://github.com/bbende/nifi-dependency-example 2. http://nifi.rocks/developing-a-custom-apache-nifi-controller-service/ On Thu, Mar 8, 2018 at 1:47 AM Sivaprasanna wrote: > Hey Varun, > > It would br helpful if you could share the error stacktrace. By the way, > you can use DBCP Connection Pool for connecting to databases. > > On Thu, 8 Mar 2018 at 12:14 PM, varun yadav > wrote: > > > Hii, > > > > I am trying to create a custom controller service for connection to a > local > > database. > > I am getting so many issues in that. If you can help with a few documents > > or videos, it will be very helpful for me to proceed. > > > > Thanks Varun > > >
Re: Custom controller service
Varun, You didn't say what type of database is it. Can you share that info? On Thu, Mar 8, 2018 at 1:15 AM, varun yadav wrote: > Hii, > > I am trying to create a custom controller service for connection to a local > database. > I am getting so many issues in that. If you can help with a few documents > or videos, it will be very helpful for me to proceed. > > Thanks Varun >
Re: Custom Controller Service Debugging
Brian,
I just ran into this exact scenario today while testing a custom controller
service.
The cause of the problem was that my custom service NAR, did not have a NAR
dependency on the service API NAR because I had accidentally marked the API
NAR as provided in the pom file. Once I removed the provided scope
everything worked.
You can check the MANIFEST file in your custom service NAR... under your
NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it should
have a "Nar-Dependency-Id" on your API NAR. If it is missing that, that is
likely the problem.
-Bryan
On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli
wrote:
> NiFi Gurus,
>
> Regarding Controller Services, I'm seeing some strange behavior regarding a
> custom Controller Service only showing its ID in a Processor and was
> wondering if anyone's encountered similar issues. Seeing this on the
> latest develop branch, 0.3.0-SNAPSHOT.
>
> Steps taken:
> 1. Create instance of Controller Service in the Controller Settings >
> Controller Services tab
> 1a. Enable that instance of the Controller Service
> 2. Add new Processor on canvas that will use Controller Service
> 3. Configure Processor
> 3a. Existing Controller Service does not show in property options. Only "No
> value" and "Create new service..."
> 3b. Clicking "Create new service..." opens modal with the custom Controller
> Service as the only item in the dropdown
> 3c. Clicking "Create" button succeeds, but puts the UUID of the Controller
> Service in the Value of the Processor Property.
>
> Observed state:
> - Processor is still invalid
> - The ID in the Processor Property matches the ID of the second (disabled)
> instance of the Controller Service in the Controller Settings modal.
>
> There are no errors in nifi-app.log - only contains log messages from the
> StandardControllerServiceProvider that it "Created Controller Service of
> type [my-implementation-class] with identifier
> [UUID-that-shows-in-the-GUI]"
>
> Adding a DetectDuplicate processor with the
> DistributedMapCacheClientService works just fine, so it'd hard to point
> fingers at the framework. Any thoughts on where to begin debugging and/or
> loggers to turn up would be greatly appreciated!
>
> Thanks,
> Brian
>
Re: Custom Controller Service Debugging
Bryan,
Thank you!! Adding a NAR dependency to my standard-processors NAR did the
trick to get NiFi to recognize the service for these processors and
auto-create a new instance of the Controller Service on template import. I
owe ya a beer or six the next time our paths cross.
How did you figure out the required NAR dependency? Any great debugging
tricks to share, or was it your existing depth of NiFi knowledge that led
you to that solution?
As a follow-on to that, I'm seeing that Controller Services created via
Template instantiation are created in "Disabled" state. Is that expected?
Is there any way to drag-and-drop a template with controller services onto
a canvas and have the newly created controller services enabled?
Thanks again,
Brian
On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende wrote:
> Brian,
>
> I just ran into this exact scenario today while testing a custom controller
> service.
>
> The cause of the problem was that my custom service NAR, did not have a NAR
> dependency on the service API NAR because I had accidentally marked the API
> NAR as provided in the pom file. Once I removed the provided scope
> everything worked.
>
> You can check the MANIFEST file in your custom service NAR... under your
> NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it should
> have a "Nar-Dependency-Id" on your API NAR. If it is missing that, that is
> likely the problem.
>
> -Bryan
>
>
>
> On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli
> wrote:
>
> > NiFi Gurus,
> >
> > Regarding Controller Services, I'm seeing some strange behavior
> regarding a
> > custom Controller Service only showing its ID in a Processor and was
> > wondering if anyone's encountered similar issues. Seeing this on the
> > latest develop branch, 0.3.0-SNAPSHOT.
> >
> > Steps taken:
> > 1. Create instance of Controller Service in the Controller Settings >
> > Controller Services tab
> > 1a. Enable that instance of the Controller Service
> > 2. Add new Processor on canvas that will use Controller Service
> > 3. Configure Processor
> > 3a. Existing Controller Service does not show in property options. Only
> "No
> > value" and "Create new service..."
> > 3b. Clicking "Create new service..." opens modal with the custom
> Controller
> > Service as the only item in the dropdown
> > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> Controller
> > Service in the Value of the Processor Property.
> >
> > Observed state:
> > - Processor is still invalid
> > - The ID in the Processor Property matches the ID of the second
> (disabled)
> > instance of the Controller Service in the Controller Settings modal.
> >
> > There are no errors in nifi-app.log - only contains log messages from the
> > StandardControllerServiceProvider that it "Created Controller Service of
> > type [my-implementation-class] with identifier
> > [UUID-that-shows-in-the-GUI]"
> >
> > Adding a DetectDuplicate processor with the
> > DistributedMapCacheClientService works just fine, so it'd hard to point
> > fingers at the framework. Any thoughts on where to begin debugging
> and/or
> > loggers to turn up would be greatly appreciated!
> >
> > Thanks,
> > Brian
> >
>
--
Brian Ghigiarelli
570-878-9139
Re: Custom Controller Service Debugging
Brian,
The NAR dependency stuff all breaks down to providing classpath isolation
and not polluting the classpath of downstream NARs with dependencies
required for Controller Service implementation. We try to keep Controller
Service interfaces lean by not depending on external libs in the interface
(generally a good practice anyways). Both the implementation of the
Controller Service and the any Processor NAR that want to consume the
Controller Service needs to depend on the Controller Service API NAR (where
the interface lives). This established a common parent ClassLoader which
allows the service to be access across NARs. If your building Processor
NARs that depend on custom Controller Services and standard Controller
Services, you'll want to have your custom Controller Services API NAR
depend on the standard Controller Services API NAR. The NAR dependency tree
(if you will) maps to a ClassLoader hierarchy within NiFi.
Controller Services being added in a disabled state is remaining consistent
with Processors (and other components) in template. Also, because sensitive
properties are not included in templates, they may require the user to
complete the configuration prior to enabling.
Matt
On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli
wrote:
> Bryan,
>
> Thank you!! Adding a NAR dependency to my standard-processors NAR did the
> trick to get NiFi to recognize the service for these processors and
> auto-create a new instance of the Controller Service on template import. I
> owe ya a beer or six the next time our paths cross.
>
> How did you figure out the required NAR dependency? Any great debugging
> tricks to share, or was it your existing depth of NiFi knowledge that led
> you to that solution?
>
> As a follow-on to that, I'm seeing that Controller Services created via
> Template instantiation are created in "Disabled" state. Is that expected?
> Is there any way to drag-and-drop a template with controller services onto
> a canvas and have the newly created controller services enabled?
>
> Thanks again,
> Brian
>
> On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende wrote:
>
> > Brian,
> >
> > I just ran into this exact scenario today while testing a custom
> controller
> > service.
> >
> > The cause of the problem was that my custom service NAR, did not have a
> NAR
> > dependency on the service API NAR because I had accidentally marked the
> API
> > NAR as provided in the pom file. Once I removed the provided scope
> > everything worked.
> >
> > You can check the MANIFEST file in your custom service NAR... under your
> > NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it should
> > have a "Nar-Dependency-Id" on your API NAR. If it is missing that, that
> is
> > likely the problem.
> >
> > -Bryan
> >
> >
> >
> > On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli
> > wrote:
> >
> > > NiFi Gurus,
> > >
> > > Regarding Controller Services, I'm seeing some strange behavior
> > regarding a
> > > custom Controller Service only showing its ID in a Processor and was
> > > wondering if anyone's encountered similar issues. Seeing this on the
> > > latest develop branch, 0.3.0-SNAPSHOT.
> > >
> > > Steps taken:
> > > 1. Create instance of Controller Service in the Controller Settings >
> > > Controller Services tab
> > > 1a. Enable that instance of the Controller Service
> > > 2. Add new Processor on canvas that will use Controller Service
> > > 3. Configure Processor
> > > 3a. Existing Controller Service does not show in property options. Only
> > "No
> > > value" and "Create new service..."
> > > 3b. Clicking "Create new service..." opens modal with the custom
> > Controller
> > > Service as the only item in the dropdown
> > > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> > Controller
> > > Service in the Value of the Processor Property.
> > >
> > > Observed state:
> > > - Processor is still invalid
> > > - The ID in the Processor Property matches the ID of the second
> > (disabled)
> > > instance of the Controller Service in the Controller Settings modal.
> > >
> > > There are no errors in nifi-app.log - only contains log messages from
> the
> > > StandardControllerServiceProvider that it "Created Controller Service
> of
> > > type [my-implementation-class] with identifier
> > > [UUID-that-shows-in-the-GUI]"
> > >
> > > Adding a DetectDuplicate processor with the
> > > DistributedMapCacheClientService works just fine, so it'd hard to point
> > > fingers at the framework. Any thoughts on where to begin debugging
> > and/or
> > > loggers to turn up would be greatly appreciated!
> > >
> > > Thanks,
> > > Brian
> > >
> >
>
>
>
> --
> Brian Ghigiarelli
> 570-878-9139
>
Re: Custom Controller Service Debugging
Brian,
Glad it worked!
I think it was just a combination of things... I had previously spent a lot
of time studying the existing dependency setup in nifi-standard-bundle, as
well as the dev
guide section on NARs [1]. Also, someone else had ran into a similar
problem recently and it turned out to be a dependency problem as well.
There should be a Maven archetype for Controller Services [2] coming soon
which might help nail down the correct dependency setup, also after seeing
your email last night I put together a bare-bones project [3] that has a
custom processor using a custom controller service (both created using
the archetypes) to serve as an example.
-Bryan
[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
[2] https://issues.apache.org/jira/browse/NIFI-736
[3] https://github.com/bbende/nifi-dependency-example
On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman
wrote:
> Brian,
>
> The NAR dependency stuff all breaks down to providing classpath isolation
> and not polluting the classpath of downstream NARs with dependencies
> required for Controller Service implementation. We try to keep Controller
> Service interfaces lean by not depending on external libs in the interface
> (generally a good practice anyways). Both the implementation of the
> Controller Service and the any Processor NAR that want to consume the
> Controller Service needs to depend on the Controller Service API NAR (where
> the interface lives). This established a common parent ClassLoader which
> allows the service to be access across NARs. If your building Processor
> NARs that depend on custom Controller Services and standard Controller
> Services, you'll want to have your custom Controller Services API NAR
> depend on the standard Controller Services API NAR. The NAR dependency tree
> (if you will) maps to a ClassLoader hierarchy within NiFi.
>
> Controller Services being added in a disabled state is remaining consistent
> with Processors (and other components) in template. Also, because sensitive
> properties are not included in templates, they may require the user to
> complete the configuration prior to enabling.
>
> Matt
>
> On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli
> wrote:
>
> > Bryan,
> >
> > Thank you!! Adding a NAR dependency to my standard-processors NAR did the
> > trick to get NiFi to recognize the service for these processors and
> > auto-create a new instance of the Controller Service on template
> import. I
> > owe ya a beer or six the next time our paths cross.
> >
> > How did you figure out the required NAR dependency? Any great debugging
> > tricks to share, or was it your existing depth of NiFi knowledge that led
> > you to that solution?
> >
> > As a follow-on to that, I'm seeing that Controller Services created via
> > Template instantiation are created in "Disabled" state. Is that
> expected?
> > Is there any way to drag-and-drop a template with controller services
> onto
> > a canvas and have the newly created controller services enabled?
> >
> > Thanks again,
> > Brian
> >
> > On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende wrote:
> >
> > > Brian,
> > >
> > > I just ran into this exact scenario today while testing a custom
> > controller
> > > service.
> > >
> > > The cause of the problem was that my custom service NAR, did not have a
> > NAR
> > > dependency on the service API NAR because I had accidentally marked the
> > API
> > > NAR as provided in the pom file. Once I removed the provided scope
> > > everything worked.
> > >
> > > You can check the MANIFEST file in your custom service NAR... under
> your
> > > NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it should
> > > have a "Nar-Dependency-Id" on your API NAR. If it is missing that, that
> > is
> > > likely the problem.
> > >
> > > -Bryan
> > >
> > >
> > >
> > > On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <
> briang...@gmail.com>
> > > wrote:
> > >
> > > > NiFi Gurus,
> > > >
> > > > Regarding Controller Services, I'm seeing some strange behavior
> > > regarding a
> > > > custom Controller Service only showing its ID in a Processor and was
> > > > wondering if anyone's encountered similar issues. Seeing this on the
> > > > latest develop branch, 0.3.0-SNAPSHOT.
> > > >
> > > > Steps taken:
> > > > 1. Create instance of Controller Service in the Controller Settings >
> > > > Controller Services tab
> > > > 1a. Enable that instance of the Controller Service
> > > > 2. Add new Processor on canvas that will use Controller Service
> > > > 3. Configure Processor
> > > > 3a. Existing Controller Service does not show in property options.
> Only
> > > "No
> > > > value" and "Create new service..."
> > > > 3b. Clicking "Create new service..." opens modal with the custom
> > > Controller
> > > > Service as the only item in the dropdown
> > > > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> > > Controller
> > > > Service in the Value of the Processor Property.
> > > >
> > > > Observed
Re: Custom Controller Service Debugging
Matt and Bryan, Thanks for the detailed explanation. The class loading aspect makes perfect sense, and yes, importing as disabled is understood as well. I do think I've come across a small bug with automatically creating Controller Services on template imports though. NiFi is not automatically creating Controller Services for Processors inside of Process Groups on template imports. Can one of you check me on this before we write up a Jira ticket? Steps to reproduce: 1. Start fresh (clean canvas, no controller services) 2. Create Process Group and go into it 3. Add DetectDuplicate processor to Process Group 4. Configure DetectDuplicate processor with a new DistributedMapCacheClientService and enable that service so that the DetectDuplicate processor is valid and ready to run 5. Navigate to root canvas (see the Process Group) 6. Create template from root canvas (name: "Controller Service Test") 7. Delete everything from the canvas and disable + delete the controller service so that we're essentially back to the start 8. Drag and drop the "Controller Service Test" template onto the canvas (expecting that everything will be auto-created) 9. Navigate into the Process Group and notice that the DetectDuplicate processor is invalid because the DistributedMapCacheClientService ID cannot be found As an additional test, I created a template with a processor at the top-level and one inside the Process Group that are using the same Controller Service, and that works just fine. My guess is that the top-level processor is used to create the Controller Service and the inner-processor picks it up by that same ID. I can provide a sample template in a Jira ticket if that would help. Thanks, Brian On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende wrote: > Brian, > > Glad it worked! > > I think it was just a combination of things... I had previously spent a lot > of time studying the existing dependency setup in nifi-standard-bundle, as > well as the dev > guide section on NARs [1]. Also, someone else had ran into a similar > problem recently and it turned out to be a dependency problem as well. > > There should be a Maven archetype for Controller Services [2] coming soon > which might help nail down the correct dependency setup, also after seeing > your email last night I put together a bare-bones project [3] that has a > custom processor using a custom controller service (both created using > the archetypes) to serve as an example. > > -Bryan > > [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars > [2] https://issues.apache.org/jira/browse/NIFI-736 > [3] https://github.com/bbende/nifi-dependency-example > > > > On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman > wrote: > > > Brian, > > > > The NAR dependency stuff all breaks down to providing classpath isolation > > and not polluting the classpath of downstream NARs with dependencies > > required for Controller Service implementation. We try to keep Controller > > Service interfaces lean by not depending on external libs in the > interface > > (generally a good practice anyways). Both the implementation of the > > Controller Service and the any Processor NAR that want to consume the > > Controller Service needs to depend on the Controller Service API NAR > (where > > the interface lives). This established a common parent ClassLoader which > > allows the service to be access across NARs. If your building Processor > > NARs that depend on custom Controller Services and standard Controller > > Services, you'll want to have your custom Controller Services API NAR > > depend on the standard Controller Services API NAR. The NAR dependency > tree > > (if you will) maps to a ClassLoader hierarchy within NiFi. > > > > Controller Services being added in a disabled state is remaining > consistent > > with Processors (and other components) in template. Also, because > sensitive > > properties are not included in templates, they may require the user to > > complete the configuration prior to enabling. > > > > Matt > > > > On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli > > wrote: > > > > > Bryan, > > > > > > Thank you!! Adding a NAR dependency to my standard-processors NAR did > the > > > trick to get NiFi to recognize the service for these processors and > > > auto-create a new instance of the Controller Service on template > > import. I > > > owe ya a beer or six the next time our paths cross. > > > > > > How did you figure out the required NAR dependency? Any great debugging > > > tricks to share, or was it your existing depth of NiFi knowledge that > led > > > you to that solution? > > > > > > As a follow-on to that, I'm seeing that Controller Services created via > > > Template instantiation are created in "Disabled" state. Is that > > expected? > > > Is there any way to drag-and-drop a template with controller services > > onto > > > a canvas and have the newly created controller services enabled? > > > > > > Thanks again, > > > Brian > > > > > > On
Re: Custom Controller Service Debugging
Brian, I see the same behavior regarding Controller Service creation when referenced by Processor's in sub Groups. Please file the JIRA ticket and it will be addressed in our release. Thanks! Matt On Fri, Aug 14, 2015 at 11:30 AM, Brian Ghigiarelli wrote: > Matt and Bryan, > > Thanks for the detailed explanation. The class loading aspect makes > perfect sense, and yes, importing as disabled is understood as well. > > I do think I've come across a small bug with automatically creating > Controller Services on template imports though. NiFi is not automatically > creating Controller Services for Processors inside of Process Groups on > template imports. Can one of you check me on this before we write up a Jira > ticket? > > Steps to reproduce: > 1. Start fresh (clean canvas, no controller services) > 2. Create Process Group and go into it > 3. Add DetectDuplicate processor to Process Group > 4. Configure DetectDuplicate processor with a new > DistributedMapCacheClientService and enable that service so that the > DetectDuplicate processor is valid and ready to run > 5. Navigate to root canvas (see the Process Group) > 6. Create template from root canvas (name: "Controller Service Test") > 7. Delete everything from the canvas and disable + delete the controller > service so that we're essentially back to the start > 8. Drag and drop the "Controller Service Test" template onto the canvas > (expecting that everything will be auto-created) > 9. Navigate into the Process Group and notice that the DetectDuplicate > processor is invalid because the DistributedMapCacheClientService ID cannot > be found > > As an additional test, I created a template with a processor at the > top-level and one inside the Process Group that are using the same > Controller Service, and that works just fine. My guess is that the > top-level processor is used to create the Controller Service and the > inner-processor picks it up by that same ID. > > I can provide a sample template in a Jira ticket if that would help. > > Thanks, > Brian > > On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende wrote: > > > Brian, > > > > Glad it worked! > > > > I think it was just a combination of things... I had previously spent a > lot > > of time studying the existing dependency setup in nifi-standard-bundle, > as > > well as the dev > > guide section on NARs [1]. Also, someone else had ran into a similar > > problem recently and it turned out to be a dependency problem as well. > > > > There should be a Maven archetype for Controller Services [2] coming soon > > which might help nail down the correct dependency setup, also after > seeing > > your email last night I put together a bare-bones project [3] that has a > > custom processor using a custom controller service (both created using > > the archetypes) to serve as an example. > > > > -Bryan > > > > [1] > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars > > [2] https://issues.apache.org/jira/browse/NIFI-736 > > [3] https://github.com/bbende/nifi-dependency-example > > > > > > > > On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman > > wrote: > > > > > Brian, > > > > > > The NAR dependency stuff all breaks down to providing classpath > isolation > > > and not polluting the classpath of downstream NARs with dependencies > > > required for Controller Service implementation. We try to keep > Controller > > > Service interfaces lean by not depending on external libs in the > > interface > > > (generally a good practice anyways). Both the implementation of the > > > Controller Service and the any Processor NAR that want to consume the > > > Controller Service needs to depend on the Controller Service API NAR > > (where > > > the interface lives). This established a common parent ClassLoader > which > > > allows the service to be access across NARs. If your building Processor > > > NARs that depend on custom Controller Services and standard Controller > > > Services, you'll want to have your custom Controller Services API NAR > > > depend on the standard Controller Services API NAR. The NAR dependency > > tree > > > (if you will) maps to a ClassLoader hierarchy within NiFi. > > > > > > Controller Services being added in a disabled state is remaining > > consistent > > > with Processors (and other components) in template. Also, because > > sensitive > > > properties are not included in templates, they may require the user to > > > complete the configuration prior to enabling. > > > > > > Matt > > > > > > On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli < > briang...@gmail.com> > > > wrote: > > > > > > > Bryan, > > > > > > > > Thank you!! Adding a NAR dependency to my standard-processors NAR did > > the > > > > trick to get NiFi to recognize the service for these processors and > > > > auto-create a new instance of the Controller Service on template > > > import. I > > > > owe ya a beer or six the next time our paths cross. > > > > > > > > How did you figure out the required NAR dependency?
Re: Custom Controller Service Debugging
Cool - thanks for verifying that, Matt! https://issues.apache.org/jira/browse/NIFI-854 On Fri, Aug 14, 2015 at 11:43 AM, Matt Gilman wrote: > Brian, > > I see the same behavior regarding Controller Service creation when > referenced by Processor's in sub Groups. Please file the JIRA ticket and it > will be addressed in our release. Thanks! > > Matt > > On Fri, Aug 14, 2015 at 11:30 AM, Brian Ghigiarelli > wrote: > > > Matt and Bryan, > > > > Thanks for the detailed explanation. The class loading aspect makes > > perfect sense, and yes, importing as disabled is understood as well. > > > > I do think I've come across a small bug with automatically creating > > Controller Services on template imports though. NiFi is not > automatically > > creating Controller Services for Processors inside of Process Groups on > > template imports. Can one of you check me on this before we write up a > Jira > > ticket? > > > > Steps to reproduce: > > 1. Start fresh (clean canvas, no controller services) > > 2. Create Process Group and go into it > > 3. Add DetectDuplicate processor to Process Group > > 4. Configure DetectDuplicate processor with a new > > DistributedMapCacheClientService and enable that service so that the > > DetectDuplicate processor is valid and ready to run > > 5. Navigate to root canvas (see the Process Group) > > 6. Create template from root canvas (name: "Controller Service Test") > > 7. Delete everything from the canvas and disable + delete the controller > > service so that we're essentially back to the start > > 8. Drag and drop the "Controller Service Test" template onto the canvas > > (expecting that everything will be auto-created) > > 9. Navigate into the Process Group and notice that the DetectDuplicate > > processor is invalid because the DistributedMapCacheClientService ID > cannot > > be found > > > > As an additional test, I created a template with a processor at the > > top-level and one inside the Process Group that are using the same > > Controller Service, and that works just fine. My guess is that the > > top-level processor is used to create the Controller Service and the > > inner-processor picks it up by that same ID. > > > > I can provide a sample template in a Jira ticket if that would help. > > > > Thanks, > > Brian > > > > On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende wrote: > > > > > Brian, > > > > > > Glad it worked! > > > > > > I think it was just a combination of things... I had previously spent a > > lot > > > of time studying the existing dependency setup in nifi-standard-bundle, > > as > > > well as the dev > > > guide section on NARs [1]. Also, someone else had ran into a similar > > > problem recently and it turned out to be a dependency problem as well. > > > > > > There should be a Maven archetype for Controller Services [2] coming > soon > > > which might help nail down the correct dependency setup, also after > > seeing > > > your email last night I put together a bare-bones project [3] that has > a > > > custom processor using a custom controller service (both created using > > > the archetypes) to serve as an example. > > > > > > -Bryan > > > > > > [1] > > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars > > > [2] https://issues.apache.org/jira/browse/NIFI-736 > > > [3] https://github.com/bbende/nifi-dependency-example > > > > > > > > > > > > On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman > > > wrote: > > > > > > > Brian, > > > > > > > > The NAR dependency stuff all breaks down to providing classpath > > isolation > > > > and not polluting the classpath of downstream NARs with dependencies > > > > required for Controller Service implementation. We try to keep > > Controller > > > > Service interfaces lean by not depending on external libs in the > > > interface > > > > (generally a good practice anyways). Both the implementation of the > > > > Controller Service and the any Processor NAR that want to consume the > > > > Controller Service needs to depend on the Controller Service API NAR > > > (where > > > > the interface lives). This established a common parent ClassLoader > > which > > > > allows the service to be access across NARs. If your building > Processor > > > > NARs that depend on custom Controller Services and standard > Controller > > > > Services, you'll want to have your custom Controller Services API NAR > > > > depend on the standard Controller Services API NAR. The NAR > dependency > > > tree > > > > (if you will) maps to a ClassLoader hierarchy within NiFi. > > > > > > > > Controller Services being added in a disabled state is remaining > > > consistent > > > > with Processors (and other components) in template. Also, because > > > sensitive > > > > properties are not included in templates, they may require the user > to > > > > complete the configuration prior to enabling. > > > > > > > > Matt > > > > > > > > On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli < > > briang...@gmail.com> > > > > wrote: > > > > > > > > >
