Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch
Peter, Looks like this won't make it into 1.10, but it's a simple patch to apply if you download the source code for 1.10 and apply it manually. Mike On Tue, Oct 22, 2019 at 2:46 PM Matt Burgess wrote: > For saving space (if you want to run from the assembly/target directory) > you can add a -Ddir-only and it won’t build the tar.gz. > > > > > Sent from my iPhone > > On Oct 22, 2019, at 2:25 PM, Mike Thomsen wrote: > > > If you run `mvn clean install -DskipTests=true` from the root of the > source folder, you'll get a tar.gz build in $ROOT/nifi-assembly/target. I'd > recommend testing against that as it'll be faster. > > On Tue, Oct 22, 2019 at 1:56 PM Peter Moberg > wrote: > >> Mike, >> >> thanks for putting together that PR. I have built everything successfully >> but I haven't been able to test this yet since I haven't built the new >> Docker image. >> >> I assume you guys just use the ‘dockermaven’ folder to build the >> Dockerfile with src artifacts? >> >> My current dev machine is running out of disk space so I am spending some >> time trying to get it back to a runnable state. Hopefully I can build the >> PR and test it in Kubernetes this week sometime. >> >> Thanks, >> >> Peter >> On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen , >> wrote: >> >> As a compromise, I upgraded to the latest 5.X client and manually >> incremented Apache HttpClient to 4.5.10. PR is here: >> >> https://github.com/apache/nifi/pull/3828 >> >> There are integration tests for that package that automatically startup >> and provision an ES node, and they all passed with this configuration. >> Hopefully this PR will fix your issue since it appears to be an external >> dependency causing it. >> >> On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen >> wrote: >> >>> There's no hard and fast reason to stay with 5.X there, so you can build >>> your own copy of 1.9.2 with that dependency upgraded if you want to try it >>> out. I'll try to find time to test that change on 1.10.0-SNAPSHOT. >>> >>> On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg >>> wrote: >>> The certs in the TrustStore are marked as trusted. The host name specified in the ClientServiceImpl is: https://quickstart-es-http.es-cluster:9200 The CN field of the server certificate is: https://quickstart-es-http.es-cluster.es.local So at first glance it looks like the issue would be that the CN field differs from the host name. However, Im not getting an error message saying that the HostName doesn’t match the CN. According to the RFC spec first mentioned by Andy the CN field should only be used if the SubjectiveAlternativeName is empty. The Server Cert has the SAN set to: quickstart-es-http.es-cluster What is really strange to me is why I can connect using the QueryElasticSearch processor with the same SSLContextService and hostname but when I use the ClientServicesImpl it will not work. So I did some more investigation and it looks like the QueryES processor uses the okHTTP library and ClientServicesImpl uses the Elastic Search RestClient which in turn uses the HTTPClient from Apache. There was an issue with the HTTPClient library in v4.5.2 ( https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t do the right hostname verification. The version of Apache Nifi that Im using is 1.9.2 and it seems to use v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of Apache HTTPClient. https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties But again if it was a HostName validation issue the error message should read more like: “Certificate for doesn’t match common name of certificate subject…” On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen , wrote: I'm far from a SSL/TLS expert, but let's get these out of the way: 1. Did you mark the server's cert as "trusted" when you created the trust store with keytool? 2. Are you sure that you're specifying the same hostname in the client service that is in the CN field in the server's cert? FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use it to generate valid certificates. It's meant for NiFi, but the certs it generates should be a solid step up from self-signed certs if you're not able to access a company CA. On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg wrote: > Nope. No good. I even dump the network traffic and analyzed it in > Wireshark. The ES server sends back two certificates (server + self-signed > one) and both of them are present in my TrustStore. I am specifying both a > TrustStore and a Keystore now but it still gives the error that it can’t > find the certificate. > > Thanks, > > Peter > On Oct 18, 2019, 4:44 PM -0500, Peter Moberg ,
Re: Unit test is getting failed for custom ,Awscredentialprovidercontrolerservice
Thanks Bryan, got ur point.
Now everything is working fine.
On Tue, 22 Oct, 2019, 7:46 PM Bryan Bende, wrote:
> You shouldn't be modifying the service code to make the test pass.
>
> You need to set whatever properties are needed to make it valid by
> using Runner.setProperty(service, property name, value)
>
> On Tue, Oct 22, 2019 at 6:54 AM Otto Fowler
> wrote:
> >
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/credentials/provider/service/AWSCredentialsProviderControllerServiceTest.java
> >
> > It may be helpful for you to look at other things in the code base. Here
> are the tests for the current service
> >
> >
> >
> >
> > On October 21, 2019 at 22:31:38, sanjeet rath (rath.sanj...@gmail.com)
> wrote:
> >
> > Hi Team,
> >
> > Our project structure we have a custom controller service
> ,Awscredentialprovidercontrolerservice controller service to connect AWS
> with our defined 5 attributes means properties.(This is present in a
> separate project NIFI-AWS-CUSTOM_PING_CONTROLER).
> >
> > In NiFi UI this controller service is working fine .But in Unit testing
> I am using bellow code to test.
> >
> > @Test
> > public void test Awscredentialprovidercontrolerservice() {
> >
> > final TestRunner runner = TestRunners.newTestRunner(new
> puts3Object);
> > final Awscredentialprovidercontrolerservice serviceimpl= new
> Awscredentialprovidercontrolerservice()
> > Runner.setproperty(…) //Setting my 5 properties which I have created for
> my custom controller service
> > Runner.enablecontrolerservice(serviceimpl)
> > // will do assert }
> >
> > Here in enabling controller service gives null pointer exception in the
> custom validate method , its excepting all the default Properties also need
> to be declared like Accesskey, secret key etc(which is present in the
> default Awscredentialprovidercontrolerservice class) in my custom
> controller Awscredentialprovidercontrolerservice.
> >
> > After declaring the default properties in my custom
> Awscredentialprovidercontrolerservice unit test is working fine.But problem
> is these parameters are appearing in the NiFi UI of my custom
> Awscredentialprovidercontrolerservice.
> >
> > So I have 2 option, either after declaring of the default properties, is
> there any way to stop displaying I Nifi UI.
> >
> > Or As its working fine in UI flow without setting default properties in
> custom Awscredentialprovidercontrolerservice.so should I set something in
> Unit test case to make it passed
> >
> > Thanks & Regards
> > --
> > Sanjeet Kumar Rath,
> > mob- +91 8777577470
> >
>
Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch
For saving space (if you want to run from the assembly/target directory) you can add a -Ddir-only and it won’t build the tar.gz. Sent from my iPhone > On Oct 22, 2019, at 2:25 PM, Mike Thomsen wrote: > > > If you run `mvn clean install -DskipTests=true` from the root of the source > folder, you'll get a tar.gz build in $ROOT/nifi-assembly/target. I'd > recommend testing against that as it'll be faster. > >> On Tue, Oct 22, 2019 at 1:56 PM Peter Moberg wrote: >> Mike, >> >> thanks for putting together that PR. I have built everything successfully >> but I haven't been able to test this yet since I haven't built the new >> Docker image. >> >> I assume you guys just use the ‘dockermaven’ folder to build the Dockerfile >> with src artifacts? >> >> My current dev machine is running out of disk space so I am spending some >> time trying to get it back to a runnable state. Hopefully I can build the PR >> and test it in Kubernetes this week sometime. >> >> Thanks, >> >> Peter >>> On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen , >>> wrote: >>> As a compromise, I upgraded to the latest 5.X client and manually >>> incremented Apache HttpClient to 4.5.10. PR is here: >>> >>> https://github.com/apache/nifi/pull/3828 >>> >>> There are integration tests for that package that automatically startup and >>> provision an ES node, and they all passed with this configuration. >>> Hopefully this PR will fix your issue since it appears to be an external >>> dependency causing it. >>> On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen wrote: There's no hard and fast reason to stay with 5.X there, so you can build your own copy of 1.9.2 with that dependency upgraded if you want to try it out. I'll try to find time to test that change on 1.10.0-SNAPSHOT. > On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg > wrote: > The certs in the TrustStore are marked as trusted. > > The host name specified in the ClientServiceImpl is: > https://quickstart-es-http.es-cluster:9200 > > The CN field of the server certificate is: > https://quickstart-es-http.es-cluster.es.local > > So at first glance it looks like the issue would be that the CN field > differs from the host name. However, Im not getting an error message > saying that the HostName doesn’t match the CN. According to the RFC spec > first mentioned by Andy the CN field should only be used if the > SubjectiveAlternativeName is empty. > > The Server Cert has the SAN set to: quickstart-es-http.es-cluster > > What is really strange to me is why I can connect using the > QueryElasticSearch processor with the same SSLContextService and hostname > but when I use the ClientServicesImpl it will not work. > > So I did some more investigation and it looks like the QueryES processor > uses the okHTTP library and ClientServicesImpl uses the Elastic Search > RestClient which in turn uses the HTTPClient from Apache. > > There was an issue with the HTTPClient library in v4.5.2 > (https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t > do the right hostname verification. > > The version of Apache Nifi that Im using is 1.9.2 and it seems to use > v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 > of Apache HTTPClient. > > https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties > > But again if it was a HostName validation issue the error message should > read more like: “Certificate for doesn’t match common name of > certificate subject…” > >> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen , >> wrote: >> I'm far from a SSL/TLS expert, but let's get these out of the way: >> >> 1. Did you mark the server's cert as "trusted" when you created the >> trust store with keytool? >> 2. Are you sure that you're specifying the same hostname in the client >> service that is in the CN field in the server's cert? >> >> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can use >> it to generate valid certificates. It's meant for NiFi, but the certs it >> generates should be a solid step up from self-signed certs if you're not >> able to access a company CA. >> >>> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg >>> wrote: >>> Nope. No good. I even dump the network traffic and analyzed it in >>> Wireshark. The ES server sends back two certificates (server + >>> self-signed one) and both of them are present in my TrustStore. I am >>> specifying both a TrustStore and a Keystore now but it still gives the >>> error that it can’t find the certificate. >>> >>> Thanks, >>> >>> Peter On Oct 18, 2019, 4:44 PM -0500, Peter Moberg , wrote: Think I might have found
Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch
If you run `mvn clean install -DskipTests=true` from the root of the source folder, you'll get a tar.gz build in $ROOT/nifi-assembly/target. I'd recommend testing against that as it'll be faster. On Tue, Oct 22, 2019 at 1:56 PM Peter Moberg wrote: > Mike, > > thanks for putting together that PR. I have built everything successfully > but I haven't been able to test this yet since I haven't built the new > Docker image. > > I assume you guys just use the ‘dockermaven’ folder to build the > Dockerfile with src artifacts? > > My current dev machine is running out of disk space so I am spending some > time trying to get it back to a runnable state. Hopefully I can build the > PR and test it in Kubernetes this week sometime. > > Thanks, > > Peter > On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen , > wrote: > > As a compromise, I upgraded to the latest 5.X client and manually > incremented Apache HttpClient to 4.5.10. PR is here: > > https://github.com/apache/nifi/pull/3828 > > There are integration tests for that package that automatically startup > and provision an ES node, and they all passed with this configuration. > Hopefully this PR will fix your issue since it appears to be an external > dependency causing it. > > On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen > wrote: > >> There's no hard and fast reason to stay with 5.X there, so you can build >> your own copy of 1.9.2 with that dependency upgraded if you want to try it >> out. I'll try to find time to test that change on 1.10.0-SNAPSHOT. >> >> On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg >> wrote: >> >>> The certs in the TrustStore are marked as trusted. >>> >>> The host name specified in the ClientServiceImpl is: >>> https://quickstart-es-http.es-cluster:9200 >>> >>> The CN field of the server certificate is: >>> https://quickstart-es-http.es-cluster.es.local >>> >>> So at first glance it looks like the issue would be that the CN field >>> differs from the host name. However, Im not getting an error message saying >>> that the HostName doesn’t match the CN. According to the RFC spec first >>> mentioned by Andy the CN field should only be used if the >>> SubjectiveAlternativeName is empty. >>> >>> The Server Cert has the SAN set to: quickstart-es-http.es-cluster >>> >>> What is really strange to me is why I can connect using the >>> QueryElasticSearch processor with the same SSLContextService and hostname >>> but when I use the ClientServicesImpl it will not work. >>> >>> So I did some more investigation and it looks like the QueryES processor >>> uses the okHTTP library and ClientServicesImpl uses the Elastic Search >>> RestClient which in turn uses the HTTPClient from Apache. >>> >>> There was an issue with the HTTPClient library in v4.5.2 ( >>> https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it didn’t >>> do the right hostname verification. >>> >>> The version of Apache Nifi that Im using is 1.9.2 and it seems to use >>> v.5.6.8 of Elastic Search client libraries and that in turn uses v4.5.2 of >>> Apache HTTPClient. >>> >>> >>> https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties >>> >>> But again if it was a HostName validation issue the error message should >>> read more like: “Certificate for doesn’t match common name of >>> certificate subject…” >>> >>> On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen , >>> wrote: >>> >>> I'm far from a SSL/TLS expert, but let's get these out of the way: >>> >>> 1. Did you mark the server's cert as "trusted" when you created the >>> trust store with keytool? >>> 2. Are you sure that you're specifying the same hostname in the client >>> service that is in the CN field in the server's cert? >>> >>> FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you can >>> use it to generate valid certificates. It's meant for NiFi, but the certs >>> it generates should be a solid step up from self-signed certs if you're not >>> able to access a company CA. >>> >>> On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg >>> wrote: >>> Nope. No good. I even dump the network traffic and analyzed it in Wireshark. The ES server sends back two certificates (server + self-signed one) and both of them are present in my TrustStore. I am specifying both a TrustStore and a Keystore now but it still gives the error that it can’t find the certificate. Thanks, Peter On Oct 18, 2019, 4:44 PM -0500, Peter Moberg , wrote: Think I might have found the issue. Will report tonight. Mike, please don’t spend any time debugging this because I think it might be an issue on my side. Appreciate all the help so far. Thanks, Peter On Oct 18, 2019, 2:21 PM -0500, Peter Moberg , wrote: Here it is: 2019-10-18 18:47:02,548 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.LookupRecord LookupRecord[id=df596687-016d-1000--65536eb2] Failed to process
Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch
Mike, thanks for putting together that PR. I have built everything successfully but I haven't been able to test this yet since I haven't built the new Docker image. I assume you guys just use the ‘dockermaven’ folder to build the Dockerfile with src artifacts? My current dev machine is running out of disk space so I am spending some time trying to get it back to a runnable state. Hopefully I can build the PR and test it in Kubernetes this week sometime. Thanks, Peter On Oct 20, 2019, 7:11 AM -0500, Mike Thomsen , wrote: > As a compromise, I upgraded to the latest 5.X client and manually incremented > Apache HttpClient to 4.5.10. PR is here: > > https://github.com/apache/nifi/pull/3828 > > There are integration tests for that package that automatically startup and > provision an ES node, and they all passed with this configuration. Hopefully > this PR will fix your issue since it appears to be an external dependency > causing it. > > > On Sun, Oct 20, 2019 at 5:26 AM Mike Thomsen wrote: > > > There's no hard and fast reason to stay with 5.X there, so you can build > > > your own copy of 1.9.2 with that dependency upgraded if you want to try > > > it out. I'll try to find time to test that change on 1.10.0-SNAPSHOT. > > > > > > > On Sun, Oct 20, 2019 at 1:52 AM Peter Moberg > > > > wrote: > > > > > The certs in the TrustStore are marked as trusted. > > > > > > > > > > The host name specified in the ClientServiceImpl is: > > > > > https://quickstart-es-http.es-cluster:9200 > > > > > > > > > > The CN field of the server certificate is: > > > > > https://quickstart-es-http.es-cluster.es.local > > > > > > > > > > So at first glance it looks like the issue would be that the CN field > > > > > differs from the host name. However, Im not getting an error message > > > > > saying that the HostName doesn’t match the CN. According to the RFC > > > > > spec first mentioned by Andy the CN field should only be used if the > > > > > SubjectiveAlternativeName is empty. > > > > > > > > > > The Server Cert has the SAN set to: quickstart-es-http.es-cluster > > > > > > > > > > What is really strange to me is why I can connect using the > > > > > QueryElasticSearch processor with the same SSLContextService and > > > > > hostname but when I use the ClientServicesImpl it will not work. > > > > > > > > > > So I did some more investigation and it looks like the QueryES > > > > > processor uses the okHTTP library and ClientServicesImpl uses the > > > > > Elastic Search RestClient which in turn uses the HTTPClient from > > > > > Apache. > > > > > > > > > > There was an issue with the HTTPClient library in v4.5.2 > > > > > (https://issues.apache.org/jira/browse/HTTPCLIENT-1802) where it > > > > > didn’t do the right hostname verification. > > > > > > > > > > The version of Apache Nifi that Im using is 1.9.2 and it seems to use > > > > > v.5.6.8 of Elastic Search client libraries and that in turn uses > > > > > v4.5.2 of Apache HTTPClient. > > > > > > > > > > https://github.com/elastic/elasticsearch/blob/v5.6.8/buildSrc/version.properties > > > > > > > > > > But again if it was a HostName validation issue the error message > > > > > should read more like: “Certificate for doesn’t match common > > > > > name of certificate subject…” > > > > > > > > > > On Oct 19, 2019, 7:39 AM -0500, Mike Thomsen > > > > > , wrote: > > > > > > I'm far from a SSL/TLS expert, but let's get these out of the way: > > > > > > > > > > > > 1. Did you mark the server's cert as "trusted" when you created the > > > > > > trust store with keytool? > > > > > > 2. Are you sure that you're specifying the same hostname in the > > > > > > client service that is in the CN field in the server's cert? > > > > > > > > > > > > FWIW, if you grab the NiFi TLS Toolkit from nifi.apache.org, you > > > > > > can use it to generate valid certificates. It's meant for NiFi, but > > > > > > the certs it generates should be a solid step up from self-signed > > > > > > certs if you're not able to access a company CA. > > > > > > > > > > > > > On Sat, Oct 19, 2019 at 2:06 AM Peter Moberg > > > > > > > wrote: > > > > > > > > Nope. No good. I even dump the network traffic and analyzed it > > > > > > > > in Wireshark. The ES server sends back two certificates (server > > > > > > > > + self-signed one) and both of them are present in my > > > > > > > > TrustStore. I am specifying both a TrustStore and a Keystore > > > > > > > > now but it still gives the error that it can’t find the > > > > > > > > certificate. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Peter > > > > > > > > On Oct 18, 2019, 4:44 PM -0500, Peter Moberg > > > > > > > > , wrote: > > > > > > > > > Think I might have found the issue. Will report tonight. > > > > > > > > > > > > > > > > > > Mike, please don’t spend any time debugging this because I > > > > > > > > > think it might be an issue on my side. Appreciate all the > > > > > > > > > help so
Re: Merge content Defrag with high activity
Thanks for the answer. When I read the source code, I saw ow it is complexe and powerful. In fact, i need multiples tasks, and I may loop on the "same" file. To summarize, it is something like : Object as X version. For first version : * Fork the object to one branch in order to do an external call and get the result. * Fork the object to do "nothing" then merge the result from first fork into it Increase the version number and go do it again. And so loop to all versions. And for "performance", I try to do this on 50 concurrent objects. In most case, it works during the merge. But sometimes... the bin was fired as "ready" but did not reached the expected content. Anyway, as you said, I had to work on all parameters. I change the fragment identifier in order to have the version number, and not only a value that is commun to all versions. I also tried to set current task, for the merge to 1 and set a Run Schedule to 0.1 But it was "pretty slow", almost not as fast as I expected. But now, working with a new identifier, and set maximum bean number relly greater than expected possible (6x more) I works as expected. I will survey anyway. Thanks Etienne Jouvin Le mar. 22 oct. 2019 à 16:59, Joe Witt a écrit : > Hello > > You should only have 1 or a few tasks at most for this processor. > Scheduling can be frequent but choosing different options and seeing for > your case is best. > > This processor is relatively difficult to configure correctly as it is a > complex case and has powerful options. What you will need to watch out for > is the maximum number of bins it can track at once. If each bin is to hold > at least and at most 2 things and lots of data is arriving then what you > need are lots of bins so focus on that setting. > > Thanks > > On Tue, Oct 22, 2019 at 10:49 AM Etienne Jouvin > wrote: > >> Hi, >> >> Here is the case. >> >> High activity, and use a MergeContent action. >> I setup the mergeContent with 300 concurrent action and no schedule, >> meaning Run Schedule set to 0. >> >> >> Minimum Number of Entries : 2 >> Maximum Number of Entries : 2 >> >> >> No limit on the size. >> >> >> In some case, I reach exception : >> because the expected number of fragments is 2 but found only 1 fragments >> >> >> >> What I believe is that I am reaching side effet. >> May be, I have multiple execution at the same time, and some bins are >> considered as fulled and returned to be proceeded. But when returned, the >> object does not contains all expected flowfiles and during the execution, >> function processBins in class BinFiles, I am reaching the exception. >> >> >> It seems that I manage to skip this error when setting concurrent task to >> 1. >> But, it slow down a little the process. >> >> Should I keep 300 concurrent tasks, and set some schedule, something like >> 0.1 second ? >> >> Regards >> >> Etienne Jouvin >> >
Re: Merge content Defrag with high activity
Hello You should only have 1 or a few tasks at most for this processor. Scheduling can be frequent but choosing different options and seeing for your case is best. This processor is relatively difficult to configure correctly as it is a complex case and has powerful options. What you will need to watch out for is the maximum number of bins it can track at once. If each bin is to hold at least and at most 2 things and lots of data is arriving then what you need are lots of bins so focus on that setting. Thanks On Tue, Oct 22, 2019 at 10:49 AM Etienne Jouvin wrote: > Hi, > > Here is the case. > > High activity, and use a MergeContent action. > I setup the mergeContent with 300 concurrent action and no schedule, > meaning Run Schedule set to 0. > > > Minimum Number of Entries : 2 > Maximum Number of Entries : 2 > > > No limit on the size. > > > In some case, I reach exception : > because the expected number of fragments is 2 but found only 1 fragments > > > > What I believe is that I am reaching side effet. > May be, I have multiple execution at the same time, and some bins are > considered as fulled and returned to be proceeded. But when returned, the > object does not contains all expected flowfiles and during the execution, > function processBins in class BinFiles, I am reaching the exception. > > > It seems that I manage to skip this error when setting concurrent task to > 1. > But, it slow down a little the process. > > Should I keep 300 concurrent tasks, and set some schedule, something like > 0.1 second ? > > Regards > > Etienne Jouvin >
Merge content Defrag with high activity
Hi, Here is the case. High activity, and use a MergeContent action. I setup the mergeContent with 300 concurrent action and no schedule, meaning Run Schedule set to 0. Minimum Number of Entries : 2 Maximum Number of Entries : 2 No limit on the size. In some case, I reach exception : because the expected number of fragments is 2 but found only 1 fragments What I believe is that I am reaching side effet. May be, I have multiple execution at the same time, and some bins are considered as fulled and returned to be proceeded. But when returned, the object does not contains all expected flowfiles and during the execution, function processBins in class BinFiles, I am reaching the exception. It seems that I manage to skip this error when setting concurrent task to 1. But, it slow down a little the process. Should I keep 300 concurrent tasks, and set some schedule, something like 0.1 second ? Regards Etienne Jouvin
Re: Unit test is getting failed for custom ,Awscredentialprovidercontrolerservice
You shouldn't be modifying the service code to make the test pass.
You need to set whatever properties are needed to make it valid by
using Runner.setProperty(service, property name, value)
On Tue, Oct 22, 2019 at 6:54 AM Otto Fowler wrote:
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/credentials/provider/service/AWSCredentialsProviderControllerServiceTest.java
>
> It may be helpful for you to look at other things in the code base. Here are
> the tests for the current service
>
>
>
>
> On October 21, 2019 at 22:31:38, sanjeet rath (rath.sanj...@gmail.com) wrote:
>
> Hi Team,
>
> Our project structure we have a custom controller service
> ,Awscredentialprovidercontrolerservice controller service to connect AWS
> with our defined 5 attributes means properties.(This is present in a separate
> project NIFI-AWS-CUSTOM_PING_CONTROLER).
>
> In NiFi UI this controller service is working fine .But in Unit testing I am
> using bellow code to test.
>
> @Test
> public void test Awscredentialprovidercontrolerservice() {
>
> final TestRunner runner = TestRunners.newTestRunner(new puts3Object);
> final Awscredentialprovidercontrolerservice serviceimpl= new
> Awscredentialprovidercontrolerservice()
> Runner.setproperty(…) //Setting my 5 properties which I have created for my
> custom controller service
> Runner.enablecontrolerservice(serviceimpl)
> // will do assert }
>
> Here in enabling controller service gives null pointer exception in the
> custom validate method , its excepting all the default Properties also need
> to be declared like Accesskey, secret key etc(which is present in the default
> Awscredentialprovidercontrolerservice class) in my custom controller
> Awscredentialprovidercontrolerservice.
>
> After declaring the default properties in my custom
> Awscredentialprovidercontrolerservice unit test is working fine.But problem
> is these parameters are appearing in the NiFi UI of my custom
> Awscredentialprovidercontrolerservice.
>
> So I have 2 option, either after declaring of the default properties, is
> there any way to stop displaying I Nifi UI.
>
> Or As its working fine in UI flow without setting default properties in
> custom Awscredentialprovidercontrolerservice.so should I set something in
> Unit test case to make it passed
>
> Thanks & Regards
> --
> Sanjeet Kumar Rath,
> mob- +91 8777577470
>
Re: Unit test is getting failed for custom ,Awscredentialprovidercontrolerservice
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/credentials/provider/service/AWSCredentialsProviderControllerServiceTest.java
It may be helpful for you to look at other things in the code base. Here
are the tests for the current service
On October 21, 2019 at 22:31:38, sanjeet rath (rath.sanj...@gmail.com)
wrote:
Hi Team,
Our project structure we have a custom controller service
,Awscredentialprovidercontrolerservice controller service to connect AWS
with our defined 5 attributes means properties.(This is present in a
separate project NIFI-AWS-CUSTOM_PING_CONTROLER).
In NiFi UI this controller service is working fine .But in Unit testing I
am using bellow code to test.
@Test
public void test Awscredentialprovidercontrolerservice() {
final TestRunner runner = TestRunners.newTestRunner(new
puts3Object);
final Awscredentialprovidercontrolerservice serviceimpl= new
Awscredentialprovidercontrolerservice()
Runner.setproperty(…) //Setting my 5 properties which I have created for my
custom controller service
Runner.enablecontrolerservice(serviceimpl)
// will do assert }
Here in enabling controller service gives null pointer exception in the
custom validate method , its excepting all the default Properties also need
to be declared like Accesskey, secret key etc(which is present in the
default Awscredentialprovidercontrolerservice class) in my
custom controller Awscredentialprovidercontrolerservice.
After declaring the default properties in my custom
Awscredentialprovidercontrolerservice unit test is working fine.But problem
is these parameters are appearing in the NiFi UI of my custom
Awscredentialprovidercontrolerservice.
So I have 2 option, either after declaring of the default properties, is
there any way to stop displaying I Nifi UI.
Or As its working fine in UI flow without setting default properties in
custom Awscredentialprovidercontrolerservice.so should I set something in
Unit test case to make it passed
Thanks & Regards
--
Sanjeet Kumar Rath,
mob- +91 8777577470
