Re: Re: NiFi backpressure not work
Lei Please do not post to both users and dev list. Dropping dev. Yes these scenarios are based on limitations of a given processor implementation or the nature of a given protocol/mechanism. NiFi enforces back pressure by slowing/stopping scheduling a component and making fact of back pressure available to components. I dont know about this particular processor but it likely needs to be updated so the max amount of data it will return in any one cycle is small enough or controllable enough to avoud what youre seeing. Thanks Joe On Wed, Oct 16, 2019 at 3:06 AM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > CaptureChangeMySQLSelf is from CaptureChangeMySQL and I fix a small bug > about binlog parsing. But i am sure there's no change that will affect the > backpressure mechanism. > > Actually I don't quite understand it. > It is related with the implementation of the processor? In order to use > the backpressure mechanism, the processor must do something? > In my case, what it will happen if the queue continue to grow(for example > to 100G)? Something bad will happen? > > Thanks, > Lei > > > -- > wangl...@geekplus.com.cn > > > *From:* josef.zahn...@swisscom.com > *Date:* 2019-10-16 14:17 > *To:* users@nifi.apache.org; d...@nifi.apache.org > *Subject:* Re: NiFi backpressure not work > > Hi Lei > > > > I assume it’s the same as for the ListSFTP. As I don’t know the > CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. > There with one single execution of the processor you get as much flowfiles > as you have files on your disk. If it finds 1000 it sends out 1000 > flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The > processor can’t hold back some flows and knows nothing about your queue, so > it sends out everything in one “batch” per execution – even if it is more > than the queue size. Hope this explains the behavior a bit. > > > > Cheers > > > > > > *From: *"wangl...@geekplus.com.cn" > *Reply to: *"users@nifi.apache.org" > *Date: *Wednesday, 16 October 2019 at 08:10 > *To: *users , dev > *Subject: *NiFi backpressure not work > > > > As the picture shows. > > I have set the backpressure object threshold to 1, but picture shows > there's more than 1,000,000. Why this happens? > > > > > > > > Thanks, Lei > -- > > wangl...@geekplus.com.cn > >
Re: Re: NiFi backpressure not work
Thanks very much. I got it. wangl...@geekplus.com.cn From: Joe Witt Date: 2019-10-16 17:24 To: users@nifi.apache.org Subject: Re: Re: NiFi backpressure not work Lei Please do not post to both users and dev list. Dropping dev. Yes these scenarios are based on limitations of a given processor implementation or the nature of a given protocol/mechanism. NiFi enforces back pressure by slowing/stopping scheduling a component and making fact of back pressure available to components. I dont know about this particular processor but it likely needs to be updated so the max amount of data it will return in any one cycle is small enough or controllable enough to avoud what youre seeing. Thanks Joe On Wed, Oct 16, 2019 at 3:06 AM wangl...@geekplus.com.cn wrote: CaptureChangeMySQLSelf is from CaptureChangeMySQL and I fix a small bug about binlog parsing. But i am sure there's no change that will affect the backpressure mechanism. Actually I don't quite understand it. It is related with the implementation of the processor? In order to use the backpressure mechanism, the processor must do something? In my case, what it will happen if the queue continue to grow(for example to 100G)? Something bad will happen? Thanks, Lei wangl...@geekplus.com.cn From: josef.zahn...@swisscom.com Date: 2019-10-16 14:17 To: users@nifi.apache.org; d...@nifi.apache.org Subject: Re: NiFi backpressure not work Hi Lei I assume it’s the same as for the ListSFTP. As I don’t know the CaptureChangeMySQLSelf processor I try to explain it for the ListSFTP. There with one single execution of the processor you get as much flowfiles as you have files on your disk. If it finds 1000 it sends out 1000 flowfiles. If it finds 1’000’000 it sends out 1’000’000 flowfiles. The processor can’t hold back some flows and knows nothing about your queue, so it sends out everything in one “batch” per execution – even if it is more than the queue size. Hope this explains the behavior a bit. Cheers From: "wangl...@geekplus.com.cn" Reply to: "users@nifi.apache.org" Date: Wednesday, 16 October 2019 at 08:10 To: users , dev Subject: NiFi backpressure not work As the picture shows. I have set the backpressure object threshold to 1, but picture shows there's more than 1,000,000. Why this happens? Thanks, Lei wangl...@geekplus.com.cn
Can balance by attribute guarantee the order of the FlowFile？
Since data will be balanced to different node, the order after balanced will be the same as before? Thanks, Lei wangl...@geekplus.com.cn
Re: Can balance by attribute guarantee the order of the FlowFile？
Lei The order won't necessarily be the same. You'd want EnforceOrder to follow the load balanced connection most likely. It is important to keep in mind the queues are basically insertion order and the system is inherently multi-threaded so the data can be shuffled in such cases. You can carefully design your flow and use mechanisms like EnforceOrder to help. Thanks On Wed, Oct 16, 2019 at 8:25 AM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > > Since data will be balanced to different node, the order after balanced > will be the same as before? > > Thanks, > Lei > > -- > wangl...@geekplus.com.cn >
What UDFs are supported by QueryRecord?
Hi Folks,
I'd like to use a SQL statement in QueryRecords like this uses the extract
the last 10 chars of field and compare it to today, similar to the line
below:
SELECT substring( '/index', 10, -1) as expired FROM FLOWFILE WHERE expired
<= '${now():format("-MM-dd")}'
This statement is not work and I can't find a list of QueryRecord supported
UDF or if (nd how) I can use RecordPath functions.
Is there any documentation where I can see the available UDFs?
Regards,
Re: What UDFs are supported by QueryRecord?
Ok, figured out a way to do it. I noticed that QueryRecord uses Apache
Calcite so I tried some of Calcite functions until I got into this
statement that seems to work:
SELECT index FROM FLOWFILE WHERE CAST( SUBSTRING(index FROM
CHAR_LENGTH(index)-9) AS DATE) <= CURRENT_DATE
Anyhow, I would still like to know if (and how) I could use either NiFi
expression or RecordPath functions in a QueryRecord statement.
Thanks in advance,
Em qua, 16 de out de 2019 às 14:30, Eric Chaves escreveu:
> Hi Folks,
>
> I'd like to use a SQL statement in QueryRecords like this uses the extract
> the last 10 chars of field and compare it to today, similar to the line
> below:
>
> SELECT substring( '/index', 10, -1) as expired FROM FLOWFILE WHERE expired
> <= '${now():format("-MM-dd")}'
>
> This statement is not work and I can't find a list of QueryRecord
> supported UDF or if (nd how) I can use RecordPath functions.
>
> Is there any documentation where I can see the available UDFs?
>
> Regards,
>
>
Re: High CPU consumption
Thank you for that tip, Andy! This is actually a bug I've wanted to track down and fix but it's in parts of the NiFi codebase I'm really not familiar with and wasn't sure how to start ... if you can connect me with someone who knows that area (scheduling and clusters, mainly) I would be happy to see if it's something I can patch! On Tue, Oct 15, 2019 at 2:43 PM Andy LoPresto wrote: > Evan, > > Thanks for sharing that diagnosing technique. While ideally we would have > other controls to prevent excess CPU usage, this seems like a useful tool > which could be automated using NiPyAPI [1] to perform a “bisect” command. > I’ve used this for git commit searching as well as side-effect unit test > identification. > > [1] https://nipyapi.readthedocs.io/en/latest/readme.html > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Oct 15, 2019, at 1:40 PM, Evan Reynolds wrote: > > I have found two issues that can cause high CPU when idle (high being > about 200% CPU when idle.) I haven’t verified these with 1.9.2, but it > doesn’t hurt to tell you. > > If you are using ports, make sure each input port is connected. If you > have a one that isn’t connected, that can bring your CPU to 200% and stay > there. > > If you have any processors that are set to run on the primary node of a > cluster, that can also take your CPU to 200%. I know RouteOnAttribute will > do that (again, haven’t tested 1.9.2, but it was a problem for me for a > bit!) The fix – either don’t run it on the primary node, or else set the > run schedule to 5 seconds or something instead of 0. > > To find out if this is the case – well, this is what I did. It worked, and > wasn’t that hard, though isn’t exactly elegant. > > Back up your flowfile (flow.xml.gz) > Stop all your processors and see what your CPU does > Start half of them and watch your CPU – basically do a binary search. If > your CPU stays reasonable, then whatever group you started is fine. If not, > then start stopping things until your CPU becomes reasonable. > Eventually you’ll find a processor that spikes your CPU when you start it > and then you can figure out what to do about that processor. Record which > processor it is and how you altered it to bring CPU down. > Repeat, as there may be more than one processor spiking CPU. > Stop NiFi and restore your flowfile by copying it back in place – since > you were running around stopping things, this just makes sure everything is > correctly back to where it should be > > Then use the list of processors and fixes to make changes. > > --- > > Evan Reynolds > e...@usermind.com > > > *From: *Jon Logan > *Reply-To: *"users@nifi.apache.org" > *Date: *Sunday, October 13, 2019 at 6:12 PM > *To: *"users@nifi.apache.org" > *Subject: *Re: High CPU consumption > > That isn't logback, that lists all jars on your classpath, the first of > which happens to be logback. > > If you send a SIGKILL3 (you can send it in HTOP) it will dump the thread > stacks to stdout (probably the bootstrap log)...but that's just for one > instant, and may or may not be helpful. > > On Sun, Oct 13, 2019 at 8:58 PM Luis Carmona > wrote: > > hi Aldrin, > > thanks a lot, by now I'm trying to learn how to make the profiling you > mentioned. > > One more question: Is it normal that the father java process has very low > consumption while the child process related to logback jar is the one that > is eating up all the CPU ? > Please take a look at the attached image. > > Thanks, > > LC > > -- > *From: *"Aldrin Piri" > *To: *"users" > *Sent: *Sunday, October 13, 2019 9:30:47 PM > *Subject: *Re: High CPU consumption > > Luis, please feel free to give us some information on your flow so we can > help you track down problematic areas as well. > > On Sun, Oct 13, 2019 at 3:56 PM Jon Logan wrote: > > You should put a profiler on it to be sure. > Just because your processors aren't processing data doesn't mean they are > idle though -- many have to poll for new data, especially sources -- ex. > connecting to Kafka, etc, will itself consume some CPU. > > But again, you should attach a profiler before participating in a wild > goose chase of performance issues. > > On Sun, Oct 13, 2019 at 12:20 PM Luis Carmona > wrote: > > HI, > > I've struggling to reduce my nifi installation CPU consumption. Even in > idle state, all processors running but no data flowing, it is beyond 60% > CPU consumption, with peaks of 200%. > > What I've done so far > - Read and followed every instruction/post about tuning NIFI I've found > googling. > - Verify scheduling is 1s for most consuming processors: http processors, > wait/notify, jolt, etc. > - Scratch my head... > > But nothing seems to have a major effect on the issue. > > Can anyone give me some precise directions or tips about how to solve this > please ? > Is
Re: High CPU consumption
Hi Luis, Can you describe the part of the flow that turned out to be a problem a little more? Was it a port on the root canvas used for s2s that was then connected into a process group where everything inside was disabled? And what did you do to solve the problem, did you stop the port? Thanks, Bryan On Wed, Oct 16, 2019 at 3:15 PM Evan Reynolds wrote: > > Thank you for that tip, Andy! > > This is actually a bug I've wanted to track down and fix but it's in parts of > the NiFi codebase I'm really not familiar with and wasn't sure how to start > ... if you can connect me with someone who knows that area (scheduling and > clusters, mainly) I would be happy to see if it's something I can patch! > > On Tue, Oct 15, 2019 at 2:43 PM Andy LoPresto wrote: >> >> Evan, >> >> Thanks for sharing that diagnosing technique. While ideally we would have >> other controls to prevent excess CPU usage, this seems like a useful tool >> which could be automated using NiPyAPI [1] to perform a “bisect” command. >> I’ve used this for git commit searching as well as side-effect unit test >> identification. >> >> [1] https://nipyapi.readthedocs.io/en/latest/readme.html >> >> >> Andy LoPresto >> alopre...@apache.org >> alopresto.apa...@gmail.com >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Oct 15, 2019, at 1:40 PM, Evan Reynolds wrote: >> >> I have found two issues that can cause high CPU when idle (high being about >> 200% CPU when idle.) I haven’t verified these with 1.9.2, but it doesn’t >> hurt to tell you. >> >> If you are using ports, make sure each input port is connected. If you have >> a one that isn’t connected, that can bring your CPU to 200% and stay there. >> >> If you have any processors that are set to run on the primary node of a >> cluster, that can also take your CPU to 200%. I know RouteOnAttribute will >> do that (again, haven’t tested 1.9.2, but it was a problem for me for a >> bit!) The fix – either don’t run it on the primary node, or else set the run >> schedule to 5 seconds or something instead of 0. >> >> To find out if this is the case – well, this is what I did. It worked, and >> wasn’t that hard, though isn’t exactly elegant. >> >> Back up your flowfile (flow.xml.gz) >> Stop all your processors and see what your CPU does >> Start half of them and watch your CPU – basically do a binary search. If >> your CPU stays reasonable, then whatever group you started is fine. If not, >> then start stopping things until your CPU becomes reasonable. >> Eventually you’ll find a processor that spikes your CPU when you start it >> and then you can figure out what to do about that processor. Record which >> processor it is and how you altered it to bring CPU down. >> Repeat, as there may be more than one processor spiking CPU. >> Stop NiFi and restore your flowfile by copying it back in place – since you >> were running around stopping things, this just makes sure everything is >> correctly back to where it should be >> >> Then use the list of processors and fixes to make changes. >> >> --- >> >> Evan Reynolds >> e...@usermind.com >> >> >> From: Jon Logan >> Reply-To: "users@nifi.apache.org" >> Date: Sunday, October 13, 2019 at 6:12 PM >> To: "users@nifi.apache.org" >> Subject: Re: High CPU consumption >> >> That isn't logback, that lists all jars on your classpath, the first of >> which happens to be logback. >> >> If you send a SIGKILL3 (you can send it in HTOP) it will dump the thread >> stacks to stdout (probably the bootstrap log)...but that's just for one >> instant, and may or may not be helpful. >> >> On Sun, Oct 13, 2019 at 8:58 PM Luis Carmona wrote: >> >> hi Aldrin, >> >> thanks a lot, by now I'm trying to learn how to make the profiling you >> mentioned. >> >> One more question: Is it normal that the father java process has very low >> consumption while the child process related to logback jar is the one that >> is eating up all the CPU ? >> Please take a look at the attached image. >> >> Thanks, >> >> LC >> >> >> From: "Aldrin Piri" >> To: "users" >> Sent: Sunday, October 13, 2019 9:30:47 PM >> Subject: Re: High CPU consumption >> >> Luis, please feel free to give us some information on your flow so we can >> help you track down problematic areas as well. >> >> On Sun, Oct 13, 2019 at 3:56 PM Jon Logan wrote: >> >> You should put a profiler on it to be sure. >> Just because your processors aren't processing data doesn't mean they are >> idle though -- many have to poll for new data, especially sources -- ex. >> connecting to Kafka, etc, will itself consume some CPU. >> >> But again, you should attach a profiler before participating in a wild goose >> chase of performance issues. >> >> On Sun, Oct 13, 2019 at 12:20 PM Luis Carmona >> wrote: >> >> HI, >> >> I've struggling to reduce my nifi installation CPU consumption. Even in idle >
Re: High CPU consumption
Hi Bryan, lamentably I didn't keep a copy of the template, didn't think could be useful. But Im pretty sure to remember two escenarios. First one an input port in a Process Group which was straight connected to 2 flows (no condition in between), first connection to an active processor and the other connection was to a disabled processor, leading nowhere (was just something I tested and forgot to delete, but was disabled). Second one, was quite similar but with an output port, as it got out from a process group, had two connections one to an active processor and the other to a disabled processor. What I did was stop all the canvas, and then start one by one the Groups, Once I detected the high consuming group, started the processors inside one bye one, then I noticed it didn't have any further effect, so thought might be the port and the connections described above. Stopped the whole thing again, deleted the useless processor and connections and voilá the consumption problem was highly reduced. I'll try to reproduce the scenario, and I if I get it I will send you the template. Regards, LC - Original Message - From: "Bryan Bende" To: "users" Sent: Wednesday, October 16, 2019 4:32:06 PM Subject: Re: High CPU consumption Hi Luis, Can you describe the part of the flow that turned out to be a problem a little more? Was it a port on the root canvas used for s2s that was then connected into a process group where everything inside was disabled? And what did you do to solve the problem, did you stop the port? Thanks, Bryan On Wed, Oct 16, 2019 at 3:15 PM Evan Reynolds wrote: > > Thank you for that tip, Andy! > > This is actually a bug I've wanted to track down and fix but it's in parts of > the NiFi codebase I'm really not familiar with and wasn't sure how to start > ... if you can connect me with someone who knows that area (scheduling and > clusters, mainly) I would be happy to see if it's something I can patch! > > On Tue, Oct 15, 2019 at 2:43 PM Andy LoPresto wrote: >> >> Evan, >> >> Thanks for sharing that diagnosing technique. While ideally we would have >> other controls to prevent excess CPU usage, this seems like a useful tool >> which could be automated using NiPyAPI [1] to perform a “bisect” command. >> I’ve used this for git commit searching as well as side-effect unit test >> identification. >> >> [1] https://nipyapi.readthedocs.io/en/latest/readme.html >> >> >> Andy LoPresto >> alopre...@apache.org >> alopresto.apa...@gmail.com >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Oct 15, 2019, at 1:40 PM, Evan Reynolds wrote: >> >> I have found two issues that can cause high CPU when idle (high being about >> 200% CPU when idle.) I haven’t verified these with 1.9.2, but it doesn’t >> hurt to tell you. >> >> If you are using ports, make sure each input port is connected. If you have >> a one that isn’t connected, that can bring your CPU to 200% and stay there. >> >> If you have any processors that are set to run on the primary node of a >> cluster, that can also take your CPU to 200%. I know RouteOnAttribute will >> do that (again, haven’t tested 1.9.2, but it was a problem for me for a >> bit!) The fix – either don’t run it on the primary node, or else set the run >> schedule to 5 seconds or something instead of 0. >> >> To find out if this is the case – well, this is what I did. It worked, and >> wasn’t that hard, though isn’t exactly elegant. >> >> Back up your flowfile (flow.xml.gz) >> Stop all your processors and see what your CPU does >> Start half of them and watch your CPU – basically do a binary search. If >> your CPU stays reasonable, then whatever group you started is fine. If not, >> then start stopping things until your CPU becomes reasonable. >> Eventually you’ll find a processor that spikes your CPU when you start it >> and then you can figure out what to do about that processor. Record which >> processor it is and how you altered it to bring CPU down. >> Repeat, as there may be more than one processor spiking CPU. >> Stop NiFi and restore your flowfile by copying it back in place – since you >> were running around stopping things, this just makes sure everything is >> correctly back to where it should be >> >> Then use the list of processors and fixes to make changes. >> >> --- >> >> Evan Reynolds >> e...@usermind.com >> >> >> From: Jon Logan >> Reply-To: "users@nifi.apache.org" >> Date: Sunday, October 13, 2019 at 6:12 PM >> To: "users@nifi.apache.org" >> Subject: Re: High CPU consumption >> >> That isn't logback, that lists all jars on your classpath, the first of >> which happens to be logback. >> >> If you send a SIGKILL3 (you can send it in HTOP) it will dump the thread >> stacks to stdout (probably the bootstrap log)...but that's just for one >> instant, and may or may not be helpful. >> >> On Sun, Oct 13, 2019 at 8:58 PM Luis
Re: Re: Can balance by attribute guarantee the order of the FlowFile？
Hi Joe, In high data volume circumstance, seems it is impossible to keep order accross the whole cluster. Even if EnforceOrder is used, it can be only scheduled on primary node and becomes a performance bettleneck. I tried to set the concurrency to 1 from the begining and not load balance. But if the execution set to all nodes, there will be one thread on every node. There will be also order issues in the final. Is there any method that can let the processor run only on one specified node? Thanks, Lei wangl...@geekplus.com.cn Sender: Joe Witt Send Time: 2019-10-16 20:27 Receiver: users Subject: Re: Can balance by attribute guarantee the order of the FlowFile？ Lei The order won't necessarily be the same. You'd want EnforceOrder to follow the load balanced connection most likely. It is important to keep in mind the queues are basically insertion order and the system is inherently multi-threaded so the data can be shuffled in such cases. You can carefully design your flow and use mechanisms like EnforceOrder to help. Thanks On Wed, Oct 16, 2019 at 8:25 AM wangl...@geekplus.com.cn wrote: Since data will be balanced to different node, the order after balanced will be the same as before? Thanks, Lei wangl...@geekplus.com.cn
ElasticSearchClientServiceImpl not working for secured ElasticSearch
I have an Elastic Search cluster that is setup with SSL. It uses a self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow that has the PutElasticSearchHttp component. I have setup a SSLContextService for that component where I have specified a trust store that has the self-signed cert from ES. I specify an https endpoint to access Elastic Search and Im having no issues populating my Elastic Search instance using this flow. I have another flow where I want to do some lookups. So I have been using the LookupRecord processor. That one I have associated with an ElasticSearchClientServiceImpl which I have setup to point to the same SSLContextService as used above. I specified the same HTTPS Url (triple checked this). However, when I run this second Flow I am not able to verify the ES server's self-signed certificate. I check the nifi-app.log and it says: Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target I am a bit surprised that I am not able to verify the same server certificate in the two different flows. Completely stuck on this so if anyone have any pointers please let me know. Thanks, Peter
