Re: Retry on HTTP processor preserving order of flow file
Hi Eduardo, Thanks for the suggestions, I finally ended up testing it in a different way. 1. Having the failure/retry relationship connected back to invoke http processor with a backpressure value as 1. 2. So on any failure all messages will get queued up before the invokehttp processor. 3. When we fix the issue with failing endpoints, the flow miraculously keeps it's Order. 4. I tested the API by failing it for more than 2 hours, thay I want to make sure if it fails for days. It should keep the order. 5. I am also planning for sending a mail if the queue is having a single flowfile Do you see any issues with this method ? On Sun, 5 Jul 2020 at 19:57, Eduardo Fontes wrote: > Hi Midhu, > > I can't think in other solution that guarantees the order using only NiFi. > Even if you put a funnel before the InvokeHTTP, receiving the main input > and the retries with prioritizers in the queue, the order could not be > guaranteed.Using other tools, you can do it by, for exemple, using an > external queue (like Kafka), you could consume from it every time the > InvokeHTTP succeed or failed with retries. > > Eduardo Fontes > Data Engineer > > > On Fri, Jul 3, 2020 at 12:53 AM Midhun Mohan > wrote: > > > Hi Eduardo, > > > > It seems success on 100% success scenario, but 50 % failure scenario it > > seems to be mixing up the sequence.. So do you have any other suggestion > > than using wait/notify . > > > > On Wed, 1 Jul 2020 at 05:53, Eduardo Fontes > > wrote: > > > > > Hi Mithun, > > > > > > First of all, did you verify if the sequence is ok in a 100% success > > > scenario? If not, you could use the prioritizers to correct this. > > > If you need to confirm every HTTP call (including retries) before > process > > > the next in the queue, you could use a Wait/Notify. But be advised that > > > this approach will slow down the throughput of your flow. > > > > > > Eduardo Fontes > > > Data Engineer > > > > > > > > > On Tue, Jun 30, 2020 at 8:58 AM Midhun Mohan > > > wrote: > > > > > > > Hi all, > > > > > > > > I was trying to figure out a way to do a retry on http processor on > > > > following scenario > > > > > > > > 1. The flow file should preserve the order , since the order of data > is > > > > critical for the API > > > > > > > > > > > > I am using the retry processor to do the 3 times default retry with > 20 > > > sec > > > > delay. All the In and Out queue is first in first out priority. > > > > > > > > > > > > But on my realtime load test it seems the Order is not preserved. I > am > > > > getting a mix of responses. > > > > > > > > Please help me to figure it out > > > > > > > > > > > > Thanks in advance > > > > > > > > -- > > > > > > > > > > > > Regards, > > > > Midhun Mohan > > > > > > > > > > > > > -- > > > > > > Regards, > > Midhun Mohan > > > -- Regards, Midhun Mohan
[RESULT] [VOTE] Release Apache NiFi Registry 0.7.0
Apache NiFi Community, I am pleased to announce that the nifi-registry-0.7.0 release of Apache NiFi Registry passes with: 10 +1 (binding) votes 5 +1 (non-binding) votes 0 0 votes 0 -1 votes Thanks to all who helped make this release possible. Here is the PMC vote thread: https://mail-archives.apache.org/mod_mbox/nifi-dev/202007.mbox/%3CCALo_M19OZcbZOG7VeEcVZ%2BMtuu6ehoaEcL%3D1868ZAHReiLi16A%40mail.gmail.com%3E On Mon, Jul 20, 2020 at 9:10 AM Bryan Bende wrote: > +1 (binding) > > > On Sat, Jul 18, 2020 at 5:16 PM Arpad Boda > wrote: > >> +1 (binding) >> Verified build on Java11, signatures, hashes. Lic and notice looks good. >> Started, was able to create buckets and save flow. >> >> On Sat, Jul 18, 2020 at 3:19 PM Tony Kurc wrote: >> >> > +1 (binding) >> > >> > Built on an ubuntu 16.04 x64 w/openjdk 11 without issues. Verified >> > signatures and hashes. Saw no issues with LICENSE or NOTICE. >> > >> > Tony >> > >> > On Fri, Jul 17, 2020 at 10:10 PM Marc Parisi >> wrote: >> > >> > > +1 binding >> > > >> > > Verified sigs and hashes >> > > >> > > Built and run on Linux LXC (Java 11) w/ Secured Apache NiFi built and >> run >> > > in the same container. All looked good >> > > >> > > Perused L&N and it looked good. >> > > >> > > Thanks, >> > > Marc >> > > >> > > On Fri, Jul 17, 2020 at 5:48 PM Peter Turcsanyi > > >> > > wrote: >> > > >> > > > +1 (non-binding) >> > > > >> > > > - verified signatures and checksums >> > > > - built with Java 8 (empty local maven repo, OS: Ubuntu 18) >> > > > - run NiFi Registry and connected to it from NiFi 1.11.4 (unsecure), >> > > > performed some tests >> > > > - download the convenience binary and tested a few things on it too >> > > > >> > > > Thanks, >> > > > Peter >> > > > >> > > > On Fri, Jul 17, 2020 at 7:20 PM Marton Szasz >> > wrote: >> > > > >> > > > > +1 (non-binding) >> > > > > >> > > > > Followed the verification guide, migrated a test deployment using >> > > > > GitFlowPersistenceProvider from 0.6.0, then tested MiNiFi C++ C2 >> flow >> > > > > update using the new registry version. >> > > > > >> > > > > Thanks, >> > > > > Marton >> > > > > >> > > > > On Fri, 17 Jul 2020 at 18:36, Mark Payne >> > wrote: >> > > > > >> > > > > > +1 (binding) >> > > > > > >> > > > > > Verified full build with contrib check, deployed and performed >> some >> > > > > tests, >> > > > > > verifying that the FlowFiles concurrency and outbound policy was >> > > > properly >> > > > > > stored and retrieved. >> > > > > > >> > > > > > Thanks >> > > > > > -Mark >> > > > > > >> > > > > > >> > > > > > > On Jul 17, 2020, at 11:55 AM, Pierre Villard < >> > > > > > pierre.villard...@gmail.com> wrote: >> > > > > > > >> > > > > > > +1 (binding) >> > > > > > > >> > > > > > > * went through the usual release verification steps >> (signatures, >> > > > build, >> > > > > > > contrib check, licenses) >> > > > > > > * deployed a local NR 0.7.0 instance with a single local NiFi >> > > > > > > 1.12.0-SNAPSHOT (non secure) >> > > > > > > >> > > > > > > Thanks Bryan for taking care of this! >> > > > > > > >> > > > > > > Le ven. 17 juil. 2020 à 17:46, Matt Burgess < >> > mattyb...@apache.org> >> > > a >> > > > > > écrit : >> > > > > > > >> > > > > > >> +1 (binding), verified hashes and commit ID, tested a few >> things >> > > on >> > > > an >> > > > > > >> unsecured instance. Thanks for RM'ing Bryan! >> > > > > > >> >> > > > > > >> On Wed, Jul 15, 2020 at 11:32 AM Bryan Bende < >> bbe...@apache.org >> > > >> > > > > wrote: >> > > > > > >>> >> > > > > > >>> Hello, >> > > > > > >>> >> > > > > > >>> I am pleased to be calling this vote for the source release >> of >> > > > Apache >> > > > > > >> NiFi >> > > > > > >>> Registry 0.7.0. >> > > > > > >>> >> > > > > > >>> The source zip, including signatures, digests, etc. can be >> > found >> > > > at: >> > > > > > >>> >> > > > > >> > https://repository.apache.org/content/repositories/orgapachenifi-1161 >> > > > > > >>> >> > > > > > >>> The source being voted upon and the convenience binaries >> can be >> > > > found >> > > > > > at: >> > > > > > >>> >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.7.0/ >> > > > > > >>> >> > > > > > >>> A helpful reminder on how the release candidate verification >> > > > process >> > > > > > >> works: >> > > > > > >>> >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate >> > > > > > >>> >> > > > > > >>> The Git tag is nifi-registry-0.7.0-RC1 >> > > > > > >>> The Git commit ID is >> c8f26039712354b94c4d458b7ea491316c6facac >> > > > > > >>> >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://gitbox.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=c8f26039712354b94c4d458b7ea491316c6facac >> > > > > > >>> >> > > > > > >>> Checksums of nifi-registry-0.7.0-source-release.zip: >> > > > > > >>> SH
Re: [VOTE] Release Apache NiFi Registry 0.7.0
+1 (binding) On Sat, Jul 18, 2020 at 5:16 PM Arpad Boda wrote: > +1 (binding) > Verified build on Java11, signatures, hashes. Lic and notice looks good. > Started, was able to create buckets and save flow. > > On Sat, Jul 18, 2020 at 3:19 PM Tony Kurc wrote: > > > +1 (binding) > > > > Built on an ubuntu 16.04 x64 w/openjdk 11 without issues. Verified > > signatures and hashes. Saw no issues with LICENSE or NOTICE. > > > > Tony > > > > On Fri, Jul 17, 2020 at 10:10 PM Marc Parisi > wrote: > > > > > +1 binding > > > > > > Verified sigs and hashes > > > > > > Built and run on Linux LXC (Java 11) w/ Secured Apache NiFi built and > run > > > in the same container. All looked good > > > > > > Perused L&N and it looked good. > > > > > > Thanks, > > > Marc > > > > > > On Fri, Jul 17, 2020 at 5:48 PM Peter Turcsanyi > > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > - verified signatures and checksums > > > > - built with Java 8 (empty local maven repo, OS: Ubuntu 18) > > > > - run NiFi Registry and connected to it from NiFi 1.11.4 (unsecure), > > > > performed some tests > > > > - download the convenience binary and tested a few things on it too > > > > > > > > Thanks, > > > > Peter > > > > > > > > On Fri, Jul 17, 2020 at 7:20 PM Marton Szasz > > wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > Followed the verification guide, migrated a test deployment using > > > > > GitFlowPersistenceProvider from 0.6.0, then tested MiNiFi C++ C2 > flow > > > > > update using the new registry version. > > > > > > > > > > Thanks, > > > > > Marton > > > > > > > > > > On Fri, 17 Jul 2020 at 18:36, Mark Payne > > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Verified full build with contrib check, deployed and performed > some > > > > > tests, > > > > > > verifying that the FlowFiles concurrency and outbound policy was > > > > properly > > > > > > stored and retrieved. > > > > > > > > > > > > Thanks > > > > > > -Mark > > > > > > > > > > > > > > > > > > > On Jul 17, 2020, at 11:55 AM, Pierre Villard < > > > > > > pierre.villard...@gmail.com> wrote: > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > * went through the usual release verification steps > (signatures, > > > > build, > > > > > > > contrib check, licenses) > > > > > > > * deployed a local NR 0.7.0 instance with a single local NiFi > > > > > > > 1.12.0-SNAPSHOT (non secure) > > > > > > > > > > > > > > Thanks Bryan for taking care of this! > > > > > > > > > > > > > > Le ven. 17 juil. 2020 à 17:46, Matt Burgess < > > mattyb...@apache.org> > > > a > > > > > > écrit : > > > > > > > > > > > > > >> +1 (binding), verified hashes and commit ID, tested a few > things > > > on > > > > an > > > > > > >> unsecured instance. Thanks for RM'ing Bryan! > > > > > > >> > > > > > > >> On Wed, Jul 15, 2020 at 11:32 AM Bryan Bende < > bbe...@apache.org > > > > > > > > wrote: > > > > > > >>> > > > > > > >>> Hello, > > > > > > >>> > > > > > > >>> I am pleased to be calling this vote for the source release > of > > > > Apache > > > > > > >> NiFi > > > > > > >>> Registry 0.7.0. > > > > > > >>> > > > > > > >>> The source zip, including signatures, digests, etc. can be > > found > > > > at: > > > > > > >>> > > > > > > > https://repository.apache.org/content/repositories/orgapachenifi-1161 > > > > > > >>> > > > > > > >>> The source being voted upon and the convenience binaries can > be > > > > found > > > > > > at: > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.7.0/ > > > > > > >>> > > > > > > >>> A helpful reminder on how the release candidate verification > > > > process > > > > > > >> works: > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > > > >>> > > > > > > >>> The Git tag is nifi-registry-0.7.0-RC1 > > > > > > >>> The Git commit ID is c8f26039712354b94c4d458b7ea491316c6facac > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://gitbox.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=c8f26039712354b94c4d458b7ea491316c6facac > > > > > > >>> > > > > > > >>> Checksums of nifi-registry-0.7.0-source-release.zip: > > > > > > >>> SHA256: > > > > > > acc6b21444d229d78b8a604750231e1c7e80495c9cccfb59ec043ebe952fd2a8 > > > > > > >>> SHA512: > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > 428839e0c861095547e328fcfda42cb3f2a87c0b03f7205d6878f8b7267975fe2ad051d4b5f980cac8f47a249c8ee21686c8d909f275454e25d8a654099bd683 > > > > > > >>> > > > > > > >>> Release artifacts are signed with the following key: > > > > > > >>> https://people.apache.org/keys/committer/bbende.asc > > > > > > >>> > > > > > > >>> KEYS file available here: > > > > > > >>> https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > >>> > > > > > > >>> 19 issues were closed/resolved for t
