The Provenance feature can sometimes present a very wide range of results and I've found it is sometimes difficult to see my content and attributes for the flowfile of interest. Queueing up the results after a processor limits the review set to just those results. I've found it to be easier to get at what I want to see during test and development. I will be the first to own up to my own ignorance though Mark <lol>, and will try harder to use the provenance features instead.
Our flow is not actually that high. This threshold has been reached following several a few days of processing. and this is perhaps the biggest reason I asked about Request Expiration: I was operating under the impression that Request Expiration would force the removal of all those older than 10 minutes, regardless of connection state or anything. Could it be that these "split paths" / queued results I created put some sort of hold on the flowfile that prevents the removal from the context map, regardless of the fact that I have indeed Http Responded? Thank you again for your insights. I am learning a great deal about what this is doing under the hood, and hope to learn why, exactly, Request Expiration isn't working as I anticipated. I must be doing something that negatively impacts its purpose. Jim On Wed, Feb 22, 2017 at 10:36 AM, Mark Payne <marka...@hotmail.com> wrote: > Jim, > > re: two Success paths - Yes, you should send only one of them to the > HandleHttpResponse. I'm curious though - why use > a disabled processor and queue data up instead of using the Data > Provenance feature? > > Yes, StandardHttpContextMap should be removing any entries on its own that > exceed the timeout. How many requests per > second are you seeing? I am assuming that you are receiving a pretty high > rate if it is at the point of containing over 10K entries > with a 10 minute timeout. If you are not seeing that many requests, then > there may be something else going on there. > > Thanks > -Mark > > > > On Feb 22, 2017, at 10:31 AM, James McMahon <jsmcmah...@gmail.com> wrote: > > Additional questions about this. Immediately following my > HandleHttpRequest processor, I have an ExecuteScript processor that then > sends flowfile copies out two Success paths. one path eventually culminates > in a HandleHttpResponse that has the aforementioned auto-termintaion of > Success and Failure results. The second path is to a MonitorActivity > processor that is disabled, to permit me to queue up and review incoming > flowfile results after ExecuteScript during dev and test. Does that second > path also have to send a response? Isn't it enough that the ContextMap is > cleared by the response from the first path? > > Second question: how does this ever happen? Doesn't the Request Expiration > I set on the StandradHttpContextMap force the obliteration of all entries > that age beyond that point? > > Jim > > On Wed, Feb 22, 2017 at 10:13 AM, James McMahon <jsmcmah...@gmail.com> > wrote: > >> I may well have that Mark. I have a number of paths where I have >> HandleHttpResponse that auto terminates Failures. That would cause such a >> problem, wouldn't it? >> >> How do people handle this situation: app does a POST, and so we handle >> the request. App closes or timesout for whatever the reason may be. The >> HanldeHttpResponse is unable to reply. Should those not be auto terminated? >> >> In a situation like this then Mark, are these the steps to recover? >> 1. HanldeHttpResponse at end of all paths >> 2. do not autoterminate failure conditions >> 3. DELETE the StandardHttpContextMap (to clear the log jam) >> 4. Recreate it fresh, which I presume creates it empty (I hope) >> >> What else must I do to recover? And how do I properly handle those >> "broken connection" situations? >> >> On Wed, Feb 22, 2017 at 10:06 AM, Mark Payne <marka...@hotmail.com> >> wrote: >> >>> Jim, >>> >>> You likely have a path through your flow where you are receiving an HTTP >>> Request via HandleHttpRequest >>> but you never respond via a HandleHttpResponse. When using these >>> processors, it's important that every >>> incoming FlowFile go to a HandleHttpResponse processor. Do you have some >>> path in your flow where you >>> are not responding to the request? >>> >>> Thanks >>> -Mark >>> >>> >>> > On Feb 22, 2017, at 9:58 AM, James McMahon <jsmcmah...@gmail.com> >>> wrote: >>> > >>> > I am getting the following errors when my users attempt to use curl or >>> python to post to my HandleHttpRequest processor (cannot export actual >>> messages, must select pieces and retype here): >>> > WARNING >>> > Received request from [IP address is here] but could not process it >>> because too many requests are already outstaning; responding with >>> SERVICE_UNAVAILABLE >>> > ERROR >>> > ...claim=StandardContentClaim.... >>> > transfer relationship not specified >>> > >>> > None of my apps can post to NiFi. >>> > >>> > I have a StandradSSLContextService and a standradHttpContextMap, both >>> of which are enabled. I suspect I may have inadvertently caused this >>> problem by setting my ContextMap parameters badly. Here are those params: >>> > Maximum Outstanding Requests: 10000 >>> > Request Expiration 10 min >>> > >>> > I've looked across my workflow and no flowfiles are queued up. So my >>> expectation is that there should be ample space in my ContextMap. But these >>> errors indicate otherwise. How do I fix this? >>> > Thanks very much in advance for your help. >>> > Jim >>> >>> >> > >
