Hi Michal, Regarding the session.commit... most processors extend from AbstractProcessor which takes care of the commit [1]. If you extend from AbstractSessionFactoryProcessor, or directly implement the Processor interface with no parent class, then it is up to you to call commit.
For the specific elastic search situation, I'm not sure if penalizing makes sense in all those cases. The catch block for IdentifierNotFoundException has a penalize, but this doesn't seem like something that would resolve itself if tried again, similar for MalformedRecordException in the catch block below. [1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java#L28 On Tue, May 7, 2019 at 6:10 AM Michal Klempa <[email protected]> wrote: > > Hi, > I came across an issue, which looks similar to > https://stackoverflow.com/questions/51891413/nifi-processor-task-count-is-very-high-what-must-be-the-reason > Many tasks/time on downstream processor. > > In my case, we are using PutElasticsearchHttpRecord and the issue is, > when RecordReader cannot parse input records (e.g. invalid JSON). > We fall into catch block: > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L455 > > Where the FlowFile is penalized and transferred to FAILURE. > > Now the questions: > 1. is penalization meant for retry-able errors? According to > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#penalization-vs-yielding > it seems like the answer is Yes, but the example given is also with > FAILURE relationship (not RETRY relationship) > 2. I can't find any session.commit in processor code. Is it OK? May it > be, that the problem happens when session is not committed? Does NiFi > any auto-commit then? > 3. is there any design document for new processor regarding Penalize + > RETRY/FAILURE relationship? > > Thanks. > Michal
