Created https://issues.apache.org/jira/browse/NIFI-3082
-----Original Message----- From: Matt Burgess [mailto:[email protected]] Sent: Monday, November 07, 2016 2:53 PM To: [email protected] Subject: Re: PutElasticsearchHttp error behaviour I agree, this should be caught and the flow file routed to failure. Do you mind filing a Jira for this? Thanks, Matt On Mon, Nov 7, 2016 at 2:40 PM, Gaspar, Carson <[email protected]> wrote: > Yes, I know why it failed, and how to fix it. What I don’t understand > is why it was retried forever, causing backpressure and stopping other > updates. > That’s really unacceptable. > > > > From: Pierre Villard [mailto:[email protected]] > Sent: Monday, November 07, 2016 1:15 PM > > > To: [email protected] > Subject: Re: PutElasticsearchHttp error behaviour > > > > Hi Gaspar, > > I believe this is because 4294967295 is a too big value that cannot be > contained in an integer. I am not a ES expert but you should look at > your applicationId type if you can change it. > > Hope this helps. > > > > 2016-11-07 19:09 GMT+01:00 Gaspar, Carson <[email protected]>: > > It stays in the queue on a field type error, e.g.: > [2016-11-07 12:37:17,263][DEBUG][action.bulk ] [Cybele] > [estreamer_index_2016.11.07][0] failed to execute bulk item (index) > index > {[estreamer_index_2016.11.07][estreamer_doc][AVg_3ZmnIxO_esWdRpuM], source > ...data... > MapperParsingException[failed to parse [applicationId]]; nested: > NumberFormatException[For input string: "4294967295"]; > at > org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:329) > at > org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:311) > at > org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:438) > at > org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:264) > at > org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:124) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309) > at > org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:529) > at > org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:506) > at > org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:214) > at > org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:223) > at > org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:327) > at > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:120) > at > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68) > at > org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:648) > at > org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) > at > org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279) > at > org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271) > at > org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77) > at > org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376) > at > org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NumberFormatException: For input string: "4294967295" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:583) > at java.lang.Integer.parseInt(Integer.java:615) > at > org.elasticsearch.common.xcontent.support.AbstractXContentParser.intValue(AbstractXContentParser.java:127) > at > org.elasticsearch.index.mapper.core.IntegerFieldMapper.innerParseCreateField(IntegerFieldMapper.java:293) > at > org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:241) > at > org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:321) > ... 22 more > > > -----Original Message----- > From: Koji Kawamura [mailto:[email protected]] > Sent: Monday, November 07, 2016 12:10 AM > To: [email protected] > Subject: Re: PutElasticsearchHttp error behaviour > > Hello Gaspar, > > I looked at the PutElasticsearchHttp code, it distinguishes following > two types of error: > > 1. Error occurs when the processor tries to connect Elasticsearch, but > it couldn't. E.g. the specified Elasticsearch URL is not correct and > java.net.ConnectException is thrown. > When this happens, the processor throws ProcessException, and NiFi > framework keeps the FlowFile in the queue, so that it can be retried later. > This indicates that DataFlowManager(user) has to update the processor > configuration to make it work properly. > > 2. The processor successfully sends a request to Elasticsearch, but > Elasticsearch returns error response. This happens with operations > such as updating with a document Id that doesn't exist in the target index. > When this happens, the processor routes the incoming FlowFile which > caused the error, to failure relationship. > > I think the error you encountered the former type of error above (#1). > If the error was returned from Elasticsearch, it should be routed to > failure relationship as you suspected. Would you please let us know > about the specific error you got? A StackTrace would be helpful. > > BTW, PutElasticsearchHttp also has REL_RETRY relationship, but it > seems it's not used currently. > > Thanks! > Koji > > On Thu, Nov 3, 2016 at 5:47 AM, Gaspar, Carson <[email protected]> wrote: >> When PutElasticsearchHttp gets an error from Elasticsearch, it >> _appears_ to never dequeue the problem flow file and just try it over >> and over again. It certainly doesn’t route the problem flow file to >> the retry or failure queues. This seems wrong to me – am I missing >> something? >> >> Nifi is from the Hortonworks >> nifi_2_0_0_0_579-1.0.0.2.0.0.0-579.el6.x86_64 >> RPM. >> >> > >
