[ https://issues.apache.org/jira/browse/METRON-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657159#comment-16657159 ]
ASF GitHub Bot commented on METRON-1829: ---------------------------------------- Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1239#discussion_r226725656 --- Diff: metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java --- @@ -118,12 +119,18 @@ public void commit(BulkWriterResponse response) { public void error(String sensorType, Throwable e, Iterable<Tuple> tuples, MessageGetStrategy messageGetStrategy) { LOG.error(format("Failing %d tuple(s); sensorType=%s", Iterables.size(tuples), sensorType), e); - MetronError error = new MetronError() - .withSensorType(Collections.singleton(sensorType)) - .withErrorType(Constants.ErrorType.INDEXING_ERROR) - .withThrowable(e); - tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t))); - handleError(tuples, error); + tuples.forEach(t -> { + MetronError error = new MetronError() + .withSensorType(Collections.singleton(sensorType)) + .withErrorType(Constants.ErrorType.INDEXING_ERROR) + .withThrowable(e) + .addRawMessage(messageGetStrategy.get(t)); --- End diff -- Thinking more about this some more... I feel that what I have described in METRON-1832 is a pre-existing condition. I don't think this PR makes that pre-existing condition any worse. This PR is a solid, necessary fix. I am in favor of merging this and I will open a discuss thread on a resolution for METRON-1832. > Large Error Message Causes Slow Search Performance > -------------------------------------------------- > > Key: METRON-1829 > URL: https://issues.apache.org/jira/browse/METRON-1829 > Project: Metron > Issue Type: Bug > Reporter: Ryan Merriman > Priority: Major > > Errors that occur during batch writes in the index topologies (batch and RA) > are written to Elasticsearch as a single, large error message, with a field > for each failed message. For example, if the batch size is 5000, a single > error message will be created with 5000 fields `raw_message_0`, > `raw_message_1`, .., `raw_message_4999`. With such large messages, searching > the error index in Elasticsearch is excessively slow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)