We absolutely should be acking the dropped messages otherwise they'll be in a replay loop. Not acking is a flat-out bug IMO.
On Wed, Dec 19, 2018 at 2:37 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > When a message is filtered by the message filtering mechanism, we > explicitly drop the message (and presumably ack it in Storm), as explained > here - > > https://github.com/apache/metron/tree/master/metron-platform/metron-parsing#filtered > . > When using the REGEX_SELECT field transformation (see here - > > https://github.com/apache/metron/tree/master/metron-platform/metron-parsing#fieldtransformation-configuration > ) > with the kafka.topicField option for parser-chaining, it's unclear to me > whether we expect the same behavior (drop message, ack it). The > interpretation I get from this example in the parser-chaining doc > > https://github.com/apache/metron/tree/master/use-cases/parser_chaining#the-pix_syslog_router-parser > suggests to me that the approach we take for messages with message > filtering is the correct one, however in testing an example with dropped > messages, we appear not to ack those dropped messages. > > Before I go creating a fix I thought it best to summarize and confirm my > expectations on this functionality. Messages from a REGEX_SELECT that don't > match a pattern, and therefore don't get a value assigned to their output > topic value, should be dropped and acked. > > *Example:* > { > "parserClassName": "org.apache.metron.parsers.GrokParser", > "sensorTopic": "myInTopic", > ... > "parserConfig": { > ..., > "kafka.topicField": "output_topic" > }, > "fieldTransformations": [ > { > "input": [ > "message" > ], > "output": [ > "output_topic" > ], > "transformation": "REGEX_SELECT", > "config": { > "world": "^Hello " > } > }, > ... > } > > *Input Records:* > "...sshd[32469]: Hello..." > "...sshd[30432]: Bye..." > > *Output:* > Kafka topic = "world" (as determined by the REGEX_SELECT pattern match that > sets the "output_topic" property used by kafka.topicField) > 1 record present > contents of that record = our record with "Hello" in it > 1 record is dropped ("Bye" record) and will not be forwarded any further > through the pipeline. >