[
https://issues.apache.org/jira/browse/METRON-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012880#comment-16012880
]
ASF GitHub Bot commented on METRON-936:
---------------------------------------
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/585
@mmiklavc it depends on which test case you're talking about. We have two
modes of operation in the pcap topology and 2 test cases in the integration
test and these are defined by the flux property `kafka.pcap.ts_scheme`. These
modes define the deserialization logic used in the topology to convert kafka
key/values to bytes suitable for writing to HDFS:
* `FROM_PACKET`: which expects a fully-formed packet (with headers) and
parses the packet and extracts the timestamp from the value. This is a legacy
mode, which functioned with pycapa prior to rewriting. We should eventually
deprecate this and remove it. This is associated with the
`FromPacketDeserializer`
* `FROM_KEY` : which expects raw data and a timestamp from the key. This
is by far the dominant mode of operation and the one you will see in `pycapa`
or `fastcapa`. This is associated with the `FromKeyDeserializer`
It appears that you are doing the null check in the `HDFSWriterCallback`.
I would recommend doing this null check in `FromKeyDeserializer` as a null key
is not an illegal state for the `FromPacketDeserializer`.
> Fixes to pcap for performance and testing
> -----------------------------------------
>
> Key: METRON-936
> URL: https://issues.apache.org/jira/browse/METRON-936
> Project: Metron
> Issue Type: Bug
> Reporter: Michael Miklavcic
> Assignee: Michael Miklavcic
>
> Default in pcap.properties for kafka.pcap.start=END, and it's also not used
> in the Flux file. topology.workers should also be provided via the properties
> file.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
