> Stephane: How can I debug this?
We created the PARSER* functions to help debug issues like this.
Unfortunately, it does not work in this case because the bug is in the
Enveloping, not in the Parser itself. This is gap in the PARSER_*
functionality that I think we should address to make debugging issues like
this simpler.
Just to give you an idea of how debugging this *should* work, here is a
play-by-play.
(1) Launch the REPL.
[root@node1 ~]# source /etc/default/metron
[root@node1 ~]# cd $METRON_HOME
[root@node1 0.7.1]# bin/stellar -z $ZOOKEEPER
(2) Setup your parser configuration. Paste your configuration in the
editor session that opens.
[Stellar]>>> conf := SHELL_EDIT()
{
"parserClassName": "org.apache.metron.parsers.json.JSONMapParser",
"filterClassName": null,
"sensorTopic": "my_topic",
...
(3) Define your test message that you want to parse.
[Stellar]>>> msg := SHELL_EDIT()
{
"_index": "indexing",
"_type": "Event",
...
(4) Initialize the parser with your configuration.
[Stellar]>>> p := PARSER_INIT("my", conf)
Parser{0 successful, 0 error(s)}
(5) Parse the message.
[Stellar]>>> PARSER_PARSE(p, msg)
[{"original_string":"{\"_type\":\"Event\",\"_source\":{\"dst\":\"127.0.0.1\",\"devTimeEpoch\":\"1512437340000\",\"dstPort\":\"0\",\"srcPort\":\"80\",\"src\":\"194.51.198.185\"},\"_id\":\"AWAkTAefYn0uCUpkHmCy\",\"_index\":\"indexing\",\"_score\":1}","_index":"indexing","_type":"Event","guid":"1c00f4d4-16f7-4f24-8056-31742ffe06d1","_id":"AWAkTAefYn0uCUpkHmCy","_score":1,"timestamp":1556206301709,"source.type":"my"}]
[Stellar]>>> p
Parser{1 successful, 0 error(s)}
(6) You can see that the JSON Map Parser was able to successfully parse the
message content just fine. That was the first thing that made me point my
finger at the Enveloping, rather than the parser.
But that's the problem with the PARSER_* functions. Storm could NOT parse
this. Unfortunately, the PARSER_* functions do not mimic exactly what
happens in Storm, which is a gap that we should address. Specifically,
none of the Enveloping strategy stuff can be tested in the REPL.
(7) Using "?" is also very helpful when in the REPL.
[Stellar]>>> ?PARSER_INIT
PARSER_INIT
Description: Initialize a parser to parse the raw telemetry produced by a
sensor.
Arguments:
sensorType - The type of sensor to parse.
config - [Optional] The parser configuration. If not provided, the
configuration will be retrieved from Zookeeper.
Returns: A parser that can be used to parse sensor telemetry with
`PARSER_PARSE`.
[Stellar]>>> ?PARSER_PARSE
PARSER_PARSE
Description: Parses the raw telemetry produced by a sensor.
Arguments:
parser - The parser created with PARSER_INIT.
input - A telemetry message or list of telemetry messages to parse.
Returns: A list of messages that result from parsing the input telemetry.
If the input cannot be parsed, a message encapsulating the error is
returned as part of that list.
On Thu, Apr 25, 2019 at 10:48 AM <stephane.d...@orange.com> wrote:
> Hello,
>
>
>
> I’m trying to load some JSON data which has the following structure (this
> is a sample):
>
>
>
> {
>
> "_index": "indexing",
>
> "_type": "Event",
>
> "_id": "AWAkTAefYn0uCUpkHmCy",
>
> "_score": 1,
>
> "_source": {
>
> "dst": "127.0.0.1",
>
> "devTimeEpoch": "1512437340000",
>
> "dstPort": "0",
>
> "srcPort": "80",
>
> "src": "194.51.198.185"
>
> }
>
> }
>
>
>
> In my file, everything is on the same line. My parser config is the
> following:
>
>
>
> {
>
> "parserClassName": "org.apache.metron.parsers.json.JSONMapParser",
>
> "filterClassName": null,
>
> "sensorTopic": "my_topic",
>
> "outputTopic": null,
>
> "errorTopic": null,
>
> "writerClassName": null,
>
> "errorWriterClassName": null,
>
> "readMetadata": true,
>
> "mergeMetadata": true,
>
> "numWorkers": 2,
>
> "numAckers": null,
>
> "spoutParallelism": 1,
>
> "spoutNumTasks": 1,
>
> "parserParallelism": 2,
>
> "parserNumTasks": 2,
>
> "errorWriterParallelism": 1,
>
> "errorWriterNumTasks": 1,
>
> "spoutConfig": {},
>
> "securityProtocol": null,
>
> "stormConfig": {},
>
> "parserConfig": {
>
> },
>
> "fieldTransformations": [
>
> {
>
> "transformation":"RENAME",
>
> "config": {
>
> "dst": "ip_dst_addr",
>
> "src": "ip_src_addr",
>
> "srcPort": "ip_src_port",
>
> "dstPort": "ip_dst_port",
>
> "devTimeEpoch": "timestamp"
>
> }
>
> }
>
> ],
>
> "cacheConfig": {},
>
> "rawMessageStrategy": "ENVELOPE",
>
> "rawMessageStrategyConfig": {
>
> "messageField": "_source"
>
> }
>
> }
>
>
>
> But in Storm I get the following errors:
>
>
>
> 2019-04-25 16:45:22.225 o.a.s.d.executor Thread-5-parserBolt-executor[8 8]
> [ERROR]
>
> java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to
> java.lang.String
>
> at
> org.apache.metron.common.message.metadata.EnvelopedRawMessageStrategy.get(EnvelopedRawMessageStrategy.java:78)
> ~[stormjar.jar:?]
>
> at
> org.apache.metron.common.message.metadata.RawMessageStrategies.get(RawMessageStrategies.java:54)
> ~[stormjar.jar:?]
>
> at
> org.apache.metron.common.message.metadata.RawMessageUtil.getRawMessage(RawMessageUtil.java:55)
> ~[stormjar.jar:?]
>
> at
> org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:251)
> [stormjar.jar:?]
>
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
>
>
>
>
>
> How can I debug this?
>
>
>
> Thanks
>
>
>
> Stéphane
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
> Thank you.
>
>
