[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-31 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222640686 Try pulling data from that broker using the console consumer --- If your project is set up for it, you can reply to this email and have your reply appear on G

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-31 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222638653 You sure Kafka is still up? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project do

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222590012 Now getting the following error on the bro topology trying to enrich: 2016-05-31 04:24:05.212 o.a.k.c.n.Selector [WARN] Error in I/O with ip

Re: [GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread Casey Stella
If you turn on debug logs for org.apache.metron, you should be able to see exactly when Ack and errors happen. On Mon, May 30, 2016 at 19:34 james-sirota wrote: > Github user james-sirota commented on the pull request: > > > https://github.com/apache/incubator-metron/pull/127#issuecomment-2225658

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222565824 Another interesting thing that I think is a problem is that I sent it exactly 30 tuples. The spout acked 60 tuples (somehow doubled the count) and when i

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222564455 Tracing through the log i got this. Let me bounce hbase and see if that helps 2016-05-30 23:09:55.612 o.a.h.h.z.RecoverableZooKeeper [WARN] Possibly

Re: [GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread Casey Stella
Yeah the writer is called in process of the parser bolt, just like it was legacy. I'd like a follow on jira splitting them up. Make sure you send in a fair number of messages. The UI doesn't seem to ack before 20 messages I have found. On Mon, May 30, 2016 at 18:57 james-sirota wrote: > Github u

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222563571 I was able to get past the previous error by uploading a new common jar. Now when the topology comes up it processes the CSV no problem. But, I only hav

Re: [GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread Casey Stella
Actually also the common jar. What may have happened is that you only copied up the parsers jar and the zk_utils utility runs from the common jar, which had old code. That would explain that field not being able to be deserialized by jackson. On Mon, May 30, 2016 at 13:59 Casey Stella wrote: > Yo

Re: [GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread Casey Stella
You can see from https://github.com/cestella/incubator-metron/blob/METRON-174/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/SensorParserConfig.java that it has the setter and getter for writerClassName Hmm try once more with a clean mvn clean package of the bra

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222534631 I built the jar from the branch and copied it out to an existing AWS cluster that I had. --- If your project is set up for it, you can reply to this ema

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222534318 Looks like it can't find the writerClassname field. Are you sure you ran a build from this branch before the deploy? On Mon, May 30, 2016 at 13:48 James

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222533633 /usr/metron/0.1BETA/bin/zk_load_configs.sh -m DUMP -z 1xxx:2181 log4j:WARN No appenders could be found for logger (org.apache.curator.framework.imps.Cu

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222452433 If you did push the config before trying to start the parser, then please confirm that the `user` topology is in zookeeper via inspecting the output of `/usr

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-30 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222452241 @james-sirota Did you push the new parser config to zookeeper via `/usr/metron/0.1BETA/bin/zk_load_configs.sh -m PUSH -z node1:2181 -i /usr/metron/0.1BETA/con

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-29 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222414251 On AWS the following did not work: /usr/metron/0.1BETA/bin/start_parser_topology.sh -s user -k xxx:9092 -z xxx:2181 I got a: 41

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-29 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222411369 FYI...for some reason the kafka topic does not always get auto created. I can't figure out what options cause it to not auto create. Also, some times ka

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-29 Thread cestella
Github user cestella commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222364875 In order to validate this, you can do the following: * Configure a new parser, in this example I'll call it a `user` parser and we'll parse some CSV data t

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-29 Thread james-sirota
Github user james-sirota commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-222348337 Where are the docs for validating this? I would like to stream in an enrichment --- If your project is set up for it, you can reply to this email and h

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-25 Thread merrimanr
Github user merrimanr commented on the pull request: https://github.com/apache/incubator-metron/pull/127#issuecomment-221736442 I like it. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-25 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/127#discussion_r64669773 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -60,7 +93,25 @@ public void prepare(

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-25 Thread cestella
Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/127#discussion_r64669063 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -60,7 +93,25 @@ public void prepare(M

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-25 Thread merrimanr
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/127#discussion_r64668872 --- Diff: metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java --- @@ -60,7 +93,25 @@ public void prepare(

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

2016-05-20 Thread cestella
GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/127 METRON-174 Storm consumption of hbase enrichment reference data We should support streaming enrichment data into kafka and writing it out to HBase in a format suitable to be used in eith