[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[cestella]

Github user cestella commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222638653
  
You sure Kafka is still up?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[james-sirota]

Github user james-sirota commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222590012
  
Now getting the following error on the bro topology trying to enrich:

2016-05-31 04:24:05.212 o.a.k.c.n.Selector [WARN] Error in I/O with ip 
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
~[?:1.8.0_40]
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) 
~[?:1.8.0_40]
at org.apache.kafka.common.network.Selector.poll(Selector.java:238) 
[stormjar.jar:?]
at 
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) 
[stormjar.jar:?]
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) 
[stormjar.jar:?]
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) 
[stormjar.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]

I am giving it the same kafka broker that i gave to the user topology, 
which worked 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[Casey Stella]

If you turn on debug logs for org.apache.metron, you should be able to see
exactly when Ack and errors happen.
On Mon, May 30, 2016 at 19:34 james-sirota  wrote:

> Github user james-sirota commented on the pull request:
>
>
> https://github.com/apache/incubator-metron/pull/127#issuecomment-222565824
>
> Another interesting thing that I think is a problem is that I sent it
> exactly 30 tuples.  The spout acked 60 tuples (somehow doubled the count)
> and when it failed the number of failed acks was 100.  Not sure where the
> other 40 tuples came from, but i think it may be replaying them.  We need
> to check all our topologies for this behavior.  We need to make sure that
> failed tuples are not being replayed
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[james-sirota]

Github user james-sirota commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222565824
  
Another interesting thing that I think is a problem is that I sent it 
exactly 30 tuples.  The spout acked 60 tuples (somehow doubled the count) and 
when it failed the number of failed acks was 100.  Not sure where the other 40 
tuples came from, but i think it may be replaying them.  We need to check all 
our topologies for this behavior.  We need to make sure that failed tuples are 
not being replayed 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[Casey Stella]

Yeah the writer is called in process of the parser bolt, just like it was
legacy. I'd like a follow on jira splitting them up.

Make sure you send in a fair number of messages. The UI doesn't seem to ack
before 20 messages I have found.
On Mon, May 30, 2016 at 18:57 james-sirota  wrote:

> Github user james-sirota commented on the pull request:
>
>
> https://github.com/apache/incubator-metron/pull/127#issuecomment-222563571
>
> I was able to get past the previous error by uploading a new common
> jar.  Now when the topology comes up it processes the CSV no problem.  But,
> I only have the spout and the parser bolt come up.  The writer does not
> come up.  Also, I can't get the parser to ack anything.  My file looks like
> this:
>
> {
> "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
>,"writerClassName" :
> "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
>,"sensorTopic":"user"
>,"parserConfig":
>{
>  "shew.table" : "enrichment"
> ,"shew.cf" : "t"
> ,"shew.keyColumns" : "user"
> ,"shew.enrichmentType" : "user"
> ,"columns" : {
> "user" : 0
>,"ip" : 1
>  }
>}
> }
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[james-sirota]

Github user james-sirota commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222563571
  
I was able to get past the previous error by uploading a new common jar.  
Now when the topology comes up it processes the CSV no problem.  But, I only 
have the spout and the parser bolt come up.  The writer does not come up.  
Also, I can't get the parser to ack anything.  My file looks like this:

{
"parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
   ,"writerClassName" : 
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
   ,"sensorTopic":"user"
   ,"parserConfig":
   {
 "shew.table" : "enrichment"
,"shew.cf" : "t"
,"shew.keyColumns" : "user"
,"shew.enrichmentType" : "user"
,"columns" : {
"user" : 0
   ,"ip" : 1
 }
   }
}


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[Casey Stella]

You can see from
https://github.com/cestella/incubator-metron/blob/METRON-174/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/SensorParserConfig.java
that it has the setter and getter for writerClassName

Hmm try once more with a clean mvn clean package of the branch and make
sure you replace the parsers jar on the cluster.
On Mon, May 30, 2016 at 13:54 cestella  wrote:

> Github user cestella commented on the pull request:
>
>
> https://github.com/apache/incubator-metron/pull/127#issuecomment-222534318
>
> Looks like it can't find the writerClassname field. Are you sure you
> ran a
> build from this branch before the deploy?
> On Mon, May 30, 2016 at 13:48 James Sirota 
> wrote:
>
> > /usr/metron/0.1BETA/bin/zk_load_configs.sh -m DUMP -z 1xxx:2181
> > log4j:WARN No appenders could be found for logger
> > (org.apache.curator.framework.imps.CuratorFrameworkImpl).
> > log4j:WARN Please initialize the log4j system properly.
> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for
> > more info.
> > GLOBAL Config: global
> > {
> > "es.clustername": "metron",
> > "es.ip": "xxx",
> > "es.port": "9300",
> > "es.date.format": ".MM.dd.HH"
> > }
> >
> > PARSER Config: websphere
> > {
> >
> >
> "parserClassName":"org.apache.metron.parsers.websphere.GrokWebSphereParser",
> > "sensorTopic":"websphere",
> > "parserConfig":
> > {
> > "grokPath":"/patterns/websphere",
> > "patternLabel":"WEBSPHERE",
> > "timestampField":"timestamp_string",
> > "dateFormat":" MMM dd HH:mm:ss"
> > }
> > }
> >
> > PARSER Config: bluecoat
> > {
> >
> "parserClassName":"org.apache.metron.parsers.bluecoat.BasicBluecoatParser",
> > "sensorTopic":"bluecoat",
> > "parserConfig": {}
> > }
> >
> > PARSER Config: squid
> > {
> > "parserClassName": "org.apache.metron.parsers.GrokParser",
> > "sensorTopic": "squid",
> > "parserConfig": {
> > "grokPath": "/patterns/squid",
> > "patternLabel": "SQUID_DELIMITED",
> > "timestampField": "timestamp"
> > }
> > }
> >
> > Exception in thread "main" java.lang.RuntimeException: Unable to
> load {
> >
> >
> > "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
> > ,"writerClassName" :
> > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
> > ,"sensorTopic":"user"
> > ,"parserConfig":
> > {
> > "shew.table" : "enrichment"
> > ,"shew.cf" : "t"
> > ,"shew.keyColumns" : "user"
> > ,"shew.enrichmentType" : "user"
> > ,"columns" : {
> > "user" : 0
> > ,"ip" : 1
> > }
> > }
> > }
> >
> > at
> org.apache.metron.common.configuration.ConfigurationType.lambda$static$1(ConfigurationType.java:47)
> > at
> org.apache.metron.common.configuration.ConfigurationType$$Lambda$9/1684106402.apply(Unknown
> Source)
> > at
> org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:78)
> > at
> org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$0(ConfigurationsUtils.java:272)
> > at
> org.apache.metron.common.configuration.ConfigurationsUtils$$Lambda$7/785992331.visit(Unknown
> Source)
> > at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:264)
> > at
> org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:251)
> > at
> org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:271)
> > at
> org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:115)
> > at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:177)
> > at
> org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:161)
> > at
> org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:198)
> >
> > Caused by:
> > com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
> > Unrecognized field "writerClassName" (class
> > org.apache.metron.common.configuration.SensorParserConfig), not
> marked as
> > ignorable (3 known properties: , "parserConfig", "parserClassName",
> > "sensorTopic"])
> > at Source: java.io.StringReader@23bb844
> > <
> https://github.com/java.io.StringReader/incubator-metron/commit/23bb8443>;
> > line: 3, column: 26
> > at
> >
> com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:79)
> > at
> >
> com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:555)
> > at
> >
> 

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[james-sirota]

Github user james-sirota commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222534631
  
I built the jar from the branch and copied it out to an existing AWS 
cluster that I had. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[cestella]

Github user cestella commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222534318
  
Looks like it can't find the writerClassname field. Are you sure you ran a
build from this branch before the deploy?
On Mon, May 30, 2016 at 13:48 James Sirota  wrote:

> /usr/metron/0.1BETA/bin/zk_load_configs.sh -m DUMP -z 1xxx:2181
> log4j:WARN No appenders could be found for logger
> (org.apache.curator.framework.imps.CuratorFrameworkImpl).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> GLOBAL Config: global
> {
> "es.clustername": "metron",
> "es.ip": "xxx",
> "es.port": "9300",
> "es.date.format": ".MM.dd.HH"
> }
>
> PARSER Config: websphere
> {
>
> 
"parserClassName":"org.apache.metron.parsers.websphere.GrokWebSphereParser",
> "sensorTopic":"websphere",
> "parserConfig":
> {
> "grokPath":"/patterns/websphere",
> "patternLabel":"WEBSPHERE",
> "timestampField":"timestamp_string",
> "dateFormat":" MMM dd HH:mm:ss"
> }
> }
>
> PARSER Config: bluecoat
> {
> 
"parserClassName":"org.apache.metron.parsers.bluecoat.BasicBluecoatParser",
> "sensorTopic":"bluecoat",
> "parserConfig": {}
> }
>
> PARSER Config: squid
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "squid",
> "parserConfig": {
> "grokPath": "/patterns/squid",
> "patternLabel": "SQUID_DELIMITED",
> "timestampField": "timestamp"
> }
> }
>
> Exception in thread "main" java.lang.RuntimeException: Unable to load {
>
>
> "parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
> ,"writerClassName" :
> "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
> ,"sensorTopic":"user"
> ,"parserConfig":
> {
> "shew.table" : "enrichment"
> ,"shew.cf" : "t"
> ,"shew.keyColumns" : "user"
> ,"shew.enrichmentType" : "user"
> ,"columns" : {
> "user" : 0
> ,"ip" : 1
> }
> }
> }
>
> at 
org.apache.metron.common.configuration.ConfigurationType.lambda$static$1(ConfigurationType.java:47)
> at 
org.apache.metron.common.configuration.ConfigurationType$$Lambda$9/1684106402.apply(Unknown
 Source)
> at 
org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:78)
> at 
org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$0(ConfigurationsUtils.java:272)
> at 
org.apache.metron.common.configuration.ConfigurationsUtils$$Lambda$7/785992331.visit(Unknown
 Source)
> at 
org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:264)
> at 
org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:251)
> at 
org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:271)
> at 
org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:115)
> at 
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:177)
> at 
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:161)
> at 
org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:198)
>
> Caused by:
> com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
> Unrecognized field "writerClassName" (class
> org.apache.metron.common.configuration.SensorParserConfig), not marked as
> ignorable (3 known properties: , "parserConfig", "parserClassName",
> "sensorTopic"])
> at Source: java.io.StringReader@23bb844
> 
;
> line: 3, column: 26
> at
> 
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:79)
> at
> 
com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:555)
> at
> 
com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:708)
> at
> 
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1160)
> at
> 
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:315)
> at
> 
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121)
> at
> 
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2888)
> at
> 
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2034)
> at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:71)
> 

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[james-sirota]

Github user james-sirota commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222533633
  
/usr/metron/0.1BETA/bin/zk_load_configs.sh -m DUMP -z 1xxx:2181
log4j:WARN No appenders could be found for logger 
(org.apache.curator.framework.imps.CuratorFrameworkImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
GLOBAL Config: global
{
  "es.clustername": "metron",
  "es.ip": "xxx",
  "es.port": "9300",
  "es.date.format": ".MM.dd.HH"
}

PARSER Config: websphere
{
  
"parserClassName":"org.apache.metron.parsers.websphere.GrokWebSphereParser",
  "sensorTopic":"websphere",
  "parserConfig":
  {
"grokPath":"/patterns/websphere",
"patternLabel":"WEBSPHERE",
"timestampField":"timestamp_string",
"dateFormat":" MMM dd HH:mm:ss"
  }
}

PARSER Config: bluecoat
{
"parserClassName":"org.apache.metron.parsers.bluecoat.BasicBluecoatParser",
"sensorTopic":"bluecoat",
"parserConfig": {}
}

PARSER Config: squid
{
  "parserClassName": "org.apache.metron.parsers.GrokParser",
  "sensorTopic": "squid",
  "parserConfig": {
"grokPath": "/patterns/squid",
"patternLabel": "SQUID_DELIMITED",
"timestampField": "timestamp"
  }
}

Exception in thread "main" java.lang.RuntimeException: Unable to load {
"parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
   ,"writerClassName" : 
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
   ,"sensorTopic":"user"
   ,"parserConfig":
   {
 "shew.table" : "enrichment"
,"shew.cf" : "t"
,"shew.keyColumns" : "user"
,"shew.enrichmentType" : "user"
,"columns" : {
"user" : 0
   ,"ip" : 1
 }
   }
}

at 
org.apache.metron.common.configuration.ConfigurationType.lambda$static$1(ConfigurationType.java:47)
at 
org.apache.metron.common.configuration.ConfigurationType$$Lambda$9/1684106402.apply(Unknown
 Source)
at 
org.apache.metron.common.configuration.ConfigurationType.deserialize(ConfigurationType.java:78)
at 
org.apache.metron.common.configuration.ConfigurationsUtils.lambda$dumpConfigs$0(ConfigurationsUtils.java:272)
at 
org.apache.metron.common.configuration.ConfigurationsUtils$$Lambda$7/785992331.visit(Unknown
 Source)
at 
org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:264)
at 
org.apache.metron.common.configuration.ConfigurationsUtils.visitConfigs(ConfigurationsUtils.java:251)
at 
org.apache.metron.common.configuration.ConfigurationsUtils.dumpConfigs(ConfigurationsUtils.java:271)
at 
org.apache.metron.common.cli.ConfigurationManager.dump(ConfigurationManager.java:115)
at 
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:177)
at 
org.apache.metron.common.cli.ConfigurationManager.run(ConfigurationManager.java:161)
at 
org.apache.metron.common.cli.ConfigurationManager.main(ConfigurationManager.java:198)
Caused by: 
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized 
field "writerClassName" (class 
org.apache.metron.common.configuration.SensorParserConfig), not marked as 
ignorable (3 known properties: , "parserConfig", "parserClassName", 
"sensorTopic"])
 at [Source: java.io.StringReader@23bb8443; line: 3, column: 26] (through 
reference chain: 
org.apache.metron.common.configuration.SensorParserConfig["writerClassName"])
at 
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:79)
at 
com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:555)
at 
com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:708)
at 
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1160)
at 
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:315)
at 
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:121)
at 
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2888)
at 
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2034)
at org.apache.metron.common.utils.JSONUtils.load(JSONUtils.java:71)
at 
org.apache.metron.common.configuration.ConfigurationType.lambda$static$1(ConfigurationType.java:45)
... 11 more

So looks like it can't load it.

Looking in the jar it seems to exist

jar -tf 

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[james-sirota]

Github user james-sirota commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222414251
  
On AWS the following did not work:

/usr/metron/0.1BETA/bin/start_parser_topology.sh -s user -k xxx:9092 -z 
xxx:2181

I got a:

41  [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager - State 
change: CONNECTED
java.lang.NullPointerException
at 
org.apache.metron.parsers.topology.ParserTopologyBuilder.build(ParserTopologyBuilder.java:57)
at 
org.apache.metron.parsers.topology.ParserTopologyCLI.main(ParserTopologyCLI.java:232)

Looking at the file here:

https://github.com/cestella/incubator-metron/blob/METRON-174/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java

 looks like it chokes on the sensor topic.  However, listing the topics:

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --list  --zookeeper 
xxx:2181
bro
enrichments
pcap
snort
user
yaf

looks like i have it correct.  My parser topology config looks as follows:

{
"parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
   ,"writerClassName" : 
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
   ,"sensorTopic":"user"
   ,"parserConfig":
   {
 "shew.table" : "enrichment"
,"shew.cf" : "t"
,"shew.keyColumns" : "user"
,"shew.enrichmentType" : "user"
,"columns" : {
"user" : 0
   ,"ip" : 1
 }
   }
}

And is located under /usr/metron/0.1BETA/config/zookeeper/parsers/user.json

Any suggestions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[james-sirota]

Github user james-sirota commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222411369
  
FYI...for some reason the kafka topic does not always get auto created.  I 
can't figure out what options cause it to not auto create.  Also, some times 
kafka throws an error the first time you push into a topic.  However, when you 
run the same command again it works just fine.  Maybe something to keep in mind 
for the future is that we can't rely on the kafka producer to reliably create a 
topic 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[cestella]

Github user cestella commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-222364875
  
In order to validate this, you can do the following:
* Configure a new parser, in this example I'll call it a `user` parser and 
we'll parse some CSV data to map `username` to `ip` by creating a file 
`/usr/metron/0.1BETA/config/zookeeper/enrichment/user.json` with

```
{
"parserClassName" : "org.apache.metron.parsers.csv.CSVParser"
   ,"writerClassName" : 
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter"
   ,"sensorTopic":"user"
   ,"parserConfig":
   {
 "shew.table" : "enrichment"
,"shew.cf" : "t"
,"shew.keyColumns" : "user"
,"shew.enrichmentType" : "user"
,"columns" : {
"user" : 0
   ,"ip" : 1
 }
   }
}
```
* Add a new `user` enrichment type to `bro` data by adding `ip_src_addr` to 
`hbaseEnrichment` and associating `user` as a field type for `ip_src_addr` in  
`/usr/metron/0.1BETA/config/zookeeper/enrichment/bro.json` like so
```
{
  "index": "bro",
  "batchSize": 5,
  "enrichment": {
"fieldMap": {
  "geo": [
"ip_dst_addr",
"ip_src_addr"
  ],
  "host": [
"host"
  ],
  "hbaseEnrichment" : [ "ip_src_addr" ]
},
   "fieldToTypeMap":
   {
  "ip_src_addr" : [ "user"]
   }
  },
  "threatIntel":{
"fieldMap":
{
  "hbaseThreatIntel": ["ip_dst_addr", "ip_src_addr"]
},
"fieldToTypeMap":
{
  "ip_dst_addr" : [ "malicious_ip" ]
,"ip_src_addr" : [ "malicious_ip" ]
}
  }
}```
* Create the Kafka Queue as in the tutorials
* Using `/usr/metron/0.1BETA/bin/zk_load_configs.sh` push up the config you 
just created. `/usr/metron/0.1BETA/bin/zk_load_configs.sh -m PUSH -z node1:2181 
-i /usr/metron/0.1BETA/config/zookeeper`
* Create some reference CSV reference data with that looks like 
`jsirota,192.168.168.1` into a csv file named `user.csv`
* Use the kafka console producer to push data into the `user` topic via  
`cat user.csv | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh 
--broker-list node1:6667 --topic user`
* You should be able to check that the data gets into HBase by doing a 
`scan 'enrichment'` from the `hbase shell`
* You should also be able to check, after new data has been run through, 
that the data is enriched in elasticsearch.  I would suggest bouncing the 
enrichment topology to ensure that stale data in the caches get flushed, but 
that is not strictly necessary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[merrimanr]

Github user merrimanr commented on the pull request:

https://github.com/apache/incubator-metron/pull/127#issuecomment-221736442
  
I like it.  +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/127#discussion_r64669063
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -60,7 +93,25 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
 super.prepare(stormConf, context, collector);
 this.collector = collector;
 parser.init();
-writer.init();
+
+if(isBulk) {
+  writerTransformer = config -> new ParserWriterConfiguration(config);
--- End diff --

The config object can change if zookeeper is updated, so we want an
indirection here.
On Wed, May 25, 2016 at 19:11 merrimanr  wrote:

> In
> 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
> 
> :
>
> > @@ -60,7 +93,25 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
> >  super.prepare(stormConf, context, collector);
> >  this.collector = collector;
> >  parser.init();
> > -writer.init();
> > +
> > +if(isBulk) {
> > +  writerTransformer = config -> new 
ParserWriterConfiguration(config);
>
> Curious why a Function is used here. Why not just instantiate a
> WriterConfiguration object for each case and pass that to
> messageWriter.init? Is there a benefit to doing it this way?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub
> 

>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request: METRON-174 Storm consumption of hba...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/127#discussion_r64668872
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -60,7 +93,25 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
 super.prepare(stormConf, context, collector);
 this.collector = collector;
 parser.init();
-writer.init();
+
+if(isBulk) {
+  writerTransformer = config -> new ParserWriterConfiguration(config);
--- End diff --

Curious why a Function is used here.  Why not just instantiate a 
WriterConfiguration object for each case and pass that to messageWriter.init?  
Is there a benefit to doing it this way?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---