[jira] [Updated] (METRON-1533) Create KAFKA_FIND Stellar Function
[ https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1533: --- Fix Version/s: (was: Next + 1) 0.6.0 > Create KAFKA_FIND Stellar Function > -- > > Key: METRON-1533 > URL: https://issues.apache.org/jira/browse/METRON-1533 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > Fix For: 0.6.0 > > > When creating enrichments, I often find that I want to validate that the > enrichment I just created was successful on the live, incoming stream of > telemetry. My workflow looks something like this. > 1. Create and test the enrichment that I want to create. > {code:java} > [Stellar]>>> ip_src_addr := "72.34.49.86" > 72.34.49.86 > [Stellar]>>> geo := GEO_GET(ip_src_addr) > {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, > latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, > longitude=-118.2512} > {code} > 2. That looks good to me. Now let's add that to my Bro telemetry. > {code:java} > [Stellar]>>> conf := SHELL_EDIT(conf) > { > "enrichment" : { > "fieldMap": { > "stellar": { > "config": [ >"geo := GEO_GET(ip_src_addr)" > ] > } > } > }, > "threatIntel": { > } > } > [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro") > {code} > > 3. It looks like that worked, but did that really work? > At this point, I would run KAFKA_GET as many times as it takes to retrieve a > Bro message. You would just have to get lucky and hope that the enrichment > worked and secondly that you would pull down a Bro message (as opposed to a > different sensor). > > I would rather have a function that lets me only pull back the messages that > I care about. In this case I could either retrieve only Bro messages. > {code:java} > KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro') > {code} > Or I could look for messages that contain geolocation data. > {code:java} > KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m)) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1533) Create KAFKA_FIND Stellar Function
[ https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1533: --- Fix Version/s: Next + 1 > Create KAFKA_FIND Stellar Function > -- > > Key: METRON-1533 > URL: https://issues.apache.org/jira/browse/METRON-1533 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > Fix For: Next + 1 > > > When creating enrichments, I often find that I want to validate that the > enrichment I just created was successful on the live, incoming stream of > telemetry. My workflow looks something like this. > 1. Create and test the enrichment that I want to create. > {code:java} > [Stellar]>>> ip_src_addr := "72.34.49.86" > 72.34.49.86 > [Stellar]>>> geo := GEO_GET(ip_src_addr) > {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, > latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, > longitude=-118.2512} > {code} > 2. That looks good to me. Now let's add that to my Bro telemetry. > {code:java} > [Stellar]>>> conf := SHELL_EDIT(conf) > { > "enrichment" : { > "fieldMap": { > "stellar": { > "config": [ >"geo := GEO_GET(ip_src_addr)" > ] > } > } > }, > "threatIntel": { > } > } > [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro") > {code} > > 3. It looks like that worked, but did that really work? > At this point, I would run KAFKA_GET as many times as it takes to retrieve a > Bro message. You would just have to get lucky and hope that the enrichment > worked and secondly that you would pull down a Bro message (as opposed to a > different sensor). > > I would rather have a function that lets me only pull back the messages that > I care about. In this case I could either retrieve only Bro messages. > {code:java} > KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro') > {code} > Or I could look for messages that contain geolocation data. > {code:java} > KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m)) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1533) Create KAFKA_FIND Stellar Function
[ https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1533: --- Description: When creating enrichments, I often find that I want to validate that the enrichment I just created was successful on the live, incoming stream of telemetry. My workflow looks something like this. 1. Create and test the enrichment that I want to create. {code:java} [Stellar]>>> ip_src_addr := "72.34.49.86" 72.34.49.86 [Stellar]>>> geo := GEO_GET(ip_src_addr) {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, longitude=-118.2512} {code} 2. That looks good to me. Now let's add that to my Bro telemetry. {code:java} [Stellar]>>> conf := SHELL_EDIT(conf) { "enrichment" : { "fieldMap": { "stellar": { "config": [ "geo := GEO_GET(ip_src_addr)" ] } } }, "threatIntel": { } } [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro") {code} 3. It looks like that worked, but did that really work? At this point, I would run KAFKA_GET as many times as it takes to retrieve a Bro message. You would just have to get lucky and hope that the enrichment worked and secondly that you would pull down a Bro message (as opposed to a different sensor). I would rather have a function that lets me only pull back the messages that I care about. In this case I could either retrieve only Bro messages. {code:java} KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro') {code} Or I could look for messages that contain geolocation data. {code:java} KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m)) {code} was: When creating enrichments, I often find that I want to validate that the enrichment I just created was successful on the live, incoming stream of telemetry. My workflow looks something like this. 1. Create and test the enrichment that I want to create. {code:java} [Stellar]>>> ip_src_addr := "72.34.49.86" 72.34.49.86 [Stellar]>>> geo := GEO_GET(ip_src_addr) {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, longitude=-118.2512} {code} 2. That looks good to me. Now let's add that to my Bro telemetry. {code:java} [Stellar]>>> conf := SHELL_EDIT(conf) { "enrichment" : { "fieldMap": { "stellar": { "config": [ "geo := GEO_GET(ip_src_addr)" ] } } }, "threatIntel": { } } [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro") {code} 3. It looks like that worked, but did that really work? At this point, I would run KAFKA_GET as many times as it takes to retrieve a Bro message. You would just have to get lucky and hope that the enrichment worked and secondly that you would pull down a Bro message (as opposed to a different sensor). I would rather have a function that lets me only pull back the messages that I care about. In this case I could either retrieve only Bro messages. {code:java} KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro') {code} Or I could look for messages that contain geolocation data. {code:java} KAFKA_FIND('indexing', m -> MAP_EXISTS('geo', m)) {code} > Create KAFKA_FIND Stellar Function > -- > > Key: METRON-1533 > URL: https://issues.apache.org/jira/browse/METRON-1533 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > > When creating enrichments, I often find that I want to validate that the > enrichment I just created was successful on the live, incoming stream of > telemetry. My workflow looks something like this. > 1. Create and test the enrichment that I want to create. > {code:java} > [Stellar]>>> ip_src_addr := "72.34.49.86" > 72.34.49.86 > [Stellar]>>> geo := GEO_GET(ip_src_addr) > {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, > latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, > longitude=-118.2512} > {code} > 2. That looks good to me. Now let's add that to my Bro telemetry. > {code:java} > [Stellar]>>> conf := SHELL_EDIT(conf) > { > "enrichment" : { > "fieldMap": { > "stellar": { > "config": [ >"geo := GEO_GET(ip_src_addr)" > ] > } > } > }, > "threatIntel": { > } > } > [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro") > {code} > > 3. It looks like that worked, but did that really work? > At this point, I would run KAFKA_GET as many times as it takes to retrieve a > Bro message. You would just have to get lucky and hope that the enrichment > worked and secondly that you would pull down a Bro message (as opposed to a > different sensor). > > I would rather have a function that lets me only pull back the messages that > I care ab
[jira] [Updated] (METRON-1533) Create KAFKA_FIND Stellar Function
[ https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1533: --- Description: When creating enrichments, I often find that I want to validate that the enrichment I just created was successful on the live, incoming stream of telemetry. My workflow looks something like this. 1. Create and test the enrichment that I want to create. {code:java} [Stellar]>>> ip_src_addr := "72.34.49.86" 72.34.49.86 [Stellar]>>> geo := GEO_GET(ip_src_addr) {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, longitude=-118.2512} {code} 2. That looks good to me. Now let's add that to my Bro telemetry. {code:java} [Stellar]>>> conf := SHELL_EDIT(conf) { "enrichment" : { "fieldMap": { "stellar": { "config": [ "geo := GEO_GET(ip_src_addr)" ] } } }, "threatIntel": { } } [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro") {code} 3. It looks like that worked, but did that really work? At this point, I would run KAFKA_GET as many times as it takes to retrieve a Bro message. You would just have to get lucky and hope that the enrichment worked and secondly that you would pull down a Bro message (as opposed to a different sensor). I would rather have a function that lets me only pull back the messages that I care about. In this case I could either retrieve only Bro messages. {code:java} KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro') {code} Or I could look for messages that contain geolocation data. {code:java} KAFKA_FIND('indexing', m -> MAP_EXISTS('geo', m)) {code} was: When creating enrichments, I often find that I want to validate that the enrichment I just created was successful on the live, incoming stream of telemetry. My workflow looks something like this. 1. Create and test the enrichment that I want to create. {code:java} [Stellar]>>> ip_src_addr := "72.34.49.86" 72.34.49.86 [Stellar]>>> geo := GEO_GET(ip_src_addr) {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, longitude=-118.2512} {code} 2. That looks good to me. Now let's add that to my Bro telemetry. {code:java} [Stellar]>>> conf := SHELL_EDIT(conf) { "enrichment" : { "fieldMap": { "stellar": { "config": [ "geo := GEO_GET(ip_src_addr)" ] } } }, "threatIntel": { } } [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro") {code} 3. It looks like that worked, but did that really work? At this point, I would run KAFKA_GET as many times as it takes to retrieve a Bro message. You would just have to get lucky and hope that the enrichment worked and secondly that you would pull down a Bro message (as opposed to a different sensor). 4. I would rather have a function that lets me only pull back the messages that I care about. In this case I could either retrieve only Bro messages. {code:java} KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro') {code} Or I could look for messages that contain geolocation data. {code:java} KAFKA_FIND('topic', m -> MAP_EXISTS('geo', m)) {code} > Create KAFKA_FIND Stellar Function > -- > > Key: METRON-1533 > URL: https://issues.apache.org/jira/browse/METRON-1533 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > > When creating enrichments, I often find that I want to validate that the > enrichment I just created was successful on the live, incoming stream of > telemetry. My workflow looks something like this. > 1. Create and test the enrichment that I want to create. > {code:java} > [Stellar]>>> ip_src_addr := "72.34.49.86" > 72.34.49.86 > [Stellar]>>> geo := GEO_GET(ip_src_addr) > {country=US, dmaCode=803, city=Los Angeles, postalCode=90014, > latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361, > longitude=-118.2512} > {code} > 2. That looks good to me. Now let's add that to my Bro telemetry. > {code:java} > [Stellar]>>> conf := SHELL_EDIT(conf) > { > "enrichment" : { > "fieldMap": { > "stellar": { > "config": [ >"geo := GEO_GET(ip_src_addr)" > ] > } > } > }, > "threatIntel": { > } > } > [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro") > {code} > > 3. It looks like that worked, but did that really work? > At this point, I would run KAFKA_GET as many times as it takes to retrieve a > Bro message. You would just have to get lucky and hope that the enrichment > worked and secondly that you would pull down a Bro message (as opposed to a > different sensor). > > I would rather have a function that lets me only pull back the messages that > I care about. I