[jira] [Commented] (METRON-2343) Bro Kafka plugin - ability to dynamically modify JSON
[ https://issues.apache.org/jira/browse/METRON-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111669#comment-17111669 ] Jon Zeolla commented on METRON-2343: [~ReignInChaos] this is now in master thanks to [~otto] > Bro Kafka plugin - ability to dynamically modify JSON > - > > Key: METRON-2343 > URL: https://issues.apache.org/jira/browse/METRON-2343 > Project: Metron > Issue Type: Wish >Affects Versions: 0.3.0 >Reporter: Rich Irwin >Assignee: Otto Fowler >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Desire to have the ability to modify Bro log JSON and add a field prior to > producing to Kafka. There is an ability to add a field to the actual Bro > log, however, this could be cumbersome on disk space. Furthermore, the field > looking to be added only pertains to the destined data lake for analytical > purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [metron-bro-plugin-kafka] asfgit closed pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON
asfgit closed pull request #46: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON
JonZeolla commented on pull request #46: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631134572 LGTM +1 thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (METRON-2356) Metron kafka plugin should have btests for plugin json output
Otto Fowler created METRON-2356: --- Summary: Metron kafka plugin should have btests for plugin json output Key: METRON-2356 URL: https://issues.apache.org/jira/browse/METRON-2356 Project: Metron Issue Type: Improvement Reporter: Otto Fowler Although the plugin understands mocking ( IE can be tested without actually calling Kafka ) there is value in understanding what JSON is sent to kafka for validation of configurations, like adding static fields through configuration. We should change the plugin such that it still understands mocking, but while mocking may still have additional configuration to triggers it to log the json output instead of sending it to kafka. however we implement it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON
ottobackwards commented on pull request #46: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631089959 https://issues.apache.org/jira/browse/METRON-2356 <-- @JonZeolla This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON
ottobackwards commented on pull request #46: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631090146 I think that jira captures why the change is beyond this pr This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON
JonZeolla commented on pull request #46: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631050619 Looks good. Noticed we are lacking btests - I'm fine with having that as a follow-on. +1 pending JIRA creation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on a change in pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON
ottobackwards commented on a change in pull request #46: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46#discussion_r427535179 ## File path: README.md ## @@ -224,6 +224,22 @@ event zeek_init() =-10 } ``` +### Example 6 - Add static values to each outgoing Kafka message +It is possible to define name value pairs and have them added to each outgoing Kafka json message when tagged_json is set to true. Each will be added to the root json object. +* the Kafka::additional_message_values table can be configured with each name and value +* based on the following configuration, each outgoing message will have "FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME": "SECOND_STATIC_VALUE" added. +```bash +@load packages +redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG); +redef Kafka::topic_name = "zeek"; +redef Kafka::tag_json = T; +redef Kafka::kafka_conf = table(["metadata.broker.list"] = "kafka-1:9092,kafka-2:9092"); +redef Kafka::additional_message_values = table(["FIRST_STATIC_NAME"] = "FIRST_STATIC_VALUE", ["SECOND_STATIC_NAME"] = "SECOND_STATIC_VALUE"); Review comment: follow on This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON
JonZeolla commented on a change in pull request #46:
URL:
https://github.com/apache/metron-bro-plugin-kafka/pull/46#discussion_r427518350
##
File path: scripts/init.zeek
##
@@ -53,6 +53,14 @@ export {
["metadata.broker.list"] = "localhost:9092"
)
+## Key value pairs that will be added to outgoing messages at the root level
+## for example: ["bro_server"] = "this_server_name"
Review comment:
Let's use `zeek` in examples instead of the old `bro`
##
File path: README.md
##
@@ -224,6 +224,22 @@ event zeek_init() =-10
}
```
+### Example 6 - Add static values to each outgoing Kafka message
Review comment:
The notes are specific to the prior example, so please shift this down
in the README
##
File path: scripts/init.zeek
##
@@ -53,6 +53,14 @@ export {
["metadata.broker.list"] = "localhost:9092"
)
+## Key value pairs that will be added to outgoing messages at the root level
+## for example: ["bro_server"] = "this_server_name"
+## will results in a "bro_server":"this_server_name" field added to the
outgoing
Review comment:
Same as above comment
##
File path: README.md
##
@@ -224,6 +224,22 @@ event zeek_init() =-10
}
```
+### Example 6 - Add static values to each outgoing Kafka message
+It is possible to define name value pairs and have them added to each outgoing
Kafka json message when tagged_json is set to true. Each will be added to the
root json object.
+* the Kafka::additional_message_values table can be configured with each
name and value
+* based on the following configuration, each outgoing message will have
"FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME":
"SECOND_STATIC_VALUE" added.
+```bash
Review comment:
This is `zeek` not `bash`
##
File path: scripts/init.zeek
##
@@ -53,6 +53,14 @@ export {
["metadata.broker.list"] = "localhost:9092"
)
+## Key value pairs that will be added to outgoing messages at the root level
+## for example: ["bro_server"] = "this_server_name"
+## will results in a "bro_server":"this_server_name" field added to the
outgoing
+## json
+## note this depends on tag_json being T
+const additional_message_values: table[string] of string = table(
Review comment:
Can't we just shrink this to one line via `table() `?
##
File path: scripts/init.zeek
##
@@ -53,6 +53,14 @@ export {
["metadata.broker.list"] = "localhost:9092"
)
+## Key value pairs that will be added to outgoing messages at the root level
Review comment:
Please align indentation with other items
##
File path: README.md
##
@@ -224,6 +224,22 @@ event zeek_init() =-10
}
```
+### Example 6 - Add static values to each outgoing Kafka message
+It is possible to define name value pairs and have them added to each outgoing
Kafka json message when tagged_json is set to true. Each will be added to the
root json object.
+* the Kafka::additional_message_values table can be configured with each
name and value
+* based on the following configuration, each outgoing message will have
"FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME":
"SECOND_STATIC_VALUE" added.
+```bash
+@load packages
+redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG,
FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG,
Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG,
RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);
+redef Kafka::topic_name = "zeek";
+redef Kafka::tag_json = T;
+redef Kafka::kafka_conf = table(["metadata.broker.list"] =
"kafka-1:9092,kafka-2:9092");
+redef Kafka::additional_message_values = table(["FIRST_STATIC_NAME"] =
"FIRST_STATIC_VALUE", ["SECOND_STATIC_NAME"] = "SECOND_STATIC_VALUE");
Review comment:
We should consider a more illustrative example, like:
```
@if ( Cluster::is_enabled() )
redef Kafka::additional_message_values = table(["CLUSTER_ENABLED"] = "True",
["NODE_ID"] = Cluster::node);
@else
redef Kafka::additional_message_values = table(["CLUSTER_ENABLED"] =
"False");
@endif
```
Warning: did not test. May need to pull `node$id` for the NODE_ID value
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] ottobackwards opened a new pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON
ottobackwards opened a new pull request #46: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46 Add the ability to define static json fields and values to be added to the top level object when tagged_json is T. ### Testing - run_end_to_end.sh - look at any of the kafka output logs in the testing output directory for: "FIRST_STATIC_NAME" : "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME":"SECOND_STATIC_VALUE" in the json. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed via: ``` bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [-] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (METRON-2343) Bro Kafka plugin - ability to dynamically modify JSON
[ https://issues.apache.org/jira/browse/METRON-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned METRON-2343: --- Assignee: Otto Fowler > Bro Kafka plugin - ability to dynamically modify JSON > - > > Key: METRON-2343 > URL: https://issues.apache.org/jira/browse/METRON-2343 > Project: Metron > Issue Type: Wish >Affects Versions: 0.3.0 >Reporter: Rich Irwin >Assignee: Otto Fowler >Priority: Major > > Desire to have the ability to modify Bro log JSON and add a field prior to > producing to Kafka. There is an ability to add a field to the actual Bro > log, however, this could be cumbersome on disk space. Furthermore, the field > looking to be added only pertains to the destined data lake for analytical > purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #39: Feat/Add sensor_name to the tagged json
JonZeolla commented on pull request #39: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/39#issuecomment-630714126 Hi, checking in to see how things are coming along and if you were interested in contributing to the package This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #40: METRON-2309 Add a Kafka "metadata.broker.list" for each log writer filter.
JonZeolla commented on pull request #40: URL: https://github.com/apache/metron-bro-plugin-kafka/pull/40#issuecomment-630712041 As much as I would love to get this in, I have to mark it as abandoned. I'll be closing this soon if there's no response This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
