from:"GitBox"

[GitHub] [metron] JonZeolla commented on pull request #1282: METRON-1881: Update the bro plugin used in metron development

2021-02-25 Thread GitBox



JonZeolla commented on pull request #1282:
URL: https://github.com/apache/metron/pull/1282#issuecomment-785972885


   Closing due to the official decision to move Apache Metron to the ASF attic



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] JonZeolla closed pull request #1282: METRON-1881: Update the bro plugin used in metron development

2021-02-25 Thread GitBox



JonZeolla closed pull request #1282:
URL: https://github.com/apache/metron/pull/1282


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla closed pull request #40: METRON-2309 Add a Kafka "metadata.broker.list" for each log writer filter.

2020-11-25 Thread GitBox



JonZeolla closed pull request #40:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/40


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-10-22 Thread GitBox



JonZeolla commented on pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-714611136


   Hi @vikgmdev yeah unfortunately those are known issues.  If you manually 
download 
[nitroba](https://github.com/open-nsm/course/blob/master/pcaps/nitroba.pcap?raw=true)
 and put it in the [pcap 
directory](https://github.com/apache/metron-bro-plugin-kafka/blob/92b85e5e00cc2fd9023ea7f53466db6592eb6634/docker/run_end_to_end.sh#L193),
 and change the [zeek 
version](https://github.com/apache/metron-bro-plugin-kafka/blob/92b85e5e00cc2fd9023ea7f53466db6592eb6634/docker/docker-compose.yml#L32)
 to `3.1.5` it should be able to get through the tests until we can figure out 
a workaround for the [change](https://github.com/zeek/zeek/issues/977) which is 
causing the tests to exit due to pcap issues.  Sorry for the hassle with 
this... We plan to make this a non-issue as a part of future work.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev edited a comment on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-10-10 Thread GitBox

vikgmdev edited a comment on pull request #47:
URL:
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-706576041

Hey, @JonZeolla I encountered two issues when running the end to end
`docker/` tests...

The link to download the
[nitroba.pcap](http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap)
is no longer available:
```
--2020-10-10 11:26:40--
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
Resolving downloads.digitalcorpora.org (downloads.digitalcorpora.org)...
129.174.125.204
Connecting to downloads.digitalcorpora.org
(downloads.digitalcorpora.org)|129.174.125.204|:80... failed: Connection timed
out.
Retrying.
```
I commented that line to unblock me and keep with the test.

But after that, the script is crashing and throwing this error:
```
...
===
Configuring kafka plugin
configured the kafka plugin
MADE
/home/vick/Documents/Anubis/metron-bro-plugin-kafka/docker/test_output/sáb_10_oct_2020_11_32_09_CDT/exercise-traffic_pcap
Running docker_execute_process_data_dir with
CONTAINER_NAME = metron-bro-plugin-kafka_zeek_1
PCAP_FILE_NAME = exercise-traffic.pcap
OUTPUT_DIRECTORY_NAME = exercise-traffic_pcap
===
executing process_data_file.sh in the zeek docker container

PCAP_FILE_NAME = exercise-traffic.pcap
OUTPUT_DIRECTORY_NAME = exercise-traffic_pcap

WARNING: No Site::local_nets have been defined. It's usually a good idea to
define your local networks.
1258563753.320650 fatal error: failed to read a packet from
/root/data/example-traffic/exercise-traffic.pcap: truncated dump file; tried to
read 1434 captured bytes, only got 415
```
so not sure if it's an actual bug on the test or if it's just a step I
missed from my setup.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-10-10 Thread GitBox

vikgmdev commented on pull request #47:
URL:
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-706576041

Hey, @JonZeolla I encountered two issues when running the end to end
`docker/` tests...

PCAP_FILE_NAME = exercise-traffic.pcap
OUTPUT_DIRECTORY_NAME = exercise-traffic_pcap

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-10-09 Thread GitBox



JonZeolla commented on pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-705538753


   Thanks @vikgmdev looking good. Have you tested with the end to end `docker/` 
tests?  I think after the initial spin-up you'll need to configure the 
environment using `docker_execute_create_topic_in_kafka.sh` (because the topics 
don't auto-create on publish yet), update the zeek scripts with what you have 
in the README, and then re-run to ensure it works.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-10-08 Thread GitBox



JonZeolla commented on pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-705538753


   Thanks @vikgmdev looking good. Have you tested with the end to end `docker/` 
tests?  I think after the initial spin-up you'll need to configure the 
environment using `docker_execute_create_topic_in_kafka.sh` (because the topics 
don't auto-create on publish yet), update the zeek scripts with what you have 
in the README, and then re-run to ensure it works.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-10-01 Thread GitBox



vikgmdev commented on pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-702370870


   Hey, @JonZeolla just noticed I had the changes since a few months ago but 
haven't pushed them before. We'll wait for you re-review. Thanks.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-10-01 Thread GitBox



vikgmdev commented on a change in pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r498486281



##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+const stream_name: string = sub(stream_string, /::/, "_");
+
+# lowercase the whole string for nomalization
+const topic_name_lower: string = to_lower(stream_name);
+
+# remove the _log at the of each topic name
+const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+# replace `_` by `-` for compatibility with acceptable Kafka topic naes
+const topic_name: string = sub(topic_name_under, /_/, "-");
+
+if (|Kafka::logs_to_send| == 0 || stream_id in Kafka::logs_to_send)

Review comment:
   Hey, @JonZeolla I just improve this part of the example by removing the 
condition, it still works the same. I also had some confusions about that line 
`|Kafka::logs_to_send| == 0 || ` but now is all clear so it isn't need at all.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-09-23 Thread GitBox



vikgmdev commented on pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-697312918


   Hi @JonZeolla hadn't the chance to push the updates. Let me retake the issue 
and I'll push the updates soon. Thanks a lot for your patience.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-09-23 Thread GitBox



JonZeolla commented on pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-697271265


   Hi @vikgmdev thanks again for the contribution.  Do you have an update?  If 
not we will unfortunately need to mark this as abandoned



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #48: METRON-2360 BRO-PLUGIN: does not build with 3.2.1

2020-09-22 Thread GitBox



JonZeolla commented on pull request #48:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/48#issuecomment-696788728


   Tests are broken because `diff` isn't installed in our zeek docker image; we 
need to add `diffutils`.  Since it's only required for the tests which are run 
during `zkg install`, I suggest throwing it in the `zkg` `RUN` block.  After 
doing that I was able to get e2e all the way to the expected error due to the 
fix for zeek/zeek#977:
   ```bash
   1258563753.320650 fatal error: failed to read a packet from 
/root/data/example-traffic/exercise-traffic.pcap: truncated dump file; tried to 
read 1434 captured bytes, only got 415
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] asfgit closed pull request #48: METRON-2360 BRO-PLUGIN: does not build with 3.2.1

2020-09-22 Thread GitBox



asfgit closed pull request #48:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/48


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] asfgit closed pull request #48: METRON-2360 BRO-PLUGIN: does not build with 3.2.1

2020-09-22 Thread GitBox



asfgit closed pull request #48:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/48


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #48: METRON-2360 BRO-PLUGIN: does not build with 3.2.1

2020-09-22 Thread GitBox



JonZeolla commented on pull request #48:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/48#issuecomment-696788728


   Tests are broken because `diff` isn't installed in our zeek docker image; we 
need to add `diffutils`.  Since it's only required for the tests which are run 
during `zkg install`, I suggest throwing it in the `zkg` `RUN` block.  After 
doing that I was able to get e2e all the way to the expected error due to the 
fix for zeek/zeek#977:
   ```bash
   1258563753.320650 fatal error: failed to read a packet from 
/root/data/example-traffic/exercise-traffic.pcap: truncated dump file; tried to 
read 1434 captured bytes, only got 415
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards opened a new pull request #48: METRON-2360 BRO-PLUGIN: does not build with 3.2.1

2020-09-16 Thread GitBox



ottobackwards opened a new pull request #48:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/48


   Cleanup namespace and includes change to newer zeek.
   
   NOTE: new zeek does not like our pcaps used with docker testing.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] pbhalesain closed pull request #1583: Merge pull request #1 from apache/master

2020-08-07 Thread GitBox



pbhalesain closed pull request #1583:
URL: https://github.com/apache/metron/pull/1583


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] pbhalesain opened a new pull request #1583: Merge pull request #1 from apache/master

2020-08-07 Thread GitBox



pbhalesain opened a new pull request #1583:
URL: https://github.com/apache/metron/pull/1583


   merge with original
   
   ## Contributor Comments
   [Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]
   
   
   ## Pull Request Checklist
   
   Thank you for submitting a contribution to Apache Metron.  
   Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
   Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  
   
   
   In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be 
created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
   - [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   
   ### For code changes:
   - [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
   - [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
   - [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
 ```
 mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
 ```
   
   - [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] Have you verified the basic functionality of the build by building and 
running locally with Vagrant full-dev environment or the equivalent?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered by building and verifying the site-book? If not then run the 
following commands and the verify changes via 
`site-book/target/site/index.html`:
   
 ```
 cd site-book
 mvn site
 ```
   
   - [ ] Have you ensured that any documentation diagrams have been updated, 
along with their source files, using [draw.io](https://www.draw.io/)? See 
[Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 for instructions.
   
    Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   It is also recommended that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-06-12 Thread GitBox



JonZeolla commented on pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-643479496


   Hi @vikgmdev any update on this?  Thanks!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] sziszo commented on pull request #1576: METRON-2336 Stack advisor provides some components multiple times (santal)

2020-06-08 Thread GitBox



sziszo commented on pull request #1576:
URL: https://github.com/apache/metron/pull/1576#issuecomment-640424335


   @mmiklavc Is there anything further to do to merge?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-22 Thread GitBox



JonZeolla commented on a change in pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r429223738



##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+const stream_name: string = sub(stream_string, /::/, "_");
+
+# lowercase the whole string for nomalization
+const topic_name_lower: string = to_lower(stream_name);
+
+# remove the _log at the of each topic name
+const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+# replace `_` by `-` for compatibility with acceptable Kafka topic naes

Review comment:
   I'm fine with that but if that's the approach the above :: substitution 
should go direct to -





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] asfgit closed pull request #1582: METRON-2358:Fix typo in descriptions.

2020-05-22 Thread GitBox



asfgit closed pull request #1582:
URL: https://github.com/apache/metron/pull/1582


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] ottobackwards commented on pull request #1582: METRON-2358:Fix typo in descriptions.

2020-05-22 Thread GitBox



ottobackwards commented on pull request #1582:
URL: https://github.com/apache/metron/pull/1582#issuecomment-632665258


   +1 thanks!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-22 Thread GitBox



ottobackwards commented on a change in pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r429198652



##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+const stream_name: string = sub(stream_string, /::/, "_");
+
+# lowercase the whole string for nomalization
+const topic_name_lower: string = to_lower(stream_name);
+
+# remove the _log at the of each topic name
+const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+# replace `_` by `-` for compatibility with acceptable Kafka topic naes

Review comment:
   I think it complicates things, we want this to be as simple as possible.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-22 Thread GitBox



JonZeolla commented on a change in pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r429162878



##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+const stream_name: string = sub(stream_string, /::/, "_");
+
+# lowercase the whole string for nomalization
+const topic_name_lower: string = to_lower(stream_name);
+
+# remove the _log at the of each topic name
+const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+# replace `_` by `-` for compatibility with acceptable Kafka topic naes

Review comment:
   As an example it makes sense to leave but maybe be more clear on the 
situation.  Can you add a version pinned link to Kafka documentation or code 
into your comment?  





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] LeeJiangchuan opened a new pull request #1582: METRON-2358:Fix typo in descriptions.

2020-05-22 Thread GitBox



LeeJiangchuan opened a new pull request #1582:
URL: https://github.com/apache/metron/pull/1582


   ## Contributor Comments
   I modified some misspelled word. So this pull request is a trivial rework 
without any test coverage
   
   
   ## Pull Request Checklist
   
   Thank you for submitting a contribution to Apache Metron.  
   Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
   Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  
   
   
   In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be 
created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
   - [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   
   ### For code changes:
   - [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
   - [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
   - [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
 ```
 mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
 ```
   
   - [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] Have you verified the basic functionality of the build by building and 
running locally with Vagrant full-dev environment or the equivalent?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered by building and verifying the site-book? If not then run the 
following commands and the verify changes via 
`site-book/target/site/index.html`:
   
 ```
 cd site-book
 mvn site
 ```
   
   - [ ] Have you ensured that any documentation diagrams have been updated, 
along with their source files, using [draw.io](https://www.draw.io/)? See 
[Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 for instructions.
   
    Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   It is also recommended that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] ottobackwards commented on pull request #1581: [MINOR] study

2020-05-21 Thread GitBox



ottobackwards commented on pull request #1581:
URL: https://github.com/apache/metron/pull/1581#issuecomment-632196780


   Hi @LeeJiangchuan, thanks for the contribution.
   Can you change your pr title to start with the jira number?   METRON-.   
If you have not opened a jira for this change, we do require it.
   
   This is a good resource: 
https://cwiki.apache.org/confluence/display/METRON/Metron+Development+Environment+Setup+Instructions#MetronDevelopmentEnvironmentSetupInstructions-HowtoStartContributing
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-21 Thread GitBox



vikgmdev commented on a change in pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r428765442



##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+const stream_name: string = sub(stream_string, /::/, "_");
+
+# lowercase the whole string for nomalization
+const topic_name_lower: string = to_lower(stream_name);
+
+# remove the _log at the of each topic name
+const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+# replace `_` by `-` for compatibility with acceptable Kafka topic naes

Review comment:
   Yes, I referred to that but you're right it's more a personal 
preference, should I remove it from the example?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-21 Thread GitBox



vikgmdev commented on a change in pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r428759769



##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG

Review comment:
   On it, thanks.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-21 Thread GitBox



vikgmdev commented on a change in pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r428759024



##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.

Review comment:
   Oh, yup, I refer to **e.g.** sorry for the typo and I should improve my 
grammatic.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-21 Thread GitBox



ottobackwards commented on pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#issuecomment-632103636


   I have added Victor to the jira contributors role and assigned the jira to 
him



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-21 Thread GitBox



JonZeolla commented on a change in pull request #47:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/47#discussion_r428631726



##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+const stream_name: string = sub(stream_string, /::/, "_");
+
+# lowercase the whole string for nomalization
+const topic_name_lower: string = to_lower(stream_name);
+
+# remove the _log at the of each topic name
+const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+# replace `_` by `-` for compatibility with acceptable Kafka topic naes

Review comment:
   I assume you are referring to 
[this](https://github.com/apache/kafka/blob/d63e0181bb7b9b4f5ed088abc00d7b32aeb0/core/src/main/scala/kafka/admin/TopicCommand.scala#L198-L199)?
  Because otherwise, _ appears to be an allowed character in a topic name.

##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG

Review comment:
   Nit: Please keep the indentation consistent.

##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+const stream_name: string = sub(stream_string, /::/, "_");
+
+# lowercase the whole string for nomalization
+const topic_name_lower: string = to_lower(stream_name);
+
+# remove the _log at the of each topic name
+const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+# replace `_` by `-` for compatibility with acceptable Kafka topic naes

Review comment:
   naes typo

##
File path: README.md
##
@@ -179,6 +179,48 @@ event zeek_init() =-10
 }
 ```
 
+ Dynamically send each zeek log to a topic with its same name.
+
+ * ej. `CONN::LOG` logs are sent to the `conn` topic or `Known::CERTS_LOG` to 
the `known-certs` topic.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::logs_to_send = set(DHCP::LOG, RADIUS::LOG, DNS::LOG);
+redef Kafka::topic_name = "";
+redef Kafka::tag_json = T;
+
+event zeek_init() =-10
+{
+for (stream_id in Log::active_streams) {
+# Convert stream type enum to string
+const stream_string: string = fmt("%s", stream_id);
+
+# replace `::` by `_` from the log string name
+   # ej. CONN::LOG to CONN_LOG or Known::CERTS_LOG to Known_CERTS_LOG
+const stream_name: string = sub(stream_string, /::/, "_");
+
+# lowercase the whole string for nomalization
+const topic_name_lower: string = to_lower(stream_name);
+
+# remove the _log at the of each topic name
+const topic_name_under: string = sub(topic_name_lower, /_log$/, "");
+
+# replace `_` by `-` for compatibility with acceptable Kafka topic naes
+const topic_name: string = sub(topic_name_under, /_/, "-");
+
+if (|Kafka::logs_to_send| == 0 || stream_id in

[GitHub] [metron] LeeJiangchuan opened a new pull request #1581: [MINOR] study

2020-05-21 Thread GitBox



LeeJiangchuan opened a new pull request #1581:
URL: https://github.com/apache/metron/pull/1581


   ## Contributor Comments
   [Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]
   
   I modified a misspelled word. So this pull request is a trivial rework/ code 
cleanup without ant test coverage
   ## Pull Request Checklist
   
   Thank you for submitting a contribution to Apache Metron.  
   Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
   Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  
   
   
   In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be 
created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
   - [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   
   ### For code changes:
   - [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
   - [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
   - [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
 ```
 mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
 ```
   
   - [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] Have you verified the basic functionality of the build by building and 
running locally with Vagrant full-dev environment or the equivalent?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered by building and verifying the site-book? If not then run the 
following commands and the verify changes via 
`site-book/target/site/index.html`:
   
 ```
 cd site-book
 mvn site
 ```
   
   - [ ] Have you ensured that any documentation diagrams have been updated, 
along with their source files, using [draw.io](https://www.draw.io/)? See 
[Metron Development 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines)
 for instructions.
   
    Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   It is also recommended that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev opened a new pull request #47: METRON-2357: Extends example 4 with a dynamic version

2020-05-20 Thread GitBox



vikgmdev opened a new pull request #47:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/47


   ## Contributor Comments
   [Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]
   
   - Extends the current [Example 
4](https://github.com/apache/metron-bro-plugin-kafka#example-4---send-each-zeek-log-to-a-unique-topic)
  to include a dynamic version.
   
   It will demonstrate how to automatically send each zeek log to a topic with 
the same name.
   
   For instance the `CONN::LOG` log to be sent to the `conn` topic or 
`Known::CERTS_LOG` to the `known-certs` topic without defining a `Log::Filter` 
for each of those.
   
   
   ## Pull Request Checklist
   
   Thank you for submitting a contribution to Apache Metron's Bro kafka writer 
plugin.
   
   In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? If not one needs to be 
created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
   - [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   ### For code changes:
   - [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
   - [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
   - [ ] Have you ensured that the full suite of tests and checks have been 
executed via:
 ```
 bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH
 ```
   - [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] Have you verified the basic functionality of the build by building and 
running locally with Apache Metron's [Vagrant full-dev 
environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6)
 or the equivalent?
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on pull request #39: Feat/Add sensor_name to the tagged json

2020-05-20 Thread GitBox



vikgmdev commented on pull request #39:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/39#issuecomment-631637805


   Hey @ottobackwards, I saw the changed this morning, It's exactly what I was 
looking for thanks for that!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #39: Feat/Add sensor_name to the tagged json

2020-05-20 Thread GitBox



ottobackwards commented on pull request #39:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/39#issuecomment-631628406


   @vikgmdev https://github.com/apache/metron-bro-plugin-kafka/pull/46 has 
landed and will give you the functionality you are looking for I think



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] vikgmdev commented on pull request #39: Feat/Add sensor_name to the tagged json

2020-05-20 Thread GitBox



vikgmdev commented on pull request #39:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/39#issuecomment-631575681


   Hi @JonZeolla, to be honest, I don't know too much about C++ but It's a good 
opportunity to learn, so take me into account for contributions, do you have a 
backlog or from where I can start?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] asfgit closed pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

2020-05-19 Thread GitBox



asfgit closed pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

2020-05-19 Thread GitBox



JonZeolla commented on pull request #46:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631134572


   LGTM +1 thanks



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

2020-05-19 Thread GitBox



ottobackwards commented on pull request #46:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631089959


   https://issues.apache.org/jira/browse/METRON-2356 <-- @JonZeolla 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

2020-05-19 Thread GitBox



ottobackwards commented on pull request #46:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631090146


   I think that jira captures why the change is beyond this pr



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

2020-05-19 Thread GitBox



JonZeolla commented on pull request #46:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/46#issuecomment-631050619


   Looks good.  Noticed we are lacking btests - I'm fine with having that as a 
follow-on.  +1 pending JIRA creation



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on a change in pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

2020-05-19 Thread GitBox



ottobackwards commented on a change in pull request #46:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/46#discussion_r427535179



##
File path: README.md
##
@@ -224,6 +224,22 @@ event zeek_init() =-10
 }
 ```
 
+### Example 6 - Add static values to each outgoing Kafka message
+It is possible to define name value pairs and have them added to each outgoing 
Kafka json message when tagged_json is set to true.  Each will be added to the 
root json object.
+* the Kafka::additional_message_values table can be configured with each 
name and value
+* based on the following configuration, each outgoing message will have 
"FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME": 
"SECOND_STATIC_VALUE" added.
+```bash
+@load packages
+redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, 
FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, 
Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, 
RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);
+redef Kafka::topic_name = "zeek";
+redef Kafka::tag_json = T;
+redef Kafka::kafka_conf = table(["metadata.broker.list"] = 
"kafka-1:9092,kafka-2:9092");
+redef Kafka::additional_message_values = table(["FIRST_STATIC_NAME"] = 
"FIRST_STATIC_VALUE", ["SECOND_STATIC_NAME"] = "SECOND_STATIC_VALUE");

Review comment:
   follow on





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

2020-05-19 Thread GitBox



JonZeolla commented on a change in pull request #46:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/46#discussion_r427518350



##
File path: scripts/init.zeek
##
@@ -53,6 +53,14 @@ export {
 ["metadata.broker.list"] = "localhost:9092"
 ) 
 
+##  Key value pairs that will be added to outgoing messages at the root level
+##  for example:  ["bro_server"] = "this_server_name"

Review comment:
   Let's use `zeek` in examples instead of the old `bro`

##
File path: README.md
##
@@ -224,6 +224,22 @@ event zeek_init() =-10
 }
 ```
 
+### Example 6 - Add static values to each outgoing Kafka message

Review comment:
   The notes are specific to the prior example, so please shift this down 
in the README

##
File path: scripts/init.zeek
##
@@ -53,6 +53,14 @@ export {
 ["metadata.broker.list"] = "localhost:9092"
 ) 
 
+##  Key value pairs that will be added to outgoing messages at the root level
+##  for example:  ["bro_server"] = "this_server_name"
+##  will results in a  "bro_server":"this_server_name" field added to the 
outgoing

Review comment:
   Same as above comment

##
File path: README.md
##
@@ -224,6 +224,22 @@ event zeek_init() =-10
 }
 ```
 
+### Example 6 - Add static values to each outgoing Kafka message
+It is possible to define name value pairs and have them added to each outgoing 
Kafka json message when tagged_json is set to true.  Each will be added to the 
root json object.
+* the Kafka::additional_message_values table can be configured with each 
name and value
+* based on the following configuration, each outgoing message will have 
"FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME": 
"SECOND_STATIC_VALUE" added.
+```bash

Review comment:
   This is `zeek` not `bash`

##
File path: scripts/init.zeek
##
@@ -53,6 +53,14 @@ export {
 ["metadata.broker.list"] = "localhost:9092"
 ) 
 
+##  Key value pairs that will be added to outgoing messages at the root level
+##  for example:  ["bro_server"] = "this_server_name"
+##  will results in a  "bro_server":"this_server_name" field added to the 
outgoing
+##  json
+##  note this depends on tag_json being T
+const additional_message_values: table[string] of string = table(

Review comment:
   Can't we just shrink this to one line via `table() `?

##
File path: scripts/init.zeek
##
@@ -53,6 +53,14 @@ export {
 ["metadata.broker.list"] = "localhost:9092"
 ) 
 
+##  Key value pairs that will be added to outgoing messages at the root level

Review comment:
   Please align indentation with other items

##
File path: README.md
##
@@ -224,6 +224,22 @@ event zeek_init() =-10
 }
 ```
 
+### Example 6 - Add static values to each outgoing Kafka message
+It is possible to define name value pairs and have them added to each outgoing 
Kafka json message when tagged_json is set to true.  Each will be added to the 
root json object.
+* the Kafka::additional_message_values table can be configured with each 
name and value
+* based on the following configuration, each outgoing message will have 
"FIRST_STATIC_NAME": "FIRST_STATIC_VALUE", "SECOND_STATIC_NAME": 
"SECOND_STATIC_VALUE" added.
+```bash
+@load packages
+redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, 
FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, 
Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, 
RFB::LOG, Stats::LOG, CaptureLoss::LOG, SIP::LOG);
+redef Kafka::topic_name = "zeek";
+redef Kafka::tag_json = T;
+redef Kafka::kafka_conf = table(["metadata.broker.list"] = 
"kafka-1:9092,kafka-2:9092");
+redef Kafka::additional_message_values = table(["FIRST_STATIC_NAME"] = 
"FIRST_STATIC_VALUE", ["SECOND_STATIC_NAME"] = "SECOND_STATIC_VALUE");

Review comment:
   We should consider a more illustrative example, like:
   ```
   @if ( Cluster::is_enabled() )
   redef Kafka::additional_message_values = table(["CLUSTER_ENABLED"] = "True", 
["NODE_ID"] = Cluster::node);
   @else
   redef Kafka::additional_message_values = table(["CLUSTER_ENABLED"] = 
"False");
   @endif
   ```
   
   Warning: did not test.  May need to pull `node$id` for the NODE_ID value





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards opened a new pull request #46: METRON-2343 Bro Kafka plugin - ability to dynamically modify JSON

2020-05-19 Thread GitBox



ottobackwards opened a new pull request #46:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/46


   Add the ability to define static json fields and values to be added to
   the top level object when tagged_json is T.
   
   ### Testing
   - run_end_to_end.sh
   - look at any of the kafka output logs in the testing output directory for:
   "FIRST_STATIC_NAME" : "FIRST_STATIC_VALUE", 
"SECOND_STATIC_NAME":"SECOND_STATIC_VALUE"
   in the json.
   
   
   ## Pull Request Checklist
   
   Thank you for submitting a contribution to Apache Metron's Bro kafka writer 
plugin.
   
   In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? If not one needs to be 
created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
   - [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   - [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   ### For code changes:
   - [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
   - [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
   - [x] Have you ensured that the full suite of tests and checks have been 
executed via:
 ```
 bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH
 ```
   - [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
   - [-] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [-] Have you verified the basic functionality of the build by building and 
running locally with Apache Metron's [Vagrant full-dev 
environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6)
 or the equivalent?
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #39: Feat/Add sensor_name to the tagged json

2020-05-19 Thread GitBox



JonZeolla commented on pull request #39:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/39#issuecomment-630714126


   Hi, checking in to see how things are coming along and if you were 
interested in contributing to the package 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #40: METRON-2309 Add a Kafka "metadata.broker.list" for each log writer filter.

2020-05-19 Thread GitBox



JonZeolla commented on pull request #40:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/40#issuecomment-630712041


   As much as I would love to get this in, I have to mark it as abandoned.  
I'll be closing this soon if there's no response



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] ottobackwards commented on pull request #45: METRON-2353: Plugin e2e tests should support multiple brokers

2020-05-18 Thread GitBox



ottobackwards commented on pull request #45:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/45#issuecomment-630470979


   Great work @JonZeolla, thanks!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] asfgit closed pull request #45: METRON-2353: Plugin e2e tests should support multiple brokers

2020-05-18 Thread GitBox



asfgit closed pull request #45:
URL: https://github.com/apache/metron-bro-plugin-kafka/pull/45


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #45: METRON-2353: Plugin e2e tests should support multiple brokers

2020-05-18 Thread GitBox



JonZeolla commented on pull request #45:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/45#issuecomment-630408339


   @ottobackwards ready for review now.  In troubleshooting some issues running 
this up locally I found that I was hitting the consumer timeout; hence 
increasing it from 1000 to 5000 based on the additional resources needed for 
multiple brokers/partitions.  However, I found many other small bugs along the 
way which are fixed in this PR.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] asfgit closed pull request #1580: METRON-2348 fix prepare-commit

2020-05-17 Thread GitBox



asfgit closed pull request #1580:
URL: https://github.com/apache/metron/pull/1580


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] JonZeolla commented on pull request #1580: METRON-2348 fix prepare-commit

2020-05-17 Thread GitBox



JonZeolla commented on pull request #1580:
URL: https://github.com/apache/metron/pull/1580#issuecomment-629842633


   +1 stands



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] ottobackwards commented on pull request #1580: METRON-2348 fix prepare-commit

2020-05-17 Thread GitBox



ottobackwards commented on pull request #1580:
URL: https://github.com/apache/metron/pull/1580#issuecomment-629804321


   @JonZeolla one last check please



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron] JonZeolla commented on pull request #1580: METRON-2348 fix prepare-commit

2020-05-06 Thread GitBox



JonZeolla commented on pull request #1580:
URL: https://github.com/apache/metron/pull/1580#issuecomment-624664769


   +1



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on pull request #44: METRON-2270 Update to reflect bro project rename

2020-05-06 Thread GitBox



JonZeolla commented on pull request #44:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/44#issuecomment-624657441


   That's just a side effect of our merging process, which doesn't use the 
github built-ins.  This was squash merged as 
587e9dac9b915f7d0c1116a4432447b5d92e meaning it's in master.  At some point 
probably not too far out I expect to cut a 1.0.0 release



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [metron-bro-plugin-kafka] dcode commented on pull request #44: METRON-2270 Update to reflect bro project rename

2020-05-06 Thread GitBox



dcode commented on pull request #44:
URL: 
https://github.com/apache/metron-bro-plugin-kafka/pull/44#issuecomment-624639843


   @JonZeolla thanks for the heads up. @rocknsm does still use this plugin. 
Looks like this PR was closed without merging though? Is this still happening?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 3514 matches

Mail list logo