[Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka
[ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1543: -- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > Kafka Logger - Writes Bro Logs to Kafka > --- > > Key: BIT-1543 > URL: https://bro-tracker.atlassian.net/browse/BIT-1543 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro >Reporter: Nick Allen >Assignee: Robin Sommer > > As part of the Apache Metron project, we needed a way to send Bro logs to > Kafka. From my research it seems like this is a common request. I'd rather > give this code back to the Bro community than maintain it as part of Apache > Metron. > This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as > simple as adding the following Bro script. > {{ > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > redef Kafka::topic_name = "bro"; > redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092" > ); > }} > This plugin has the following features. > * The user can specify a subset of all logs that should be sent to kafka. For > example, to only send conn, http, and dns logs, specify the following. > {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > }} > * Full configurability of Kafka connectivity. Any configuration setting > accepted by the librdkafka library can be passed to the plugin to tune how > the logs are sent to Kafka. > {{redef Kafka::kafka_conf = table( >["metadata.broker.list"] = "localhost:9092", >["client.id"] = "bro" > ); > }} > * The plugin will wait a configurable period of time (for example, 3 seconds) > after shutdown to attempt to send any queued messages to Kafka. > {{redef Kafka::max_wait_on_shutdown = 3000; > }} > * There are two message formats to choose from. By default, the standard Bro > JSON format is used. There is an alternative 'tagged JSON' format that is > provided by the plugin. Currently, all messages are sent to a single Bro > topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log > stream the message originated from. This format prepends the log stream > identifier to the JSON message. > {{{'conn': { ... }} > {'http': { ... }} > {'dns': { ... > To enable this alternative format, simply specify the following. > {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Fw: Broker raw throughput
Hi: Forwarding reply to the bro-dev list: original was mistakenly posted elsewhere (sorry about that). Leaving original message content inline for context. From: Matthias Vallentinon behalf of Matthias Vallentin Sent: Thursday, March 10, 2016 1:46 PM To: Clark, Gilbert Subject: Re: [Bro-Dev] Broker raw throughput > > Sorry if this is a stupid question, but what are the performance > > requirements for broker, exactly? > >We have too little experience to tell what we need Right, this was my question. The less that broker's requirements are documented and understood, the more difficult it becomes to evaluate whether or not broker will fit with an intended use-case, I think. To me, the performance numbers themselves don't matter as much as managing expectations does: should I *expect* to be able to pass all of my events through broker? >and where we hit bottlenecks. > > This is why we compare a worst-case scenario across >multiple libraries and see where we find low-hanging fruits for >optimization potential. Got it. I think that's what confused me ... >> That's *good enough* for most things, but it's also still quite >> possible to do better: I've built DSP applications on top of DPDK that >> do fine with ~14 Mpps, which is itself pretty slow when compared to >> results reported by e.g. [2]. > >It's not about going as fast as possible. We're looking to achieve good >performance in the common case, *without* reducing a high level of >abstraction. ... because some of those messaging libraries operate at different levels of abstraction than others, which is going to drive the performance to some extent ... > >> Realistically, depending on what broker is intended to support, maybe >> 200k messages / second is fine: > >Agreed. At this point, this rate is certainly fine for our >worst-case-single-int-per-message benchmarks. > >> TL/DR: I'm of the opinion that optimization is fun, but I also would >> feel kind of bad watching CAF go too far down a (very scary) rabbit >> hole just to support one (albeit very large and rather cool) >> application ... > >I don't think we want to do that either, this must be a >misunderstanding. The optimizations we've been looking at are >application-independent. ... so it looks like it was indeed a misunderstanding on my part. Sorry about that. Trying to express things a slightly different way, I was concerned that the different numbers from the different libraries were being interpreted as an apples-to-apples comparison. Modifying CAF to achieve the same results as e.g. 0mq would, at some point and in some way, eventually require modifying CAF to be more like 0mq. I don't think that would be good, because 0mq and CAF aren't (and shouldn't be, in my humble opinion) the same thing. > We have looked at a very specific workload to >bound worst-case performance. I'm very happy with the recent >improvements that will ship with CAF 0.15. Definitely. Performance improvements are always good :) Cheers, Gilbert ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly
[ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=24803#comment-24803 ] Justin Azoff commented on BIT-1545: --- The other thing to keep in mind is how this affects missed_bytes and capture loss. When I do shunting with the Arista I allow control packets through which lets most counters work, the only issue is the missed_bytes ends up being huge because bro thinks we are dropping all the packets. > SSH connection not recording entire flow correctly > -- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro >Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card >Reporter: Jason Carr >Assignee: Johanna Amann > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while > running with broctl but it does log to weird.log and ssh.log but nothing to > conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with > an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log > output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it > works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly
[ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=24802#comment-24802 ] Johanna Amann commented on BIT-1545: Yes, that was pretty much the outcome of our discussion. The SSH case is fixed now (the merged patch only removes the SSH analyzer - all counting stays intact), and I was mistaken about the other protocols, they do not do it. For external shunting (which is not part of Bro yet, but will be soon), we have a way to get some information from the switches (if they support that). I just have to get that into conn log. We also discussed that adding a character to the connection history for "connection was shunted" would be a good idea, to indicate that the numbers are only a guess. > SSH connection not recording entire flow correctly > -- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro >Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card >Reporter: Jason Carr >Assignee: Johanna Amann > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while > running with broctl but it does log to weird.log and ssh.log but nothing to > conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with > an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log > output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it > works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1540) Ifconfig is hardcoded in BroControl
[ https://bro-tracker.atlassian.net/browse/BIT-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=24801#comment-24801 ] Jon Schipp commented on BIT-1540: - I added support for the ip tool in branch topic/jschipp/broctl-ip-support > Ifconfig is hardcoded in BroControl > --- > > Key: BIT-1540 > URL: https://bro-tracker.atlassian.net/browse/BIT-1540 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl >Affects Versions: git/master >Reporter: Johanna Amann >Assignee: Jon Schipp > Fix For: 2.5 > > > From the mailing list: > {quote} > Hi Folks, > On later versions of Linux distros iproute2 replaces ifconfig with ip > Starting at line 601 at > https://github.com/bro/broctl/blob/master/BroControl/config.py > It looks like ifconfig is hard-written into the logic. Probably needs a > patch to check for the ip command. > Cheers, > Harry > {quote} > We should probably check for the presence of the ip utility and use that, if > present. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1545) SSH connection not recording entire flow correctly
[ https://bro-tracker.atlassian.net/browse/BIT-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=24800#comment-24800 ] Vern Paxson commented on BIT-1545: -- I'm definitely a fan of at least adding transparency that the value has not been properly tracked! It would also be good to understand in what shunting situations one can still afford to track such values; and I would hope that even if there's full (blind) shunting, the FIN/RSTs that terminate the connection are still captured, so one can make a guess based on sequence numbers. (Likewise, we'd want this annotated as a guess and not a directly measured value.) > SSH connection not recording entire flow correctly > -- > > Key: BIT-1545 > URL: https://bro-tracker.atlassian.net/browse/BIT-1545 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro >Affects Versions: git/master, 2.4 > Environment: Ubuntu 14.04 LTS, myricom 10g capture card >Reporter: Jason Carr >Assignee: Johanna Amann > Labels: logging > Fix For: 2.5 > > Attachments: ssh-port22.pcap > > > Making a connection out to a server via ssh does not write to conn.log while > running with broctl but it does log to weird.log and ssh.log but nothing to > conn.log. > While running bro -C -r ssh-port22.pcap, a partial log entry is listed with > an incorrect and very low number of packets and bytes. > It was determined that disabling the SSH analyzer gets the correct conn.log > output. > Analyzer::disable_analyzer(Analyzer::ANALYZER_SSH); > Testing on try.bro.org, 2.4+ and master has this problem but 2.3 and below it > works as expected. > Attached is the SSH connection outbound pcap. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py
[ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp updated BIT-1498: Description: When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. The patch is trivial: --- a/BroControl/ssh_runner.py +++ b/BroControl/ssh_runner.py @@ -108,6 +108,7 @@ class SSHMaster: self.base_cmd = [ "ssh", "-o", "BatchMode=yes", +"-q", host, ] self.need_connect = True was: When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. *test* The patch is trivial: --- a/BroControl/ssh_runner.py +++ b/BroControl/ssh_runner.py @@ -108,6 +108,7 @@ class SSHMaster: self.base_cmd = [ "ssh", "-o", "BatchMode=yes", +"-q", host, ] self.need_connect = True > add '-q' to ssh execution in ssh_runner.py > -- > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl >Affects Versions: 2.4 >Reporter: scampbell >Assignee: Jon Schipp >Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be > displayed in the broctl command. In the event that they can not be > configured away on the sshd end using '-q' avoids displaying the banner on > the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > +"-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py
[ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp updated BIT-1498: Description: When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. *test* The patch is trivial: --- a/BroControl/ssh_runner.py +++ b/BroControl/ssh_runner.py @@ -108,6 +108,7 @@ class SSHMaster: self.base_cmd = [ "ssh", "-o", "BatchMode=yes", +"-q", host, ] self.need_connect = True was: When using broctl in an environment with login banners, they will be displayed in the broctl command. In the event that they can not be configured away on the sshd end using '-q' avoids displaying the banner on the client side. The patch is trivial: --- a/BroControl/ssh_runner.py +++ b/BroControl/ssh_runner.py @@ -108,6 +108,7 @@ class SSHMaster: self.base_cmd = [ "ssh", "-o", "BatchMode=yes", +"-q", host, ] self.need_connect = True > add '-q' to ssh execution in ssh_runner.py > -- > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl >Affects Versions: 2.4 >Reporter: scampbell >Assignee: Jon Schipp >Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be > displayed in the broctl command. In the event that they can not be > configured away on the sshd end using '-q' avoids displaying the banner on > the client side. > *test* > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > +"-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1549) broctl top command doesn't work on OS X 10.10 or newer
Daniel Thayer created BIT-1549: -- Summary: broctl top command doesn't work on OS X 10.10 or newer Key: BIT-1549 URL: https://bro-tracker.atlassian.net/browse/BIT-1549 Project: Bro Issue Tracker Issue Type: Task Components: BroControl Reporter: Daniel Thayer On OS X Mavericks, the broctl top command was working, but on Yosemite (and El Capitan), it no longer works. The reason is that the "-stats vprvt" option of the top command always prints "N/A". -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py
[ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Schipp reassigned BIT-1498: --- Assignee: Jon Schipp > add '-q' to ssh execution in ssh_runner.py > -- > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl >Affects Versions: 2.4 >Reporter: scampbell >Assignee: Jon Schipp >Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be > displayed in the broctl command. In the event that they can not be > configured away on the sshd end using '-q' avoids displaying the banner on > the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > +"-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1549) broctl top command doesn't work on OS X 10.10 or newer
[ https://bro-tracker.atlassian.net/browse/BIT-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer updated BIT-1549: --- Fix Version/s: 2.5 > broctl top command doesn't work on OS X 10.10 or newer > -- > > Key: BIT-1549 > URL: https://bro-tracker.atlassian.net/browse/BIT-1549 > Project: Bro Issue Tracker > Issue Type: Task > Components: BroControl >Reporter: Daniel Thayer > Fix For: 2.5 > > > On OS X Mavericks, the broctl top command was working, but on Yosemite > (and El Capitan), it no longer works. The reason is that the > "-stats vprvt" option of the top command always prints "N/A". -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1498) add '-q' to ssh execution in ssh_runner.py
[ https://bro-tracker.atlassian.net/browse/BIT-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Thayer reassigned BIT-1498: -- Assignee: (was: Daniel Thayer) > add '-q' to ssh execution in ssh_runner.py > -- > > Key: BIT-1498 > URL: https://bro-tracker.atlassian.net/browse/BIT-1498 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BroControl >Affects Versions: 2.4 >Reporter: scampbell >Priority: Trivial > Labels: broctl > Fix For: 2.5 > > > When using broctl in an environment with login banners, they will be > displayed in the broctl command. In the event that they can not be > configured away on the sshd end using '-q' avoids displaying the banner on > the client side. > The patch is trivial: > --- a/BroControl/ssh_runner.py > +++ b/BroControl/ssh_runner.py > @@ -108,6 +108,7 @@ class SSHMaster: > self.base_cmd = [ > "ssh", > "-o", "BatchMode=yes", > +"-q", > host, > ] > self.need_connect = True -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-014#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [Auto] Merge Status
Open Merge Requests === IDComponentReporterAssignee Updated For VersionPrioritySummary --- -- -- - -- -- BIT-1547 [1] BroControl Justin AzoffJustin Azoff 2016-03-08 2.5 Normal broctl sets the same state variables over and over BIT-1543 [2] Bro Nick Allen Robin Sommer 2016-03-08 - Normal Kafka Logger - Writes Bro Logs to Kafka BIT-1507 [3] Bro Jan Grashoefer Seth Hall 2016-01-25 - Low Intel framework does not match mail addresses properly Open GitHub Pull Requests = Issue ComponentUserUpdated Title --- -- -- - #52 [4] bro J-Gras [5] 2016-01-18 Fixed matching mail address intel [6] #19 [7] bro-plugins nickwallen [8] 2016-03-09 [BIT-1543] Kafka Logger - Write Bro Logs to Kafka [9] #18 [10] bro-plugins jshlbrd [11]2016-03-03 SSDP analyzer [12] [1] BIT-1547 https://bro-tracker.atlassian.net/browse/BIT-1547 [2] BIT-1543 https://bro-tracker.atlassian.net/browse/BIT-1543 [3] BIT-1507 https://bro-tracker.atlassian.net/browse/BIT-1507 [4] Pull Request #52 https://github.com/bro/bro/pull/52 [5] J-Gras https://github.com/J-Gras [6] Merge Pull Request #52 with git pull --no-ff --no-commit https://github.com/J-Gras/bro.git topic/jgras/bit-1507 [7] Pull Request #19 https://github.com/bro/bro-plugins/pull/19 [8] nickwallen https://github.com/nickwallen [9] Merge Pull Request #19 with git pull --no-ff --no-commit https://github.com/nickwallen/bro-plugins.git add-kafka-plugin [10] Pull Request #18 https://github.com/bro/bro-plugins/pull/18 [11] jshlbrd https://github.com/jshlbrd [12] Merge Pull Request #18 with git pull --no-ff --no-commit https://github.com/jshlbrd/bro-plugins-1.git topic/jshlbrd/ssdp ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev