[Bro-Dev] Broker cluster discovery and load balancing
I've been working on an application that will fire a large volume of events into bro through broker. Basically, I want to fire events into Broker and load balance them across a pool of workers without the need for more than one worker to handle the same event. I'm having difficulty determining the most right way to accomplish a couple tasks. I have looked at the implementations of publish_hrw and publish_rr in bro. I could easily implement those features in my application if that is the recommended way to handle this issue. Unfortunately, that leaves me with another unfortunate problem. I have been unsuccessful in determining how to 'discover' members of a Bro cluster via Broker. Is there a way to do discovery, or do I need to know who the cluster members are and what port they are listening on via a broctl configuration equivalent? ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Config Framework Feedback
I agree that more complex code on my side would definitely make this a non-issue. In my case, I was setting up something in bro_init that would be used repeatedly throughout execution, and you're right--my code never handled option updates. In this case, I opted to just use old redefable consts, as they will perform exactly as expected for any downstream users (changes require a bro restart). Honestly, I only brought this up at all because I can definitely see a person new to scripting wondering why their options do not have the expected values at bro_init time and troubleshooting for hours. Maybe it would be worthwhile to just add a note in the documentation for the config framework about it and leave it at that? From: Azoff, Justin S Sent: Tuesday, October 30, 2018 3:41:00 PM To: Hosom, Stephen M Cc: bro-dev@bro.org Subject: Re: [Bro-Dev] Config Framework Feedback Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. > On Oct 30, 2018, at 2:09 PM, Hosom, Stephen M wrote: > > I bumped into a limitation that was a little frustrating to work around with > the config framework. > > > It is inconvenient that values stored in files read by adding to > Config::config_files are not available within the bro_init event. After > reviewing how the Config framework works, I understand why this is the case, > but it means that if you want to use configuration values on startup, you're > not guaranteed to be working with the anticipated value. > > > I think the introduction of an event that ensures that configuration files > have been read by the time that the event fires might be worthwhile. I was > wondering if anyone had any thoughts on how to resolve/work-around this issue. This is an issue, but probably not the one you are thinking of. even if configuration files were fully read by the time bro_init was ran, you'd still need to handle the value changing at runtime. If you need to run code when an option changes, you can use the Option::set_change_handler to make bro raise an event. Using this instead of bro_init would fix the startup problem, and handle it changing at any time. There's a test that uses this in testing/btest/core/option-priorities.bro and a few other files. — Justin Azoff ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Config Framework Feedback
I bumped into a limitation that was a little frustrating to work around with
the config framework.
It is inconvenient that values stored in files read by adding to
Config::config_files are not available within the bro_init event. After
reviewing how the Config framework works, I understand why this is the case,
but it means that if you want to use configuration values on startup, you're
not guaranteed to be working with the anticipated value.
I think the introduction of an event that ensures that configuration files have
been read by the time that the event fires might be worthwhile. I was wondering
if anyone had any thoughts on how to resolve/work-around this issue.
known.dat:
Known::KnownServers 10.230.21.220,10.230.21.221
try-config.bro:
module Known;
redef Config::config_files += {"Known.dat"};
export {
option KnownServers: set[addr] = set();
}
event bro_init() {
print KnownServers;
}
event bro_done() {
print KnownServers;
}
=== output ===
bro -r stream-1.pcap ./try-config.bro
{
}
{
10.230.21.220,
10.230.21.221
}
Thanks,
Stephen
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Moving to GitHub?
Well... in looking at how I set this up internally, I now realize that this feature is being EOL and will no longer work as of January 31, 2019. I guess it would have to be webhooks. On 5/16/18, 11:30 AM, "bro-dev-boun...@bro.org on behalf of Hosom, Stephen M" wrote: I too would miss the commit / change notifications, however, I think that this can be set up in GitHub in some way. We use Slack / Email internally for commit notifications. For Slack, we use webhooks to send notifications. Email commit notifications are actually a builtin integration in GitHub... You can set them up under Settings -> Integrations & Services -> Add service -> Email. On 5/15/18, 8:41 PM, "bro-dev-boun...@bro.org on behalf of Johanna Amann" wrote: Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. > What do people think? Any support, or concerns? I am in favor. The only thing I would miss are the immediate change notifications by email - I really like those... Johanna ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Moving to GitHub?
I too would miss the commit / change notifications, however, I think that this can be set up in GitHub in some way. We use Slack / Email internally for commit notifications. For Slack, we use webhooks to send notifications. Email commit notifications are actually a builtin integration in GitHub... You can set them up under Settings -> Integrations & Services -> Add service -> Email. On 5/15/18, 8:41 PM, "bro-dev-boun...@bro.org on behalf of Johanna Amann" wrote: Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. > What do people think? Any support, or concerns? I am in favor. The only thing I would miss are the immediate change notifications by email - I really like those... Johanna ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] 2.5.1 release?
Battelle's test cluster is currently on Master. I'll forward any issues that we have. -Original Message- From: bro-dev-boun...@bro.org [mailto:bro-dev-boun...@bro.org] On Behalf Of Robin Sommer Sent: Monday, May 15, 2017 11:18 AM To: Vlad Grigorescu Cc: bro-dev@bro.org Subject: Re: [Bro-Dev] 2.5.1 release? Message received from outside the Battelle network. Carefully examine it before you open any links or attachments. On Sat, May 13, 2017 at 00:28 -0500, you wrote: > We'll look at upgrading our test cluster (and UIUC's test cluster) to > master. Sounds good, let us know how that is going. Robin -- Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker data store API
The number of key/values would depend on the scale of the environment in the case of the authentication framework. In my last implementation... it was one record per user/host pair... which could scale into the tens of thousands of key/value pairs pretty quickly. I haven't looked at that stuff in a while since I'm eagerly awaiting your rewrite of the Broker APIs :) From: Matthias Vallentin [matth...@vallentin.net] on behalf of Matthias Vallentin [vallen...@icir.org] Sent: Monday, July 25, 2016 11:25 AM To: Hosom, Stephen M Cc: Siwek, Jon; bro-dev@bro.org Subject: Re: [Bro-Dev] Broker data store API > I can't speak to whether or not it is 'needed', but I have had desire > to use it in the past... The only thing preventing me from doing it > was the fact that Broker is currently a fast moving target. Good to know. Scott Campbell also uses the current version of Broker in his projects and mentioned the need for a scalable and performing storage backend. > Generally speaking, I was wanting to do it so that I could save state > between cluster restarts (specifically for authentication data). How many keys to you anticipate in your data store? And what's the rate of updates? Any ballpark estimate would be useful here. Given the interest in a scalable backend, I will bring back support for a RocksDB backend. Matthias ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker data store API
I can't speak to whether or not it is 'needed', but I have had desire to use it in the past... The only thing preventing me from doing it was the fact that Broker is currently a fast moving target. Generally speaking, I was wanting to do it so that I could save state between cluster restarts (specifically for authentication data). From: bro-dev-boun...@bro.org [bro-dev-boun...@bro.org] on behalf of Matthias Vallentin [vallen...@icir.org] Sent: Sunday, July 24, 2016 2:43 PM To: Siwek, Jon Cc: bro-dev@bro.org Subject: Re: [Bro-Dev] Broker data store API > Not sure about the choice of RocksDB in particular — could have just > been that it happened to pop up on people’s radar at the right time. It's certainly an industrial-strength key-value, so I think it's solid choice for those with better performance when needing persistence. > Given those historical reasons for it existing, would make sense to me > if it were temporarily ignored or removed completely (unless there’s > people already invested in using it). My plan was to put on hold for now, just to have less moving parts. It's great that you've already invested the time to understand the API and come up with an implementation. Same for SQLite. It took me only a day to convert your backend code and read up on SQLite here and there. I would imagine it will be the same for RocksDB. That said, adding backends is fortunately a quite mechanical task. It's easy to ship as an incremental release. I'm curious to find out what types of backends they would like to see and use once they build broker-enabled applications. Matthias ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
