Re: [Bro-Dev] CBAN design proposal
On Thu, May 26, 2016 at 17:41 +, you wrote: > I imagined it being part of the nightly process that does quality/metric > gathering. Yeah, makes sense. > Is it an important use case for the client to be able to interact w/ > other repos that are structured like the one the Bro Team will be > hosting? Seems less likely that people will want to do that > especially if the CBAN repository is easy enough for people to submit Yeah, agree it's less important with that liberal model. I would still like to support it though unless it's a major pain. > But maybe there’s a slicker thing you could add to the Bro language > itself to let users specify a custom namespace mapping for certain > file paths allowing them to resolve conflicts themselves. Conflict reporting sounds good, renaming could get confusing. > The client's “update” command will do a “git pull” in the parent repo > and then a “git fetch” in any installed submodules. So does that mean the client ignores the version that the central CBAN parent repository points to? Say Seth finished version 1.0 of his cool-plugin and files a merge request. We add the submodule to CBAN; it will point to 1.0. Then Seth updates to 1.1 on his end. CBAN would still point to 1.0, but clients would just move their local clones to 1.1, ignoring the parent repository's state. Is that what you have in mind? Robin -- Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN design proposal
On Thu, May 26, 2016 at 15:57 +, you wrote: > But we are wrapping everything in the Bro plugin architecture, right? I think we'll need to play with that a bit still to see how exactly a minimal script module would be distributed. The binary plugins require a bit more structure, and are loaded a bit differently, than something that has just scripts would need. Maybe we can simplify the plugin layout a bit to support that case better, not sure. That said, even if using all the same structure, in my head I still associate "plugin" with one of the binary extensions (log writer, packet source) etc. and that's also how the documentation uses that term currently. Robin -- Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN design proposal
On Thu, May 26, 2016 at 14:07 -0700, you wrote: > Is it necessary to enforce this? Intuitively, I would just leave > namespacing to the user. No need to enforce, but it would be good to have some guidelines at least. And part of the guidelines should be keeping the "Bro" namespace reserved. > "extension" comes to mind, or maybe "bundle." I like "bundle". Robin -- Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN design proposal
Or just bro-ports On May 26, 2016, at 4:52 PM, Jan Grashöfer wrote: >> Here are some bro'ish suggestions (not all are serious ones): >> >> - brow (part of the logo already) >> - broom (sweep new code into bro) >> - broil (Bake your enhancements into bro) >> - broad (extends Bro in a broader sense) >> - broem (pleasing in a literary manner) >> - bropane (hot stuff for your bro) >> - brorito (original scripts from Mexico!) >> - brotein (makes you grow like a bro) >> - hebro (how about catering to Isreal?) >> - bronx (or to NYC?) >> >> And some random word forging: >> >> - bkg (instead of FreeBSD's pkg) >> - lion (animal I like) >> - crank (crank up that Bro) >> - scrit (tweak on "script") >> - berk (a Bro perk...or short for Berkeley) >> - bip (pip for Bro) >> - bang (starts with a "b" and sounds hip) >> - beco (Bro ecosystem) > > I like bkg, broil and brow. What about bob (helps you building up you > Bro ;)? > ___ > bro-dev mailing list > bro-dev@bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN design proposal
> Here are some bro'ish suggestions (not all are serious ones): > > - brow (part of the logo already) > - broom (sweep new code into bro) > - broil (Bake your enhancements into bro) > - broad (extends Bro in a broader sense) > - broem (pleasing in a literary manner) > - bropane (hot stuff for your bro) > - brorito (original scripts from Mexico!) > - brotein (makes you grow like a bro) > - hebro (how about catering to Isreal?) > - bronx (or to NYC?) > > And some random word forging: > > - bkg (instead of FreeBSD's pkg) > - lion (animal I like) > - crank (crank up that Bro) > - scrit (tweak on "script") > - berk (a Bro perk...or short for Berkeley) > - bip (pip for Bro) > - bang (starts with a "b" and sounds hip) > - beco (Bro ecosystem) I like bkg, broil and brow. What about bob (helps you building up you Bro ;)? ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Configurable &write_expire interval
Hi, I ran into an issue while trying to make the &write_expire interval configurable: Using a redefable constant does not work here, as the expression only gets evaluated when the table is initialized and thus later redefs do not influence the value. I thought about circumventing this by setting the value to 0 and maintain an extra variable to check against in my expire_func and return the right value. Unfortunately this won't work with write/read_expire as a write or read will reset the expiration to the initial value of 0. A solution could be to evaluate the interval expression every time it is used inside the table implementation. The drawback would be that there is no fixed value for serialization (I am not sure about the effects here). Another solution would be to provide a bif (or implement a language feature) to change the expire_time value from inside the expire_func. There was a somehow similar discussion about per item expiration (see http://mailman.icsi.berkeley.edu/pipermail/bro-dev/2016-April/011731.html) in which Robin came up with the solution of multiple tables with different expiration values. Again this would be a solution but doesn't feel right (duplicate code, static and somehow counterintuitive for the user). By the way: This is again about intel expiration. This time I thought I'll keep it simple and just have one user-defined expiration value for all items :D Maybe I am missing something regarding the loading sequence of scripts and this problem could be solved easier. So I am open for any suggestions or feedback! Best regards, Jan ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN design proposal
> - having both "upgrade" vs "update" commands sounds confusing, I'm > sure I would constantly mix up the two. Suggest to rename one of > them. I think this comes from Homebrew (and maybe other package managers as well). I kinda got used to it in this context, but occasionally still trip over it. So yeah, I also think we should use either "update" or "upgrade," but not both. > - What's the policy for namespaces (both script-land and plugin-land), > and what's the relationship to the CBAN path? Should people use > their GitHub name as namespace for their module? On the one hand > that would be an easy way to avoid conflicts. On the other hand that > makes forking difficult. Is it necessary to enforce this? Intuitively, I would just leave namespacing to the user. Clearly, to be interoperable, a convention could be using your github handle, but I don't see it being a necessity. If multiple devs want to collaborate and enhance the same namespace, we should not make this a cumbersome due to namespace hassles. > - Originally I wanted to suggest allowing more than one plugin per git > repository but I really like the idea to leverage submodules, so I'm > skipping that suggestion. :-) How do submodules scale? Or asked differently, what are the number (order of magnitude) of submodules we're expecting? > - Using a gist for meta data sounds good. We should then also have a > nice permanent *.bro.org URL redirecting there so that we hide the > actual location a bit for easier relocation. Is the idea that all meta data will reside in a single file? What format would that file be, YAML? If each user repo has 20 lines of meta data, then the file would be 2KB to track 100 repositories. Seems fine to me, but I wonder how much will end up in there to get a tractable upper bound. > - Any thoughts on distributing precompiled plugins? It would be nice > if, say, I could just install the Mac version of plugin XYZ without > having to compile it locally. The authors could optionally offer > selected prebuilds for whatever platforms they want to support. > Probably more a a feature for CBAN v2, but would be wort keeping > mind how binaries would fit in. Binaries are a big project, I believe. Not only do we need to support differently platforms, but also different OS versions and distributions. It would probably mean we have to invest into a bigger Jenkins setup, and then push the binaries into a CDN after a successful build. > - Terminology 1: agree that we should find a better name for CBAN. Here are some bro'ish suggestions (not all are serious ones): - brow (part of the logo already) - broom (sweep new code into bro) - broil (Bake your enhancements into bro) - broad (extends Bro in a broader sense) - broem (pleasing in a literary manner) - bropane (hot stuff for your bro) - brorito (original scripts from Mexico!) - brotein (makes you grow like a bro) - hebro (how about catering to Isreal?) - bronx (or to NYC?) And some random word forging: - bkg (instead of FreeBSD's pkg) - lion (animal I like) - crank (crank up that Bro) - scrit (tweak on "script") - berk (a Bro perk...or short for Berkeley) - bip (pip for Bro) - bang (starts with a "b" and sounds hip) - beco (Bro ecosystem) > The best alternative names I can come up with are "module" or > "package", which are also overloaded already, but at least more > generic. Of the two, I prefer "module" because "package" to me reminds me of a compiled thing, which is not always the case. Other than that, "extension" comes to mind, or maybe "bundle." > - We could create a public mailing list for CBAN for all discussions > of individual plugins/modules, including in particular for decisions > to remove something. Would be good to keep this all open and > transparent. ...and a Gitter channel, once we have converged on a name :-). Matthias ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN design proposal
> On May 26, 2016, at 10:46 AM, Robin Sommer wrote: > > - Let's keep BroControl integration in mind at leat. I agree that a > standalone client makes most sense right now, but if there's > something we can do that will make it easier for BroControl later to > integrate, that's worth preparing for. > > - One idea along those integration lines (BroControl and even > elsewhere): would be nice to structure the client so that the main > functionality resides in a Python module with a well-defined API. > The client itself would then become just a small command-line > frontend to that library. Ack. > - Where's that backend-side "out-of-band process" located? Is that > part of the client as well? Or maybe at least part of that Python > module? Would be nice to keep it easy to operate for CBAN-like > repositories that people maintain externally. I imagined it being part of the nightly process that does quality/metric gathering. The code to do it could be part of the client, but I expect the aggregation process could be time consuming enough that a user wouldn’t be happy w/ it. Especially if CBAN grows to contain many submodules, I don’t think it would scale well for each client to have to do the metadata aggregation themselves. Is it an important use case for the client to be able to interact w/ other repos that are structured like the one the Bro Team will be hosting? Seems less likely that people will want to do that especially if the CBAN repository is easy enough for people to submit to and the client can handle the case of adding individual external repos just fine. The metadata from individual, externally added repos would be searchable along with the bulk/aggregated metadata. e.g. all the submodules a user has locally installed have the latest metadata available, but all the submodules the user has not installed will fall back on the aggregated metadata that was fetched the last time they ran an “update” command. > - having both "upgrade" vs "update" commands sounds confusing, I'm > sure I would constantly mix up the two. Suggest to rename one of > them. I do confuse them too, but it’s also what some other package managers use (e.g. home-brew and apt-get). I’ll look at others package managers for more examples to see if I find something. > - What's the policy for namespaces (both script-land and plugin-land), > and what's the relationship to the CBAN path? Should people use > their GitHub name as namespace for their module? On the one hand > that would be an easy way to avoid conflicts. On the other hand that > makes forking difficult. Didn’t really have plans/ideas for a namespace policy, but checking for conflicts could be added to the list of things the automated nightly quality/metrics thing checks for. But maybe there’s a slicker thing you could add to the Bro language itself to let users specify a custom namespace mapping for certain file paths allowing them to resolve conflicts themselves. > - How do updating a plugin works now? If the author just updates the > remote code, the submodules won't move, so is there an additional > step there? The client's “update” command will do a “git pull” in the parent repo and then a “git fetch” in any installed submodules. Any submodules that are not installed are a no-op because they are just pointers that got updated by the “git pull”. This means the client should now be aware of updates to all the plugins they have installed, but are still using the old version until they choose to perform the “upgrade” command which then moves the installed submodules to the latest version. Or at least it will look something like that. I haven’t had a chance to test if there will need to be a separate “working copy” of submodules the user chooses to install. > - Any thoughts on distributing precompiled plugins? It would be nice > if, say, I could just install the Mac version of plugin XYZ without > having to compile it locally. The authors could optionally offer > selected prebuilds for whatever platforms they want to support. > Probably more a a feature for CBAN v2, but would be wort keeping > mind how binaries would fit in. Didn’t think much about it yet, but I expect a plugin to be able to define its own build steps. For precompiled plugins they could likely just do a no-op for the build step. Maybe there’s other install paths that need to get set up that I’m forgetting, but that shouldn't be too much extra to get working. Let me know if you had other thoughts that would make this more complicated than I’m expecting. > - Terminology 2: Using "plugin" as the entity name for everything is > confusing I think, as right now I think most people understand it as > something that gets compiled; nobody calls a script a plugin (unless > it's a plugin for a specific script-framework; but that's even more > confusing then :). The best alternative names I can come up with are > "module" or "package
Re: [Bro-Dev] CBAN design proposal
CBAN is a good name. It’s short, easy to say, and what the acronym means, the “Comprehensive Bro Archive Network”, makes sense. But I don’t really have a dog in this fight. -- Jeannette Dopheide Training and Outreach Coordinator National Center for Supercomputing Applications University of Illinois at Urbana-Champaign On 5/26/16, 10:57 AM, "bro-dev-boun...@bro.org on behalf of Slagell, Adam J" wrote: > On May 26, 2016, at 10:46 AM, Robin Sommer wrote: > > - Terminology 1: agree that we should find a better name for CBAN. BroForge? ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN design proposal
> On May 26, 2016, at 10:46 AM, Robin Sommer wrote: > > - Terminology 1: agree that we should find a better name for CBAN. BroForge? > > - Terminology 2: Using "plugin" as the entity name for everything is > confusing I think, as right now I think most people understand it as > something that gets compiled; nobody calls a script a plugin (unless > it's a plugin for a specific script-framework; but that's even more > confusing then :). The best alternative names I can come up with are > "module" or "package", which are also overloaded already, but at > least more generic. Other suggestions? > But we are wrapping everything in the Bro plugin architecture, right? I guess I I don’t think of plugins as necessarily binary, but maybe we shift nomenclature completely and start calling plugins packages across our documentation. But if we refer to the same thing as a plugin sometimes and a package other times, I think it gets even more confusing. > - We could create a public mailing list for CBAN for all discussions > of individual plugins/modules, including in particular for decisions > to remove something. Would be good to keep this all open and > transparent. I like the idea of a CBAN list. -- Adam J. Slagell Chief Information Security Officer Director, Cybersecurity Division National Center for Supercomputing Applications University of Illinois at Urbana-Champaign www.slagell.info "Under the Illinois Freedom of Information Act (FOIA), any written communication to or from University employees regarding University business is a public record and may be subject to public disclosure." ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN design proposal
On Wed, May 25, 2016 at 17:59 +, you wrote: > https://www.bro.org/development/projects/cban3.html Looks great to me. Unsorted thoughts/notes: - Let's keep BroControl integration in mind at leat. I agree that a standalone client makes most sense right now, but if there's something we can do that will make it easier for BroControl later to integrate, that's worth preparing for. - One idea along those integration lines (BroControl and even elsewhere): would be nice to structure the client so that the main functionality resides in a Python module with a well-defined API. The client itself would then become just a small command-line frontend to that library. - Where's that backend-side "out-of-band process" located? Is that part of the client as well? Or maybe at least part of that Python module? Would be nice to keep it easy to operate for CBAN-like repositories that people maintain externally. - having both "upgrade" vs "update" commands sounds confusing, I'm sure I would constantly mix up the two. Suggest to rename one of them. - What's the policy for namespaces (both script-land and plugin-land), and what's the relationship to the CBAN path? Should people use their GitHub name as namespace for their module? On the one hand that would be an easy way to avoid conflicts. On the other hand that makes forking difficult. - Originally I wanted to suggest allowing more than one plugin per git repository but I really like the idea to leverage submodules, so I'm skipping that suggestion. :-) - How do updating a plugin works now? If the author just updates the remote code, the submodules won't move, so is there an additional step there? - Using a gist for meta data sounds good. We should then also have a nice permanent *.bro.org URL redirecting there so that we hide the actual location a bit for easier relocation. - Any thoughts on distributing precompiled plugins? It would be nice if, say, I could just install the Mac version of plugin XYZ without having to compile it locally. The authors could optionally offer selected prebuilds for whatever platforms they want to support. Probably more a a feature for CBAN v2, but would be wort keeping mind how binaries would fit in. - Another maybe-v2 idea: if a plugin listed its configuration options in a well-defined manner, tools like BroControl could pick up on that and offer a configuration UI without peopling having to write script code. - Terminology 1: agree that we should find a better name for CBAN. - Terminology 2: Using "plugin" as the entity name for everything is confusing I think, as right now I think most people understand it as something that gets compiled; nobody calls a script a plugin (unless it's a plugin for a specific script-framework; but that's even more confusing then :). The best alternative names I can come up with are "module" or "package", which are also overloaded already, but at least more generic. Other suggestions? - We could create a public mailing list for CBAN for all discussions of individual plugins/modules, including in particular for decisions to remove something. Would be good to keep this all open and transparent. - I'm offended that my plugin is just "nice", but Seth's is *cool*! Robin -- Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Option -z
> If one > could express such analyses easily with a few lines of script code, > that would be quite powerful for doing script inspection that's also > easy to customize. Well sure, but it's not clear one can get to that point without some significant work under the hood anyway in terms of the features needed to make the script-level expression a few lines of code. Vern ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Option -z
On Thu, May 26, 2016 at 07:41 -0700, you wrote: > I wonder if they don't use it because it's not on their radar. It's > actually pretty handy, I see that in principle but hardcoding the functionality in C++-land doesn't seem to be the ideal way to go about things like this. If one could express such analyses easily with a few lines of script code, that would be quite powerful for doing script inspection that's also easy to customize. Robin -- Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Option -z
> Just removing this specific use > of finding NOTICEs, which doesn't seem anybody has been using in a > long time. I wonder if they don't use it because it's not on their radar. It's actually pretty handy, a way of telling when you think the set of NOTICEs should be X, but it's actually X'. Can help with writing documentation or finding dead code (of a form), or telling just what happens due to the hierarchy of @load's that a script pulls in. Vern ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Option -z
> On May 26, 2016, at 10:15 AM, Robin Sommer wrote: > > > > On Wed, May 25, 2016 at 20:56 -0700, you wrote: > >> Well it's there in CHANGES, per the appended. But yeah looks like it never >> went anywhere beyond the original instigation, so I think removing it is >> okay. > > Ah, I didn't realize this is what originally introduced the whole > traversal machinery. That infrastructure is used in a few places now, > and I'm not planing on touching that. Just removing this specific use > of finding NOTICEs, which doesn't seem anybody has been using in a > long time. > > Robin It also has a minor issue that prevents it from being more useful, it outputs AddressScan instead of the fully namespaced Scan::AddressScan -- - Justin Azoff ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Option -z
On Wed, May 25, 2016 at 20:56 -0700, you wrote: > Well it's there in CHANGES, per the appended. But yeah looks like it never > went anywhere beyond the original instigation, so I think removing it is okay. Ah, I didn't realize this is what originally introduced the whole traversal machinery. That infrastructure is used in a few places now, and I'm not planing on touching that. Just removing this specific use of finding NOTICEs, which doesn't seem anybody has been using in a long time. Robin -- Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
