Re: [Bro-Dev] Building bro 2.6 with static broker/caf libraries
Johanna has been working on this recently. We may end up needing to do a minor release update just to fix this because it appears that it's going to be a bit of a packaging problem on a number of OSs and distros otherwise. .Seth On 5 Dec 2018, at 22:03, Craig Leres wrote: > I've read up on cmake variable scope but I can't figure out how to > build > bro with static libraries. The bro-bundled caf already has > CAF_BUILD_STATIC_ONLY which I'm pretty sure works but I'm unable to > turn > it on when building caf as part of a bro build. > > For example I'd like is to optionally (i.e. from a -D argument to > cmake > itself) be able to turn on CAF_BUILD_STATIC_ONLY for > aux/broker/3rdparty/caf/CMakeLists.txt but nothing I've tried in the > top > level CMakeLists.txt is seen when the caf CMakeLists.txt is being > evaluated. > > (I'm working on updating the FreeBSD port to 2.6 and can't install > things like libcaf_io.so in /usr/local/lib because they conflict with > libraries potentially installed by the devel/caf port.) > > Craig > ___ > bro-dev mailing list > bro-dev@bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] attributes & named types
On 5 Nov 2018, at 11:40, Robin Sommer wrote: > I like the "=T|F" syntax to control this more directly, as long as > "" remains being equivalent to "=T". Oh, I like that too and it dovetails very nicely with the existing syntax. I remember those discussions we had about back then and I remember being the one that pushed for it but even then I didn't feel particularly comfortable with it (as I recall you feeling). .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] "bro" name is non-inclusive? (Contribution is rejected based )
Hi Vitaly! We renamed Bro to Zeek a couple of days ago. You should be good to reach out to the person that runs the ua-parser project and let them know so that your parser can be included on their webpage. If you need to give evidence of the name change, it's provided here: http://blog.bro.org/2018/10/renaming-bro-project_11.html Thanks for writing the uap plugin! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com > On Mar 9, 2018, at 12:55 AM, Vitaly Repin wrote: > > Hello, > > I would like to bring your attention to one strange and unexpected problem I > faced > while trying to include user agent parser for bro I have developed (plugin: > https://github.com/vitalyrepin/uap-bro ) into the list of uap implementations > https://github.com/ua-parser > > The req > -- > WBR & WBW, Vitaly > ___ > bro-dev mailing list > bro-dev@bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] DHCP event removal
On 18 Jun 2018, at 15:09, Alan Commike wrote: > With the default TSV, any change can break export into the various > back-end log stores and SIEMs. When adding new fields, it would be > nice to see them added to the end of the Info structure. This was a complete rework on the logs and scripts so the structure is completely different. Unfortunately it wasn't just one of the cases where a field or two was added. I don't think that assuming the order of fields is ever a safe assumption. It's why we shipped a version of bro-cut with Bro 2.0. We wanted to encourage people to refer to fields by the field name rather than the ordinal position of the field. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] DHCP event removal
Thanks Vlad! .Seth On 16 Jun 2018, at 9:07, Vlad Grigorescu wrote: > Yep, already working on it. :-) > > On Sat, Jun 16, 2018 at 6:26 AM, Seth Hall wrote: > >> >> On 15 Jun 2018, at 17:22, Azoff, Justin S wrote: >> >>> The fix is a little trickier, you can't handle both events because the >>> DHCP::Msg type no longer exists and you need to wrap the old event >>> with >>> >>> @ifdef (DHCP::Msg) >>> @endif >> >> Ugh, I didn't consider that. Bro has re-reached the point where >> touching any number of things can set off an avalanche of problems like >> this. >> >> Anyone on this thread up for submitting a patch which makes Bro cope >> with the changes automatically? You can then even mark the old events >> as deprecated. :) >> >> .Seth >> >> -- >> Seth Hall * Corelight, Inc * www.corelight.com >> ___ >> bro-dev mailing list >> bro-dev@bro.org >> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev >> -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] DHCP event removal
On 15 Jun 2018, at 17:22, Azoff, Justin S wrote: > The fix is a little trickier, you can't handle both events because the > DHCP::Msg type no longer exists and you need to wrap the old event > with > > @ifdef (DHCP::Msg) > @endif Ugh, I didn't consider that. Bro has re-reached the point where touching any number of things can set off an avalanche of problems like this. Anyone on this thread up for submitting a patch which makes Bro cope with the changes automatically? You can then even mark the old events as deprecated. :) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] DHCP event removal
On 15 Jun 2018, at 20:02, Michał Purzyński wrote: > Hey, I use the dhcp analyzer because i cannot count on our dhcp logs. > Not just that, I do some detection around it. How much trouble is it to migrate your scripts to what's in Bro master? .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] DHCP event removal
On 15 Jun 2018, at 0:30, Michael Dopheide wrote: > How was the determination made that it's not widely used? I don't > recall a survey on the bro/bro-dev lists and there's clearly instances > of it's use when searching github. I didn't dig around much which I made that change. Generally the DHCP analyzer hasn't been something that I've heard of much use (although Vlad did point out that I broke one of my own scripts!). How many scripts do you think it impacted? I suspect it's fairly few and we probably just need to put a change note in the release to say that developers will need to handle a new event to get the data. On the upside, you can handle both the old events and the new and they shouldn't impact each other (if you want to make a script work on multiple releases). .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Moving to GitHub?
On 15 May 2018, at 20:19, Robin Sommer wrote: > This has been coming up in various contexts & subgroups of people, and > I wanted to send it out as a proposal to gather some broader feedback: > Do we want to move Bro's git repositories and tickets to GitHub? I like the idea. It'll be nice to have one process for handling everything instead of the multiple avenues for tickets and merge requests that we have today. It seems that we should even be able to solve the issue with the immediate diff emails by using webhooks with an AWS lambda function. I'm not sure if we'd be able to do all of the git work from that though. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Final Broker branch testing
On 3 May 2018, at 17:20, Jon Siwek wrote: > Otherwise, I've stabilized some unit tests and made a merge request > [1] > for the broker branch. I'm really late on this, but congratulations Jon! That's super exciting and it's been a long slog to getting to this point but now I can't wait for you to be able to turn your attention to other stuff soon. :) Thanks! .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] timer delays between different events for same connection
On 13 Apr 2018, at 0:30, Aashish Sharma wrote: > So I am seeing some weird stuff in my sample pcap of scanners. May be > too > obvious and I am just not seeing why/how of it. It's a straight forward answer but not completely obvious. :) > Q. Why would connection_attempt event kick in after 36 minutes and 6 > seconds ? ( > 06:13:48 - 05:37:42 ) ? I bet that you have a jump in timestamps in your pcap. Since Bro's internal clock is driven forward by seeing timestamps associated with packets it's possible that your pcap has a 36 minute jump in timestamps so Bro couldn't have possibly expired anything in the time before that because from its perspective there was an immediate jump in time. You don't normally experience the effects of this behavior in traffic you're sniffing live because you will tend to have many packets every second so you see Bro's clock driven forward in very tiny increments as you would expect. If you go a long time without receiving a packet is when stuff gets tricky. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro SMB1 Issue in smb_cmd.log
This is probably a bug. That smb torture pcap is a notoriously bad example (although it does exhibit some far, far edge case type of behavior). I deliberately did not use that pcap as an example while I was writing the SMB analyzer because it sent me down a lot of rabbit holes that didn't provide much benefit for the first run at the SMB analyzer. If you identify the bug, please report back. My experience is that just running down these bugs to the exact failure can take quite a while. .Seth On 23 Feb 2018, at 10:09, Fernandez, Mark I wrote: Bro-Dev Group, ISSUE: I encountered an issue where Bro is not logging some rather significant SMB1 commands in the smb_cmd.log file. I understand that some SMB commands are deliberately omitted from the log (such as Negotiate Protocol, Session Setup, and Tree Connect); however, I observe that an instance of NT Create and Delete are not being recorded. I also understand that some SMB messages are deliberately omitted based on the status code; but the status codes ire STATUS_SUCCESS, so it should be logged. In this particular traffic sample, there are more than 100+ SMB messages going back and forth in the TCP stream, but only first several are recorded in smb_cmd.log, then it stops. Please help. Bro Version: I am using the Bro v2.5.1 docker image I pulled from the following URL: https://hub.docker.com/r/rsmmr/hilti/ PCAP File: I downloaded the "smbtorture" pcap file from the Wireshark public repository, at the URL: https://wiki.wireshark.org/SampleCaptures?action=AttachFile=get=sm btorture.cap.gz The issue I observe corresponds to stream #1 extracted from the file above, via filter: 'tcp.stream eq 1'. I attached a PCAP file containing stream #1 only. PCAP Analysis of SMB Messages: From the PCAP file, using Wireshark, the following sequence of SMB Messages are observed (summarized below as Request & Response pairs): (01) Negotiate Protocol Req & Resp (02) Session Setup AndX Req & Resp [x2] (03) Tree Connect AndX Req & Resp (04) Delete Req & Resp [file \torture_qfileinfo.txt] (05) NT Create AndX Req & Resp [fid 4000, file \torture_qfileinfo.txt] (06) Write AndX Req & Resp (07) Trans2 Req & Resp (08) Set Information2 Req & Resp (09) Query Information2 Req & Resp (10) Query Information Req & Resp (11) Query Information2 Req & Resp (12) Trans2 Req & Resp [x57] (13) Close Req & Resp [fid 4000] (14) NT Create AndX Req & Resp [fid 4001, file TORTUR~1.TXT] (15) Close Req & Resp [fid 4001] (16) Delete Req & Resp [file \torture_qfileinfo.txt -> formerly fid 4000] (17) Tree Disconnect Bro Analysis of smb_cmd.log: The Bro smb_cmd.log records events (04) - (10). I understand that events (01) - (03) are deliberately omitted from the log, but I am concerned that nothing is logged after event (10), Query Information Req & Resp. I think this is an important issue because the smb_cmd.log fails to record two significant events in this TCP stream: (i) A second file is created in step (14) (ii) The first file (create in step [05]) is deleted in step (16) The SMB messages look well-formed in Wireshark. Nothing seems to be wrong. The SMB status code is STATUS_SUCCESS for the requests and the responses, so it should be logged. Artifacts: Attached are the following artifacts to help you reproduce the issue: (a) ws_smbtorture_stream001.pcap (pcap of stream #1 only) (b) test.bro script (c) smb_cmd.log (d) smb_files.log (e) files.log (f) conn.log (g) packet_filter.log Not sure what is going wrong. Please help. Cheers, Mark ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Queueing in Broker?
On 13 Feb 2018, at 11:35, Robin Sommer wrote: > One more Broker idea: I'm thinking we should add a queuing mechanism > to Broker that buffers outgoing messages for a while when a peer goes > down. Once it comes back up, we'd pass them on. That way an endpoint > could restart for example without us loosing data. Yes! > I'm not immediately sure how/where we'd integrate that. For outgoing > messages, we could add it to the transparent reconnect. However, for > incoming connections, where the local endpoint doesn't have a notion > of "that peer should be coming back", it might not be as straight > forward? I can imagine being able to define queue length and queue (byte) size for consumers and producers might be interesting. As a producer: Keep up to 1000 messages and/or 1MByte of data. As a consumer: Only be willing to receive up to the 1000 most recent message or up to 1MByte of data. I still haven't spent time with the broker API to see if these thoughts actually make sense though. :) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Shipping CAF with Broker?
On 13 Feb 2018, at 11:31, Robin Sommer wrote: > We could even go a step further and compile CAF statically into > libbroker, so that in the end from a user's perspective all they deal > with is Broker: if they link against it, they get everything they > need. > > Would that make sense? I think we're quite a ways off from CAF being generally packaged with most operating systems (especially having the correct version to work with broker). I think that including it with broker and building libcaf into libbroker statically makes sense too. I've been concerned about the difficulty of building Bro from source too. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Logging TCP server banners
This fits with a feature that I've been talking to several people about
for quite a while which would make a bit of the beginning of each
direction of a connection available in script-land. That would help
with your problem a bit, but it sounds like since there is a particular
packet that you want, you may want to write your own analyzer that gets
the exact data that you are looking for because you should be able to do
packet level stuff easily there.
.Seth
On 12 Feb 2018, at 15:19, Pierre LALET wrote:
> Hi everyone,
>
> [This mail has been sent to bro@ first, but I think I might have more
> luck (and answers) here. Sorry for the inconvenience to those who have
> already read it.]
>
> For a network recon framework I am working on (https://ivre.rocks/ --
> for those interested), I would like to log each "TCP server banner"
> Bro sees.
>
> I call "TCP server banner" the first chunk of data a server sends,
> before the client has sent data (if the client sends data before the
> server, I don't want to log anything).
>
> Here is what I have done so far (`PassiveRecon` is my module's name):
>
> ```
> export {
> redef tcp_content_deliver_all_resp = T;
>
> [...]
> }
>
> [...]
>
> event tcp_contents(c: connection, is_orig: bool, seq: count, contents:
> string)
> {
> if (! is_orig && seq == 1 && c$orig$num_pkts == 2)
> {
> Log::write(PassiveRecon::LOG, [$ts=c$start_time,
>$host=c$id$resp_h,
>$srvport=c$id$resp_p,
>$recon_type=TCP_SERVER_BANNER,
>$value=contents]);
> }
> }
> ```
>
> Basically, I consider we have a "TCP server banner" when `is_orig` is
> false, when `seq` equals 1 and when we have seen exactly two packets
> from the client (which should be a SYN and the first ACK).
>
> This solution generally works **but** I sometimes log a data chunk
> when I should not, particularly if I have missed part of the
> traffic.
>
> As an example, the following Scapy script creates a PCAP file that
> would trick my script into logging a "TCP server banner" while the
> client has actually sent some data (and we have missed an ACK packet,
> left as a comment in the script):
>
> ```
> wrpcap("test.cap", [
> Ether() / IP(dst="1.2.3.4", src="5.6.7.8") /
> TCP(dport=80, sport=5678, flags="S", ack=0, seq=555678),
> Ether() / IP(src="1.2.3.4", dst="5.6.7.8") /
> TCP(sport=80, dport=5678, flags="SA", seq=111234, ack=555679),
> # Ether() / IP(dst="1.2.3.4", src="5.6.7.8") /
> # TCP(dport=80, sport=5678, flags="A", ack=111235, seq=555679),
> Ether() / IP(dst="1.2.3.4", src="5.6.7.8") /
> TCP(dport=80, sport=5678, flags="PA", ack=111235, seq=555679) /
> "DATA",
> Ether() / IP(src="1.2.3.4", dst="5.6.7.8") /
> TCP(sport=80, dport=5678, flags="PA", seq=111235, ack=555683) /
> "DATA"
> ])
> ```
>
> Is there a way to know that I have not missed any packet from the
> client and/or a way to know that the client has not sent any data on
> the connection (like an equivalent of the `seq` parameter, but for the
> `ack`)?
>
> Also, when `seq` equals 1, am I certain that I have not missed any
> packet from the server?
>
> One more question: is there a better, cleaner, etc. way to do what I'm
> trying to do?
>
> Thanks a lot,
>
> Pierre
>
> --
> Pierre
> http://pierre.droids-corp.org/
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
--
Seth Hall * Corelight, Inc * www.corelight.com
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro DCE-RPC Fix for AlterContext and AlterContextResponse Parsers
Cool, thanks Mark! .Seth On 8 Feb 2018, at 7:16, Fernandez, Mark I wrote: > Hi Seth, > > Yes, I have a pcap containing the ALTER_CONTEXT req/resp packets. I > will start working on the bug fix and submit to BIT, with pcap and > test script, hopefully soon. > > Cheers, > Mark > > -Original Message- > From: Seth Hall [mailto:s...@corelight.com] > Sent: Saturday, February 3, 2018 10:46 PM > To: Fernandez, Mark I <mfernan...@mitre.org> > Cc: bro-dev@bro.org > Subject: Re: [Bro-Dev] Bro DCE-RPC Fix for AlterContext and > AlterContextResponse Parsers > > > > On 2 Feb 2018, at 9:54, Fernandez, Mark I wrote: > >> 5. Bro Issue Tracker >> >> I plan to submit this to Bro Issue Tracker. Just wanted to give you >> a >> heads up here. > > Thanks Mark! Those were probably my comments. Unfortunately there > were > a number of areas where I just ran out of steam doing investigations > into why things were happening the way they were so this investigation > is deeply appreciated. > > Do you have PCAPs with ALTER_CONTEXT messages in them? Because this > is > difficult-to-understand change without seeing actual traffic it would > be > best if you were able to submit the changes along with tests. > > Thanks, >.Seth > > -- > Seth Hall * Corelight, Inc * www.corelight.com -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro DCE-RPC Fix for AlterContext and AlterContextResponse Parsers
On 2 Feb 2018, at 9:54, Fernandez, Mark I wrote: > 5. Bro Issue Tracker > > I plan to submit this to Bro Issue Tracker. Just wanted to give you a > heads up here. Thanks Mark! Those were probably my comments. Unfortunately there were a number of areas where I just ran out of steam doing investigations into why things were happening the way they were so this investigation is deeply appreciated. Do you have PCAPs with ALTER_CONTEXT messages in them? Because this is difficult-to-understand change without seeing actual traffic it would be best if you were able to submit the changes along with tests. Thanks, .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro DCE-RPC Analyzer Questions
On 31 Jan 2018, at 12:14, Fernandez, Mark I wrote: > Am I close to the right answer for sending data to the at-svc parser? Yep, I don't know the "best" answer, but I think you're on the right track with either direction that you outlined. If that code just magically existed with either of your outlined implementations I don't think we could resist merging it in. ;) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro DCE-RPC Analyzer Questions
The original idea was to get extensive parsing in place for DCE-RPC messages by parsing the IDL files for those services. Someone in the community had hoped to take it on, but hasn't had time yet to complete it. If you'd be interested in discussing that, I think that could be a huge addition to Bro (hundreds of new events!). The At service parsing file is still there because I didn't want to lose track of it but I think there was some slight architectural change that needed to happen before I could pass data to it. I don't think that data is even going to that parser, it's not just that there aren't events. I'd have to refer back to the code to see what exactly is wrong though. As for an approach to this problem right now, I'd prefer to see parsing done in the core. Architecturally we try to avoid passing unparsed data to Bro script land because of performance concerns and we generally don't have the intrinsic tools to be able to do parsing well in Bro scripts. .Seth On 25 Jan 2018, at 12:28, Fernandez, Mark I wrote: Bro-Dev Group, I am doing a little research into using Bro to log and analyze specific Microsoft DCE-RPC interfaces and methods. I notice that the Bro events for 'dce_rpc_request' and 'dce_rpc_response' provide the length of the RCP data stub (aka 'stub_len'). I found reference that these events previously provided a byte string containing the stub data itself, but at some point it was reduced to just the stub_len instead. I have a few questions that I hope you could answer: 1. What was the reason you decided to remove the stub data from the events and pass only the stub length? 1. On github, I see a BinPAC file for the RPC 'At' service (bro/src/analyzerprotocol/dce-rpc/endpoint-atsvc.pac), but there are no events generated by it. I think this would be very useful for my project. What is the reason that you have the analyzer, but no events for scriptland? 1. I have a use case, for a very few, limited number of RPC interfaces/methods, where I need to receive the stub data in scriptland for logging and analysis. How do you recommend I approach this scenario? I see a couple options: 1. I could customize the DCE-RPC analyzer to pass the sub data for *ALL* 'dce_rpc_request' and 'dce_rpc_response' events; or 2. I could customize the DCE-RPC analyzer to create new events specifically for the interfaces/methods (aka UUIDs/OpNums) that I care about. 3. Other ideas? I think both (a) and (b) will achieve the desired result; but there are trade-offs, pros and cons. I wonder which option would have a more negative impact on Bro performance? I imagine the reason you stopped passing stub data for all events was due to the performance hit, so I want to approach this in the best way possible. I appreciate your feedback. Cheers! Mark ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Merged branches deletion
Thanks for sweeping through all of these branches! I think I have a few extra branches that I could get rid of that haven't been merged too, this is a good reminder that I should get those cleaned up. .Seth On 30 Jan 2018, at 18:36, Johanna Amann wrote: > Hi, > > I am going to delete these (merged) branches thursday, unless someone > feels especially attached to them: > > topic/dnthayer/ticket1821 > topic/dnthayer/ticket1836 > topic/dnthayer/ticket1863 > topic/jazoff/contentline-limit > topic/jazoff/fix-gridftp > topic/jazoff/fix-intel-error > topic/jazoff/speedup-for > topic/robin/broker-logging > topic/robin/event-args > topic/robin/plugin-version-check > topic/seth/add-file-lookup-functions > topic/seth/input-thread-behavior > topic/seth/remove-dns-weird > topic/vladg/bit-1838 > > Johanna > ___ > bro-dev mailing list > bro-dev@bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] 'async' update and proposal
On 29 Jan 2018, at 12:00, Robin Sommer wrote: > Actually, I think one > possible solution has been floating around for a while already: event > *scopes* that express serialization requirements in terms of shared > context. I like this idea a lot! Do you foresee that causing trouble if we went that direction though? It seems like it could cause trouble by causing events to backup waiting for some other event to finish executing. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] SMB transaction messages pull request
Thanks Jon! I do apologize Jeffrey, the pull request was my responsibility and I've been meaning to get to it. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com > On Jan 2, 2018, at 11:58 AM, Jon Siwek <jsi...@corelight.com> wrote: > > On Fri, Dec 29, 2017 at 2:19 AM, Bencteux Jeffrey > <jeffrey.benct...@ssi.gouv.fr> wrote: >> I made a pull request a while ago to add/update messages for the SMB >> analyzer and I did not get no feedback. Is there something wrong with >> it? I'd be happy to modify it to fit your requirements if necessary. >> >> >> You can find it here : https://github.com/bro/bro/pull/119. > > Thanks for the PR, likely just no one had time to look yet. I'll do > that shortly. > > - Jon > ___ > bro-dev mailing list > bro-dev@bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Scientific notation?
On 6 Nov 2017, at 15:03, Jim Mellander wrote: > How about a user redef'able format string for doubles in logs? Even > more > flexible would be to make it a function. Let the user decide the > format > they need, and adapt their scripts accordingly, with the default being > the > current behavior. Ah, I like that idea. I think the current logs (both JSON formatted and normal Bro format) are fine for most people, but for the people that actually want doubles displayed differently this could give them that option. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Scientific notation?
Right now, Bro will print scientific notation in JSON logs but we've always tended to avoid it in the standard Bro log format. What does everyone think about switching to allow scientific notation in the standard log format? Daniel recently did some exploration of various versions of awk and they all support scientific notation (I think that was part of my concern a long time ago). Thoughts? .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] New CAF release for new Broker
On 29 Sep 2017, at 12:28, Robin Sommer wrote: > For those tracking the new Broker version in the topic/actor-system > branch: A new CAF release 0.15.4 is out now that's incorporating > everything that code requires, so no need to use the CAF "develop" > branch anymore. Woohoo! It's getting closer. :) .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] NETMAP plugin
I *finally* pushed out the Bro NETMAP plugin into the Bro package manager. This is the same thing that was in the old plugins repository that has now been deprecated. I would appreciate if people would try it if you use NETMAP. It can be installed like this: bro-pkg refresh bro-pkg install bro/bro-netmap You can read some more directions about how to use it in the repository here: https://github.com/bro/bro-netmap .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Configuration framework syntax proposal
On 22 Sep 2017, at 16:26, Jan Grashöfer wrote:
>> module Foo;
>>
>> export {
>>
>> ## The username for our new feature.
>> ##
>> ## Display: User Name
>> option user_name: string;
>>
>> }
>
> I really like that syntax!
I agree. That looks really nice!
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Configuration framework syntax proposal
I just wanted to note that after keeping up on this thread that I agree with those same points. :) .Seth On 21 Sep 2017, at 12:50, Johanna Amann wrote: >> I think that it's important to have this behavior come with a >> reasonable >> default. I think that whatever we choose should just work out of the >> box. >> For example, I think the existing construct should continue to work: >> >>> const user_name: string > > I agree; note that what I proposed preserves this compatibility (it > does > not change anything at all about redefs). The feature that the > configuration feature wants to bring is the ability to change options > during runtime - which cannot be accomplished with redefs. redef-able > consts will still have their place afterwards (for everything that > still > cannot be changed during runtime). > >> At the end of the day, what we're discussing is how a developer >> should >> document and expose a feature to an end-user. If, as a developer, I >> choose >> a bad variable name, then I'm not providing a good experience for the >> end-user, but that's my decision. I don't think that forcing >> developers to >> essentially add documentation via syntactic sugar is the right >> approach. If >> their variable names are confusing, people are less likely to use >> their >> script. >> >> I think that a lot of what users might want to re-configure is too >> complex >> to be explained through a variable name anyway. We already have a >> system in >> place to document variables, and I think that we need to rely on that >> instead of focusing so much on which name is exposed to the user. > > I agree with this. > >> As we look at moving Bro scripts to packages, I think we need to look >> at >> how other package repositories have handled similar configuration >> options. >> Puppet Forge, for instance, has a types tab which documents the names >> of >> the parameters, and what they do: >> https://forge.puppet.com/puppetlabs/mysql/types This would be pretty >> easy >> to do with the Broxygen documentation, and a UI could also expose >> this. > > Yup, I also agree with this. > > Johanna > ___ > bro-dev mailing list > bro-dev@bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Configuration framework syntax proposal
On 21 Sep 2017, at 4:17, Jan Grashöfer wrote: > While I agree that there are two (more or less) distinct groups and > that > the notion of configuration should be separated from the notion of > brogramming, I don't think that anyone would profit from introducing > something like ="just.another.name". Because of the additional > name mapping, this will make scripts harder to understand for > brogrammers (especially for beginners), whereas the benefit for users > would be minimal. I don't think that this proposal makes scripts any harder to understand for programmers than things are currently. From a programmers perspective, they would still use variables the same way they currently do. The only difference is that a user might not be using redef to change values, but the programmer doesn't see that happening anyway. It also doesn't change anything about what configuration values the programmer makes available since the programmer is already forced to expose their configuration through the export section. >> Invariably, there will be variable names chosen within software that >> will be short and convenient or long and explanatory but may not end >> up >> being just right if someone simply wants to configure a behavior. > > I think that would just be bad coding style. That's a fair statement. In an ideal world, all programmers would name their variables perfectly. I know that the opportunity for poor naming is still present with the attribute value, but for me at least, it puts me in a different frame of mind because this is the name that I'm exposing to users as a configuration option. It almost forces people to step out of the programming mentality. > So that is definitively a valid point! But instead of coming up with a > "new language element" I would prefer to add support for multi-level > namespaces into Bro. Yes, please do this! :) > That all said, what's about the poor users? Instead of throwing a huge > config file of key-value-pairs at someone who does not know about the > internals of the corresponding scripts, I would provide a UI to them > (maybe someone comes up with a bro-package...). The UI could display > all > options in a structured way and further support configuration by also > showing the corresponding documentation for each option. This would > allow a clean cut between users and brogrammers without intermingling > both worlds. Yep, this notion of making things abstract-able into easy configuration interfaces and/or good documentation (using the inline broxygen comments) was always in the proposal, Johanna pointed it out in the original code sample. There is just something about the idea of exposing variable names to users (even if it's wrapped in a gui) that is intensely unpalatable to me. It's pretty much unheard of among other types of software. It would be like exposing internal variable names to command line programs instead of abstracting it into easy flags (i.e. -a or --help) or, if in a gui a text entry box had a label next to it like "GUI::My_Program::user_name" instead of showing "Username". Sometimes abstraction like this isn't warranted, but I think it has to be done here. Bro needs to turn into a platform that treats users as first class citizens in the community and we need to acknowledge that there will be a day that they won't be reading script source code and they won't want to be exposed to programmer-isms. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Configuration framework syntax proposal
On 20 Sep 2017, at 20:19, Johanna Amann wrote: > technically - yes. Though I am not quite sure that I like it :). I also don't like it. I think with this proposal there is some recognition that our community has separate and distinct parts (and some overlap between them). The people that program Bro scripts and the people that use Bro scripts. I feel like there are benefits to getting a chance to separate the notion of configuration away from the notion of programming. Invariably, there will be variable names chosen within software that will be short and convenient or long and explanatory but may not end up being just right if someone simply wants to configure a behavior. There's also the problem of single level namespaces which will limit the expressiveness and depth that you could possibly give through configuration keys. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] ASCII response filetype
On 18 Sep 2017, at 16:43, Keith Lehigh wrote: > Hi Folks, > I’ve been mulling over an addition to the file mime type > signature that consists of “1 to 16 ASCII readable characters”. > 16 is an arbitrary length cutoff. The purpose of this signature would > be to log instances where a short status code is returned by a web > service. I see lots of responses like “[]” or “OK” or > “Success” and currently these are logged in files.log as unknown > file types. I think Bro would be improved by logging a filetype for > these responses. What about creating a mime type for an enumerated list of all of the ones you find? With a pattern like /^(OK|Success|0|1)$/ That was you could avoid other short responses from getting caught up in the net. I also suspect that [] should be something different because if you see that over HTTP, it's probably in most cases just an empty JSON array. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Adding a non-Bro script packages to bro-pkg
On 13 Sep 2017, at 12:54, Krenz, Mark Steven wrote: > Hello Bro devs. I was talking with Seth yesterday at Bro Con about how > an independent command line script for working with Bro logs could fit > into bro-pkg. Yeah, I think that including command line tools as a component of Bro Packages makes sense. I always forget how many little tools are laying around that various people have written to process logs. Having those in the central repository would be really nice. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] bro-pkg dependencies ?
On 8 Sep 2017, at 13:29, Aashish Sharma wrote: > Can we specify dependent packages in bro-pkg and would bro-pkg go and > resolve > (install) those dependencies by itself ? Yep. Here's one that does.. https://github.com/corelight/top-dns/blob/master/bro-pkg.meta > Also, can we make the bro-pkg dump some output (notes) before? or > after? pkg > installation - something like see this file for details etc ? bro-pkg tells you that it's going to install dependencies and asks if you want to proceed. If you want to see how it works, install the corelight/top-dns package. .Seth -- Seth Hall * Corelight, Inc * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] input-framework file locations
On 25 Aug 2017, at 16:56, Aashish Sharma wrote:
> global smtp_indicator_feed= fmt
> ("%s/feeds/smtp_malicious_indicators.out",@DIR) ;
>
> Problem is: @DIR gives the path of the directory where script is
> residing.
>
> So when I do broctl install - all the scripts go into :
> ../spool/installed-scripts-do-not-touch/
Huh, that's definitely a problem that I can see limiting people. What
you might want to do is reference a particular directory and having
instructions for people that they need to make it writable by the user
running the Bro process (and the directory could be redef-able).
Alternately, it looks like you're only using that to persist state
across executions. Is that right? If you're doing that, then you could
possibly get away with storing in $TMP.
Once Broker is in Bro, you can use Broker data stores to store and
retrieve your data.
.Seth
--
Seth Hall * Corelight, Inc * www.corelight.com
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] send_id (Re: [Bro-Commits] [git/bro] topic/jsiwek/actor-system: Finish port of control framework to use broker. (8dddae1))
I believe that Robin meant the intel framework instead of sumstats. (Hopefully this avoids some confusion) .Seth On Sat, Aug 26, 2017 at 11:12 AM Robin Sommer <ro...@icir.org> wrote: > Jon, replacing send_id() may indeed work better with an extension at > the C++/Broker level. I'd like to avoid introducing new dependencies > on Bro's serialization code, as I'm very much hoping that once the old > communication code code goes we won't need that serialization layer > anymore either (I know we're using it for opaque values over Broker > too, but that's quite contained and should be easy to replace). > > Some thoughts here: > > - I'm thinking the best approach may be a new Bro-specific >message for Broker, similar to the log-create/write messages. >We can add that to the Bro shim in Broker. It'd send the name >of the ID and the new value as a broker::data instance, and the >receiving Bro updates the value as the message comes in. > > - There's one larger problem with replacing send_id() though: the >old communication system has logic to send large values >incrementally, so that send_id() won't block stuff. As Seth >reminded me the SumStats framework is relying on that quite >extensively for sending tables around. Incremental operation is >something we don't have with Broker. I think that's ok, we can >replace the few existing use cases of sending large values with >something else. For SumStats that should probably be data >stores. I don't remember if there even are further instances of >this, my guess is no (I don't think we need to consider >broctl's configuration updating here; those values are small >and a non-incrememtnal send_id() is fine). > > - Another approach for broctl's updating could be switching to >the upcoming configuration framework, which takes a different >approach to dynamic reconfiguration. However, it's a bit out >still until we can that switch completely, so for now providing >a substitute for send_id() that can cover the simple uses cases >looks like the best way to go. > > Robin > > On Fri, Aug 25, 2017 at 13:15 -0500, Jonathan Siwek wrote: > > > commit 8dddae17db9340f5261d11382aa9b67e965d5fef > > Author: Jon Siwek <jsi...@illinois.edu> > > Date: Fri Aug 25 13:15:00 2017 -0500 > > > > Finish port of control framework to use broker. > > > > To replace usages of the send_id BIF, I had to make 3 new BIFs > > > > - serialize > > - deserialize > > - update_ID > > > > Using those, any Bro value can be explicitly converted to a string > > of bytes, sent to a peer via a Broker event, unserialized to a Bro > > val on the other side and then installed into a global identifier > > via its name. > > > > I think this may be the most straightforward way to get this to > > work for now, at least without changing significantly the Broker > > internals or messaging format. It mostly boils down to not being > > able to deserialize into a Bro value with the 'any' type, at least > > not without also carrying Bro's actual type information somewhere > > inside Broker's serialized message. > > > > And I think deserializing into 'any' would be needed because it's > > not possible to e.g. explicitly enumerate all possible types in a Bro > > script and have a particular event signature to use for any given > one. > > That's not possible because there's infinite ways you can create > > composite types (tables, sets, vectors, etc). > > -- > Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin > ___ > bro-dev mailing list > bro-dev@bro.org > http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev > -- Seth Hall * Corelight, Inc * s...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] clusterization issue: logger node vs manager node or both ?
On Thu, Jun 1, 2017 at 1:12 PM, Aashish Sharma <asha...@lbl.gov> wrote: > I can surely do "Cluster::local_node_type() == Cluster::LOGGER" and then > events logger2manager_events and logger2worker_events etc etc so on so forth. I *strongly* recommend not running code on the logger. The whole point of the logger is that it doesn't have any script execution tasks to take care of and it's solely dedicated to logging. What's the problem you're trying to solve by running code there? .Seth -- Seth Hall * Corelight, Inc * s...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] 2.5.1 release?
> On May 12, 2017, at 4:34 PM, Robin Sommer <ro...@icir.org> wrote: > > I just went through the CHANGES since 2.5 and > I can see snapshotting current master as 2.5.1 if folks here are fine > with doing a broader release (which will need a bit more testing than > just a bugfix update would). Any opinions? I'd be fine with that. I think master is quite stable right now anyway. .Seth -- Seth Hall * Corelight, Inc * s...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] can I send an opaque of bloomfilter over Cluster::manager2worker_event ?
> On May 1, 2017, at 2:41 PM, Aashish Sharma <asha...@lbl.gov> wrote: > > >>const global_hash_seed: string = "" > > Yes, with setting of global_hash_seed, bloomfilter movement across workers is > working fine and as expected, I see from initial tests. Speaking of this, I think we still need to have broctl autogenerate a file with this configured to a random value when it starts up (if that file doesn't already exist). That way everyones cluster will end up with a random value that stays consistent across restarts. .Seth -- Seth Hall * Corelight, Inc * s...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] ConfigurePackaging in plugins?
I'm casting around for thoughts on adding a mechanism to add the ConfigurePackaging cmake packaging mechanism to plugins without having to replicate the cmake script in the main cmake repository or making near-clones of it. Is there some way we could use that script from the main Bro repository without needing to include it with the plugin? .Seth ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] [desired broker api as oppose to whats in known-hosts.bro]
> On Mar 6, 2017, at 11:43 AM, Robin Sommer <ro...@icir.org> wrote: > > Whether we want to go further, like with Aashish's proposal, we > can see then, I'm still a bit undecided there. Either way, a better > Bro-side UI for Broker is coming, we just need to get the various > pieces in place first that people are working on currently before we > can move forward. I'm playing with rewriting some of the scripts for 2.6 right now with events because I think that the broker api is too complicated and could potentially have too many side effects. (like overuse of "when" for instance). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Bro packages with plugins
I was emailing with Justin this morning and he reminded me that we still have the issue with people that install Bro from binary packages and their inability to install packages with binary plugins. I don't know if I'll be able to kick start a brief discussion on this right now or not, but I'll throw a couple of ideas out there to see if any stick. - Include header files with the binary package installs. PROS - Solves the problem. Package users could install plugins. CONS - "Feels" nasty since we don't have a fully formalized API and we'd just have to include all Bro header files. - Requires users to have a compiler. - Create mechanism to release binary plugins in the bro package manager. PROS - Super easy installation for users. CONS - Here be (library linking) dragons. I don't know how many we could avoid with static linking. - We would probably have to create more project infrastructure to build packages. Other thoughts? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Splitting up init-bare?
> On Feb 10, 2017, at 1:03 PM, Johanna Amann <joha...@icir.org> wrote: > >> For the protocols, I could see having a file such as >> protocols/kerberos/bare.bro which defines the appropriate types which are >> currently in init-bare. > > That sounds like a good idea - I am not a big fan of the fact that a lot > of the protocol dependent datatypes are in init-bare currently. If we started structuring the analyzers internally more like external plugins with the scripts and everything in them, it would feel more comfortable to me. It seems like we'd be able to keep all of a protocol ephemera tied closely with it. Would that work? I know that internal and external plugins have some differences, but I don't know if that means we're limited in a bit in how we handle script land required data structures for analyzers. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker's remote logging (BIT-1784)
> On Jan 31, 2017, at 5:41 PM, Robin Sommer <ro...@icir.org> wrote: > > For RemoteSerializer, log events and log > filters run only on the originating nodes; those guys make all > decisions about what's getting logged exactly and they then send that > on to the manager, which just writes out the data it receives. Just to pile onto this, I think it should be this way too. I'd really like to avoid script code executing on the logger. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] [Bro] ActiveHTTP
> On Jan 28, 2017, at 9:15 AM, Dave Crawford <b...@pingtrip.com> wrote: > > And the second print doesn’t execute: > > $ bro -r test.pcap local ../test.bro > > Entering the ActiveHTTP::Request when() block... > > I have ‘exit_only_after_terminate’ set to true so it just hangs at this point > until I ctrl-c and I see the tmp files deleted. Following on this ticket from the main Bro list, I wonder if we could change the behavior of Bro slightly to make what Dave tried work? I *think* the problem here is that once the packets run out, Bro's internal clock stops moving forward which causes all sorts of trouble for timers and other stuff I'm sure. What does everyone think about making the clock continue to move forward even after the packet source runs dry? This especially makes sense when someone uses pseudo-realtime because we can keep moving the clock at the rate it was moving (but not jump to current time, we'd just do subtraction based on the time when the packet source ran dry). The main problem I see with this idea is if someone reads a PCAP at full speed, what rate do we make the clock continue ticking? Does this idea make sense at all? I think we've had too many new Bro programmers get frustrated with this behavior which worries me a little bit. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] plugins/hooks test fail in the new year
> On Jan 12, 2017, at 8:21 AM, Johanna Amann <joha...@icir.org> wrote: > > While I know this is cute, currently this will make the test fail on every > year > change. Do we perhaps want to either remove this, or at least move this > somewhere > outside of base/? We can remove it. The whole thing with the mugs kind of fell flat anyway. :( .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] [Proposal] Language extensions for better Broker support
> On Dec 15, 2016, at 8:01 PM, Robin Sommer <ro...@icir.org> wrote: > > On Thu, Dec 15, 2016 at 12:04 -0500, you wrote: > >> force people to cast the value to an error type, maybe like this... >> >> if ( v as error == Error::Success ) > > Mind elaboratong why you think we need to force the cast? That seems > to take away some of the appeal of the solution. I don't know. It just feels too magic otherwise, but I guess it's no different than what the system is already doing in it's normal automatic type casting situations so I don't think I can really complain here. >> I find "Error::Success" really unintuitive and kind of funny too. :) > > Yeah, agree, that's part of the question what namespace to use. > "Broker::Success" would certainly be nicer. Yep. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] [Proposal] Language extensions for better Broker support
> On Dec 14, 2016, at 10:27 AM, Robin Sommer <ro...@icir.org> wrote: > >local v = Broker::lookup(h, 42) # Continues to return 'opaque of > Broker::Data' > >if ( v == Error::Success ) >print (v as string); This is a neat trick and would be nice if all Bro types could be compared against error enums. > Seth, a question: do you have more in mind by "nicely generalized > error handling" than this? Do you see a way to generalize this to > purely script-level logic (i.e., no opaques, just normal Bro types > being passed around)? I don't have any concrete suggestions beyond or alternate to the one you suggested above. I actually really like the look but I think I need to try writing some code like that to see if I find issues with it. One thing I see immediately is that it seems like we're want to force people to cast the value to an error type, maybe like this... if ( v as error == Error::Success ) I find "Error::Success" really unintuitive and kind of funny too. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] [Proposal] Language extensions for better Broker support
> On Dec 13, 2016, at 2:51 PM, Matthias Vallentin <vallen...@icir.org> wrote: > >> Maybe the conversion to bool that I proposed originally should really >> be a conversion to a dedicated error type, so that one can >> differentiate what happened. > > I like that. I like that too. Having nicely generalized error handling in Bro would be such a huge benefit for script authors. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] bro-pkg support for centralized package structures?
> On Oct 24, 2016, at 10:22 AM, Siwek, Jon <jsi...@illinois.edu> wrote: > > I think supporting that would significantly change the design and goals of > the package manager, so I wonder what ideas/thoughts others have about the > current model vs. the proposed one? I don't think we should change the design of the package manager based on that use case. I think that the user is currently experiencing a moment in time where virtually no packages exist yet and assuming that no packages will exist in the future based on that fact. All of the packages that the user placed into their big script repository are scripts that hopefully will eventually be turned into packages and more easily installed. The user could still package all of their scripts into a single repository and have a script mechanism for choosing which scripts to actually load on different systems (i.e. @if directives choosing to load certain scripts on different systems, etc). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] bro-pkg -> bropkg
> On Sep 22, 2016, at 10:45 AM, Seth Hall <s...@icir.org> wrote: > > Yeah, I think changing bro-cut is too far gone at this point. I was only > asking about bro-pkg because it's so new and so few people have it installed. Quick follow up. I'm fine leaving things as-is. It's certainly easier and we can move on. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] bro-pkg -> bropkg
> On Sep 16, 2016, at 4:57 PM, Robin Sommer <ro...@icir.org> wrote: > > No strong preference either way on my end, but let's not touch bro-cut > please, that's used in too many scripts. Yeah, I think changing bro-cut is too far gone at this point. I was only asking about bro-pkg because it's so new and so few people have it installed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] ICAP Analyzer Design Guidance
> On Sep 21, 2016, at 5:03 PM, Fernandez, Mark I <mfernan...@mitre.org> wrote: > > Design guidance: is it acceptable to redefine/extend the ‘conn_id’ record as > described above? You probably don't want to extend the conn_id record. There are some cases where it can cause trouble doing lookups because the conn_id is used at a table index in a lot of places. Is there somewhere else you could stash the information that you need? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] bro-pkg -> bropkg
What does everyone think about making a change to rename bro-pkg to bropkg? I think we're early enough that we could probably get away with it and it fits with more with existing tools like broctl. I find it difficult to type the hyphen too for some reason. Thoughts? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Updating NEWS for 2.5
> On Aug 31, 2016, at 11:42 AM, Jan Grashöfer <jan.grashoe...@gmail.com> wrote: > >> Jan, would you mind if we posted this directly to the Bro blog? > > I thought that was the original idea. I would be glad if the post is > shared on the Bro blog once we have a final version. Hah, ignore me. I was reading Michal's email poorly. Everyone seems to be on the same page. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Updating NEWS for 2.5
> On Aug 30, 2016, at 8:10 PM, Michał Purzyński <michalpurzyns...@gmail.com> > wrote: > > Feedback shared privately. > > Come on, let's read it and polish it a bit. It's already very good! Let's wait until the 2.5 release. Seems like it might be a bit premature to post it now since the 2.5 release hasn't happened yet. :) Jan, would you mind if we posted this directly to the Bro blog? (obviously with all credit given to you!) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] SMB in master
> On Aug 8, 2016, at 7:11 PM, Robin Sommer <ro...@icir.org> wrote: > > How much of a performance impact would you expect? I'm seeing these > runtime increases (with a debug version): I don't really know what to expect from this in terms of performance yet. I've heard anecdotally that things are working ok, but I don't have any clue what would feel right or wrong. > [ 0%] coverage.bare-mode-errors ... failed Whoops. Fixed. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Kerberos changes
I just pushed some changes to the GSSAPI and Kerberos analyzers in the master branch. Bro is now handling kerberos carried over GSSAPI although I don't think there is any script support for how it's used yet so it's likely you won't see any change in your krb.log. If anyone sees any trouble, let me know. Thanks, .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] package manager progress
> On Jul 27, 2016, at 12:15 PM, Johanna Amann <joha...@icir.org> wrote: > > And to add a me three to this - I am also with him on this one. On top of > things - I might misremember this, but didn't we plan package names to > include the github user name at one point of time? So a package name would be > user/redis, for example, and there also could be user2/redis? I may have lost track of the design so I don't know where things stand now, but I think this would make sense too. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] package manager progress
> On Jul 25, 2016, at 4:49 PM, Azoff, Justin S <jaz...@illinois.edu> wrote: > > In one aspect the pktsrc- prefix acts like a tag, but can also help > disambiguate plugins... i.e., a redis log writer plugin vs. a redis data > store plugin vs. a redis protocol analyzer. I actually don't like this that much because some of these can cross boundaries and do all sorts of different things in a single plugin. It makes more sense to me to leave the naming open. If people want to name a plugin with a prefix, they're free to, but I wouldn't want to discourage people from maintaining individual plugins that provide a variety of features. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Remove application/pkix-cert from files.log?
What does everyone think of making some change for 2.5 so that certificates from SSL aren't logged in the files.log by default? I've heard grumblings about the number of certs that show up from quite a few people and personally noticed that the number of certificates will dwarf all other files types pretty badly which makes the output look a bit weird since very few people are ever interested in looking at those files in the files.log. Certificates would still be passed through the files framework, so it's not an architectural change, it would all be related to just not doing the log. There is one minor issue that this brings up though in that right now certificate hashes are all given in the files.log. We could move them elsewhere like x509.log or ssl.log, but I'm curious if anyone had thoughts on what they think would be most useful? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Unified scan.bro script
> On Jul 11, 2016, at 8:44 PM, Azoff, Justin S <jaz...@illinois.edu> wrote: > > It is.. amazing! The unified code is simpler, uses less memory, puts less > load on sumstats, generates nicer notice messages, and detects attackers > scanning across multiple victims AND ports. Nice job Justin! Perhaps this begs the question if we should use this version in Bro? We do have a tendency to make design decisions so that Bro works the best that it can with minimal configuration for even the largest sites. I think the notices are very reasonable and have the additional benefit of being a single noticed to watch for for "scanning". Having to watch for two different notices always felt a bit unnatural. I think that I personally care about scans, not the type of scan being performed (although there may be some nuance to that that someone is taking advantage of?). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro plugins + broctl plugins?
> On Jun 23, 2016, at 2:33 PM, Daniel Thayer <dntha...@illinois.edu> wrote: > > Could you specify what the problem is with the current implementation in git > master? I see it now. I'll close again. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro plugins + broctl plugins?
> On Jun 23, 2016, at 2:33 PM, Daniel Thayer <dntha...@illinois.edu> wrote: > > Could you specify what the problem is with the current implementation in git > master? Oh, I looked for a commit that implements this and couldn't find it. Adam didn't seem to indicate that the work had been done when he closed it either so I thought it might have been a mistake. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro plugins + broctl plugins?
> On Jun 23, 2016, at 12:52 PM, Slagell, Adam J <slag...@illinois.edu> wrote: > > https://bro-tracker.atlassian.net/browse/BIT-1551 I reopened this ticket since it looks like it was wrongly closed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Bro plugins + broctl plugins?
> On Jun 23, 2016, at 12:52 PM, Slagell, Adam J <slag...@illinois.edu> wrote: > > https://bro-tracker.atlassian.net/browse/BIT-1551 Great! Thanks Adam (and Daniel!). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Bro plugins + broctl plugins?
Has any movement been made on the ability to add broctl plugins into bro plugins? I know we talked about it a few times, and it's sort of becoming necessary are more packet source plugins are showing up in the bro-plugins repository. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] CBAN naming
> On Jun 6, 2016, at 3:09 PM, Siwek, Jon <jsi...@illinois.edu> wrote: > > If we switch the design to instead user the super-container format, then > that’s not an issue for me anymore because the relationship changes from “is > a plugin” to “may have a plugin”. I like the positioning of this because suddenly it suddenly feels very natural to explain the contents of a package (or whatever it ends up getting called). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] New proposal (Re: CBAN naming)
> On Jun 9, 2016, at 5:32 PM, Siwek, Jon <jsi...@illinois.edu> wrote: > > I like the “packages” + “package-manager” combo that Johanna suggests. I like that too. It feels nice and clean. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Flare removal
> On May 20, 2016, at 6:02 PM, Azoff, Justin S <jaz...@illinois.edu> wrote: > > I think Robins branch doesn't fix the problem because I don't think the > flares were really the issue.. I think bro started having issues because > between 2.3 and 2.4 traffic volumes increased, cluster sizes increased, and > we added a ton of new analyzers and log files which put even more strain on > the communication system. That's an interesting idea and at least sounds as plausible as anything else I've heard. Have you tried making those small changes you outlined by themselves on the NCSA dev cluster yet? I'd be curious to hear if that fixes the problem. Or do you even see this issue on that cluster? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Flare removal
For the 2.5 release, we were hoping to understand why the topic/seth/remove-flare fixes some issues that people have been seeing with the communication code. Perhaps even more to the point we are aiming to understand why that branch fixes the problem, but Robin's branch topic/robin/no-flares-2.4.1 doesn't work. The problem that we've seen will exhibit on Linux (for some reason FreeBSD doesn't seem to be affected) and you will see high memory use on the child of your manager process. People will tend to notice it in two ways. 1. Memory exhaustion 2. Logs being written that are seconds to minutes old. This isn't exactly a request for anyone to do anything, but more a call for anyone that would like to dig around in the core to figure out what is going on here so we can get a fix merged into master. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1581) topic/seth/stats-improvement
[ https://bro-tracker.atlassian.net/browse/BIT-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1581: --- Description: This branch is ready to be merged. It makes the "misc/stats.bro" much more useful by expanding the data exposed. It still writes to stats.log by default and is a normal Bro log (to avoid the problems traditionally encountered with the misc/profiling script). The focus is to only include data from simple counters to prevent this script from blocking the main thread since simple counters are quick and easy to collect. Notes: - The "resource_usage" bif is removed in favor of a larger number of bifs for collecting data. was: This branch is ready to be merged. It makes the "misc/stats.bro" much more useful by expanding the data exposed by it to include a lot more data. It still writes to stats.log by default and is a normal Bro log (to avoid the problems traditionally encountered with the misc/profiling script). The focus is to only include data from simple counters to prevent this script from blocking the main thread since simple counters are quick and easy to collect. Notes: - The "resource_usage" bif is removed in favor of a larger number of bifs for collecting data. > topic/seth/stats-improvement > > > Key: BIT-1581 > URL: https://bro-tracker.atlassian.net/browse/BIT-1581 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro >Affects Versions: git/master >Reporter: Seth Hall > > This branch is ready to be merged. It makes the "misc/stats.bro" much more > useful by expanding the data exposed. It still writes to stats.log by > default and is a normal Bro log (to avoid the problems traditionally > encountered with the misc/profiling script). The focus is to only include > data from simple counters to prevent this script from blocking the main > thread since simple counters are quick and easy to collect. > Notes: > - The "resource_usage" bif is removed in favor of a larger number of bifs > for collecting data. -- This message was sent by Atlassian JIRA (v1000.5.0#72002) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1581) topic/seth/stats-improvement
[ https://bro-tracker.atlassian.net/browse/BIT-1581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1581: --- Status: Merge Request (was: Open) > topic/seth/stats-improvement > > > Key: BIT-1581 > URL: https://bro-tracker.atlassian.net/browse/BIT-1581 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro >Affects Versions: git/master > Reporter: Seth Hall > > This branch is ready to be merged. It makes the "misc/stats.bro" much more > useful by expanding the data exposed by it to include a lot more data. It > still writes to stats.log by default and is a normal Bro log (to avoid the > problems traditionally encountered with the misc/profiling script). The > focus is to only include data from simple counters to prevent this script > from blocking the main thread since simple counters are quick and easy to > collect. > Notes: > - The "resource_usage" bif is removed in favor of a larger number of bifs > for collecting data. -- This message was sent by Atlassian JIRA (v1000.5.0#72002) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1581) topic/seth/stats-improvement
Seth Hall created BIT-1581: -- Summary: topic/seth/stats-improvement Key: BIT-1581 URL: https://bro-tracker.atlassian.net/browse/BIT-1581 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Seth Hall This branch is ready to be merged. It makes the "misc/stats.bro" much more useful by expanding the data exposed by it to include a lot more data. It still writes to stats.log by default and is a normal Bro log (to avoid the problems traditionally encountered with the misc/profiling script). The focus is to only include data from simple counters to prevent this script from blocking the main thread since simple counters are quick and easy to collect. Notes: - The "resource_usage" bif is removed in favor of a larger number of bifs for collecting data. -- This message was sent by Atlassian JIRA (v1000.5.0#72002) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Timing regression?
> On Apr 20, 2016, at 3:49 PM, Robin Sommer <ro...@icir.org> wrote: > > No, but I also didn't look further. Could it be the new file > identifications (i.e., the regexps)? That was my thought too. I'll have to look into DFA state creations to see if we've walked into that problem again. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Timing regression?
> On Apr 14, 2016, at 2:52 PM, Robin Sommer <ro...@icir.org> wrote: > > I just ran the external testsuite for current master on my development > system, and I'm seeing some quite increased execution times: >[ 71%] tests.ipv6 ... failed (+39.8%) >[ 85%] tests.m57-long ... failed (+9.8%) Did you ever happen to figure out what was going on with this? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Test suite failures
> On Apr 13, 2016, at 4:32 PM, Seth Hall <s...@icir.org> wrote: > > It was a macro expansion name conflict. Oops! Now I noticed that you committed into fast path! We did the same fix at least. I suppose we should revert your change out of fast path now. I'll take care of that. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Test suite failures
> On Apr 13, 2016, at 12:24 PM, Thayer, Daniel N <dntha...@illinois.edu> wrote: > > Some of the failures appear to be from the RFB analyzer commits from > yesterday that > now prevent Bro from building on FreeBSD 10. I'm just about done with the fix for that! :) It was a macro expansion name conflict. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1566) RFB (VNC) protocol analyzer
[ https://bro-tracker.atlassian.net/browse/BIT-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1566: --- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) Merged with commit 9d0899325a6a4391764cc541f4c41b4353ff79e6. Thanks Martin! > RFB (VNC) protocol analyzer > --- > > Key: BIT-1566 > URL: https://bro-tracker.atlassian.net/browse/BIT-1566 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BinPAC, Bro >Reporter: Martin van Hensbergen > Assignee: Seth Hall > Labels: analyzer > Fix For: 2.5 > > > We have created a RFB protocol analyzer used for example by VNC. It contains > a binpac analyzer, bro script and two testcases. > Can be found here: > https://github.com/martinvanhensbergen/bro/tree/topic/fox/rfb > 000540645dfc406074d2a2098418711348b98079 -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-030#72002) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1568) Add rtt field to dns.log
Seth Hall created BIT-1568: -- Summary: Add rtt field to dns.log Key: BIT-1568 URL: https://bro-tracker.atlassian.net/browse/BIT-1568 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: 2.5 Reporter: Seth Hall We should add a round trip time field to dns.log. This makes it much easier to discover misbehaving or poorly performing DNS servers. -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-030#72002) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1566) RFB (VNC) protocol analyzer
[ https://bro-tracker.atlassian.net/browse/BIT-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall reassigned BIT-1566: -- Assignee: Seth Hall > RFB (VNC) protocol analyzer > --- > > Key: BIT-1566 > URL: https://bro-tracker.atlassian.net/browse/BIT-1566 > Project: Bro Issue Tracker > Issue Type: Patch > Components: BinPAC, Bro >Reporter: Martin van Hensbergen > Assignee: Seth Hall > Labels: analyzer > Fix For: 2.5 > > > We have created a RFB protocol analyzer used for example by VNC. It contains > a binpac analyzer, bro script and two testcases. > Can be found here: > https://github.com/martinvanhensbergen/bro/tree/topic/fox/rfb > 000540645dfc406074d2a2098418711348b98079 -- This message was sent by Atlassian JIRA (v7.2.0-OD-05-030#72002) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Files transferred over FTP were showing incorrect sizes. (08399da)
> On Mar 11, 2016, at 5:33 PM, Robin Sommer <ro...@icir.org> wrote: > > This seems to be causing a number of baseline mismatches in the > external test suite. I can't tell if they are legitimate, did you run > the tests? Fixed. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Files transferred over FTP were showing incorrect sizes. (08399da)
> On Mar 11, 2016, at 5:33 PM, Robin Sommer <ro...@icir.org> wrote: > > > On Fri, Mar 11, 2016 at 12:56 -0500, Seth Hall wrote: > >>Files transferred over FTP were showing incorrect sizes. > > This seems to be causing a number of baseline mismatches in the > external test suite. I can't tell if they are legitimate, did you run > the tests? Ugh, I didn't run those tests. I'll look over those soon and do any updates necessary. I'm sure that the test baselines are improved though. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1544) File analysis code fails due to CheckString
Seth Hall created BIT-1544: -- Summary: File analysis code fails due to CheckString Key: BIT-1544 URL: https://bro-tracker.atlassian.net/browse/BIT-1544 Project: Bro Issue Tracker Issue Type: Problem Components: Bro Affects Versions: git/master Reporter: Seth Hall If a `get_file_handle` function returns a string with NULL bytes in it, it causes Bro to do a reporter error due to the file analysis using CheckString to generate the file ids. -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker updates?
> On Feb 23, 2016, at 10:07 PM, Matthias Vallentin <vallen...@icir.org> wrote: > > That's currently not supported, but it's on the wish list [1]. Ah, I haven't read through the broker extensions page in much detail yet. Thanks! .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Broker updates?
If there are updates to a Broker store, is there a way that I can get evented notification that a key was modified? I'm not seeing anything that provides that functionality yet, but I need it for something I'm working on. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] get_event_peer() with Broker
> On Feb 1, 2016, at 5:45 PM, Robin Sommer <ro...@icir.org> wrote: > > I think adding such meta information would be quite valuable, however > it's actually not trivial to do that, as it would change the signature > for incoming events across the whole Broker code base, including > language bindings etc. > > Any ideas? I was a little concerned that you'd fine something like this. Could we just do what I was suggesting last week where we try to see if we can get rid of the need for that? We don't use the remote peer name for much anyway and I think that it would be good for us if we stopped using it completely. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] SMB2 - NTLM GSSAPI messages continued
> On Feb 2, 2016, at 5:38 AM, Martin van Hensbergen > <martin.vanhensber...@fox-it.com> wrote: > > 1) do we all agree that the SMB_NTLM* functions should be renamed to NTLM* or > am I missing something? Agreed. > 2) What is the best way to generate a BifEvent with SMB header and all the > parsed user/domain/workstation values that were parsed deeper inside the > protocol layer? Just generate them with the connection record as an argument and we can tie together the various protocols at the script layer. That gives you the possibility to keep the clean abstraction in the core and all of the messy cross-structure stuff can happen in scripts. > Any help on this is much appreciated; especially if you think I am > overlooking a hidden can of worms somewhere ;-) >From what you've described here and in our off-list emails, I think you're on >the right track. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] get_event_peer() with Broker
> On Feb 2, 2016, at 11:54 AM, Robin Sommer <ro...@icir.org> wrote: > > I was thinking about that but do we really want to give up the ability > to tell where some information is coming from? We can just include that information in the notices directly on the worker. There is no reason it needs to be done on the manager. I think that giving up that information is reasonable and we do it. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Jenkins errors (Re: [Bro-Commits-Internal] UnitTests - Build # 6935 - Failure!)
> On Jan 28, 2016, at 11:56 AM, Robin Sommer <ro...@icir.org> wrote: > > On Thu, Jan 28, 2016 at 03:25 -0600, jenk...@brotestbed.ncsa.illinois.edu > wrote: > >> from >> /home/jenkins/workspace/UnitTests/bro/aux/plugins/elasticsearch/build/elasticsearch.bif.cc:4: >> /home/jenkins/workspace/CompileLeakCheck/bro/src/analyzer/../SerialObj.h:172:40: >> warning: override controls (override/final) only available with -std=c++11 >> or -std=gnu++11 >> virtual bool DoSerialize(SerialInfo*) const override; \ > > There are lots of these, which I don't quite understand as c++11 > should be on by default, no? Oh, is the elasticsearch plugin being built with C++11 enabled? .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] SMB2 - NTLM GSSAPI messages
> On Jan 25, 2016, at 11:17 AM, Martin van Hensbergen > <martin.vanhensber...@fox-it.com> wrote: > > 1) (pac level) Make a separate library of the parsing of the GSSAPI blob ( as > I think this is independent of whether SMB1 or SMB2 is used ), which returns > the parsed ASN1 structure when called. Then both the SMB1 and SMB2 parser can > use these functions. Yep, that's probably the right way. We never had enough time to get that integrated more cleanly. > 2) (bro script level) Make an ASN1 parser at the bro script level that does > the parsing there. I would not opt for this route as it probably would be to > slow and then we would have two places where this parsing is done. This is almost certainly not a great idea as you learned. :) .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1510) Crash reports when no crash happened
[ https://bro-tracker.atlassian.net/browse/BIT-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=23922#comment-23922 ] Seth Hall commented on BIT-1510: I'm actually not completely sure of all of the cases where broctl ends up sending a notification, but there are a number of cases where it sends reports when it shouldn't. > Crash reports when no crash happened > > > Key: BIT-1510 > URL: https://bro-tracker.atlassian.net/browse/BIT-1510 > Project: Bro Issue Tracker > Issue Type: Problem > Components: BroControl >Affects Versions: git/master > Reporter: Seth Hall > Fix For: 2.5 > > > We need to make broctl stop sending crash reports when Bro was shutdown by a > signal. It's confusing for users because they will get these emails > sporadically when restarting Bro. > The crash report typically has the following text and no backtrace: > stderr.log > KILLED > received termination signal -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1526) Radiotap support
[ https://bro-tracker.atlassian.net/browse/BIT-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1526: --- Status: Merge Request (was: Open) > Radiotap support > > > Key: BIT-1526 > URL: https://bro-tracker.atlassian.net/browse/BIT-1526 > Project: Bro Issue Tracker > Issue Type: Improvement > Components: Bro >Affects Versions: git/master > Reporter: Seth Hall > > Radiotap support so that Bro can understand packets captured from many > wireless interfaces. It even has a test that tests 802.11 headers with and > without QoS data.. > > The branch is ready to be merged and is named: topic/seth/radiotap -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1526) Radiotap support
Seth Hall created BIT-1526: -- Summary: Radiotap support Key: BIT-1526 URL: https://bro-tracker.atlassian.net/browse/BIT-1526 Project: Bro Issue Tracker Issue Type: Improvement Components: Bro Affects Versions: git/master Reporter: Seth Hall Radiotap support so that Bro can understand packets captured from many wireless interfaces. It even has a test that tests 802.11 headers with and without QoS data.. The branch is ready to be merged and is named: topic/seth/radiotap -- This message was sent by Atlassian JIRA (v7.1.0-OD-05-006#71001) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1524) Fixing compiler warnings
[ https://bro-tracker.atlassian.net/browse/BIT-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hall updated BIT-1524: --- Status: Merge Request (was: Open) > Fixing compiler warnings > > > Key: BIT-1524 > URL: https://bro-tracker.atlassian.net/browse/BIT-1524 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro >Reporter: Seth Hall > > The topic/seth/compiler-cleanup branch in the Bro repository and the Binpac > repository fix a set of compiler warnings currently showing up in Bro. Some > of them were introduced by moving to C++11. -- This message was sent by Atlassian JIRA (v7.1.0-OD-04-012#71001) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1523) ActiveHTTP module is broken
[
https://bro-tracker.atlassian.net/browse/BIT-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Seth Hall updated BIT-1523:
---
Resolution: Invalid
Status: Closed (was: Open)
This is only a problem if you allow Bro to shutdown immediately. We don't
always develop for that scenario since it represents such a minor use of Bro.
bro active-http-test.bro exit_only_after_terminate=T
> ActiveHTTP module is broken
> ---
>
> Key: BIT-1523
> URL: https://bro-tracker.atlassian.net/browse/BIT-1523
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
>Affects Versions: 2.4
> Environment: Mac OS X 10.11.2, Bro 2.4.1 (installed from Homebrew),
> curl 7.43.0
> CentOS 7.1.1503, Bro 2.4.1 (installed from OpenSuSE build service), curl
> 7.29.0
>Reporter: Derek Ditch
> Labels: bug, script
>
> When trying a very simple script found in Seth Hall's Bro Junk Drawer [1], I
> cannot get ActiveHTTP to even perform a successful GET request. I'd like to
> use ActiveHTTP to do RESTful POSTs for a notice or given behavior.
> CentOS output:
> {code}
> [vagrant@simplerockbuild ~]$ bro active-http-test.bro
> rm: cannot remove ‘/tmp/bro-activehttp-0ZTaA97EcF9_body’: No such file or
> directory
> {code}
> Mac OS X output:
> {code}
> $ bro active-http-test.bro
> rm: /tmp/bro-activehttp-nuJUYIMCT4e_body: No such file or directory
> rm: /tmp/bro-activehttp-nuJUYIMCT4e_headers: No such file or directory
> {code}
> [1]
> https://github.com/sethhall/bro-junk-drawer/blob/master/active-http-test.bro
--
This message was sent by Atlassian JIRA
(v7.1.0-OD-04-012#71001)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1521) known services should probably ignore gridftp-data
[
https://bro-tracker.atlassian.net/browse/BIT-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=23823#comment-23823
]
Seth Hall commented on BIT-1521:
Don’t put a set intersection function as a general utility function. We
probably shouldn’t be offering that generally right now since it should be
added to the language in a way that performs better. Since you’re only using
it in a single location, just do a loop.
> known services should probably ignore gridftp-data
> --
>
> Key: BIT-1521
> URL: https://bro-tracker.atlassian.net/browse/BIT-1521
> Project: Bro Issue Tracker
> Issue Type: Improvement
> Components: Bro
>Affects Versions: 2.4
>Reporter: Justin Azoff
>Assignee: Justin Azoff
>Priority: Low
>
> known services script does
> {code}
> if ( ! addr_matches_host(id$resp_h, service_tracking) ||
> "ftp-data" in c$service || # don't include ftp data sessions
> ("DNS" in c$service && c$resp$size == 0) ) # for dns, require
> that the server talks.
> return;
> {code}
> but should probably also ignore gridftp-data. Probably a good idea to add a
> set of services that behave like ftp for it to check.
--
This message was sent by Atlassian JIRA
(v7.1.0-OD-04-012#71001)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1515) Interface setup plug-in
[ https://bro-tracker.atlassian.net/browse/BIT-1515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=23802#comment-23802 ] Seth Hall commented on BIT-1515: Should probably port that to use "ip" since "ifconfig" is deprecated on Linux. > Interface setup plug-in > --- > > Key: BIT-1515 > URL: https://bro-tracker.atlassian.net/browse/BIT-1515 > Project: Bro Issue Tracker > Issue Type: Task > Components: Bro >Reporter: Jeannette Dopheide >Assignee: Justin Azoff >Priority: Low > > Place holder ticket to remind Justin to finish the interface setup plug-in he > has been working on. -- This message was sent by Atlassian JIRA (v7.1.0-OD-04-012#71001) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Better Handling of User Agents in Software Framework
> On Dec 15, 2015, at 11:23 AM, Robin Sommer <ro...@icir.org> wrote: > > Agree. Would be good to keep it in script-land unless it indeed has a > substantial impact (and if so, maybe there are some optimizations to > short-cut common cases or so). Yep, something along these lines was/is implemented in the core, but that just made it difficult to understand and make changes to. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Better Handling of User Agents in Software Framework
> On Dec 15, 2015, at 10:18 AM, Vlad Grigorescu <v...@grigorescu.org> wrote: > > The other question I was wondering about is: should this be a BIF? > Software::parse is a rather lengthy function, with a lot of string > manipulation, which gets called rather frequently. I suspect there'd be some > performance improvements for implementing this directly as a BIF. Ah, possibly. It probably would make sense to measure that first somehow. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Better Handling of User Agents in Software Framework
> On Dec 14, 2015, at 10:51 AM, Vlad Grigorescu <v...@grigorescu.org> wrote: > > I'm not thrilled with those user agents are being handled right now, and I'm > curious to get some thoughts. Take, for example the Safari user-agent string > of: I think your proposal sounds reasonable. I’d go ahead and implement it and see what you think about overload situations since I can easily see the amount of software being tracked quickly get out of hand with that. After it’s implemented, get it running on several networks that are willing to run it and see if it causes problems for them. :) This could be a good time to also implement some better handling around software tracking to avoid obvious DoS issues by doing traffic that causes lots of state being tracked. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
