Re: [Bro-Dev] Queueing in Broker?

[Jon Siwek]

On Tue, Feb 13, 2018 at 10:35 AM, Robin Sommer  wrote:
> One more Broker idea: I'm thinking we should add a queuing mechanism
> to Broker that buffers outgoing messages for a while when a peer goes
> down. Once it comes back up, we'd pass them on. That way an endpoint
> could restart for example without us loosing data.

If the goals is to prevent loss of data, then don't we need more than
just buffering, like message acknowledgements from the peer?

e.g. you can think your peer is up, send a message, then immediately
find out it went offline and so the message got lost "in the middle".

You would also need to keep the message buffered until receiving an
ACK from *all* peers that are subscribed (and the subscription list is
a potentially moving target) ?

And if you still planned on message routing/auto-forwarding being more
widely used, I think you would want to buffer the message while the
longest subscribed *path* has a down node?

> I'm not immediately sure how/where we'd integrate that. For outgoing
> messages, we could add it to the transparent reconnect. However, for
> incoming connections, where the local endpoint doesn't have a notion
> of "that peer should be coming back", it might not be as straight
> forward?

Yeah, I'm also unclear if there's anyway you can tell if the peer is
supposed to be permanent vs. transient in come cases.

Last observation is that I think any of these types of changes would
be to the internal messaging pattern/protocol and so maybe reasonable
to change/improve in subsequent releases in a way that's transparent
to users.

- Jon
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Queueing in Broker?

[Seth Hall]

On 13 Feb 2018, at 11:35, Robin Sommer wrote:

> One more Broker idea: I'm thinking we should add a queuing mechanism
> to Broker that buffers outgoing messages for a while when a peer goes
> down. Once it comes back up, we'd pass them on. That way an endpoint
> could restart for example without us loosing data.

Yes!

> I'm not immediately sure how/where we'd integrate that. For outgoing
> messages, we could add it to the transparent reconnect. However, for
> incoming connections, where the local endpoint doesn't have a notion
> of "that peer should be coming back", it might not be as straight
> forward?

I can imagine being able to define queue length and queue (byte) size 
for consumers and producers might be interesting.

As a producer:
Keep up to 1000 messages and/or 1MByte of data.
As a consumer:
Only be willing to receive up to the 1000 most recent message or up to 
1MByte of data.

I still haven't spent time with the broker API to see if these thoughts 
actually make sense though. :)

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Shipping CAF with Broker?

[Jon Siwek]

On Tue, Feb 13, 2018 at 10:31 AM, Robin Sommer  wrote:
> I was wondering the other day if we could add CAF as submodule to
> Broker

Sounds fine to me.  Also means one less variable (CAF version) to get
under control when troubleshooting/debugging reported issues.

- Jon
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Shipping CAF with Broker?

[Johanna Amann]

On 13 Feb 2018, at 8:36, Seth Hall wrote:

> On 13 Feb 2018, at 11:31, Robin Sommer wrote:
>
>> We could even go a step further and compile CAF statically into
>> libbroker, so that in the end from a user's perspective all they deal
>> with is Broker: if they link against it, they get everything they
>> need.
>>
>> Would that make sense?
>
> I think we're quite a ways off from CAF being generally packaged with
> most operating systems (especially having the correct version to work
> with broker).  I think that including it with broker and building 
> libcaf
> into libbroker statically makes sense too.  I've been concerned about
> the difficulty of building Bro from source too.

I agree - while I see the argument of not packaging external stuff, I 
think we should make an exception here.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] 'async' update and proposal

[Johanna Amann]

On 13 Feb 2018, at 7:44, Robin Sommer wrote:

> On Thu, Feb 08, 2018 at 10:01 -0800, Johanna wrote:
>
>> I just wanted to quickly chime in here to say that I generally like 
>> the
>> idea of having these contexts.
>
> Sounds like we all like that idea. Now the question is if we want to
> wait for that to materialize (which will take quite a bit more
> brainstorming and then implementation, obviously), or if we want to
> get async in in the current state and then put that on the TODO list?
> I can see arguments either way, curious what others think.

I have one obvious fear if we put it in now. Which is that it will be 
more or less put off forever :).

And we might end up with a pretty annoying situation where people (and 
we) start using it and you end up with all these really hard to debug 
corner-cases with event ordering. This is already kind of hard to do, 
especially with protocols that have a bunch of events - I managed to get 
this wrong a few times in the case of SSL where some parts of the 
connections where not logged correctly in some circumstances.

Basically - I fear that this will make it really hard to write scripts 
that work correctly in all cases.

Johanna
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] 'async' update and proposal

[Jon Siwek]

On Tue, Feb 13, 2018 at 9:44 AM, Robin Sommer  wrote:

> Sounds like we all like that idea. Now the question is if we want to
> wait for that to materialize (which will take quite a bit more
> brainstorming and then implementation, obviously), or if we want to
> get async in in the current state and then put that on the TODO list?
> I can see arguments either way, curious what others think.

Releasing the async feature by itself sounds unsafe.  Due to the "code
understandability" concern people had, it didn't sound like there was
a big demand to start using it without a solution to that anyway, and
releasing async by itself risks that solution falling through or being
delayed/unusable for an extended period, resulting in "weird code"
being written in the meantime and possibly never rewritten later on
because "it still technically works".

I'd maybe even go about implementing context/scope by branching off
the async branch, just in case there's details that arise that make it
less suitable than originally thought and have to go back to the
drawing board.

- Jon
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Shipping CAF with Broker?

[Azoff, Justin S]

> On Feb 13, 2018, at 11:36 AM, Seth Hall  wrote:
> 
> 
> 
> On 13 Feb 2018, at 11:31, Robin Sommer wrote:
> 
>> We could even go a step further and compile CAF statically into
>> libbroker, so that in the end from a user's perspective all they deal
>> with is Broker: if they link against it, they get everything they
>> need.
>> 
>> Would that make sense?
> 
> I think we're quite a ways off from CAF being generally packaged with 
> most operating systems (especially having the correct version to work 
> with broker).  I think that including it with broker and building libcaf 
> into libbroker statically makes sense too.  I've been concerned about 
> the difficulty of building Bro from source too.
> 
>   .Seth

This will also make

pip install pybroker

feasible, which would be a huge usability improvement for integrating external 
systems with bro.


— 
Justin Azoff


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Queueing in Broker?

[Robin Sommer]

One more Broker idea: I'm thinking we should add a queuing mechanism
to Broker that buffers outgoing messages for a while when a peer goes
down. Once it comes back up, we'd pass them on. That way an endpoint
could restart for example without us loosing data.

I'm not immediately sure how/where we'd integrate that. For outgoing
messages, we could add it to the transparent reconnect. However, for
incoming connections, where the local endpoint doesn't have a notion
of "that peer should be coming back", it might not be as straight
forward?

Robin



-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Shipping CAF with Broker?

[Seth Hall]

On 13 Feb 2018, at 11:31, Robin Sommer wrote:

> We could even go a step further and compile CAF statically into
> libbroker, so that in the end from a user's perspective all they deal
> with is Broker: if they link against it, they get everything they
> need.
>
> Would that make sense?

I think we're quite a ways off from CAF being generally packaged with 
most operating systems (especially having the correct version to work 
with broker).  I think that including it with broker and building libcaf 
into libbroker statically makes sense too.  I've been concerned about 
the difficulty of building Bro from source too.

   .Seth

--
Seth Hall * Corelight, Inc * www.corelight.com
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Shipping CAF with Broker?

[Robin Sommer]

I was wondering the other day if we could add CAF as submodule to
Broker, and then just start compiling it along with everything else. A
long time ago we decided generally against shipping dependencies along
with Bro, but in this case it might make people's lives quite a bit
easier, as hardly any Bro user will have CAF installed already. And
even if they had (say from an earlier Bro version), it might not be
the right version. If we included it with Broker, things would just
work.

We could even go a step further and compile CAF statically into
libbroker, so that in the end from a user's perspective all they deal
with is Broker: if they link against it, they get everything they
need.

Would that make sense?

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] 'async' update and proposal

[Robin Sommer]

On Thu, Feb 08, 2018 at 10:01 -0800, Johanna wrote:

> I just wanted to quickly chime in here to say that I generally like the
> idea of having these contexts.

Sounds like we all like that idea. Now the question is if we want to
wait for that to materialize (which will take quite a bit more
brainstorming and then implementation, obviously), or if we want to
get async in in the current state and then put that on the TODO list?
I can see arguments either way, curious what others think.

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Logging TCP server banners

[Pierre LALET]

Hi,

On Tue, Feb 13, 2018 at 09:15:21AM +0100, Bencteux Jeffrey wrote:
> A solution could be to blacklist such connections, i-e if there is data
> sent by the client, then do not log:
> > if (! is_orig && seq == 1 && c$orig$num_pkts == 2 && c$orig$size == 
> > 0)
> 
> Another thing that comes to me is what if you miss the SYN or the
> SYN-ACK segment sent by your client? You will not log the banner so I am
> not sure about the second condition : c$orig$num_pkts == 2. I would
> remove it.

Thanks! Indeed, changing `c$orig$num_pkts == 2` to `c$orig$size == 0`
is a good move, I wish I had this idea!

> With the pcap generated with the scapy script you gave, I do not log
> anymore, however if I change it to this:
> 
> wrpcap("test.cap", [
> Ether() / IP(dst="1.2.3.4", src="5.6.7.8") /
> TCP(dport=80, sport=5678, flags="S", ack=0, seq=555678),
> Ether() / IP(src="1.2.3.4", dst="5.6.7.8") /
> TCP(sport=80, dport=5678, flags="SA", seq=111234, ack=555679),
> Ether() / IP(dst="1.2.3.4", src="5.6.7.8") /
> TCP(dport=80, sport=5678, flags="A", ack=111235, seq=555679),
> # Ether() / IP(dst="1.2.3.4", src="5.6.7.8") / no more data sent by the 
> client
> # TCP(dport=80, sport=5678, flags="PA", ack=111235, seq=555679) / "DATA",
> Ether() / IP(src="1.2.3.4", dst="5.6.7.8") /
> TCP(sport=80, dport=5678, flags="PA", seq=111235, ack=555679) / "DATA"
> ])
> 
> I do have an entry in the log.
> 
> > Also, when `seq` equals 1, am I certain that I have not missed any
> > packet from the server?
> 
> No idea about that, I think the answer is in Bro's TCP implementation in
> src/analyzer/protocol/tcp somewhere.

I think, as suggested by Seth Hall, that I would have to write my own
analyzer for that.

Thanks a lot,

Pierre

-- 
Pierre
http://pierre.droids-corp.org/
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Logging TCP server banners

[Pierre LALET]

Hi,

On Mon, Feb 12, 2018 at 05:18:05PM -0500, Seth Hall wrote:
> This fits with a feature that I've been talking to several people
> about for quite a while which would make a bit of the beginning of
> each direction of a connection available in script-land.

I think that would be great!

> That would help with your problem a bit, but it sounds like since
> there is a particular packet that you want, you may want to write
> your own analyzer that gets the exact data that you are looking for
> because you should be able to do packet level stuff easily there.

I wanted to avoid that, but actually I think you're right.

Thanks for your answer,

Pierre

-- 
Pierre
http://pierre.droids-corp.org/
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Fuzzing

[Philippe Antoine]

Hi bro team,

Do you have plans to integrate oss-fuzz ? like
https://github.com/google/oss-fuzz/issues/624 


All the best,
Philippe


signature.asc
Description: Message signed with OpenPGP
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Logging TCP server banners

[Bencteux Jeffrey]

> Another thing that comes to me is what if you miss the SYN or the
> SYN-ACK segment sent by your client? 

I meant SYN or ACK (third one in the handshake) segment sent by the
client. Sorry.


Regards,

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Logging TCP server banners

[Bencteux Jeffrey]

> I call "TCP server banner" the first chunk of data a server sends,
> before the client has sent data (if the client sends data before the
> server, I don't want to log anything).

A solution could be to blacklist such connections, i-e if there is data
sent by the client, then do not log:
> if (! is_orig && seq == 1 && c$orig$num_pkts == 2 && c$orig$size == 0)

Another thing that comes to me is what if you miss the SYN or the
SYN-ACK segment sent by your client? You will not log the banner so I am
not sure about the second condition : c$orig$num_pkts == 2. I would
remove it.

With the pcap generated with the scapy script you gave, I do not log
anymore, however if I change it to this:

wrpcap("test.cap", [
Ether() / IP(dst="1.2.3.4", src="5.6.7.8") /
TCP(dport=80, sport=5678, flags="S", ack=0, seq=555678),
Ether() / IP(src="1.2.3.4", dst="5.6.7.8") /
TCP(sport=80, dport=5678, flags="SA", seq=111234, ack=555679),
Ether() / IP(dst="1.2.3.4", src="5.6.7.8") /
TCP(dport=80, sport=5678, flags="A", ack=111235, seq=555679),
# Ether() / IP(dst="1.2.3.4", src="5.6.7.8") / no more data sent by the 
client
# TCP(dport=80, sport=5678, flags="PA", ack=111235, seq=555679) / "DATA",
Ether() / IP(src="1.2.3.4", dst="5.6.7.8") /
TCP(sport=80, dport=5678, flags="PA", seq=111235, ack=555679) / "DATA"
])

I do have an entry in the log.

> Also, when `seq` equals 1, am I certain that I have not missed any
> packet from the server?

No idea about that, I think the answer is in Bro's TCP implementation in
src/analyzer/protocol/tcp somewhere.

Regards,


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev