[Bro-Dev] Proposed IOSource reorg
I'm thinking to move the IOSource infrastructure into its own
subdirectory/namespace and turn the IOSourceRegistry into
iosource::Manager in alignment with the layout we've started to move
to with the logging/input/etc. I'd then move the classes derived from
IOSource into corresponding subdirectories, like this:
src/iosource/
src/iosource/Manager.{h,cc}
src/iosource/IOSource.{h,cc}
src/iosource/sources/pkt-src/PktSrc.{h,cc}
src/iosource/sources/pkt-src/bpf/*
src/iosource/sources/flow-src/*
src/iosource/sources/dns-mgr/*
src/iosource/sources/remote-serializer/*
The sources would turn into plugin components. New types of packet
sources (like netmap) would then go into iosource/pkt-src/foo/.
Does that make sense?
One piece where I'm unsure: would it be better to keep the remote
serializer out if this and instead do a separate serializer/ hierarchy
where all the current serialization/communication code goes?
Robin
--
Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org
ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Proposed IOSource reorg
On Dec 3, 2013, at 12:07 PM, Robin Sommer wrote: > One piece where I'm unsure: would it be better to keep the remote > serializer out if this and instead do a separate serializer/ hierarchy > where all the current serialization/communication code goes? Maybe best would be if the remote serializer code is refactored so the code that implements the IOSource interface lives in the iosource/ tree, while the code that implements Serializer interface lives in a separate serializer/ tree? Though taking the reorg and plugin adaptation one step at a time would make sense to just stick it in iosource for now and then later consider what, if any, pieces to pull out and put in serializer/. - Jon ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Proposed IOSource reorg
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Would the input framework code be morphed into iosource/sources or
continue living in it's own directory?
Re breaking out communication and serialization into it's own place,
it seems like it has a distinct function outside of i/o - it is used
by i/o, but it can work outside of the core functionality. Making
this a one step at a time reorg as Jon suggests might be a less
complex and destructive way to go about that...
cheers,
scott
On 12/3/13 12:07 PM, Robin Sommer wrote:
> I'm thinking to move the IOSource infrastructure into its own
> subdirectory/namespace and turn the IOSourceRegistry into
> iosource::Manager in alignment with the layout we've started to
> move to with the logging/input/etc. I'd then move the classes
> derived from IOSource into corresponding subdirectories, like
> this:
>
> src/iosource/ src/iosource/Manager.{h,cc}
> src/iosource/IOSource.{h,cc}
> src/iosource/sources/pkt-src/PktSrc.{h,cc}
> src/iosource/sources/pkt-src/bpf/* src/iosource/sources/flow-src/*
> src/iosource/sources/dns-mgr/*
> src/iosource/sources/remote-serializer/*
>
> The sources would turn into plugin components. New types of packet
> sources (like netmap) would then go into iosource/pkt-src/foo/.
>
> Does that make sense?
>
> One piece where I'm unsure: would it be better to keep the remote
> serializer out if this and instead do a separate serializer/
> hierarchy where all the current serialization/communication code
> goes?
>
> Robin
>
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlKeLBMACgkQK2Plq8B7ZBy2qwCghjZ2uKm+qaIT8jXs9UaqBMAR
3akAnA670VZg52SJTWDhgQVz/FqPHEYE
=jpTO
-END PGP SIGNATURE-
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Proposed IOSource reorg
On Tue, Dec 03, 2013 at 13:08 -0600, you wrote: > Would the input framework code be morphed into iosource/sources or > continue living in it's own directory? No, the input framework is separate. The threading manager could be affected (it's an IOSource as well, and the input framework uses it) but that's probably best left where it is for now as well (it's actually a similar question as with RemoteSerializer; didn't realize that yesterday). Robin -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Proposed IOSource reorg
On Tue, Dec 03, 2013 at 18:40 +, you wrote: > Maybe best would be if the remote serializer code is refactored so the > code that implements the IOSource interface lives in the iosource/ > tree, while the code that implements Serializer interface lives in a > separate serializer/ tree? Could be an option, though I'm not immediately sure how well it would split. But one step at a time sounds good in any case, so I'll go ahead with that and we can later see. Robin -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Proposed IOSource reorg
On Dec 3, 2013, at 1:07 PM, Robin Sommer wrote: >src/iosource/sources/flow-src/* To document our conversation from yesterday, flow-src should probably be thrown out and the netflow analyzer turned into a file analyzer. Extending the input framework to be able to open raw sockets would then enable us to create an input stream holding open a datagram socket and attach the netflow file analyzer to it. This would simplify the whole thing and make it possible to reuse the netflow analyzer code because we could yank netflow directly off the wire with it too (pending some analyzer infrastructure re-architecting). .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ signature.asc Description: Message signed with OpenPGP using GPGMail ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Proposed IOSource reorg
As I'm working on the reorg, I propose to do the following: - Remove flow sources completely for now. Per below, we should eventually turn them into a file analyzer and at it doesn't look worth the effort (nor the ugliness) to migrate them over to the new structure first only to throw them out later. I'd be surprised if anybody is using them anyways. - Remove the secondary path from the packet-layer code. We have discussed this before and at that time decided for keeping the code; see https://bro-tracker.atlassian.net/browse/BIT-434 However, I propose to go ahead and remove now because (1) it doesn't really fit the new structure of making the API (mostly) pcap-independent (it never really fit in well in the first place, and has made the code a lot more complex); (2) large-conns.bro seems to be the only actual use case, which we don't ship with 2.x anymore, and I'm not convinced that by itself warrants a separate data path (can we find a different solution to the problem?); and (3) it would be quite a bit of additional effort to port the code and make sure it still works (we don't have any tests, not surprisingly). Thoughts? Robin On Wed, Dec 04, 2013 at 11:12 -0500, you wrote: > > On Dec 3, 2013, at 1:07 PM, Robin Sommer wrote: > > >src/iosource/sources/flow-src/* > > To document our conversation from yesterday, flow-src should probably > be thrown out and the netflow analyzer turned into a file analyzer. > Extending the input framework to be able to open raw sockets would > then enable us to create an input stream holding open a datagram > socket and attach the netflow file analyzer to it. This would > simplify the whole thing and make it possible to reuse the netflow > analyzer code because we could yank netflow directly off the wire with > it too (pending some analyzer infrastructure re-architecting). > > .Seth > > -- > Seth Hall > International Computer Science Institute > (Bro) because everyone has a network > http://www.bro.org/ > -- Robin Sommer * Phone +1 (510) 722-6541 * ro...@icir.org ICSI/LBNL* Fax +1 (510) 666-2956 * www.icir.org/robin ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Proposed IOSource reorg
On Dec 11, 2013, at 1:48 PM, Robin Sommer wrote: > As I'm working on the reorg, I propose to do the following: Everything sound good to me. > large-conns.bro seems to be the only actual use case, which we > don't ship with 2.x anymore, and I'm not convinced that by > itself warrants a separate data path (can we find a different > solution to the problem?) I think the only reason that large-conns.bro used the secondary data path was to get access to more traffic that Bro might not be normally seeing due to being filtered out. With our change to a default open filter (and I don't see this changing anytime soon) we don't need the secondary path to let in more traffic than is normally allowed in. .Seth -- Seth Hall International Computer Science Institute (Bro) because everyone has a network http://www.bro.org/ signature.asc Description: Message signed with OpenPGP using GPGMail ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
