[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14619#comment-14619 ] Bernhard Amann commented on BIT-985: I will add this soon-ish... > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Bernhard Amann >Priority: Low > Fix For: 2.3 > > Attachments: PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.2-OD-01#6204) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-985: --- Fix Version/s: (was: 2.2) 2.3 > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Bernhard Amann >Priority: Low > Fix For: 2.3 > > Attachments: PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.2-OD-01#6204) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Siwek updated BIT-985: -- Fix Version/s: (was: 2.4) 2.5 > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.5 > > Attachments: PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20011#comment-20011 ] Johanna Amann commented on BIT-985: --- This is such a small thing that I might try to really still do it for 2.4. > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-985: -- Fix Version/s: (was: 2.5) 2.4 > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20012#comment-20012 ] scampbell commented on BIT-985: --- I have a significantly improved patch from the one that I previously attached. That one leaked memory rather enthusiastically will send over in a moment. > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] scampbell updated BIT-985: -- Attachment: input.diff > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20015#comment-20015 ] Johanna Amann commented on BIT-985: --- Thanks for the new patch. Cursory looking at it, it seems that this patch changes a lot of functionality in the Raw reader that seems to have nothing to do with skipping parts of the input file. Can you perhaps just sketch what else this patch changes? It seems to change something about how the buffering is done in the raw reader, but I am not quite sure what all this does on a first glance. > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] scampbell updated BIT-985: -- -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Absolutely - the key issues that I ran into with the first patch were dealing with file rotation under the reader and leaks in the data copying scheme. After spending a few days on the mem leak issues modifying the single use linear buffers (and mostly de-stabilizing everything), I reimplemented the whole thing as a ring buffer. My use case - reading a very rapidly moving log file - might be far enough away from the original design pattern of small reasonably static files that it is worth another type? On the other hand I might have just messed up the original work. If this makes no sense please let me know and I will look over my notes re the changes. thanks for looking into this, scott -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlUIyPMACgkQK2Plq8B7ZByVQwCghwbGlmgetHNMkxicrms6wl69 d2EAoIXsHbv1JWPeXJ5rpWv2rAlfWpPQ =bKTE -END PGP SIGNATURE- > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20018#comment-20018 ] Johanna Amann commented on BIT-985: --- Thank you for that explanation. I assume that raw_unescape_URI function made it into the patch by accident? > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] scampbell updated BIT-985: -- -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Yes - sorry about that! scott -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlUI1MoACgkQK2Plq8B7ZBxKawCgpxUNSI21dDqcDg5o49g8JKUq Q3AAoKFtR//MMCSyCke5670RdA1nGfEw =HHK7 -END PGP SIGNATURE- > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann reassigned BIT-985: - Assignee: (was: Johanna Amann) > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johanna Amann updated BIT-985: -- Status: Merge Request (was: Open) > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20314#comment-20314 ] Johanna Amann commented on BIT-985: --- Branch topic/johanna/bit0985 adds seeking functionality to raw reader. one can now add an option "offset" to the config map. Positive offsets are interpreted to be from the beginning of the file, negative from the end of the file (-1 is end of file). Only works for raw reader in streaming or manual mode. Does not work with executables. Scott, could you perhaps add a separate bug for your ring-buffer changes if you want to get them into mainline Bro? (They will not make it into 2.4 though). > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Johanna Amann >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer reassigned BIT-985: Assignee: Robin Sommer > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Robin Sommer >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20316#comment-20316 ] Robin Sommer commented on BIT-985: -- Merging. Mind documenting that option somewhere though? > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Robin Sommer >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=20317#comment-20317 ] Johanna Amann commented on BIT-985: --- I actually thought about that and was just not quite sure where - the other flags do not really seem to be documented either :/ > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Robin Sommer >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-985) 'tail -f' functionality for file reading in input framework
[ https://bro-tracker.atlassian.net/browse/BIT-985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-985: - Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) > 'tail -f' functionality for file reading in input framework > --- > > Key: BIT-985 > URL: https://bro-tracker.atlassian.net/browse/BIT-985 > Project: Bro Issue Tracker > Issue Type: New Feature > Components: Bro >Affects Versions: git/master >Reporter: scampbell >Assignee: Robin Sommer >Priority: Low > Fix For: 2.4 > > Attachments: input.diff, PATCH > > > With the current input framework, file data \-> event translation requires > that the entire data file be read at bro start time. This can be prohibitive > when the file sizes become large ( > 1GB ). > It would be great to see a file open option that would start reading at the > end of the file. -- This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
