[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1072: -- Resolution: Merged (was: Fixed) Status: Closed (was: Merge Request) merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-08#6143) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100#comment-14100 ] Bernhard Amann commented on BIT-1072: - mem-leak fix and updated documentation is in topic/bernhard/ticket1072 merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-08#6143) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1072: Status: Merge Request (was: In Progress) merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-08#6143) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1072: Status: In Progress (was: Open) merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-08#6143) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13903#comment-13903 ] Bernhard Amann commented on BIT-1072: - So - I am officially blind :) merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[
https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13900#comment-13900
]
Jon Siwek commented on BIT-1072:
Regarding to core.leaks.basic-cluster test failure, the underlying cause of the
leaks seems to be:
{noformat}
expression error in
/home/jsiwek/bro/scripts/base/frameworks/sumstats/./plugins/./hll_unique.bro,
line 56: field value missing [SumStats::rv1$hll_error_margin]
{noformat}
So seems that hook function needs better handling of uninitialized values and
maybe also in this case values are expected to be initialized and are not.
merge topic/bernhard/hyperloglog
Key: BIT-1072
URL: https://bro-tracker.atlassian.net/browse/BIT-1072
Project: Bro Issue Tracker
Issue Type: New Feature
Components: Bro
Affects Versions: git/master
Reporter: Bernhard Amann
Fix For: 2.2
Attachments: out.pdf
The branch adds support for the hyperloglog data structure.
In the branch, core/leaks/basic-cluster.bro currently faisl. However, this
seems to be unrelated to hll and just to be triggered by the addition of it
to the sumstats tests. It looks like some kind of scriptland issue. pprof
output is attached. (master, workers don't leak memory)
--
This message was sent by Atlassian JIRA
(v6.1-OD-06#6139)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13901#comment-13901 ] Bernhard Amann commented on BIT-1072: - So it was the fault of hll (or me) after all. :( Was that error reported when running bro? I do not remember ever seeing that... merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13820#comment-13820 ] Robin Sommer commented on BIT-1072: --- I fixed the bug introduced, did some more polishing, and also made the confidence a parameter to hll init. Merged into master now, but please still work on the Doxygen comments. merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13818#comment-13818 ] Bernhard Amann commented on BIT-1072: - Sorry - re-merged master merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1072: Status: Merge Request (was: Open) merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817#comment-13817 ] Robin Sommer commented on BIT-1072: --- I'm getting a number of conflicts when merging into master. Please merge the branch with master first. merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Sommer updated BIT-1072: -- Status: Open (was: Merge Request) merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819#comment-13819 ] Robin Sommer commented on BIT-1072: --- I ended up refactoring and reformatting this quite a bit, it's in topic/robin/hyperlolog-merge. However, I broke something, the tests aren't working. Need to debug that later. In the meantime, some requests/questions: - Please look over my changes and see if they make sense. (You don't need to track down the bug; I take the blame for that :). - Can you please rework the Doxygen comments in HyperLogLog.h so that the descriptions for the public methods are understandable on their own. Right now I can't really follow them as often they talk about internal parameters/functionality. What you could do is provide a short overview of the data structure parameters in the class' doc string, and then refer to that in the methods. Also, please use the @param and @return syntax. (Start from my branch with this: I already reformatted and reordered things there quite a bit.) - I don't understand what can be parameterized by the user and what not (and why not). One can give an error margin to the actor, but the confidence is a compile time constant. Also, where are the magic alpha_m values in *.cc coming from? Are these indeed always static values that don't depend on any parameters? merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1072) merge topic/bernhard/hyperloglog
[ https://bro-tracker.atlassian.net/browse/BIT-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernhard Amann updated BIT-1072: Status: Merge Request (was: Open) merge topic/bernhard/hyperloglog Key: BIT-1072 URL: https://bro-tracker.atlassian.net/browse/BIT-1072 Project: Bro Issue Tracker Issue Type: New Feature Components: Bro Affects Versions: git/master Reporter: Bernhard Amann Fix For: 2.2 Attachments: out.pdf The branch adds support for the hyperloglog data structure. In the branch, core/leaks/basic-cluster.bro currently faisl. However, this seems to be unrelated to hll and just to be triggered by the addition of it to the sumstats tests. It looks like some kind of scriptland issue. pprof output is attached. (master, workers don't leak memory) -- This message was sent by Atlassian JIRA (v6.1-OD-06#6139) ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
