[jira] [Commented] (METRON-1039) Add ZIP function to Stellar
[ https://issues.apache.org/jira/browse/METRON-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096865#comment-16096865 ] ASF GitHub Bot commented on METRON-1039: Github user jjmeyer0 commented on the issue: https://github.com/apache/metron/pull/652 @cestella open ticket for ZIP_WITH and ZIP_WITH_INDEX. To make sure we are on the same page for ZIP_WITH_INDEX, can you skim over the jira when you have time? > Add ZIP function to Stellar > --- > > Key: METRON-1039 > URL: https://issues.apache.org/jira/browse/METRON-1039 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > Stellar could use a ZIP function i.e. ZIP( [ 1, 2, 3], [ 4, 5, 6] ) == [ [1, > 4], [2, 5], [3, 6] ] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1057) Add ZIP_WITH and ZIP_WITH_INDEX Functions in Stellar
Josh Meyer created METRON-1057: -- Summary: Add ZIP_WITH and ZIP_WITH_INDEX Functions in Stellar Key: METRON-1057 URL: https://issues.apache.org/jira/browse/METRON-1057 Project: Metron Issue Type: New Feature Reporter: Josh Meyer In addition to the new `ZIP` functions in Stellar, we should add a `ZIP_WITH` and `ZIP_WITH_INDEX` functions. These have been implemented in other languages (Haskell). Below are examples of these functions. ``` ZIP_WITH([1, 2, 3], [1, 2, 3], (a, b) -> a + b) == [2, 4, 6] ZIP_WITH_INDEX([1, 2, 3]) == [[0,1], [1,2], [2,3]] ``` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-1057) Add ZIP_WITH and ZIP_WITH_INDEX Functions in Stellar
[ https://issues.apache.org/jira/browse/METRON-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Meyer updated METRON-1057: --- Description: In addition to the new `ZIP` functions in Stellar, we should add a `ZIP_WITH` and `ZIP_WITH_INDEX` functions. These have been implemented in other languages (Haskell). Below are examples of these functions. ``` ZIP_WITH([1, 2, 3], [1, 2, 3], (a, b) -> a + b) == [2, 4, 6] ZIP_WITH_INDEX([1, 2, 3]) == [[0,1], [1,2], [2,3]] ``` This was discussed shortly on this PR https://github.com/apache/metron/pull/652 was: In addition to the new `ZIP` functions in Stellar, we should add a `ZIP_WITH` and `ZIP_WITH_INDEX` functions. These have been implemented in other languages (Haskell). Below are examples of these functions. ``` ZIP_WITH([1, 2, 3], [1, 2, 3], (a, b) -> a + b) == [2, 4, 6] ZIP_WITH_INDEX([1, 2, 3]) == [[0,1], [1,2], [2,3]] ``` > Add ZIP_WITH and ZIP_WITH_INDEX Functions in Stellar > > > Key: METRON-1057 > URL: https://issues.apache.org/jira/browse/METRON-1057 > Project: Metron > Issue Type: New Feature >Reporter: Josh Meyer > > In addition to the new `ZIP` functions in Stellar, we should add a `ZIP_WITH` > and `ZIP_WITH_INDEX` functions. These have been implemented in other > languages (Haskell). Below are examples of these functions. > ``` > ZIP_WITH([1, 2, 3], [1, 2, 3], (a, b) -> a + b) == [2, 4, 6] > ZIP_WITH_INDEX([1, 2, 3]) == [[0,1], [1,2], [2,3]] > ``` > This was discussed shortly on this PR > https://github.com/apache/metron/pull/652 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-992) Create performance tuning guide
[ https://issues.apache.org/jira/browse/METRON-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096557#comment-16096557 ] ASF GitHub Bot commented on METRON-992: --- Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/614 > Create performance tuning guide > --- > > Key: METRON-992 > URL: https://issues.apache.org/jira/browse/METRON-992 > Project: Metron > Issue Type: Task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > We need a guide to outline general guidelines for tuning the Metron Storm > topologies and Kafka topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1056) Get field types from Elasticsearch
Ryan Merriman created METRON-1056: - Summary: Get field types from Elasticsearch Key: METRON-1056 URL: https://issues.apache.org/jira/browse/METRON-1056 Project: Metron Issue Type: Improvement Reporter: Ryan Merriman We need a way to retrieve field types from Elasticsearch for a specified list of indices. This will also be exposed through our indexing DAO layer with Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-992) Create performance tuning guide
[ https://issues.apache.org/jira/browse/METRON-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096268#comment-16096268 ] ASF GitHub Bot commented on METRON-992: --- Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/614 Still +1 after the recent commits. Thanks again, Mike! > Create performance tuning guide > --- > > Key: METRON-992 > URL: https://issues.apache.org/jira/browse/METRON-992 > Project: Metron > Issue Type: Task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic > > We need a guide to outline general guidelines for tuning the Metron Storm > topologies and Kafka topics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler
[ https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095973#comment-16095973 ] ASF GitHub Bot commented on METRON-1005: Github user cestella commented on the issue: https://github.com/apache/metron/pull/622 I would agree that it's easy to rewrite the old profiles in the new decodable format IF we can read the old profile keys and pull out their info (even fuzzily). > Create Decodable Row Key for Profiler > - > > Key: METRON-1005 > URL: https://issues.apache.org/jira/browse/METRON-1005 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > To be able to answer the types of questions that I outlined in METRON-450, we > need a row key that is decodable. Right now there is no logic to decode a > row key, nor is the existing row key easily decodable. > Once the row keys can be decoded, you could scan all of the row keys in the > Profiler's HBase table, decode each of them and extract things like, the > names of all your profiles, the names of entities within a profile, the > period duration of a given profile. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler
[ https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095933#comment-16095933 ] ASF GitHub Bot commented on METRON-1005: Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/622 And btw, since there is no easily expressed algorithm for the NLP part of the problem, I'm +1 on doing both a decodable rowkey and a ToC. For the existing profiles that @cestella expressed concern about, I would point out that as long as one DOES have the Profile specs still lying around, it's actually easy to re-write the old Profiles into new format with decodable rowkeys. That is a very modest-sized program, the main problem being noticing and dealing with duplicate titled Profiles with different periodDurations. But the info I pointed out in the paper helps sufficiently, I think. > Create Decodable Row Key for Profiler > - > > Key: METRON-1005 > URL: https://issues.apache.org/jira/browse/METRON-1005 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > To be able to answer the types of questions that I outlined in METRON-450, we > need a row key that is decodable. Right now there is no logic to decode a > row key, nor is the existing row key easily decodable. > Once the row keys can be decoded, you could scan all of the row keys in the > Profiler's HBase table, decode each of them and extract things like, the > names of all your profiles, the names of entities within a profile, the > period duration of a given profile. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler
[ https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095924#comment-16095924 ] ASF GitHub Bot commented on METRON-1005: Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/622 Here's what I've got on decoding old rowkeys: https://gist.github.com/mattf-horton/8e685e373b1a3fa6aeec8ef8828be096 The format of the keys is `salt (4B) + profile name (?) + entity name (?) + groupvalues (?) + period (8B)` with most of it (all but the salt and period number) in the clear as human-readable strings. Deducing periodDuration has a nice arithmetic answer, I think. The NLP issues are of course harder. Enjoy the read, it's only two pages. > Create Decodable Row Key for Profiler > - > > Key: METRON-1005 > URL: https://issues.apache.org/jira/browse/METRON-1005 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > To be able to answer the types of questions that I outlined in METRON-450, we > need a row key that is decodable. Right now there is no logic to decode a > row key, nor is the existing row key easily decodable. > Once the row keys can be decoded, you could scan all of the row keys in the > Profiler's HBase table, decode each of them and extract things like, the > names of all your profiles, the names of entities within a profile, the > period duration of a given profile. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
