Re: [DISCUSS] New Stellar Functions
I also might be worth while to overload '+' in Stellar to operate on
primitive data structures like String, List or Map as well as numeric
types. Then again, that way sometimes leads to dragons.
On Mon, Apr 10, 2017 at 3:07 PM, Matt Foley wrote:
> Hi Kyle,
> For now at least, it seems to me that something with as much core
> usefulness, and presumably non-customer-specific, as string concat should
> go into metron-common. That said, I suspect Mike is right that JOIN would
> be sufficient. But if that fits your use case poorly, just say so.
>
> --Matt
>
> On 4/10/17, 10:13 AM, "Michael Miklavcic"
> wrote:
>
> Hey Kyle,
>
> It probably belongs here -
> https://github.com/apache/incubator-metron/blob/master/
> metron-platform/metron-common/src/main/java/org/apache/
> metron/common/dsl/functions/StringFunctions.java
> There is an existing JOIN function for strings - might this suit your
> needs? I didn't see a unit test for it, so it would probably be good
> for us
> to backfill with a test here as well. I can submit a PR for it, or if
> you're already in that code, you're welcome to also -
> https://github.com/apache/incubator-metron/blob/master/
> metron-platform/metron-common/src/test/java/org/apache/
> metron/common/dsl/functions/StringFunctionsTest.java
>
> e.g.
> Object joined = run("JOIN(['A','B','C','D'], ':')",new HashedMap());
> System.out.println(joined);
> Object joined2 = run("JOIN(['A','B','C','D'], '')",new HashedMap());
> System.out.println(joined2);
>
> Output I get is:
> A:B:C:D
> ABCD
>
> AFA where to put these functions, I believe we have a number of
> options now
> that we have the ability to add to the storm topology classpath and
> sideload jars.
> - https://github.com/apache/incubator-metron/pull/204
> - https://github.com/apache/incubator-metron/pull/468
>
> I think that if the functions are unique to a customer, they should
> probably be built as a stand-alone Maven project. I believe Otto is
> working
> on this ATM if I'm not mistaken. If there is universal (across all
> functions in the system, whether parsing, analytics or otherwise) then
> they
> should probably go in with the dsl package in metron-common. At some
> point
> we might want to make Stellar its own module, but there is some work
> there.
>
> Best,
> Mike
>
>
> On Sun, Apr 9, 2017 at 2:26 PM, Kyle Richardson <
> kylerichards...@gmail.com>
> wrote:
>
> > I have the need for a new Stellar function to perform string
> concatenation.
> > I have it implemented but am curious about where new functions
> should live
> > given the new capabilities around 3rd party Stellar function
> libraries.
> >
> > So, I guess my question is, should this function live in:
> > 1) metron-common with the other string functions
> > 2) another metron project
> > 3) as a standalone project and not part of the metron source tree
> >
> > While I'm specifically asking about this case, I think it's also
> worthwhile
> > that we think about where other new functions should live in the
> long term.
> >
> > Thanks!
> >
> > -Kyle
> >
>
>
>
>
Re: [DISCUSS] New Stellar Functions
Hi Kyle,
For now at least, it seems to me that something with as much core usefulness,
and presumably non-customer-specific, as string concat should go into
metron-common. That said, I suspect Mike is right that JOIN would be
sufficient. But if that fits your use case poorly, just say so.
--Matt
On 4/10/17, 10:13 AM, "Michael Miklavcic" wrote:
Hey Kyle,
It probably belongs here -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
There is an existing JOIN function for strings - might this suit your
needs? I didn't see a unit test for it, so it would probably be good for us
to backfill with a test here as well. I can submit a PR for it, or if
you're already in that code, you're welcome to also -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
e.g.
Object joined = run("JOIN(['A','B','C','D'], ':')",new HashedMap());
System.out.println(joined);
Object joined2 = run("JOIN(['A','B','C','D'], '')",new HashedMap());
System.out.println(joined2);
Output I get is:
A:B:C:D
ABCD
AFA where to put these functions, I believe we have a number of options now
that we have the ability to add to the storm topology classpath and
sideload jars.
- https://github.com/apache/incubator-metron/pull/204
- https://github.com/apache/incubator-metron/pull/468
I think that if the functions are unique to a customer, they should
probably be built as a stand-alone Maven project. I believe Otto is working
on this ATM if I'm not mistaken. If there is universal (across all
functions in the system, whether parsing, analytics or otherwise) then they
should probably go in with the dsl package in metron-common. At some point
we might want to make Stellar its own module, but there is some work there.
Best,
Mike
On Sun, Apr 9, 2017 at 2:26 PM, Kyle Richardson
wrote:
> I have the need for a new Stellar function to perform string
concatenation.
> I have it implemented but am curious about where new functions should live
> given the new capabilities around 3rd party Stellar function libraries.
>
> So, I guess my question is, should this function live in:
> 1) metron-common with the other string functions
> 2) another metron project
> 3) as a standalone project and not part of the metron source tree
>
> While I'm specifically asking about this case, I think it's also
worthwhile
> that we think about where other new functions should live in the long
term.
>
> Thanks!
>
> -Kyle
>
Re: [DISCUSS] New Stellar Functions
I am working on that. What i’m shooting for is having archetypes for
parsers, stellar function libraries, and - metron extensions.
The stellar work will be a follow on however.
On April 10, 2017 at 13:13:46, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:
Hey Kyle,
It probably belongs here -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
There is an existing JOIN function for strings - might this suit your
needs? I didn't see a unit test for it, so it would probably be good for us
to backfill with a test here as well. I can submit a PR for it, or if
you're already in that code, you're welcome to also -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
e.g.
Object joined = run("JOIN(['A','B','C','D'], ':')",new HashedMap());
System.out.println(joined);
Object joined2 = run("JOIN(['A','B','C','D'], '')",new HashedMap());
System.out.println(joined2);
Output I get is:
A:B:C:D
ABCD
AFA where to put these functions, I believe we have a number of options now
that we have the ability to add to the storm topology classpath and
sideload jars.
- https://github.com/apache/incubator-metron/pull/204
- https://github.com/apache/incubator-metron/pull/468
I think that if the functions are unique to a customer, they should
probably be built as a stand-alone Maven project. I believe Otto is working
on this ATM if I'm not mistaken. If there is universal (across all
functions in the system, whether parsing, analytics or otherwise) then they
should probably go in with the dsl package in metron-common. At some point
we might want to make Stellar its own module, but there is some work there.
Best,
Mike
On Sun, Apr 9, 2017 at 2:26 PM, Kyle Richardson
wrote:
> I have the need for a new Stellar function to perform string
concatenation.
> I have it implemented but am curious about where new functions should
live
> given the new capabilities around 3rd party Stellar function libraries.
>
> So, I guess my question is, should this function live in:
> 1) metron-common with the other string functions
> 2) another metron project
> 3) as a standalone project and not part of the metron source tree
>
> While I'm specifically asking about this case, I think it's also
worthwhile
> that we think about where other new functions should live in the long
term.
>
> Thanks!
>
> -Kyle
>
Re: [DISCUSS] New Stellar Functions
Hey Kyle,
It probably belongs here -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
There is an existing JOIN function for strings - might this suit your
needs? I didn't see a unit test for it, so it would probably be good for us
to backfill with a test here as well. I can submit a PR for it, or if
you're already in that code, you're welcome to also -
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/StringFunctionsTest.java
e.g.
Object joined = run("JOIN(['A','B','C','D'], ':')",new HashedMap());
System.out.println(joined);
Object joined2 = run("JOIN(['A','B','C','D'], '')",new HashedMap());
System.out.println(joined2);
Output I get is:
A:B:C:D
ABCD
AFA where to put these functions, I believe we have a number of options now
that we have the ability to add to the storm topology classpath and
sideload jars.
- https://github.com/apache/incubator-metron/pull/204
- https://github.com/apache/incubator-metron/pull/468
I think that if the functions are unique to a customer, they should
probably be built as a stand-alone Maven project. I believe Otto is working
on this ATM if I'm not mistaken. If there is universal (across all
functions in the system, whether parsing, analytics or otherwise) then they
should probably go in with the dsl package in metron-common. At some point
we might want to make Stellar its own module, but there is some work there.
Best,
Mike
On Sun, Apr 9, 2017 at 2:26 PM, Kyle Richardson
wrote:
> I have the need for a new Stellar function to perform string concatenation.
> I have it implemented but am curious about where new functions should live
> given the new capabilities around 3rd party Stellar function libraries.
>
> So, I guess my question is, should this function live in:
> 1) metron-common with the other string functions
> 2) another metron project
> 3) as a standalone project and not part of the metron source tree
>
> While I'm specifically asking about this case, I think it's also worthwhile
> that we think about where other new functions should live in the long term.
>
> Thanks!
>
> -Kyle
>
[DISCUSS] New Stellar Functions
I have the need for a new Stellar function to perform string concatenation. I have it implemented but am curious about where new functions should live given the new capabilities around 3rd party Stellar function libraries. So, I guess my question is, should this function live in: 1) metron-common with the other string functions 2) another metron project 3) as a standalone project and not part of the metron source tree While I'm specifically asking about this case, I think it's also worthwhile that we think about where other new functions should live in the long term. Thanks! -Kyle
