[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/817
  
If the indexing start doesn't install the templates because ES is down or 
something, will they be installed if restarted and ES is available, or is it 
one shot, pass or fail.


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/814
  
@jjmeyer0 @cestella : Ok.
So, I added some code, where it is possible context wise to guard against 
the case where there is a single variable as the check, and no default.  This 
resolves the test issues, save for one, where there is no default and we are 
using == true.  This is more difficult.

I feel as if all of these things are wrong, because the validation by 
executing with null variables is itself logically wrong.  We have discussed 
this before, but this is really evident here.  At least I feel it is.  

I think that we should use compilation instead of validation.  We can 
decide to make it a separate option when running tests, or to make it the way 
we do all tests etc.  But if compilation does what I think it does, then I 
think it is more correct.

I would like to do that, and remove the work around I have introduced here 
( basically detecting that we are validating and have a single var that is null 
because of validation ) since changing validation would be the real complete 
answer.



---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/814
  
Because it is possible in validate() for a boolean to be null instead of 
true or false.


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/814
  
So for example:

```java
 /*
  curr is the current value on the stack.  This is the non-deferred 
actual evaluation for this expression
  and with the current context.
   */
  Token curr = instanceDeque.peek();
  if (curr != null && curr.getValue() != null && curr.getValue() 
instanceof Boolean
  && 
ShortCircuitOp.class.isAssignableFrom(token.getUnderlyingType())) {
//if we have a boolean as the current value and the next 
non-contextual token is a short circuit op
//then we need to short circuit possibly
if (token.getUnderlyingType() == BooleanArg.class) {
```

with this function: 

```java
 @Test
  @SuppressWarnings("unchecked")
  public void testVariableOnlyCheckWithDefault() {
Assert.assertEquals("a",  run("match{ foo : 'a', default : 'b' }",
new HashMap() {{
  put("foo", true);
}}));
  }
```

When we get to that point, because of the validation of vars to null `curr` 
IS null.  So even though token IS a short circuit token, we do not process it 
as we should. 


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/814
  
@jjmeyer0 thanks for the tests.  The issue is that we call validate and 
then parse.  When we call validate, we resolve all vars to NULL.  This means 
that the clauses in these failures are false.  As stated in the readme, 
something in the match must be true or you get an error.  In this case, we have 
an issue where validate is forcing us into that condition and we don't have a 
default, so it is an error.
I'll keep looking.  
If you have any thoughts, or maybe @cestella  does, I'm all ears.


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/814
  
Thanks, I added those tests to the test class, and these fail:
```java
@Test
  @SuppressWarnings("unchecked")
  public void testVariableOnlyNoDefault() {
Assert.assertEquals("a",  run("match{ foo : 'a' }",
new HashMap() {{
  put("foo", true);
}}));
  }

  @Test
  @SuppressWarnings("unchecked")
  public void testVariableEqualsCheckNoDefault() {
Assert.assertEquals("a",  run("match{ foo == true : 'a' }",
new HashMap() {{
  put("foo", true);
}}));
  }

  @Test
  @SuppressWarnings("unchecked")
  public void testVariableOnlyCheckWithDefault() {
Assert.assertEquals("a",  run("match{ foo : 'a', default: 'b' }",
new HashMap() {{
  put("foo", true);
}}));
  }
```

I will take a look.  I feel that I have similar tests to these that are 
working, so it is strange. 


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147568147
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -100,6 +102,28 @@ In the core language functions, we support basic 
functional programming primitiv
 * `FILTER` - Filters a list by a predicate in the form of a lambda 
expression.  For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` 
returns `[ 'foo' ]`
 * `REDUCE` - Applies a function over a list of input.  For instance 
`REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )` returns `6`
 
+### Stellar Language Match Expression
+
+Stellar provides the capability to write match expressions, which are 
similar to switch statements commonly found in c like languages, but more like
+Scala's match.
--- End diff --

Do you have any suggestions for a better way to describe it?  I'm not super 
happy with this either, I'll take suggestions ;)


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147568117
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MatchTest.java
 ---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.stellar.dsl.DefaultVariableResolver;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.junit.Assert;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class MatchTest {
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchLambda() {
+Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }", new 
HashMap() {{
+  put("foo", 0);
+}}));
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 0);
+}}));
+
+Assert.assertFalse(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> false, foo == 1 : ()-> true, 
default : ()-> false }",
+new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  @Ignore
+  public void testMatchMAPEvaluation() {
+
+// NOTE: THIS IS BROKEN RIGHT NOW.
+
+String expr = "match{ var1 :  MAP(['foo', 'bar'], (x) -> TO_UPPER(x)) 
}";
+
+Object o = run(expr, ImmutableMap.of("foo", "foo", "bar", "bar", 
"var1", true));
+
+Assert.assertTrue(o instanceof List);
+
+List result = (List) o;
+
+Assert.assertEquals(2, result.size());
+Assert.assertEquals("FOO", result.get(0));
+Assert.assertEquals("BAR", result.get(1));
+}
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchRegexMatch() {
+final Map variableMap = new HashMap() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
+  put("letters", "abcde");
+  put("empty", "");
+}};
+
+Assert.assertTrue(runPredicate("match{ 
REGEXP_MATCH(numbers,numberPattern): true, default : false}", new 
DefaultVariableResolver(v -> variableMap.get(v),v -> 
variableMap.containsKey(v;
+Assert.assertFalse(runPredicate("match{ 
REGEXP_MATCH(letters,numberPattern) : true, default :false}", new 
DefaultVariableResolver(v -> variableMap.get(v),v -> 
variableMap.containsKey(v;
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchBareStatements() {
+
+Assert.assertTrue(runPredicate("match { foo == 0 : bFalse, foo == 1 : 
bTrue, default : false }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertEquals("warning",
+  

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147568099
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MatchTest.java
 ---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.stellar.dsl.DefaultVariableResolver;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.junit.Assert;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class MatchTest {
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchLambda() {
+Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }", new 
HashMap() {{
+  put("foo", 0);
+}}));
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 0);
+}}));
+
+Assert.assertFalse(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> false, foo == 1 : ()-> true, 
default : ()-> false }",
+new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  @Ignore
+  public void testMatchMAPEvaluation() {
+
+// NOTE: THIS IS BROKEN RIGHT NOW.
+
+String expr = "match{ var1 :  MAP(['foo', 'bar'], (x) -> TO_UPPER(x)) 
}";
+
+Object o = run(expr, ImmutableMap.of("foo", "foo", "bar", "bar", 
"var1", true));
+
+Assert.assertTrue(o instanceof List);
+
+List result = (List) o;
+
+Assert.assertEquals(2, result.size());
+Assert.assertEquals("FOO", result.get(0));
+Assert.assertEquals("BAR", result.get(1));
+}
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchRegexMatch() {
+final Map variableMap = new HashMap() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
+  put("letters", "abcde");
+  put("empty", "");
+}};
+
+Assert.assertTrue(runPredicate("match{ 
REGEXP_MATCH(numbers,numberPattern): true, default : false}", new 
DefaultVariableResolver(v -> variableMap.get(v),v -> 
variableMap.containsKey(v;
+Assert.assertFalse(runPredicate("match{ 
REGEXP_MATCH(letters,numberPattern) : true, default :false}", new 
DefaultVariableResolver(v -> variableMap.get(v),v -> 
variableMap.containsKey(v;
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchBareStatements() {
+
+Assert.assertTrue(runPredicate("match { foo == 0 : bFalse, foo == 1 : 
bTrue, default : false }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertEquals("warning",
+  

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147568088
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MatchTest.java
 ---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.stellar.dsl.DefaultVariableResolver;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.junit.Assert;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class MatchTest {
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchLambda() {
+Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }", new 
HashMap() {{
+  put("foo", 0);
+}}));
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 0);
+}}));
+
+Assert.assertFalse(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> false, foo == 1 : ()-> true, 
default : ()-> false }",
+new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  @Ignore
+  public void testMatchMAPEvaluation() {
+
+// NOTE: THIS IS BROKEN RIGHT NOW.
--- End diff --

I actually wanted to leave this in so @cestella might help me figure out 
why it isn't working with map.  How about I leave it for now, and you hold your 
plus +1 until it is removed.  I would got to commit with this, but I want it 
there for the review.


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147568062
  
--- Diff: 
metron-stellar/stellar-common/src/main/antlr4/org/apache/metron/stellar/common/generated/Stellar.g4
 ---
@@ -272,4 +278,30 @@ lambda_variable:
   IDENTIFIER
   ;
 
+match_expr :
+  MATCH LBRACE match_clauses RBRACE #MatchClauses
+  ;
+
+match_clauses :
+  match_clause (COMMA match_clause)*
+  ;
+
+match_clause :
+  match_clause_check COLON  match_clause_action
+  ;
+
+match_clause_action :
+  match_clause_action_expr #MatchClauseAction
+  ;
 
+match_clause_action_expr :
+  transformation_expr
+  ;
+
+match_clause_check :
--- End diff --

I'll try that again.  I don't remember the specific reason why I ended up 
with it factored like this.  But this is what ended up working ;)



---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147568002
  
--- Diff: 
metron-stellar/stellar-common/src/main/antlr4/org/apache/metron/stellar/common/generated/Stellar.g4
 ---
@@ -67,6 +67,10 @@ ELSE : 'ELSE' | 'else';
 NULL : 'null' | 'NULL';
 NAN : 'NaN';
 
+AS : 'as' | 'AS';
--- End diff --

Actually, AS was for the match(longVarName as shortName){ x > 5 : shortName 
is available }
case, but since I'm not doing that, i should remove it.



---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147567984
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -100,6 +102,28 @@ In the core language functions, we support basic 
functional programming primitiv
 * `FILTER` - Filters a list by a predicate in the form of a lambda 
expression.  For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` 
returns `[ 'foo' ]`
 * `REDUCE` - Applies a function over a list of input.  For instance 
`REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )` returns `6`
 
+### Stellar Language Match Expression
+
+Stellar provides the capability to write match expressions, which are 
similar to switch statements commonly found in c like languages, but more like
+Scala's match.
+
+The syntax is:
+* `match{ logical_expression1 : evaluation expression1, 
logical_expression2 : evaluation_expression2` : A match expression with no 
default
+* `match{ logical_expression1 : evaluation expression1, 
logical_expression2 : evaluation_expression2, default : default_expression}` : 
A match expression with a default expression
+
+Where:
+
+* `logical_expression` is a Stellar expression that evaluates to true or 
false.  For instance `var > 0` or `var > 0 AND var2 == 'foo'`
--- End diff --

```bash
Please note that functions are loading lazily in the background and will be 
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...

[Stellar]>>> foo := 500
[Stellar]>>> match{ foo < 100 : THROW('oops'), foo > 200 : 'ok', default : 
THROW('exception thrown') }
ok
[Stellar]>>>
```

I don't see that.

Also, if you check the last test in TestMatch you will see I have a test 
for this kind of thing.



---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147567887
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -100,6 +102,28 @@ In the core language functions, we support basic 
functional programming primitiv
 * `FILTER` - Filters a list by a predicate in the form of a lambda 
expression.  For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` 
returns `[ 'foo' ]`
 * `REDUCE` - Applies a function over a list of input.  For instance 
`REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )` returns `6`
 
+### Stellar Language Match Expression
+
+Stellar provides the capability to write match expressions, which are 
similar to switch statements commonly found in c like languages, but more like
+Scala's match.
+
+The syntax is:
+* `match{ logical_expression1 : evaluation expression1, 
logical_expression2 : evaluation_expression2` : A match expression with no 
default
--- End diff --

oops, thanks



---

[GitHub] metron issue #817: METRON-1283: Install Elasticsearch template as a part of ...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/817
  
You're right @nickwallen , it wouldn't work. I reworked on the patch and 
here are the latest changes:
- moved the ES template installation back to the Indexing "start" method
- use a flag file to ensure the template installation runs only once. And 
so any start/restart of Indexing topology does not get affected.
- enclosed the template installation in a try/except block so that we warn 
and move on in the event of ES service not running.

I ran the following test scenarios to validate the changes:

**Full dev deployment**

- [x] Verified that the Indexing Start step also installs the ES templates. 
- [x] Verified that the flag files are created under $METRON_HOME/config
- [x] Restarted Indexing service and verified that template install is not 
called.

**12-node deployment**

- [x] Verified that the Indexing Start step also installs the ES templates. 
- [x] Verified that the flag files are created under $METRON_HOME/config
- [x] Restarted Indexing service and verified that template install is not 
called.
- [x] Stopped the ES service manually before the Indexing service starts 
and confirmed that the Indexing service startup proceeds successfully with a 
warning message indicating the template could not be installed


---

[GitHub] metron issue #814: METRON-1277 Add match statement to Stellar language

[jjmeyer0]

Github user jjmeyer0 commented on the issue:

https://github.com/apache/metron/pull/814
  
Also, I don't think there is an issue with the map function directly. It's 
actually an issue with the logical_expression `var1` and using/not using a 
`default` branch. It seems like the stack isn't properly being managed. For 
example things are being pushed (eg. when a logical_expr is just `var1`) when 
they shouldn't and popped  when they shouldn't be (eg. when there is no 
default?). I haven't pin pointed it directly yet, but I hope this helps 
somewhat. Below are a few example tests that I have created. 

```java
// Here there is a null on the stack
@Test
@SuppressWarnings("unchecked")
public void test1FailsWithMoreOnStack() {
Assert.assertEquals("a",  run("match{ foo : 'a' }",
new HashMap() {{
put("foo", true);
}}));
}
```

// Here it's an empty stack when we expect 'a' i believe.
```java
@Test
@SuppressWarnings("unchecked")
public void test1FailsEmptyStack() {
Assert.assertEquals("a",  run("match{ foo == true : 'a' }",
new HashMap() {{
put("foo", true);
}}));
}
```

```java
// This is *not* properly handled
@Test
@SuppressWarnings("unchecked")
public void test1Passes() {
Assert.assertEquals("a",  run("match{ foo : 'a', default: 'b' }",
new HashMap() {{
put("foo", true);
}}));
}
```

```java
// This is properly handled
@Test
@SuppressWarnings("unchecked")
public void test1Passes() {
Assert.assertEquals("a",  run("match{ foo == true : 'a', default: 'b' 
}",
new HashMap() {{
put("foo", true);
}}));
}
```

```java
// Example where map works
@Test
@SuppressWarnings("unchecked")
public void workingMatchWithMap() {
   Assert.assertEquals(Arrays.asList("OK", "HAHA"),  run("match{ foo > 100 
: THROW('oops'), foo > 200 : THROW('oh no'), foo >= 50 : MAP(['ok', 'haha'], 
(a) -> TO_UPPER(a)), default: 'a' }",
  new HashMap() {{
put("foo", 50);
  }}));
}
```


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147557283
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -100,6 +102,28 @@ In the core language functions, we support basic 
functional programming primitiv
 * `FILTER` - Filters a list by a predicate in the form of a lambda 
expression.  For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` 
returns `[ 'foo' ]`
 * `REDUCE` - Applies a function over a list of input.  For instance 
`REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )` returns `6`
 
+### Stellar Language Match Expression
+
+Stellar provides the capability to write match expressions, which are 
similar to switch statements commonly found in c like languages, but more like
+Scala's match.
--- End diff --

I'd be a bit hesitant to say this is similar to Scala's match. Our match 
statement doesn't really support pattern matching.


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147556465
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -100,6 +102,28 @@ In the core language functions, we support basic 
functional programming primitiv
 * `FILTER` - Filters a list by a predicate in the form of a lambda 
expression.  For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` 
returns `[ 'foo' ]`
 * `REDUCE` - Applies a function over a list of input.  For instance 
`REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )` returns `6`
 
+### Stellar Language Match Expression
+
+Stellar provides the capability to write match expressions, which are 
similar to switch statements commonly found in c like languages, but more like
+Scala's match.
+
+The syntax is:
+* `match{ logical_expression1 : evaluation expression1, 
logical_expression2 : evaluation_expression2` : A match expression with no 
default
+* `match{ logical_expression1 : evaluation expression1, 
logical_expression2 : evaluation_expression2, default : default_expression}` : 
A match expression with a default expression
+
+Where:
+
+* `logical_expression` is a Stellar expression that evaluates to true or 
false.  For instance `var > 0` or `var > 0 AND var2 == 'foo'`
--- End diff --

It seems as if the below stellar always throws an exception. Is it by 
design that `default` clauses are always evaluated? If so can you mention that 
here?

```
foo := 500
match{ foo < 100 : THROW('oops'), foo > 200 : 'ok', default : 
THROW('exception thrown') }
```


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147544977
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MatchTest.java
 ---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.stellar.dsl.DefaultVariableResolver;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.junit.Assert;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class MatchTest {
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchLambda() {
+Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }", new 
HashMap() {{
+  put("foo", 0);
+}}));
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 0);
+}}));
+
+Assert.assertFalse(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> false, foo == 1 : ()-> true, 
default : ()-> false }",
+new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  @Ignore
+  public void testMatchMAPEvaluation() {
+
+// NOTE: THIS IS BROKEN RIGHT NOW.
+
+String expr = "match{ var1 :  MAP(['foo', 'bar'], (x) -> TO_UPPER(x)) 
}";
+
+Object o = run(expr, ImmutableMap.of("foo", "foo", "bar", "bar", 
"var1", true));
+
+Assert.assertTrue(o instanceof List);
+
+List result = (List) o;
+
+Assert.assertEquals(2, result.size());
+Assert.assertEquals("FOO", result.get(0));
+Assert.assertEquals("BAR", result.get(1));
+}
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchRegexMatch() {
+final Map variableMap = new HashMap() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
+  put("letters", "abcde");
+  put("empty", "");
+}};
+
+Assert.assertTrue(runPredicate("match{ 
REGEXP_MATCH(numbers,numberPattern): true, default : false}", new 
DefaultVariableResolver(v -> variableMap.get(v),v -> 
variableMap.containsKey(v;
+Assert.assertFalse(runPredicate("match{ 
REGEXP_MATCH(letters,numberPattern) : true, default :false}", new 
DefaultVariableResolver(v -> variableMap.get(v),v -> 
variableMap.containsKey(v;
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchBareStatements() {
+
+Assert.assertTrue(runPredicate("match { foo == 0 : bFalse, foo == 1 : 
bTrue, default : false }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertEquals("warning",
+run

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147543792
  
--- Diff: 
metron-stellar/stellar-common/src/main/antlr4/org/apache/metron/stellar/common/generated/Stellar.g4
 ---
@@ -67,6 +67,10 @@ ELSE : 'ELSE' | 'else';
 NULL : 'null' | 'NULL';
 NAN : 'NaN';
 
+AS : 'as' | 'AS';
--- End diff --

I know this may be fairly unlikely, but technically this is a breaking 
change for Stellar. I believe if anyone had a variable called `as`, `AS`, 
`etc.` it would fail with a parse exception. I think we should explicitly call 
this out. Also, is this something we technically have to wait for a major 
version release to add?


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147555077
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MatchTest.java
 ---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.stellar.dsl.DefaultVariableResolver;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.junit.Assert;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class MatchTest {
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchLambda() {
+Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }", new 
HashMap() {{
+  put("foo", 0);
+}}));
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 0);
+}}));
+
+Assert.assertFalse(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> false, foo == 1 : ()-> true, 
default : ()-> false }",
+new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  @Ignore
+  public void testMatchMAPEvaluation() {
+
+// NOTE: THIS IS BROKEN RIGHT NOW.
+
+String expr = "match{ var1 :  MAP(['foo', 'bar'], (x) -> TO_UPPER(x)) 
}";
+
+Object o = run(expr, ImmutableMap.of("foo", "foo", "bar", "bar", 
"var1", true));
+
+Assert.assertTrue(o instanceof List);
+
+List result = (List) o;
+
+Assert.assertEquals(2, result.size());
+Assert.assertEquals("FOO", result.get(0));
+Assert.assertEquals("BAR", result.get(1));
+}
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchRegexMatch() {
+final Map variableMap = new HashMap() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
+  put("letters", "abcde");
+  put("empty", "");
+}};
+
+Assert.assertTrue(runPredicate("match{ 
REGEXP_MATCH(numbers,numberPattern): true, default : false}", new 
DefaultVariableResolver(v -> variableMap.get(v),v -> 
variableMap.containsKey(v;
+Assert.assertFalse(runPredicate("match{ 
REGEXP_MATCH(letters,numberPattern) : true, default :false}", new 
DefaultVariableResolver(v -> variableMap.get(v),v -> 
variableMap.containsKey(v;
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchBareStatements() {
+
+Assert.assertTrue(runPredicate("match { foo == 0 : bFalse, foo == 1 : 
bTrue, default : false }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertEquals("warning",
+run

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147543426
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -100,6 +102,28 @@ In the core language functions, we support basic 
functional programming primitiv
 * `FILTER` - Filters a list by a predicate in the form of a lambda 
expression.  For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` 
returns `[ 'foo' ]`
 * `REDUCE` - Applies a function over a list of input.  For instance 
`REDUCE([ 1, 2, 3], (sum, x) -> sum + x, 0 )` returns `6`
 
+### Stellar Language Match Expression
+
+Stellar provides the capability to write match expressions, which are 
similar to switch statements commonly found in c like languages, but more like
+Scala's match.
+
+The syntax is:
+* `match{ logical_expression1 : evaluation expression1, 
logical_expression2 : evaluation_expression2` : A match expression with no 
default
--- End diff --

There is a missing `}` here.


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147544129
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MatchTest.java
 ---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.stellar.dsl.DefaultVariableResolver;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.junit.Assert;
+import org.junit.Ignore;
+import org.junit.Test;
+
+public class MatchTest {
+
+  @Test
+  @SuppressWarnings("unchecked")
+  public void testMatchLambda() {
+Assert.assertTrue(runPredicate("match { 1 >= 0 : ()-> true }", new 
HashMap() {{
+  put("foo", 0);
+}}));
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 0);
+}}));
+
+Assert.assertFalse(
+runPredicate("match { foo == 0 : ()-> true, default : ()-> false 
}", new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(
+runPredicate("match { foo == 0 : ()-> false, foo == 1 : ()-> true, 
default : ()-> false }",
+new HashMap() {{
+  put("foo", 1);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+Assert.assertTrue(runPredicate(
+"match { foo == 0 : ()-> bFalse, foo == 1 : ()-> bTrue, default : 
()-> bFalse }",
+new HashMap() {{
+  put("foo", 1);
+  put("bFalse", false);
+  put("bTrue", true);
+}}));
+
+  }
+
+  @Test
+  @SuppressWarnings("unchecked")
+  @Ignore
+  public void testMatchMAPEvaluation() {
+
+// NOTE: THIS IS BROKEN RIGHT NOW.
--- End diff --

Can you delete this comment and add it to `@Ignore("NOTE: THIS IS BROKEN 
RIGHT NOW.")` instead? Maybe also mention that it's broken because the use of 
`MAP` is not supported right now. It may even be better to just delete the 
test, as it will probably be written when that feature is implemented. Also, 
this may just be forgotten about.


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147556583
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/StellarCompiler.java
 ---
@@ -105,59 +107,77 @@ public Object apply(ExpressionState state) {
   Deque> instanceDeque = new ArrayDeque<>();
   {
 boolean skipElse = false;
+boolean skipMatchClauses = false;
 Token token = null;
 for (Iterator> it = getTokenDeque().descendingIterator(); 
it.hasNext(); ) {
   token = it.next();
   //if we've skipped an else previously, then we need to skip the 
deferred tokens associated with the else.
-  if(skipElse && token.getUnderlyingType() == ElseExpr.class) {
-while(it.hasNext()) {
+  if (skipElse && token.getUnderlyingType() == ElseExpr.class) {
+while (it.hasNext()) {
   token = it.next();
-  if(token.getUnderlyingType() == EndConditional.class) {
+  if (token.getUnderlyingType() == EndConditional.class) {
 break;
   }
 }
 skipElse = false;
   }
+  if (skipMatchClauses && (token.getUnderlyingType() == 
MatchClauseEnd.class
+  || token.getUnderlyingType() == MatchClauseCheckExpr.class)) 
{
+while (it.hasNext()) {
--- End diff --

This will only skip one match clause after a true one is found. See below 
test for a stellar statement that fails with a `ParseException`.

```java
@Test
@SuppressWarnings("unchecked")
public void moreThanTwoConsecutiveTrueCasesWillResultInFailure() {
Assert.assertEquals("ok",  run("match{ foo < 100 : THROW('oops'), foo > 
200 : 'ok', foo > 300 : 'ok', default : 'works' }",
new HashMap() {{
put("foo", 500);
}}));
}
```


---

[GitHub] metron pull request #814: METRON-1277 Add match statement to Stellar languag...

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/814#discussion_r147556836
  
--- Diff: 
metron-stellar/stellar-common/src/main/antlr4/org/apache/metron/stellar/common/generated/Stellar.g4
 ---
@@ -272,4 +278,30 @@ lambda_variable:
   IDENTIFIER
   ;
 
+match_expr :
+  MATCH LBRACE match_clauses RBRACE #MatchClauses
+  ;
+
+match_clauses :
+  match_clause (COMMA match_clause)*
+  ;
+
+match_clause :
+  match_clause_check COLON  match_clause_action
+  ;
+
+match_clause_action :
+  match_clause_action_expr #MatchClauseAction
+  ;
 
+match_clause_action_expr :
+  transformation_expr
+  ;
+
+match_clause_check :
--- End diff --

Is this extra expression needed? Could we get rid of it? Instead of this 
just have what's below?

```
match_clause_action :
  transformation_expr #MatchClauseAction
  ;

match_clause_check :
  logical_expr #MatchClauseCheckExpr
  ;
```


---