[GitHub] incubator-metron issue #517: METRON-831: Add lambda expressions and rudiment...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/517 +1 by review. I also spun it up in full dev and tested some basic stellar statements using the new functions. Everything looks good. I think this is a great PR and will be very useful to people. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/517#discussion_r110519705 --- Diff: metron-platform/metron-common/README.md --- @@ -73,6 +73,27 @@ Below is how the `==` operator is expected to work: The `!=` operator is the negation of the above. +## Stellar Language Lambda Expressions + +Stellar provides the capability to pass lambda expressions to functions +which wish to support that layer of indirection. The syntax is: +* `(named_variables) -> stellar_expression` : Lambda expression with named variables + * For instance, the lambda expression which calls `TO_UPPER` on a named argument `x` could be be expressed as `(x) -> TO_UPPER(x)`. +* `var -> stellar_expression` : Lambda expression with a single named variable, `var` + * For instance, the lambda expression which calls `TO_UPPER` on a named argument `x` could be expressed as `x -> TO_UPPER(x)`. Note, this is more succinct but equivalent to the example directly above. +* `() -> stellar_expression` : Lambda expression with no named variables. + * If no named variables are needed, you may omit the named variable section. For instance, the lambda expression which returns a constant `false` would be `() -> false` + +where +* `named_variables` is a comma separated list of variables to use in the Stellar expression +* `stellar_expression` is an arbitrary stellar expression + + +In the core language functions, we support basic functional programming primitives such as +* `MAP` - Applies a lambda expression over a list of input. For instance `MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )` returns `[ 'FOO', 'BAR' ]` +* `FILTER` - Filters a list by a predicate in the form of a lambda expression. For instance `FILTER([ 'foo', 'bar'], (x ) -> x == 'foo' )` returns `[ 'foo' ]` +* `REDUCE` - Applies a function over a list of input. For instance `MAP([ 'foo', 'bar'], (x) -> TO_UPPER(x) )` returns `[ 'FOO', 'BAR' ]` --- End diff -- I believe you may have forgotten to update your copy/paste for the `REDUCE` documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110520142
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java
---
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.dsl.BaseStellarFunction;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.stellar.LambdaExpression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class FunctionalFunctions {
+ @Stellar(name="MAP"
+ , description="Applies lambda expression to a list of arguments.
e.g. MAP( [ 'foo', 'bar' ] , ( x ) -> TO_UPPER(x) ) would yield [ 'FOO', 'BAR'
]"
+ , params = {
+ "list - List of arguments."
+ ,"transform_expression - The lambda expression to
apply. This expression is assumed to take one argument."
+ }
+ , returns = "The input list transformed item-wise by the lambda
expression."
+ )
+ public static class Map extends BaseStellarFunction {
+
+@Override
+public Object apply(List args) {
+ List input = (List) args.get(0);
+ LambdaExpression expression = (LambdaExpression)args.get(1);
+ if(input == null || expression == null) {
+return input;
+ }
+ List ret = new ArrayList<>();
+ for(Object o : input) {
+if(o != null) {
+ ret.add(expression.apply(ImmutableList.of(o)));
+}
+ }
+ return ret;
+}
+ }
+
+ @Stellar(name="FILTER"
+ , description="Applies a filter in the form of a lambda
expression to a list. e.g. FILTER( [ 'foo', 'bar' ] , (x) -> x == 'foo') would
yield [ 'foo']"
+ , params = {
+ "list - List of arguments."
+ ,"predicate - The lambda expression to apply. This
expression is assumed to take one argument and return a boolean."
+ }
+ , returns = "The input list filtered by the predicate."
+ )
+ public static class Filter extends BaseStellarFunction {
+
+@Override
+public Object apply(List args) {
+ List input = (List) args.get(0);
+ LambdaExpression expression = (LambdaExpression) args.get(1);
+ if(input == null || expression == null) {
+return input;
+ }
+ List ret = new ArrayList<>();
+ for(Object o : input) {
+if(o == null) {
+ continue;
+}
+Object result = expression.apply(ImmutableList.of(o));
+if(result != null && result instanceof Boolean && (Boolean)result)
{
+ ret.add(o);
+}
+ }
+ return ret;
+}
+ }
+
+ @Stellar(name="REDUCE"
+ , description="Reduces a list by a binary lambda expression.
That is, the expression takes two arguments. Usage example: REDUCE( [ 1, 2, 3
] , (x, y) -> x + y) would sum the input list, yielding 6."
+ , params = {
+ "list - List of arguments."
+ ,"binary_operation - The lambda expression function
to apply to reduce the list. It is assumed that this takes two arguments, the
first being the running total and the second being an item from the list."
+ ,"initial_value - The initial value to use."
+ }
+ , returns = "The reduction of the list."
+ )
+ public static class Reduce
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/517#discussion_r110520123
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java
---
@@ -0,0 +1,121 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.dsl.functions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.dsl.BaseStellarFunction;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.stellar.LambdaExpression;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class FunctionalFunctions {
+ @Stellar(name="MAP"
+ , description="Applies lambda expression to a list of arguments.
e.g. MAP( [ 'foo', 'bar' ] , ( x ) -> TO_UPPER(x) ) would yield [ 'FOO', 'BAR'
]"
+ , params = {
+ "list - List of arguments."
+ ,"transform_expression - The lambda expression to
apply. This expression is assumed to take one argument."
+ }
+ , returns = "The input list transformed item-wise by the lambda
expression."
+ )
+ public static class Map extends BaseStellarFunction {
+
+@Override
+public Object apply(List args) {
+ List input = (List) args.get(0);
+ LambdaExpression expression = (LambdaExpression)args.get(1);
+ if(input == null || expression == null) {
+return input;
+ }
+ List ret = new ArrayList<>();
+ for(Object o : input) {
+if(o != null) {
--- End diff --
What happens if null means something to the user? They wouldn't be able to
apply null to their expression. Would this ever be a use case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/517#discussion_r110520191 --- Diff: metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/LambdaExpression.java --- @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.common.stellar; + +import org.apache.metron.common.dsl.Token; +import org.apache.metron.common.dsl.VariableResolver; + +import java.util.*; --- End diff -- Nit, explode star import --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #517: METRON-831: Add lambda expressions and r...
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/517#discussion_r110520408 --- Diff: metron-platform/metron-common/README.md --- @@ -466,6 +512,14 @@ The `!=` operator is the negation of the above. * IANA Number * Returns: The protocol name associated with the IANA number. +### `REDUCE` + * Description: Reduces a list by a binary lambda expression. That is, the expression takes two arguments. Usage example: `REDUCE( [ 1, 2, 3 ] , (x, y) -> x + y)` would sum the input list, yielding `6`. + * Input: +* list - List of arguments. +* binary_operation - The lambda expression function to apply to reduce the list. It is assumed that this takes two arguments, the first being the running total and the second being an item from the list. +* initial_value - The initial value to use. --- End diff -- This example is missing the `initial_value`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #504: Adding jmeyer to committer list on commu...
Github user jjmeyer0 closed the pull request at: https://github.com/apache/incubator-metron/pull/504 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #504: Adding jmeyer to committer list on commu...
GitHub user jjmeyer0 reopened a pull request: https://github.com/apache/incubator-metron/pull/504 Adding jmeyer to committer list on community page Added my name to community list. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jjmeyer0/incubator-metron set-committer-jj Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/504.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #504 commit 378a87020b9651fbf9a928eed995e60b46d36c4a Author: JJ <jjmey...@gmail.com> Date: 2017-04-02T16:49:54Z Updating commiter list with my name. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #504: Adding jmeyer to committer list on commu...
GitHub user jjmeyer0 opened a pull request: https://github.com/apache/incubator-metron/pull/504 Adding jmeyer to committer list on community page ## Contributor Comments [Please place any comments here. A description of the problem/enhancement, how to reproduce the issue, your testing methodology, etc.] ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron (Incubating). Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book bin/generate-md.sh mvn site:site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jjmeyer0/incubator-metron set-committer-jj Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/504.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #504 commit 378a87020b9651fbf9a928eed995e60b46d36c4a Author: JJ <jjmey...@gmail.com> Date: 2017-04-02T16:49:54Z Updating commiter list with my name. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 First off, I'd like to apologize for it taking me a little longer to retest this. I went through and tested the issues I saw again. Everything seems to be working. I think this is ready to go. It's a great feature. Thanks for implementing it! Also, since this is a big PR, don't worry about giving me credit on my commits. Especially if it makes your life easier when squashing everything. If that's allowed anyway. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr Good catch on this. I'll take a look again, but I think it's looking good! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr I had a chance to try this again. If I build from `incubator-metron` it doesn't work. If I build from `metron-interface` everything seems to be working just fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr I'm using quick-dev to do my tests. I built everything from metron root with `mvn clean install -DskipTests`. Then I setup `quick-dev`. Extracted the tar and ran the following: ``` java -jar ./lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr For some reason I am still getting the following error on the endpoints. Also, the no such method error turned into this as well. How are you running it? I'm running it using java cli. Are you using IntelliJ or something else? ``` Feb 23, 2017 12:51:24 PM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.NoSuchMethodError: org.reflections.util.ConfigurationBuilder.filterInputsBy(Lorg/apache/metron/guava/base/Predicate;)Lorg/reflections/util/ConfigurationBuilder;] with root cause java.lang.NoSuchMethodError: org.reflections.util.ConfigurationBuilder.filterInputsBy(Lorg/apache/metron/guava/base/Predicate;)Lorg/reflections/util/ConfigurationBuilder; at org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolver.resolvables(ClasspathFunctionResolver.java:167) at org.apache.metron.common.dsl.functions.resolver.BaseFunctionResolver.resolveFunctions(BaseFunctionResolver.java:119) at org.apache.metron.guava.base.Suppliers$MemoizingSupplier.get(Suppliers.java:125) at org.apache.metron.common.dsl.functions.resolver.BaseFunctionResolver.getFunctionInfo(BaseFunctionResolver.java:77) at org.apache.metron.rest.service.impl.StellarServiceImpl.getStellarFunctions(StellarServiceImpl.java:74) at org.apache.metron.rest.controller.StellarController.listFunctions(StellarController.java:71) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r102705316
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/GrokController.java
---
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.controller;
+
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.model.GrokValidation;
+import org.apache.metron.rest.service.GrokService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.ResponseEntity;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.RestController;
+
+import java.util.Map;
+
+@RestController
+@RequestMapping("/api/v1/grok")
+public class GrokController {
+
+@Autowired
+private GrokService grokService;
+
+@ApiOperation(value = "Applies a Grok statement to a sample message")
+@ApiResponse(message = "JSON results", code = 200)
+@RequestMapping(value = "/validate", method = RequestMethod.POST)
+ResponseEntity post(@ApiParam(name = "grokValidation",
value = "Object containing Grok statement and sample message", required = true)
@RequestBody GrokValidation grokValidation) throws RestException {
--- End diff --
Looks like it is working now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r101933431
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/GrokController.java
---
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.controller;
+
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.model.GrokValidation;
+import org.apache.metron.rest.service.GrokService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.ResponseEntity;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.RestController;
+
+import java.util.Map;
+
+@RestController
+@RequestMapping("/api/v1/grok")
+public class GrokController {
+
+@Autowired
+private GrokService grokService;
+
+@ApiOperation(value = "Applies a Grok statement to a sample message")
+@ApiResponse(message = "JSON results", code = 200)
+@RequestMapping(value = "/validate", method = RequestMethod.POST)
+ResponseEntity post(@ApiParam(name = "grokValidation",
value = "Object containing Grok statement and sample message", required = true)
@RequestBody GrokValidation grokValidation) throws RestException {
--- End diff --
When testing this endpoint for some reason the example below doesn't return
a result.
Example:
```
{
"sampleData": "301dd766-78c6-4c56-b8c7-178bb91fdb2c",
"statement": "UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}"
}
```
Response:
```
{
"statement": "UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}",
"sampleData": "301dd766-78c6-4c56-b8c7-178bb91fdb2c",
"results": {}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r101934020
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/StellarController.java
---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.controller;
+
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import org.apache.metron.common.field.transformation.FieldTransformations;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.model.StellarFunctionDescription;
+import org.apache.metron.rest.model.SensorParserContext;
+import org.apache.metron.rest.service.StellarService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.ResponseEntity;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.RestController;
+
+import java.util.List;
+import java.util.Map;
+
+@RestController
+@RequestMapping("/api/v1/stellar")
+public class StellarController {
+
+@Autowired
+private StellarService stellarService;
+
+ @ApiOperation(value = "Tests Stellar statements to ensure they are
well-formed")
+ @ApiResponse(message = "Returns validation results", code = 200)
+@RequestMapping(value = "/validate/rules", method = RequestMethod.POST)
+ResponseEntity<Map<String, Boolean>>
validateRules(@ApiParam(name="statements", value="List of statements to
validate", required=true)@RequestBody List statements) throws
RestException {
+return new
ResponseEntity<>(stellarService.validateRules(statements), HttpStatus.OK);
+}
+
+ @ApiOperation(value = "Executes transformations against a sample
message")
+ @ApiResponse(message = "Returns transformation results", code = 200)
+@RequestMapping(value = "/apply/transformations", method =
RequestMethod.POST)
+ResponseEntity<Map<String, Object>>
applyTransformations(@ApiParam(name="transformationValidation", value="Object
containing SensorParserConfig and sample message", required=true)@RequestBody
SensorParserContext sensorParserContext) throws RestException {
+return new
ResponseEntity<>(stellarService.applyTransformations(sensorParserContext),
HttpStatus.OK);
+}
+
+ @ApiOperation(value = "Retrieves field transformations")
+ @ApiResponse(message = "Returns a list field transformations", code =
200)
+@RequestMapping(value = "/list", method = RequestMethod.GET)
+ResponseEntity<FieldTransformations[]> list() throws RestException {
+return new ResponseEntity<>(stellarService.getTransformations(),
HttpStatus.OK);
+}
+
+ @ApiOperation(value = "Lists the Stellar functions that can be found on
the classpath")
+ @ApiResponse(message = "Returns a list of Stellar functions", code = 200)
+@RequestMapping(value = "/list/functions", method = RequestMethod.GET)
+ResponseEntity<List> listFunctions()
throws RestException {
+return new ResponseEntity<>(stellarService.getStellarFunctions(),
HttpStatus.OK);
+}
+
+ @ApiOperation(value = "Lists the simple Stellar functions (functions
with only 1 input) that can be found on the classpath")
+ @ApiResponse(message = "Returns a list of simple Stellar functions",
code = 200)
+@RequestMapping(value = "/list/simple/functions", m
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r101934168
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/SensorEnrichmentConfigController.java
---
@@ -0,0 +1,97 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.controller;
+
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.service.SensorEnrichmentConfigService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.ResponseEntity;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.RestController;
+
+import java.util.List;
+import java.util.Map;
+
+@RestController
+@RequestMapping("/api/v1/sensor/enrichment/config")
+public class SensorEnrichmentConfigController {
+
+ @Autowired
+ private SensorEnrichmentConfigService sensorEnrichmentConfigService;
+
+ @ApiOperation(value = "Updates or creates a SensorEnrichmentConfig in
Zookeeper")
+ @ApiResponses(value = { @ApiResponse(message = "SensorEnrichmentConfig
updated. Returns saved SensorEnrichmentConfig", code = 200),
+ @ApiResponse(message = "SensorEnrichmentConfig created. Returns
saved SensorEnrichmentConfig", code = 201) })
+ @RequestMapping(value = "/{name}", method = RequestMethod.POST)
+ ResponseEntity save(@ApiParam(name="name",
value="SensorEnrichmentConfig name", required=true)@PathVariable String name,
+
@ApiParam(name="sensorEnrichmentConfig", value="SensorEnrichmentConfig",
required=true)@RequestBody SensorEnrichmentConfig sensorEnrichmentConfig)
throws RestException {
+if (sensorEnrichmentConfigService.findOne(name) == null) {
+ return new ResponseEntity<>(sensorEnrichmentConfigService.save(name,
sensorEnrichmentConfig), HttpStatus.CREATED);
+} else {
+ return new ResponseEntity<>(sensorEnrichmentConfigService.save(name,
sensorEnrichmentConfig), HttpStatus.OK);
+}
+ }
+
+ @ApiOperation(value = "Retrieves a SensorEnrichmentConfig from
Zookeeper")
+ @ApiResponses(value = { @ApiResponse(message = "Returns
SensorEnrichmentConfig", code = 200),
+ @ApiResponse(message = "SensorEnrichmentConfig is missing", code
= 404) })
+ @RequestMapping(value = "/{name}", method = RequestMethod.GET)
+ ResponseEntity findOne(@ApiParam(name="name",
value="SensorEnrichmentConfig name", required=true)@PathVariable String name)
throws RestException {
+SensorEnrichmentConfig sensorEnrichmentConfig =
sensorEnrichmentConfigService.findOne(name);
+if (sensorEnrichmentConfig != null) {
+ return new ResponseEntity<>(sensorEnrichmentConfig, HttpStatus.OK);
+}
+
+return new ResponseEntity<>(HttpStatus.NOT_FOUND);
+ }
+
+ @ApiOperation(value = "Retrieves all SensorEnrichmentConfigs from
Zookeeper")
+ @ApiResponse(message = "Returns all SensorEnrichmentConfigs", code = 200)
+ @RequestMapping(method = RequestMethod.GET)
+ ResponseEntity<Map<String, SensorEnrichmentConfig>> getAll() throws
Exception {
--- End diff --
I g
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r101933196
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/HdfsController.java
---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.controller;
+
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.service.HdfsService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.ResponseEntity;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.RequestParam;
+import org.springframework.web.bind.annotation.RestController;
+
+import java.util.List;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+@RestController
+@RequestMapping("/api/v1/hdfs")
+public class HdfsController {
+
+ @Autowired
+ private HdfsService hdfsService;
+
+ @ApiOperation(value = "Reads a file from HDFS and returns the contents")
+ @ApiResponse(message = "Returns file contents", code = 200)
+ @RequestMapping(value = "/list", method = RequestMethod.GET)
+ ResponseEntity<List> list(@ApiParam(name = "path", value = "Path
to HDFS directory", required = true) @RequestParam String path) throws
RestException {
+return new ResponseEntity<>(hdfsService.list(new Path(path)),
HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Reads a file from HDFS and returns the contents")
+ @ApiResponse(message = "Returns file contents", code = 200)
+ @RequestMapping(method = RequestMethod.GET)
+ ResponseEntity read(@ApiParam(name = "path", value = "Path to
HDFS file", required = true) @RequestParam String path) throws RestException {
+String contents = hdfsService.read(new Path(path));
+if (contents != null) {
+ return new ResponseEntity<>(hdfsService.read(new Path(path)),
HttpStatus.OK);
+} else {
+ return new ResponseEntity<>(HttpStatus.NOT_FOUND);
+}
+
+ }
+
+ @ApiOperation(value = "Writes contents to an HDFS file")
+ @ApiResponse(message = "Contents were written", code = 200)
+ @RequestMapping(method = RequestMethod.POST)
+ ResponseEntity write(@ApiParam(name="path", value="Path to HDFS
file", required=true) @RequestParam String path,
--- End diff --
This is potentially a dangerous operation. Can you add to the description
that this operation will overwrite the contents of a file that already exists?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/316#discussion_r101921984 --- Diff: metron-interface/metron-rest/pom.xml --- @@ -0,0 +1,335 @@ + + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + +org.apache.metron +metron-interface +0.3.0 --- End diff -- Update to match new Metron version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/316#discussion_r101922012 --- Diff: metron-interface/pom.xml --- @@ -0,0 +1,90 @@ + + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 +metron-interface +pom +metron-interface + +org.apache.metron +Metron +0.3.0 --- End diff -- Update to match new Metron version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r101922040
--- Diff: metron-interface/metron-rest/src/test/resources/README.vm ---
@@ -0,0 +1,142 @@
+#[[#]]# Metron REST
+
+This module provides a RESTful API for interacting with Metron.
+
+#[[##]]# Prerequisites
+
+* A running Metron cluster
+* Java 8 installed
+* Storm CLI and Metron topology scripts (start_parser_topology.sh,
start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed
+
+#[[##]]# Installation
+1. Package the application with Maven:
+```
+mvn clean package
+```
+
+1. Untar the archive in the target directory. The directory structure
will look like:
+```
+bin
+ start_metron_rest.sh
+lib
+ metron-rest-$METRON_VERSION.jar
+```
+
+1. Create an `application.yml` file with the contents of
[application-docker.yml](src/main/resources/application-docker.yml).
Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc)
in properties containing `${docker.host.address}` and update the
`spring.datasource.*` properties as needed (see the [Security](#security)
section for more details).
+
+1. Start the application with this command:
+```
+./bin/start_metron_rest.sh /path/to/application.yml
+```
+
+#[[##]]# Usage
+
+The exposed REST endpoints can be accessed with the Swagger UI at
http://host:port/swagger-ui.html#/. The default port is 8080 but can be
changed in application.yml by setting "server.port" to the desired port.
+
+#[[##]]# Security
+
+The metron-rest module uses [Spring
Security](http://projects.spring.io/spring-security/) for authentication and
stores user credentials in a relational database. The H2 database is
configured by default and is intended only for development purposes. The "dev"
profile can be used to automatically load test users:
+```
+./bin/start_metron_rest.sh /path/to/application.yml
--spring.profiles.active=dev
+```
+
+For [production
use](http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/reference/htmlsingle/#boot-features-connect-to-production-database),
a relational database should be configured. For example, configuring MySQL
would be done as follows:
+
+1. Create a MySQL user for the Metron REST application
(http://dev.mysql.com/doc/refman/5.7/en/adding-users.html).
+
+1. Connect to MySQL and create a Metron REST database:
+```
+CREATE DATABASE IF NOT EXISTS metronrest
+```
+
+1. Add users:
+```
+use metronrest;
+insert into users (username, password, enabled) values
('your_username','your_password',1);
+insert into authorities (username, authority) values ('your_username',
'ROLE_USER');
+```
+
+1. Replace the H2 connection information in the application.yml file with
MySQL connection information:
+```
+spring:
+ datasource:
+driverClassName: com.mysql.jdbc.Driver
+url: jdbc:mysql://mysql_host:3306/metronrest
+username: metron_rest_user
+password: metron_rest_password
+platform: mysql
+```
+
+1. Add a dependency for the MySQL JDBC connector in the metron-rest
pom.xml:
+```
+
+ mysql
+ mysql-connector-java
+ ${mysql.client.version}
+
+```
+
+1. Follow the steps in the [Installation](#installation) section
+
+#[[##]]# API
+
+Request and Response objects are JSON formatted. The JSON schemas are
available in the Swagger UI.
+
+||
+| -- |
+#foreach( $restControllerInfo in $endpoints )
+| [ `$restControllerInfo.getMethod().toString()
$restControllerInfo.getPath()`](#$restControllerInfo.getMethod().toString().toLowerCase()-$restControllerInfo.getPath().toLowerCase().replaceAll("/",
""))|
+#end
+
+#foreach( $restControllerInfo in $endpoints )
+#[[###]]# `$restControllerInfo.getMethod().toString()
$restControllerInfo.getPath()`
+ * Description: $restControllerInfo.getDescription()
+#if($restControllerInfo.getParameterDescriptions().size() > 0)
+ * Input:
+#end
+#foreach( $parameterDescription in
$restControllerInfo.getParameterDescriptions().entrySet())
+* $parameterDescription.getKey() - $parameterDescription.getValue()
+#end
+ * Returns:
+#foreach( $response in $restControllerInfo.getResponses())
+* $response.getCode() - $response.getMessage()
+#end
+
+#end
+
+#[[##]]# Testing
+
+Profiles are includes for both the metron-docker and Quick Dev
environments.
+
+#[[###]]# metron-docker
+
+Start the [metron-docker](../../metron-docker) environment. Build the
metron-rest module and start it with the Spring
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/316#discussion_r101921992 --- Diff: metron-interface/metron-rest/src/main/resources/application.yml --- @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +metron: + version: 0.3.0 --- End diff -- Update to match new Metron version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr sorry for not commenting earlier. I think this is in a pretty good state. I do want to review this again from beginning to end after you finish the MySQL stuff. After that I'll either +1 or add some comments. It's looking good though! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/456 @ottobackwards looks good to me. Again, great idea. I think it's ready to be merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/456 Oh, absolutely. Sorry, I didn't mean to suggest to add the whole page. I just meant a check box that says something like: Have verified building and running locally by creating `full-dev` environment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/456 I agree it shouldn't be a duplicate of the dev guide. My thought on the verify builds was that quite a few PRs state they've done it. That being said, I'm not hung up on it. I'll defer to you. Other than that, I agree this is a great idea, and looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/456 Also, what do you think about adding a check box for verifying builds? https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #456: METRON-721 Add github pull request templ...
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/456#discussion_r101286501 --- Diff: .github/PULL_REQUEST_TEMPLATE.md --- @@ -0,0 +1,37 @@ +Thank you for submitting a contribution to Apache Metron (Incubating). +Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions + +In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check +the following: + +### For all changes: +- [ ] Is there a JIRA ticket associated with this PR? --- End diff -- In our dev guide it gives directions to create one if a Jira doesn't exist. Do you think it makes sense for these to match? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr the builds failing due to the `Cli to CLI` change. I've hit this before. I think it's because mac is case insensitive and gets confused. To fix it, I've had to manually change the file name in github. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r99253356
--- Diff:
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCliWrapperTest.java
---
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.service.impl;
+
+import org.apache.metron.rest.MetronRestConstants;
+import org.apache.metron.rest.RestException;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.powermock.core.classloader.annotations.PrepareForTest;
+import org.powermock.modules.junit4.PowerMockRunner;
+import org.springframework.core.env.Environment;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.HashMap;
+import java.util.Map;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.anyVararg;
+import static org.mockito.Mockito.verify;
+import static org.powermock.api.mockito.PowerMockito.mock;
+import static org.powermock.api.mockito.PowerMockito.verifyNew;
+import static org.powermock.api.mockito.PowerMockito.when;
+import static org.powermock.api.mockito.PowerMockito.whenNew;
+
+@SuppressWarnings("unchecked")
+@RunWith(PowerMockRunner.class)
+@PrepareForTest({DockerStormCLIWrapper.class, ProcessBuilder.class})
+public class StormCliWrapperTest {
--- End diff --
I know this is a very minor thing, but I think the name of this class
should change to `StormCLIWrapperTest` or all other CLI usages should be
changed to Cli. I'm not sure if we've decided on either one of these being our
standard though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r99252325
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/GrokServiceImpl.java
---
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.service.impl;
+
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import org.apache.directory.api.util.Strings;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.model.GrokValidation;
+import org.apache.metron.rest.service.GrokService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.core.env.Environment;
+import org.springframework.security.core.Authentication;
+import org.springframework.security.core.context.SecurityContextHolder;
+import org.springframework.stereotype.Service;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.StringReader;
+import java.util.Map;
+
+import static
org.apache.metron.rest.MetronRestConstants.GROK_TEMP_PATH_SPRING_PROPERTY;
+
+@Service
+public class GrokServiceImpl implements GrokService {
+
+private Environment environment;
+
+private Grok commonGrok;
+
+@Autowired
+public GrokServiceImpl(Environment environment, Grok commonGrok) {
+this.environment = environment;
+this.commonGrok = commonGrok;
+}
+
+@Override
+public Map<String, String> getCommonGrokPatterns() {
+return commonGrok.getPatterns();
+}
+
+@Override
+public GrokValidation validateGrokStatement(GrokValidation
grokValidation) throws RestException {
+Map<String, Object> results;
+try {
+String statement =
Strings.isEmpty(grokValidation.getStatement()) ? "" :
grokValidation.getStatement();
+
+Grok grok = new Grok();
+grok.addPatternFromReader(new
InputStreamReader(getClass().getResourceAsStream("/patterns/common")));
+grok.addPatternFromReader(new StringReader(statement));
+String patternLabel = statement.substring(0,
statement.indexOf(" "));
+String grokPattern = "%{" + patternLabel + "}";
+grok.compile(grokPattern);
+Match gm = grok.match(grokValidation.getSampleData());
+gm.captures();
+results = gm.toMap();
+results.remove(patternLabel);
+} catch (StringIndexOutOfBoundsException e) {
+throw new RestException("A pattern label must be included (eg.
PATTERN_LABEL %{PATTERN:field} ...)", e.getCause());
+} catch (Exception e) {
+throw new RestException(e);
+}
+grokValidation.setResults(results);
+return grokValidation;
+}
+
+@Override
+public File saveTemporary(String statement, String name) throws
RestException {
+if (statement != null) {
+try {
+File grokDirectory = new File(getTemporaryGrokRootPath());
+if (!grokDirectory.exists()) {
+ grokDirectory.mkdirs();
+}
+File path = new File(grokDirectory, name);
+FileWriter fileWriter = new FileWriter(new
File(grokDirectory, name));
--- End diff --
There may be an issue where the same `name` is passed in causing files to
be overwritten. I think we should use Java's `File. createTempFile`. Does this
make sense to do this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr you are right about guava. I'll remove my uses of it. Plus, Java has a lot of those functions built in now. No reason I shouldn't use those instead. I agree that the guava issue should be taken care of separately. Right now, I think we should allow a user to set an envitonment variable to point to which guava to use (similar to hibernate and mysql). This will make it easier for us to run it outside of our IDE. I think your changes look pretty good. Really nice clean up/tests. We've changed a lot, so I want to do another pass though. I think we should still have our `HdfsService`. Right now, I think using `FileSystem` directly works for us. Using Knox is a good idea, but it probably needs to be optional. My thought is having the service will help us achieve that. You are probably right though. If we can achieve everything with WebHDFS it's better for our service to use that. This PR is already really big though. Maybe we can have that as a take away? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #430: METRON-658: Updated Grammar to Handle More Uses...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/430 @nickwallen personally I don't think it should be a separate PR. To me it is just one of those things that should be cleaned up as we go. My reasoning is that I think we should encourage clean up as we go with minimal overhead. However, I understand that it is difficult to read on github. I do agree that I could have been more conscious of the way I made the commits though. I changed this PR to have two commits,1 with the changes and 1 with the cleanup. If this is still hard to review, I can split them into two PRs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #430: METRON-658: Updated Grammar to Handle Mo...
GitHub user jjmeyer0 opened a pull request:
https://github.com/apache/incubator-metron/pull/430
METRON-658: Updated Grammar to Handle More Uses of in/not in Expressions
This PR allows for the use of in/not in expressions in more places. For
example, the following expressions now work:
('grok' not in 'foobar') == true
'grok' not in ('foobar' == true)
'grok' not in 'foobar' == true
It is important to note that github's diff of `Stellar.g4` isn't very nice.
A vast majority of the changes are just whitespace.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jjmeyer0/incubator-metron METRON-658
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/430.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #430
commit b06217ec107eff2f8327c4ffb7fd4040849bb36e
Author: JJ <jjmey...@gmail.com>
Date: 2017-01-30T16:24:21Z
METRON-658: Updated grammar to handle more complicated in/not in statements.
commit d948bba85f3df13535d99669b9a3e955be1b4d1a
Author: JJ <jjmey...@gmail.com>
Date: 2017-01-30T20:18:55Z
Updated tests ad cleaned up some code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr as we discussed, I created a PR against yours with some unit tests for the services. Let me know if you have any questions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr this is looking good. I like how you broke out some of the services and started a constants class. There were two very minor things that I noticed. 1. `@Override` should be used wherever possible on all service class method that implement an interface. 2. I don't believe `@Service` is needed on the interfaces. I continued to test some more endpoints and things are continuing to look good. I still want to test out some more endpoints. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/316#discussion_r97171605 --- Diff: metron-interface/metron-rest-client/src/main/java/org/apache/metron/rest/model/GrokValidation.java --- @@ -0,0 +1,51 @@ +/** --- End diff -- @merrimanr i think that's a really good idea. You're right, changing the spaces would add a lot of distractions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 I've tested a few controllers (global config, grok controller, and kafka controller). Everything seems to be working pretty well. I will continue to test more, but since this is a large PR I wanted to do smaller chunks at a time. I think we should start thinking about is how we are going to architect the service layer. I'd like to discuss two things that I think will help us improve the maintainability of this. *Each service should implement an interface.** **We should discuss how to better break up services.** I think services should be conceptually simple. Mostly interact with a specific system. If we start getting into more complicated situations where multiple services are needed these should be in a separate area. I consider situations like these to be more of a service facade. A current example of this is with the `StormService`. I think the interactions with the storm API should be in its own service. Then a facade would be used with the storm service and other necessary services to give a simpler view into storm based on our context. Does this make sense? Personally I think number 1 should be done apart of this PR and number 2 should probably turn into a wider discussion. What do you all think? If others agree, I'd be glad to write up a Jira to discuss number 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r97105827
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/KafkaService.java
---
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.service;
+
+import kafka.admin.AdminUtils;
+import kafka.admin.RackAwareMode;
+import kafka.utils.ZkUtils;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.metron.rest.model.KafkaTopic;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+@Service
+public class KafkaService {
+
+@Autowired
+private ZkUtils zkUtils;
+
+@Autowired
+private KafkaConsumer<String, String> kafkaConsumer;
+
+private String offsetTopic = "__consumer_offsets";
+
+public KafkaTopic createTopic(KafkaTopic topic) {
+if (!listTopics().contains(topic.getName())) {
--- End diff --
It looks like this can throw a AdminOperationException. Is there a way we
can map these to our RestException? We may be able to do this through another
`ResponseEntityExceptionHandler`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r97079434
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/GrokService.java
---
@@ -0,0 +1,164 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.service;
+
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.parsers.GrokParser;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.model.GrokValidation;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.core.env.Environment;
+import org.springframework.security.core.Authentication;
+import org.springframework.security.core.context.SecurityContextHolder;
+import org.springframework.stereotype.Service;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.io.StringReader;
+import java.util.Map;
+
+@Service
+public class GrokService {
+
+public static final String GROK_DEFAULT_PATH_SPRING_PROPERTY =
"grok.path.default";
+public static final String GROK_TEMP_PATH_SPRING_PROPERTY =
"grok.path.temp";
+public static final String GROK_CLASS_NAME =
GrokParser.class.getName();
+public static final String GROK_PATH_KEY = "grokPath";
+public static final String GROK_STATEMENT_KEY = "grokStatement";
+public static final String GROK_PATTERN_LABEL_KEY = "patternLabel";
+
+@Autowired
+private Environment environment;
+
+@Autowired
+private Grok commonGrok;
+
+@Autowired
+private HdfsService hdfsService;
+
+public Map<String, String> getCommonGrokPatterns() {
+return commonGrok.getPatterns();
+}
+
+public GrokValidation validateGrokStatement(GrokValidation
grokValidation) throws RestException {
+Map<String, Object> results;
+try {
+Grok grok = new Grok();
+grok.addPatternFromReader(new
InputStreamReader(getClass().getResourceAsStream("/patterns/common")));
+grok.addPatternFromReader(new
StringReader(grokValidation.getStatement()));
+String patternLabel =
grokValidation.getStatement().substring(0,
grokValidation.getStatement().indexOf(" "));
+String grokPattern = "%{" + patternLabel + "}";
+grok.compile(grokPattern);
+Match gm = grok.match(grokValidation.getSampleData());
+gm.captures();
+results = gm.toMap();
+results.remove(patternLabel);
--- End diff --
I don't think I fully understand this method. Can you explain what is
supposed to be in the results part of the `GrokValidation` object. Is it
supposed to be the matched results in the sample data? If so I don't think this
will ever happen because they are removed at this line. Am I misunderstanding
this correctly or am I missing something?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96933025
--- Diff: metron-interface/metron-rest/README.md ---
@@ -0,0 +1,458 @@
+# Metron REST and Configuration UI
+
+This UI exposes and aids in sensor configuration.
+
+## Prerequisites
+
+* A running Metron cluster
+* A running instance of MySQL
+* Java 8 installed
+* Storm CLI and Metron topology scripts (start_parser_topology.sh,
start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed
+
+## Installation
+1. Package the Application with Maven:
+```
+mvn clean package
+```
+
+1. Untar the archive in the target directory. The directory structure
will look like:
+```
+bin
+ start.sh
+lib
+ metron-rest-version.jar
+```
+
+1. Install Hibernate by downloading version 5.0.11.Final from
(http://hibernate.org/orm/downloads/). Unpack the archive and set the
HIBERNATE_HOME environment variable to the absolute path of the top level
directory.
+```
+export HIBERNATE_HOME=/path/to/hibernate-release-5.0.11.Final
+```
+
+1. Install the MySQL client by downloading version 5.1.40 from
(https://dev.mysql.com/downloads/connector/j/). Unpack the archive and set the
MYSQL_CLIENT_HOME environment variable to the absolute path of the top level
directory.
+```
+export MYSQL_CLIENT_HOME=/path/to/mysql-connector-java-5.1.40
+```
+
+1. Create a MySQL user for the Config UI
(http://dev.mysql.com/doc/refman/5.7/en/adding-users.html).
+
+1. Create a Config UI database in MySQL with this command:
+```
+CREATE DATABASE IF NOT EXISTS metronrest
+```
+
+1. Create an `application.yml` file with the contents of
[application-docker.yml](src/main/resources/application-docker.yml).
Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc)
in properties containing `${docker.host.address}` and update the
`spring.datasource.username` and `spring.datasource.password` properties using
the MySQL credentials from step 4.
+
+1. Start the UI with this command:
+```
+./bin/start.sh /path/to/application.yml
+```
+
+## Usage
+
+The exposed REST endpoints can be accessed with the Swagger UI at
http://host:port/swagger-ui.html#/. The default port is 8080 but can be
changed in application.yml by setting "server.port" to the desired port. Users
can be added with this SQL statement:
+```
+use metronrest;
+insert into users (username, password, enabled) values
('your_username','your_password',1);
+insert into authorities (username, authority) values ('your_username',
'ROLE_USER');
+```
+Users can be added to additional groups with this SQL statement:
+```
+use metronrest;
+insert into authorities (username, authority) values ('your_username',
'your_group');
+```
+
+## API
+
+Request and Response objects are JSON formatted. The JSON schemas are
available in the Swagger UI.
+
+||
+| -- |
+| [ `GET /api/v1/global/config`](#get-apiv1globalconfig)|
+| [ `DELETE /api/v1/global/config`](#delete-apiv1globalconfig)|
+| [ `POST /api/v1/global/config`](#post-apiv1globalconfig)|
+| [ `GET /api/v1/grok/list`](#get-apiv1groklist)|
+| [ `POST /api/v1/grok/validate`](#post-apiv1grokvalidate)|
+| [ `GET /api/v1/kafka/topic`](#get-apiv1kafkatopic)|
+| [ `POST /api/v1/kafka/topic`](#post-apiv1kafkatopic)|
+| [ `GET /api/v1/kafka/topic/{name}`](#get-apiv1kafkatopic{name})|
+| [ `DELETE /api/v1/kafka/topic/{name}`](#delete-apiv1kafkatopic{name})|
+| [ `GET
/api/v1/kafka/topic/{name}/sample`](#get-apiv1kafkatopic{name}sample)|
+| [ `GET
/api/v1/sensor/enrichment/config`](#get-apiv1sensorenrichmentconfig)|
+| [ `GET
/api/v1/sensor/enrichment/config/list/available`](#get-apiv1sensorenrichmentconfiglistavailable)|
+| [ `DELETE
/api/v1/sensor/enrichment/config/{name}`](#delete-apiv1sensorenrichmentconfig{name})|
+| [ `POST
/api/v1/sensor/enrichment/config/{name}`](#post-apiv1sensorenrichmentconfig{name})|
+| [ `GET
/api/v1/sensor/enrichment/config/{name}`](#get-apiv1sensorenrichmentconfig{name})|
+| [ `GET /api/v1/sensor/indexing/config`](#get-apiv1sensorindexingconfig)|
+| [ `DELETE
/api/v1/sensor/indexing/config/{name}`](#delete-apiv1sensorindexingconfig{name})|
+| [ `POST
/api/v1/sensor/indexing/config/{name}`](#post-apiv1sensorindexingconfig{name})|
+| [ `GET
/api/v1/sensor/indexing/config/{name}`](#get-apiv1sensorindexingconfig{name})|
+| [ `POST /api/v1/sensor/parser/config`](#post-apiv1sensorparserconfig)|
+| [ `GET /api/v1/sensor/parser/config`](#get-apiv1sensorparserconfig)|
+| [ `GET
/api
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96929829
--- Diff:
metron-interface/metron-rest-client/src/main/java/org/apache/metron/rest/model/GrokValidation.java
---
@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.model;
+
+import java.util.Map;
+
+public class GrokValidation {
+
+private String statement;
+private String sampleData;
+private Map<String, Object> results;
+
+public String getStatement() {
+return statement;
+}
+
+public void setStatement(String statement) {
+this.statement = statement;
+}
+
+public String getSampleData() {
+return sampleData;
+}
+
+public void setSampleData(String sampleData) {
+this.sampleData = sampleData;
+}
+
+public Map<String, Object> getResults() {
+return results;
--- End diff --
This could possible result in a NPE. Does it make sense to check for this
and return an empty map if that is the case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96934260
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/GrokConfig.java
---
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.config;
+
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+import java.io.InputStreamReader;
+
+@Configuration
+public class GrokConfig {
+
+@Bean
+public Grok commonGrok() throws GrokException {
+Grok grok = new Grok();
+grok.addPatternFromReader(new
InputStreamReader(getClass().getResourceAsStream("/patterns/common")));
--- End diff --
Can the path, '/patterns/common', be extracted to a configurable property?
Would that ever be used?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/316#discussion_r96930729 --- Diff: metron-interface/metron-rest-client/src/main/java/org/apache/metron/rest/model/SensorParserConfigVersion.java --- @@ -0,0 +1,51 @@ +/** --- End diff -- 4 spaces instead of 2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96936537
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/GlobalConfigController.java
---
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.controller;
+
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.service.GlobalConfigService;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.ResponseEntity;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RequestMethod;
+import org.springframework.web.bind.annotation.RestController;
+
+import java.util.Map;
+
+@RestController
+@RequestMapping("/api/v1/global/config")
+public class GlobalConfigController {
+
+@Autowired
+private GlobalConfigService globalConfigService;
+
+@ApiOperation(value = "Creates or updates the Global Config in
Zookeeper")
+@ApiResponse(message = "Returns saved Global Config JSON", code = 200)
--- End diff --
Right now only a 201 is returned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/316#discussion_r96930355 --- Diff: metron-interface/metron-rest-client/src/main/java/org/apache/metron/rest/model/GrokValidation.java --- @@ -0,0 +1,51 @@ +/** --- End diff -- 4 spaces instead of 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96933666
--- Diff: metron-interface/metron-rest/pom.xml ---
@@ -0,0 +1,357 @@
+
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.metron
+metron-interface
+0.3.0
+
+metron-rest
+${project.parent.version}
+
+UTF-8
+
UTF-8
+1.8
+4.5
+2.7.1
+1.6.4
+1.4.1.RELEASE
+2.5.0
+5.1.40
+2.9.4
+5.0.11.Final
+
+
+
+org.springframework.boot
+spring-boot-starter-web
+
+
+ch.qos.logback
+logback-classic
+
+
+org.slf4j
+log4j-over-slf4j
+
+
+org.hibernate
+hibernate-validator
+
+
+
+
+org.springframework.boot
+spring-boot-starter-security
+
+
+org.springframework.boot
+spring-boot-starter-data-jpa
+
+
+org.hibernate
+hibernate-core
+
+
+org.hibernate
+hibernate-entitymanager
+
+
+
+
+org.hibernate
+hibernate-core
+${hibernate.version}
+provided
+
+
+mysql
+mysql-connector-java
+${mysql.client.version}
+provided
+
+
+org.apache.curator
+curator-recipes
+${curator.version}
+
+
+com.googlecode.json-simple
+json-simple
+${global_json_simple_version}
+
+
+org.antlr
+antlr4-runtime
+${antlr.version}
+
+
+com.fasterxml.jackson.core
+jackson-databind
+2.8.3
+
+
+com.fasterxml.jackson.datatype
+jackson-datatype-joda
+2.8.1
+
+
+org.apache.metron
+metron-rest-client
+${project.parent.version}
+
+
+org.apache.metron
+metron-common
+${project.parent.version}
+
+
+com.fasterxml.jackson.core
+jackson-databind
+
+
+
+
+org.apache.metron
+metron-parsers
+${project.parent.version}
+
+
+com.fasterxml.jackson.core
+jackson-databind
+
+
+org.apache.metron
+metron-profiler-client
+
+
+org.apache.metron
+metron-writer
+
+
+
+
+io.springfox
+springfox-swagger2
+${swagger.version}
+
+
+io.springfox
+springfox-swagger-ui
+${swagger.version}
+
+
+org.apache.kafka
+kafka_2.10
+${global_kafka_version}
+
+
+joda-time
+joda-time
+${joda.time.version}
+
+
+org.hibernate
+hibernate-envers
+${hibernate.version}
+provided
+
+
+io.thekraken
+grok
+0.1.0
+
+
+org.reflections
+reflections
+0.9.10
+
+
+com.google.code.findbugs
+
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96936912
--- Diff: metron-interface/metron-rest-client/pom.xml ---
@@ -0,0 +1,54 @@
+
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.metron
+metron-interface
+0.3.0
+
+metron-rest-client
+${project.parent.version}
--- End diff --
Maven complains about this being a property. It wants it to be hard coded.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96936044
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/ZookeeperConfig.java
---
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.config;
+
+import kafka.utils.ZKStringSerializer$;
+import org.I0Itec.zkclient.ZkClient;
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.Profile;
+import org.springframework.core.env.Environment;
+
+@Configuration
+@Profile("!test")
+public class ZookeeperConfig {
+
+ public static final String ZK_URL_SPRING_PROPERTY = "zookeeper.url";
+
+ @Bean(initMethod = "start", destroyMethod="close")
+ public CuratorFramework client(Environment environment) {
+RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
--- End diff --
Does it make sense for these values to be configurable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96930979
--- Diff:
metron-interface/metron-rest-client/src/main/java/org/apache/metron/rest/model/StellarFunctionDescription.java
---
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.model;
+
+public class StellarFunctionDescription {
+
+private String name;
+private String description;
+private String[] params;
+private String returns;
+
+public StellarFunctionDescription(String name, String description,
String[] params, String returns) {
+this.name = name;
+this.description = description;
+this.params = params;
+this.returns = returns;
+}
+
+public String getName() {
+return name;
+}
+
+public String getDescription() {
+return description;
+}
+
+public String[] getParams() {
+return params;
--- End diff --
Potentially causes NPE. Does it make sense to set to empty array if it's
null?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96933249
--- Diff: metron-interface/metron-rest/pom.xml ---
@@ -0,0 +1,357 @@
+
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.metron
+metron-interface
+0.3.0
+
+metron-rest
+${project.parent.version}
--- End diff --
Maven complains about this. I think maven wants us to hard code versions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96931769
--- Diff:
metron-interface/metron-rest-client/src/main/java/org/apache/metron/rest/model/TopologySummary.java
---
@@ -0,0 +1,31 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.model;
+
+public class TopologySummary {
+
+private TopologyStatus[] topologies;
+
+public TopologyStatus[] getTopologies() {
+return topologies;
+}
+
+public void setTopologies(TopologyStatus[] topologies) {
+this.topologies = topologies;
--- End diff --
Same NPE comment as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/316#discussion_r96931516
--- Diff:
metron-interface/metron-rest-client/src/main/java/org/apache/metron/rest/model/TopologyResponse.java
---
@@ -0,0 +1,42 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.model;
+
+public class TopologyResponse {
+
+ private String status;
+ private String message;
+
+ public String getStatus() {
+return status;
+ }
+
+ public String getMessage() {
+return message;
+ }
+
+ public void setSuccessMessage(String message) {
+this.status = "success";
--- End diff --
Does it make sense for success/error to be enums? It could be useful if we
need to match against the status.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/316#discussion_r96930477 --- Diff: metron-interface/metron-rest-client/src/main/java/org/apache/metron/rest/model/KafkaTopic.java --- @@ -0,0 +1,60 @@ +/** --- End diff -- 4 spaces instead of 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @ottobackwards that's a good point. To me it still may makes sense to have this API (which the Ambari view would use). I think it would be useful for clusters not using Ambari. But I suspect this is going to be a much longer discussion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr @cestella does it make sense to hold off on this for a little while? It looks like all configuration may be managed by Ambari going forward. I am getting this from the dev thread: `Ambari Metron Configuration Management consequences and call to action`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #397: METRON-627: Add HyperLogLogPlus implementation ...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/397 @mmiklavc have you rebased lately? If not, I'm sure the PR I submitted for Stellar comparisons is the culprit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #397: METRON-627: Add HyperLogLogPlus implementation ...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/397 Okay, I took a look. It's because EOL isn't defined as a fragment. If `EOL : '\n';` is changed to `fragment EOL : '\n';` it should work. Without the `fragment` keyword antlr4 makes a token for `EOL` and it gets confused on how to parse it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #397: METRON-627: Add HyperLogLogPlus implementation ...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/397 This looks like an issue with the grammar. My guess is it isn't properly handling white space at the end. If you change the variable `hllpDefaultConstructorRule` to the below, it should work. I know this isn't a permanent solution, but it narrows the problem down. ``` /** *HLLP_CARDINALITY( * HLLP_ADD( *HLLP_ADD( * HLLP_INIT(), * val1 *), *val2 * ) * )*/ @Multiline private static String hllpDefaultConstructorRule; ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #417: METRON-659 Emulate Sensors in Development Envir...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/417 @nickwallen where does the mock data come from? Did you create is? Is there any concern for licensing? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/416 +1 (non-binding). I'll create a Jira based on the last comments between @cestella and me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #416: METRON-656: Make Stellar 'in' closer to ...
Github user jjmeyer0 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/416#discussion_r96071886
--- Diff: metron-platform/metron-common/README.md ---
@@ -39,11 +40,17 @@ The following keywords need to be single quote escaped
in order to be used in St
| <= | \> | \>= |
| ? | \+ | \- |
| , | \* | / |
+| | \* | / |
Using parens such as: "foo" : "\<ok\>" requires escaping; "foo":
"\'\<ok\>\'"
+## Stellar Language Inclusion Checks (`in` and `not in`)
+1. `in` supports string contains. e.g. `'foo' in 'foobar' == true`
+2. `in` supports collection contains. e.g. `'foo' in [ 'foo', 'bar' ] ==
true`
+3. `in` supports map key contains. e.g. `'foo' in { 'foo' : 5} == true`
+4. `not in` is the negation of the in expression. e.g. `'grok' not in
'foobar' == true`
--- End diff --
Sorry, one last comment. I was a bit curious as to how the expression
`'grok' not in 'foobar' == true` would be evaluated by Stellar. I wasn't sure
if it would be `'(grok' not in 'foobar') == true` or `'grok' not in ('foobar'
== true)`. Unfortunately when I tried to run a test it said it is not a valid
expression. I think this may be an issue in the Stellar grammar. It is probably
outside the scope of this ticket, but I thought I should mention it here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/416 I found one additional corner case that may need to be addressed. It looks the like expression, `null in [ null, 'something' ]`, returns false, but should return true. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/416 Disregard number 2. It was an oversight on my part. Sorry about that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #416: METRON-656: Make Stellar 'in' closer to functio...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/416 I think this is a great feature. I do have a few suggestions/questions: 1. Does it make sense to start moving away from `StellarTest` and break out into specific test classes (eg. `StellarArithmeticTest`, `StellarPredicateProcessorTest`, etc.)? Or Should this be a task in of itself to design a good structure? 2. I think the tests that are there for `in` should also be run against `not in` as well. 3. Update the README.md to include a description of how `in` and `not in` should work. For example, I'm not really sure what I should expect from the expression: `1 in `. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr, the reasoning I used to include the http status code in the error message format is if there is a need to send these messages to a downstream system. For example, let's say a developer is integrating with the Metron API. For some reason they want to drop all error responses on a queue for processing by another system. If the status code didn't exist in the error format some context would be lost. But to your point maybe it doesn't buy us much. I don't have a strong preference either way on this one. However, maybe at some point it may be worth having a custom attribute called `code` that would allow a user to look up the errors in documentation. It could potentially show things like common causes and workarounds. That sounds like a separate PR with a lot of discussion around it though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #404: METRON-624: Updated Comparison/Equality Evaluat...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/404 Thanks @cestella. I appreciate it. If no one has already, I'll work on those functions we discussed next. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/316#discussion_r95488297 --- Diff: metron-interface/pom.xml --- @@ -0,0 +1,89 @@ + + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + 4.0.0 + metron-interface + pom + metron-interface + +org.apache.metron +Metron +0.3.0 + + Interfaces for Metron + https://metron.incubator.apache.org/ + + scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git + scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git +HEAD +https://git-wip-us.apache.org/repos/asf/incubator-metron + + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + repo + + + +metron-rest --- End diff -- The `metron-rest-client` module is missing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr yes, I was thinking of something along these lines. At a high level I was thinking that any exception that bubbled up through the controller should be mapped to an error response. I believe Spring uses `ResponseEntityExceptionHandler` to help with this (http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/reference/htmlsingle/#boot-features-error-handling). The benefits I see from doing this are keeping the error handling consistent, separating concerns, and giving better feedback to the end user. It does require more thought on what exceptions are thrown, and if the exception handler has the ability to map them to a response . Does this make sense? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user jjmeyer0 commented on the issue:
https://github.com/apache/incubator-metron/pull/316
**Project Structure:**
Personally I think the module, `metron-interface`, should be split up into
more modules. I think there should be `metron-rest` and `metron-rest-client`.
The idea would be to move the models into the client module. Specifically the
models returned by the endpoints and are used when making requests. I
personally like to to do this because it allows a user to use the client
without having to bring a ton of dependencies onto their classpath. This is
assuming there will eventually be a client module. Does this make sense? Think
it is worth doing now?
**Response Behavior:**
I think we should also decide how Metron's APIs handle errors. In the past
I have defined an error response type that was used by all endpoints. It could
potentially look something like:
{
"responseCode" : 404,
"message" : "Could not find parser config.",
'fullMessage" : "Could not find parser config with the name: [some name]"
}
Also, I personally do not like to throw exceptions of type `Exception`. I
think we could do a couple things. We could create a MetronRestException that
gets mapped to an error response. We could also have the service layer use an
`Either` type which would either return the response entity or a set of error
responses. Do you all think this is worth talking about now? I think this is
always one of those things that's tough to decide, but should be standard
across the API.
**API Documentation:**
I think it is worth documenting all the different response types for each
endpoint. For example, the endpoint `/api/v1/sensorParserConfigHistory` only
describes the response for a 200 code, but there is also a 404. This can be
done by using Swagger's `ApiResponses` annotation.
**API Structure:**
As for API structure there were a few that I thought could potentially be
changed. Below are a few examples. To me it may make sense at some point to
have a sensor endpoint. IMO breaking it up as I did below groups them a bit
more nicely (isn't an exhaustive list). What do you think?
/api/v1/globalConfig -> /api/v1/global/config
/api/v1/sensorEnrichmentConfig -> /api/v1/sensor/enrichment/config
/api/v1/sensorParserConfig -> /api/v1/sensor/parser/config
/api/v1/sensorParserConfigHistory -> /api/v1/sensor/parser/config/history
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #404: METRON-624: Updated Comparison/Equality Evaluat...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/404 @cestella if comparing floats/doubles is going to be a common thing maybe we should provide a way to do so easily. Maybe we create functions FLOAT_EQUALS(f1, f2, epsilon) and DOUBLE_EQUALS(d1, d2, epsilon). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #404: METRON-624: Updated Comparison/Equality Evaluat...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/404 @cestella I updated the comparison expressions to return false when either side is null. I'll work on the validation stuff soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #404: METRON-624: Updated Comparison/Equality ...
GitHub user jjmeyer0 opened a pull request: https://github.com/apache/incubator-metron/pull/404 METRON-624: Updated Comparison/Equality Evaluations in Stellar 1. General cleanup in Stellar.g4 - Refactored the grammar to consistently use fragments instead of inlining things. 2. Updated the way Stellar transformations were validated - Previously Stellar parsed/evaluated the transformation to validate it. In certain circumstances this would cause failures. For instances, all variables would be resolved to null which would cause an error to be thrown when evaluating transformations such as: '1 < foo', '1 < null'. Both of these statements are syntactically valid. - Now the validation just checks whether the Stellar transformation is syntactically valid. It will not send the transformation through the listener as it once did. It will not validate that variables are defined. It will not validate functions are defined. 3. Updated comparison/equality expression evaluation - Number 2 was required in order to update the comparison/equality evaluation. - Now statements such as 1 == '1' will return false. - When comparing numerical values, the standard promotion rules apply (http://docs.oracle.com/javase/specs/jls/se8/html/jls-5.html#jls-5.6.2). - When comparing non-numeric types, compareTo method is used. - When either side is null then '==' and '!=' are used. For all other comparison operators null value would throw exception. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jjmeyer0/incubator-metron METRON-624 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/404.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #404 commit c1a9a25615f2a4bc09f6c0f2eb0b8b127d1ea2ef Author: JJ <jjmey...@gmail.com> Date: 2016-12-23T18:23:18Z METRON-624: Updated Stellar to handle comparison/equality properly. These are broken up into a few categories: null comparisons, numerical comparisons, others types that can be compared, otherwise exception occurs. commit c0cc0bdcdaa16cbd1d19203dda722316a6a4ffb9 Author: JJ <jjmey...@gmail.com> Date: 2016-12-23T21:02:35Z Updated the way a Stellar transformation is validated. Before this update the listener was used to validate the Stellar transformation expression. This would cause issues because the variable map wasn't be passed in causing all variables to resolve to null which would potentially cause a failure. Particularly when trying to validate expressions such as 'null < 1' which is syntactically valid, but semantically may not be. This commit updates the validation to say whether the rule is syntactically valid. It does not try to interpret anything. It just says whether the rule conforms to the Stellar grammar or not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #392: METRON-616: Added support for float and long li...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/392 @cestella @ottobackwards is there anything else you all would like to see on this one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #392: METRON-616: Added support for float and long li...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/392 and I'll add some tests that specifically test expected type of these number literals. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #392: METRON-616: Added support for float and long li...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/392 @cestella I thought about the case where a number such as '12345678910' should be considered a long or not. The biggest reason why I chose not to go this way, was because we were basing many things on the way Java does things, and Java would throw a compile time error if this example was not followed by any l/L. Also, I had some concerns in a scenario where two integers were added, a long was automatically created from them, and then passed to a function that only accepts integers. In this case a very interesting error may occur, and confuse end users. Not sure if any of those are valid, but that was my thought process. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #392: METRON-616: Added support for double and long l...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/392 @ottobackwards Depends. It should work very similar to java. For example, doubles will not require a D/d (eg. .4, 0.5, 1., etc). All these examples are doubles. Floats require an f/F, and to be a long, it requires an l/L. It should be very similar to https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html#jls-3.10.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #392: METRON-616: Added support for double and...
GitHub user jjmeyer0 opened a pull request: https://github.com/apache/incubator-metron/pull/392 METRON-616: Added support for double and long literals in Stellar. Added support for double and long literals in Stellar. The double and long literals are based on java's longs/doubles. However, octal, binary, and hexadecimal formats were not implemented. Int literals were also updated to not allow numbers such as 0001, 009, etc. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jjmeyer0/incubator-metron METRON-616 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/392.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #392 commit f81411bfafc747aaa9ca1833fcb942e43970e310 Author: JJ <jjmey...@gmail.com> Date: 2016-12-09T11:29:16Z METRON-616: Added support for double and long literals in Stellar. The double and long literals are based on java's longs/doubles. However, octal, binary, and hexadecimal formats were not implemented. commit e806125c38703cf186898353fd0d73a2a08a0b8b Author: JJ <jjmey...@gmail.com> Date: 2016-12-09T11:58:51Z Added license to new files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #390: METRON-364: Preserve Type for Arithmetic Expres...
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/390 I believe I have updated all tests and addressed each of the concerns. Thanks again for all the help! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #390: METRON-364: Preserve Type for Arithmetic Expres...
Github user jjmeyer0 commented on the issue:
https://github.com/apache/incubator-metron/pull/390
The reason I neglected longs and floats is because I didn't see the Stellar
dsl having the ability to represent them. For example, the test below fails
with a parse exception:
`
@Test
public void testLongAddition() throws Exception {
String query = "2147483650 + 2";
Assert.assertEquals(2147483652L, run(query, new HashMap<>()));
}
`
That being said it makes sense to still handle longs and floats, if Stellar
will at some point. If not that, at the very least if this evaluator sees a
long or float it should error out.
Should I add support for longs/floats and then create a Jira for Stellar to
add them? Or am I mistaken and Stellar already has support?
Thanks for the feedback.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #206: METRON-352 Updated Deployment to use HDP 2.4
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/206 Thanks, I appreciate it. Yes, I tested this with full_dev on Vagrant. I followed this [documentation](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #206: METRON-352 Updated Deployment to use HDP 2.4
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/206 Hello David, Thanks for the feedback. I moved the group_vars to defaults. Also, I squashed my commits to make it easier to read. I also deleted the passwd_less_ssh role as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #206: [Metron-352] Updated Deployment to use HDP 2.4
Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/206 Hello, has anyone had issues with org.apache.metron.enrichment.integration.EnrichmentIntegrationTest hanging? It seems to be trhowing the same exception over and over again which cause Travis CI to kill the build. I've looked at builds that were successful. The ones I've checked also have this error, just not this many times. Also, I'm having trouble reproducing it locally. Has anyone seen this before? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #206: [Metron-352] Updated Deployment to use H...
GitHub user jjmeyer0 opened a pull request: https://github.com/apache/incubator-metron/pull/206 [Metron-352] Updated Deployment to use HDP 2.4 Change the deployment to use the Ambari 2.2.2.0 and HDP 2.4.2.0 repositories. This included consolidating some Ansible variables into group_vars and fixing typos. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jjmeyer0/incubator-metron METRON-352 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/206.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #206 commit 3292c4fa737faae8fd506bc6b0c7428238e87270 Author: JJ <jjmey...@gmail.com> Date: 2016-08-03T15:22:32Z METRON-352 Updated deployment to use HDP 2.4. Moved some configuration to group_vars to remove duplication. commit 00af3e47752ee86613b0bde4d8d43d7ac7e65f2c Author: JJ <jjmey...@gmail.com> Date: 2016-08-05T02:05:52Z Fixed java home. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
