[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r160042250 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java --- @@ -0,0 +1,106 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.metron.stellar.common.utils.validation; + +import static org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT; + +import java.lang.invoke.MethodHandles; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Optional; +import java.util.Set; +import org.apache.commons.lang.NullArgumentException; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.stellar.common.StellarProcessor; +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class StellarZookeeperBasedValidator implements StellarValidator { + + private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + private static final String FAILED_COMPILE = "Failed to compile"; + private CuratorFramework client; + + public StellarZookeeperBasedValidator(CuratorFramework client) throws NullArgumentException { +if (client == null) { + throw new NullArgumentException("client"); +} +this.client = client; + } + + + @Override + public Iterable validate(Optional writer) { +// discover all the StellarConfigurationProvider +Set providerSet = new HashSet<>(); + +for (Class c : ClassIndex.getSubclasses(StellarConfigurationProvider.class, --- End diff -- They would automatically be picked up. I'm confused. This PR as stated is to help validation of stellar configurations, where ever they may be. it is not for the user to hunt and peck and pick a configuration. So that is kind of the point. Although there have been comments about doing configuration validation with specific objects or files passed in, that is not the thrust of this pr, even if this pr enables it. ---
[GitHub] metron pull request #890: METRON-1391 Fix for README.md in Metron Management
GitHub user 1havran opened a pull request: https://github.com/apache/metron/pull/890 METRON-1391 Fix for README.md in Metron Management ## Contributor Comments This PR fixes a README.md in metron-platform/metron-management. 1. Rules expression SHELL_GET_EXPRESSION('is_local ') returns null since there is no expression called 'is_local ' with trailing space. The expression called 'is_local'. This fix removes the trailing space. 2. Triage rules fix for missing score in rule1 ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/1havran/metron fix_for_mgmt_readme Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/890.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #890 commit e94b82a7fa06d6780190bb5e238416de799d4bec Author: 1havranDate: 2018-01-06T17:57:35Z Fix for README.md in Metron Management 1. rules SHELL_GET_EXPRESSION('is_local ') returns null since there is no expression called 'is_local ' with trailing space. The expression called 'is_local'. This fix removes the trailing space. 2. triage rules fix for missing score in rule1 ---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r160030197 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java --- @@ -0,0 +1,106 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.metron.stellar.common.utils.validation; + +import static org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT; + +import java.lang.invoke.MethodHandles; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Optional; +import java.util.Set; +import org.apache.commons.lang.NullArgumentException; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.stellar.common.StellarProcessor; +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class StellarZookeeperBasedValidator implements StellarValidator { + + private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + private static final String FAILED_COMPILE = "Failed to compile"; + private CuratorFramework client; + + public StellarZookeeperBasedValidator(CuratorFramework client) throws NullArgumentException { +if (client == null) { + throw new NullArgumentException("client"); +} +this.client = client; + } + + + @Override + public Iterable validate(Optional writer) { +// discover all the StellarConfigurationProvider +Set providerSet = new HashSet<>(); + +for (Class c : ClassIndex.getSubclasses(StellarConfigurationProvider.class, --- End diff -- I think you're not understanding my point. More specifically... What happens if another implementation gets added to the classpath? > Based on your discovery logic here, just having a FilesystemConfigurationProvider on the classpath (or any other implementation) will cause the configuration in the file system to get validated. We don't want that to happen. We want the user to control that behavior. ---
[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/884 And done. Please take a look when you get a chance. ---
[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/884 Ok will do @ottobackwards ---
[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r160029261 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/specials/AssignmentCommand.java --- @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.apache.metron.stellar.common.shell.specials; + +import org.apache.metron.stellar.common.StellarAssignment; +import org.apache.metron.stellar.common.shell.StellarShellExecutor; +import org.apache.metron.stellar.common.shell.StellarResult; + +import java.util.Optional; +import java.util.function.Function; + +import static org.apache.metron.stellar.common.shell.StellarResult.error; + +/** + * A special command that allows for variable assignment. Variable + * assignment is not implemented directly within Stellar. + * + *x := 2 + 2 + */ +public class AssignmentCommand implements SpecialCommand { --- End diff -- Right, it is not. And that is one advantage of this design. Once #687 gets merged, all we have to do is remove this class. ---
[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/884 I don't thin a full design doc is what we need, just an overview. It should be in the top level of common.shell. .. Stellar can be hosted in two shell like environments, zeppelin and the cli. This packaged is factored to share the implementation details of the shell, while allowing specific integration mechanisms. The common stuff can be found HERE This includes: - Commands, which are things that do things like A,B,C - Autocomplete, which is implemented commonly HERE, and brought into the specific implementation by FOO - The executors etc AESH specific notes: Zeppelin specific notes: --- ---
[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/687 Reviewer Note: Would a default method on the interface be better for update()? ---
[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r160026668 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java --- @@ -0,0 +1,106 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.metron.stellar.common.utils.validation; + +import static org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT; + +import java.lang.invoke.MethodHandles; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Optional; +import java.util.Set; +import org.apache.commons.lang.NullArgumentException; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.stellar.common.StellarProcessor; +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class StellarZookeeperBasedValidator implements StellarValidator { + + private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + private static final String FAILED_COMPILE = "Failed to compile"; + private CuratorFramework client; + + public StellarZookeeperBasedValidator(CuratorFramework client) throws NullArgumentException { +if (client == null) { + throw new NullArgumentException("client"); +} +this.client = client; + } + + + @Override + public Iterable validate(Optional writer) { +// discover all the StellarConfigurationProvider +Set providerSet = new HashSet<>(); + +for (Class c : ClassIndex.getSubclasses(StellarConfigurationProvider.class, --- End diff -- OK, I think I understand where you are coming from. In the end, there will be more functions for validation. Those which take in a passed in object, like CONFIG_GET() -> obj, EDIT -> obj, VALIDATE_CONFIG(obj). Those functions will be user driven, ie. __**Validate this thing I gave you__**. These functions are __**Validate everything the system knows about**. And are born out of the introduction of possible errors through language changes in already deployed stellar rules. That is why we need discovery. The "job" of this function at the moment is to find everything and make sure it is still OK. ---
[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r160026564 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarExecutionListeners.java --- @@ -0,0 +1,51 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.apache.metron.stellar.common.shell; + +import org.apache.metron.stellar.common.shell.specials.SpecialCommand; +import org.apache.metron.stellar.dsl.StellarFunctionInfo; + +/** + * A listener will be notified about events that occur during the + * execution of Stellar expressions. + */ +public class StellarExecutionListeners { + + /** + * A listener that is notified when a function is defined. + */ + public interface FunctionDefinedListener { +void whenFunctionDefined(StellarFunctionInfo functionInfo); + } --- End diff -- Sorry, battle scars. ---