[jira] [Commented] (METRON-1597) EC2 10 Node Deployment - New function to check os and update *.cfg control_path
[ https://issues.apache.org/jira/browse/METRON-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501222#comment-16501222 ] ASF GitHub Bot commented on METRON-1597: Github user as22323 commented on the issue: https://github.com/apache/metron/pull/1046 Another approach I just tried (failed so far) was updating the playbook.yml directly with the following: ``` tasks: -name: OSX Execution Check shell: "echo $ANSIBLE_SSH_CONTROL_PATH" environment: ANSIBLE_SSH_CONTROL_PATH: ~/.ssh/ansible-ssh-%%C when: ansible_distribution == 'MacOSX' ``` Result:   It didn't seem to override the the local control_path. > EC2 10 Node Deployment - New function to check os and update *.cfg > control_path > --- > > Key: METRON-1597 > URL: https://issues.apache.org/jira/browse/METRON-1597 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: ClearOS release 7.4.0, CentOS 6.9, Ubuntu 18.04, Mac OS > X 10.13.3 >Reporter: ashah >Assignee: ashah >Priority: Minor > Fix For: 0.5.0 > > > This PR updates run.sh to include a function (changeAnsibleControlPath) that > will check for the OS that the run.sh file is run on. It will update the > control_path in ansible.cfg accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1046: METRON-1597 EC2 10 Node Deployment - New function to che...
Github user as22323 commented on the issue: https://github.com/apache/metron/pull/1046 Another approach I just tried (failed so far) was updating the playbook.yml directly with the following: ``` tasks: -name: OSX Execution Check shell: "echo $ANSIBLE_SSH_CONTROL_PATH" environment: ANSIBLE_SSH_CONTROL_PATH: ~/.ssh/ansible-ssh-%%C when: ansible_distribution == 'MacOSX' ``` Result:   It didn't seem to override the the local control_path. ---
[jira] [Commented] (METRON-1597) EC2 10 Node Deployment - New function to check os and update *.cfg control_path
[ https://issues.apache.org/jira/browse/METRON-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501107#comment-16501107 ] ASF GitHub Bot commented on METRON-1597: Github user as22323 commented on the issue: https://github.com/apache/metron/pull/1046 @mmiklavc. One potential solution is to use changeAnsibleControlPath() to set the ["ANSIBLE_SSH_CONTROL_PATH"](https://docs.ansible.com/ansible/2.4/config.html#envvar-ANSIBLE_SSH_CONTROL_PATH) environment variable before running the playbook. With this PR's existing code the only change that would need to be done is commenting out the #sed command between the if-fi and replacing it with the following: ``` export ANSIBLE_SSH_CONTROL_PATH=$(echo $ansiblecontrolpath) ``` With this approach the environment variable seems to override the control_path in ansible.cfg and the ansible.cfg does not get changed. I tested the line change in MacOS (ansible v 2.2.2.0) by doing the following -Intentionally put a bad control_path in ansible.cfg and run run.sh without changeAnsibleControlPath (expecting an error).  -Next I updated run.sh with the "export ANSIBLE_SSH_CONTROL_PATH" line change in changeAnsibleControlPath(). It was able to get it running past TASK [setup]  I didn't test with the Linux operating systems yet. > EC2 10 Node Deployment - New function to check os and update *.cfg > control_path > --- > > Key: METRON-1597 > URL: https://issues.apache.org/jira/browse/METRON-1597 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: ClearOS release 7.4.0, CentOS 6.9, Ubuntu 18.04, Mac OS > X 10.13.3 >Reporter: ashah >Assignee: ashah >Priority: Minor > Fix For: 0.5.0 > > > This PR updates run.sh to include a function (changeAnsibleControlPath) that > will check for the OS that the run.sh file is run on. It will update the > control_path in ansible.cfg accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1046: METRON-1597 EC2 10 Node Deployment - New function to che...
Github user as22323 commented on the issue: https://github.com/apache/metron/pull/1046 @mmiklavc. One potential solution is to use changeAnsibleControlPath() to set the ["ANSIBLE_SSH_CONTROL_PATH"](https://docs.ansible.com/ansible/2.4/config.html#envvar-ANSIBLE_SSH_CONTROL_PATH) environment variable before running the playbook. With this PR's existing code the only change that would need to be done is commenting out the #sed command between the if-fi and replacing it with the following: ``` export ANSIBLE_SSH_CONTROL_PATH=$(echo $ansiblecontrolpath) ``` With this approach the environment variable seems to override the control_path in ansible.cfg and the ansible.cfg does not get changed. I tested the line change in MacOS (ansible v 2.2.2.0) by doing the following -Intentionally put a bad control_path in ansible.cfg and run run.sh without changeAnsibleControlPath (expecting an error).  -Next I updated run.sh with the "export ANSIBLE_SSH_CONTROL_PATH" line change in changeAnsibleControlPath(). It was able to get it running past TASK [setup]  I didn't test with the Linux operating systems yet. ---
[jira] [Commented] (METRON-1598) NoClassDefFoundError when running with Elasticsearch X-Pack
[
https://issues.apache.org/jira/browse/METRON-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500878#comment-16500878
]
ASF GitHub Bot commented on METRON-1598:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1048
+1 by inspection.
> NoClassDefFoundError when running with Elasticsearch X-Pack
> ---
>
> Key: METRON-1598
> URL: https://issues.apache.org/jira/browse/METRON-1598
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running on Elasticsearch with X-Pack enabled, the following exception is
> thrown in the Metron REST logs.
> {code}
> 2018-05-23T07:43:16.000 ERROR
> [org.apache.metron.elasticsearch.utils.ElasticsearchUtils] - Failed to
> convert search request to JSON java.lang.NoClassDefFoundError:
> com/fasterxml/jackson/dataformat/smile/SmileParser at
> org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:123)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:62)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:52)
> at
> org.apache.metron.elasticsearch.utils.ElasticsearchUtils.toJSON(ElasticsearchUtils.java:277)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchRequestSubmitter.submitSearch(ElasticsearchRequestSubmitter.java:58)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:172)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:403)
> at
> org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:83)
> at
> org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
> {code}
> This does not happen with all queries, just some queries. For example, the
> following query completes successfully.
> {code}
> {
> "indices" : [],
> "query" : "*",
> "size" : 100,
> "from" : 0,
> "sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
> "fields" : null,
> "facetFields" : null
> }
> {code}
> On the other hand, this query fails with the exception shown above.
> {code}
> {
> "indices" : [],
> "query" : "*",
> "size" : 100,
> "from" : 0,
> "sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
> "fields" : null,
> "facetFields" : []
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1048: METRON-1598 NoClassDefFoundError when running with Elast...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1048 +1 by inspection. ---
[GitHub] metron issue #1043: METRON-1587: Make collection utility work for HDP search
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1043 I'm ok with full-dev off of HDP search, but that's a separate ticket than this IMO. ---
[jira] [Commented] (METRON-1587) Make collection utility work for HDP search
[ https://issues.apache.org/jira/browse/METRON-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500839#comment-16500839 ] ASF GitHub Bot commented on METRON-1587: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1043 I'm ok with full-dev off of HDP search, but that's a separate ticket than this IMO. > Make collection utility work for HDP search > --- > > Key: METRON-1587 > URL: https://issues.apache.org/jira/browse/METRON-1587 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Collection scripts need to be improved so that they use the Solr REST api > instead for collection management. Kerberos support should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1587) Make collection utility work for HDP search
[ https://issues.apache.org/jira/browse/METRON-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500825#comment-16500825 ] ASF GitHub Bot commented on METRON-1587: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1043 The latest commit adds documentation on script usage to the Solr README. > Make collection utility work for HDP search > --- > > Key: METRON-1587 > URL: https://issues.apache.org/jira/browse/METRON-1587 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Collection scripts need to be improved so that they use the Solr REST api > instead for collection management. Kerberos support should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1601) Rename metaalert "alert" nested field to "metron_alert" to avoid collision
[ https://issues.apache.org/jira/browse/METRON-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500828#comment-16500828 ] ASF GitHub Bot commented on METRON-1601: GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/1049 METRON-1601: Rename metaalert "alert" nested field to "metron_alert" to avoid collision ## Contributor Comments Currently we add a nested field called "alert" to our ES templates to enable meta alert querying. This field name is too common and should be something like "metron_alert" instead. To test this, spin up full-dev and add a meta alert via the alerts UI. You should be able to interact with it via the alerts UI as normal. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron alert_to_metron_alert Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1049.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1049 commit 01c33f1bef5349f6896d1dcd4bd37f0d332bdaef Author: cstella Date: 2018-05-29T18:15:56Z Renaming alert to metron_alert commit 2afe4377a80b9d8a451551bcbb9a98edfb09e00e Author: cstella Date: 2018-05-29T18:22:42Z Updating readme. commit 78be5089de44da1b452401a44fba1e25f960b1ea Author: cstella Date: 2018-05-29T20:03:39Z updated test commit 4f98acc9afcfb7ef5a42a2b9ba716ce43e927753 Author: cstella Date: 2018-05-29T20:41:50Z Updating test. commit ebb3b6f950d125257a7105e2a2c3a71f4c978e10 Author: cstella Date: 2018-05-29T21:19:50Z Updating test. commit 449d3f8aca422fb8b924de9e7b2106360c5813c6 Author: cstella Date: 2018-05-29T22:06:00Z changed the test dao commit d72a3f1cbb27cf38b2463eef0cc356e7ee4b040e Author: cstella Date: 2018-05-29T23:18:23Z updating alert model. commit 8aeba1f419f273a53a4a4636c21ebe46a0b242ae Author: cstella Date: 2018-05-29T23:29:27Z Missed the details component. commit
[GitHub] metron pull request #1049: METRON-1601: Rename metaalert "alert" nested fiel...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/1049 METRON-1601: Rename metaalert "alert" nested field to "metron_alert" to avoid collision ## Contributor Comments Currently we add a nested field called "alert" to our ES templates to enable meta alert querying. This field name is too common and should be something like "metron_alert" instead. To test this, spin up full-dev and add a meta alert via the alerts UI. You should be able to interact with it via the alerts UI as normal. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron alert_to_metron_alert Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1049.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1049 commit 01c33f1bef5349f6896d1dcd4bd37f0d332bdaef Author: cstella Date: 2018-05-29T18:15:56Z Renaming alert to metron_alert commit 2afe4377a80b9d8a451551bcbb9a98edfb09e00e Author: cstella Date: 2018-05-29T18:22:42Z Updating readme. commit 78be5089de44da1b452401a44fba1e25f960b1ea Author: cstella Date: 2018-05-29T20:03:39Z updated test commit 4f98acc9afcfb7ef5a42a2b9ba716ce43e927753 Author: cstella Date: 2018-05-29T20:41:50Z Updating test. commit ebb3b6f950d125257a7105e2a2c3a71f4c978e10 Author: cstella Date: 2018-05-29T21:19:50Z Updating test. commit 449d3f8aca422fb8b924de9e7b2106360c5813c6 Author: cstella Date: 2018-05-29T22:06:00Z changed the test dao commit d72a3f1cbb27cf38b2463eef0cc356e7ee4b040e Author: cstella Date: 2018-05-29T23:18:23Z updating alert model. commit 8aeba1f419f273a53a4a4636c21ebe46a0b242ae Author: cstella Date: 2018-05-29T23:29:27Z Missed the details component. commit 34c31f01b4ad585036b060f4f030bc52486f2209 Author: cstella Date: 2018-05-29T23:37:23Z updating components commit 71adb72ece3674339d8d6d555771a0dd78f1767b Author: cstella Date: 2018-06-04T17:59:35Z Merge branch 'master' into alert_to_metron_alert commit
[jira] [Created] (METRON-1601) Rename metaalert "alert" nested field to "metron_alert" to avoid collision
Casey Stella created METRON-1601: Summary: Rename metaalert "alert" nested field to "metron_alert" to avoid collision Key: METRON-1601 URL: https://issues.apache.org/jira/browse/METRON-1601 Project: Metron Issue Type: Improvement Reporter: Casey Stella Currently we add a nested field called "alert" to our ES templates to enable meta alert querying. This field name is too common and should be something like "metron_alert" instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1043: METRON-1587: Make collection utility work for HDP search
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1043 The latest commit adds documentation on script usage to the Solr README. ---
[jira] [Commented] (METRON-1589) '/api/v1/search/search' fails when 'Solr Zookeeper Urls' has comma separated multiple zookeeper urls
[
https://issues.apache.org/jira/browse/METRON-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500823#comment-16500823
]
ASF GitHub Bot commented on METRON-1589:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1040#discussion_r192870707
--- Diff:
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
---
@@ -17,47 +17,46 @@
*/
package org.apache.metron.solr.writer;
+import static org.apache.metron.solr.SolrConstants.SOLR_WRITER_NAME;
+
import com.google.common.base.Joiner;
+import java.io.IOException;
+import java.io.Serializable;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.function.Supplier;
import org.apache.commons.lang3.StringUtils;
import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.writer.BulkMessageWriter;
+import org.apache.metron.common.writer.BulkWriterResponse;
+import org.apache.metron.solr.SolrConstants;
import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpClientUtil;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer;
+import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
-import org.apache.metron.common.configuration.writer.WriterConfiguration;
-import org.apache.metron.common.writer.BulkMessageWriter;
-import org.apache.metron.common.writer.BulkWriterResponse;
-import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.response.UpdateResponse;
-import org.apache.solr.common.SolrInputDocument;
import org.json.simple.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.IOException;
-import java.io.Serializable;
-import java.lang.invoke.MethodHandles;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Collection;
-import java.util.Optional;
-import java.util.List;
-import java.util.ArrayList;
-import java.util.Set;
-import java.util.UUID;
-import java.util.function.Supplier;
-
-import static org.apache.metron.solr.SolrConstants.SOLR_WRITER_NAME;
-
public class SolrWriter implements BulkMessageWriter,
Serializable {
public static final String JAVA_SECURITY_CONFIG_PROPERTY =
"java.security.auth.login.config";
public enum SolrProperties {
-ZOOKEEPER_QUORUM("solr.zookeeper"),
+ZOOKEEPER_QUORUM(SolrConstants.SOLR_ZOOKEEPER),
--- End diff --
Yep, this is fine. +1 by inspection.
> '/api/v1/search/search' fails when 'Solr Zookeeper Urls' has comma separated
> multiple zookeeper urls
>
>
> Key: METRON-1589
> URL: https://issues.apache.org/jira/browse/METRON-1589
> Project: Metron
> Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> http://metronnode:8082/api/v1/search/search with below payload fails with
> internal server error when 'Solr Zookeeper Urls' has comma separated multiple
> zookeeper urls
> {code:java}
> {
> "indices": [],
> "facetFields": [],
> "query": "*",
> "from": 0,
> "size": 25
> }
> {code}
> {code:java}
> {"responseCode":500,"message":"Cannot connect to cluster at
> ctr-e138-1518143905142-328005-01-06.hwx.site:2181/solr,ctr-e138-1518143905142-328005-01-05.hwx.site:2181/solr:
> cluster not found/not ready","fullMessage":"SolrException: Cannot connect to
> cluster at
> ctr-e138-1518143905142-328005-01-06.hwx.site:2181/solr,ctr-e138-1518143905142-328005-01-05.hwx.site:2181/solr:
> cluster not found/not ready"}
> {code}
>
> This appears to result from here
> [SolrDao.java#L137-L144|https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrDao.java#L137-L144]
> In `getSolrClient`, `withZkHost` should be called multiple times after
> splitting the comma delimited string, per
>
[GitHub] metron pull request #1040: METRON-1589: '/api/v1/search/search' fails when '...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1040#discussion_r192870707
--- Diff:
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
---
@@ -17,47 +17,46 @@
*/
package org.apache.metron.solr.writer;
+import static org.apache.metron.solr.SolrConstants.SOLR_WRITER_NAME;
+
import com.google.common.base.Joiner;
+import java.io.IOException;
+import java.io.Serializable;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import java.util.function.Supplier;
import org.apache.commons.lang3.StringUtils;
import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.writer.BulkMessageWriter;
+import org.apache.metron.common.writer.BulkWriterResponse;
+import org.apache.metron.solr.SolrConstants;
import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpClientUtil;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer;
+import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
-import org.apache.metron.common.configuration.writer.WriterConfiguration;
-import org.apache.metron.common.writer.BulkMessageWriter;
-import org.apache.metron.common.writer.BulkWriterResponse;
-import org.apache.solr.client.solrj.SolrServerException;
-import org.apache.solr.client.solrj.response.UpdateResponse;
-import org.apache.solr.common.SolrInputDocument;
import org.json.simple.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.IOException;
-import java.io.Serializable;
-import java.lang.invoke.MethodHandles;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Collection;
-import java.util.Optional;
-import java.util.List;
-import java.util.ArrayList;
-import java.util.Set;
-import java.util.UUID;
-import java.util.function.Supplier;
-
-import static org.apache.metron.solr.SolrConstants.SOLR_WRITER_NAME;
-
public class SolrWriter implements BulkMessageWriter,
Serializable {
public static final String JAVA_SECURITY_CONFIG_PROPERTY =
"java.security.auth.login.config";
public enum SolrProperties {
-ZOOKEEPER_QUORUM("solr.zookeeper"),
+ZOOKEEPER_QUORUM(SolrConstants.SOLR_ZOOKEEPER),
--- End diff --
Yep, this is fine. +1 by inspection.
---
[jira] [Commented] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[
https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500805#comment-16500805
]
ASF GitHub Bot commented on METRON-1594:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1045#discussion_r192868140
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
---
@@ -156,33 +172,61 @@ public void configure(String sensorName,
WriterConfiguration configuration) {
}
}
+ @Override
+ public void init(Map stormConf, TopologyContext topologyContext,
WriterConfiguration config)
+ throws Exception {
+if(this.zkQuorum != null && this.brokerUrl == null) {
+ try {
+this.brokerUrl =
Joiner.on(",").join(KafkaUtils.INSTANCE.getBrokersFromZookeeper(this.zkQuorum));
+ } catch (Exception e) {
+throw new IllegalStateException("Cannot read kafka brokers from
zookeeper and you didn't specify them, giving up!", e);
+ }
+}
+this.kafkaProducer = new KafkaProducer<>(createProducerConfigs());
+ }
+
public Map createProducerConfigs() {
Map producerConfig = new HashMap<>();
producerConfig.put("bootstrap.servers", brokerUrl);
producerConfig.put("key.serializer", keySerializer);
producerConfig.put("value.serializer", valueSerializer);
producerConfig.put("request.required.acks", requiredAcks);
+producerConfig.put(ProducerConfig.BATCH_SIZE_CONFIG,
DEFAULT_BATCH_SIZE);
producerConfig.putAll(producerConfigs == null?new
HashMap<>():producerConfigs);
producerConfig = KafkaUtils.INSTANCE.normalizeProtocol(producerConfig);
return producerConfig;
}
@Override
- public void init() {
-if(this.zkQuorum != null && this.brokerUrl == null) {
+ public BulkWriterResponse write(String sensorType, WriterConfiguration
configurations,
+ Iterable tuples, List messages) {
+BulkWriterResponse writerResponse = new BulkWriterResponse();
+
+List> results = new ArrayList<>();
+int i = 0;
+for (Tuple tuple : tuples) {
+ JSONObject message = messages.get(i++);
+ Future future = kafkaProducer
+ .send(new ProducerRecord(kafkaTopic,
message.toJSONString()));
--- End diff --
It's effectively a HashMap, since that's what the JSONObject class extends.
I think the writer should basically write out whatever has been passed to it,
meaning that the responsibility would be on the invoking class to sort out
whether the HashMap has been constructed correctly in order to properly
serialize into a JSON string. I've made a conscious decision in this PR to not
make any changes to how we go about managing serialization and
deserialization. Are you worried about security threats, or just garbage data
coming from parsers and enrichments, potentially?
> KafkaWriter is asynchronous and may lose data on node failure
> -
>
> Key: METRON-1594
> URL: https://issues.apache.org/jira/browse/METRON-1594
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Currently, we do not block for data to be sent from kafka writer and we do
> not batch. Unfortunately, the send call is asynchronous, so if the node dies
> before the data is actually sent from kafka then it will drop data. It is
> highly unlikely that we will be able to make kafkawriter synchronous in a
> performant way, so we will likely need to batch in the parser and enrichment
> topologies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1045: METRON-1594: KafkaWriter is asynchronous and may ...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1045#discussion_r192868140
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java
---
@@ -156,33 +172,61 @@ public void configure(String sensorName,
WriterConfiguration configuration) {
}
}
+ @Override
+ public void init(Map stormConf, TopologyContext topologyContext,
WriterConfiguration config)
+ throws Exception {
+if(this.zkQuorum != null && this.brokerUrl == null) {
+ try {
+this.brokerUrl =
Joiner.on(",").join(KafkaUtils.INSTANCE.getBrokersFromZookeeper(this.zkQuorum));
+ } catch (Exception e) {
+throw new IllegalStateException("Cannot read kafka brokers from
zookeeper and you didn't specify them, giving up!", e);
+ }
+}
+this.kafkaProducer = new KafkaProducer<>(createProducerConfigs());
+ }
+
public Map createProducerConfigs() {
Map producerConfig = new HashMap<>();
producerConfig.put("bootstrap.servers", brokerUrl);
producerConfig.put("key.serializer", keySerializer);
producerConfig.put("value.serializer", valueSerializer);
producerConfig.put("request.required.acks", requiredAcks);
+producerConfig.put(ProducerConfig.BATCH_SIZE_CONFIG,
DEFAULT_BATCH_SIZE);
producerConfig.putAll(producerConfigs == null?new
HashMap<>():producerConfigs);
producerConfig = KafkaUtils.INSTANCE.normalizeProtocol(producerConfig);
return producerConfig;
}
@Override
- public void init() {
-if(this.zkQuorum != null && this.brokerUrl == null) {
+ public BulkWriterResponse write(String sensorType, WriterConfiguration
configurations,
+ Iterable tuples, List messages) {
+BulkWriterResponse writerResponse = new BulkWriterResponse();
+
+List> results = new ArrayList<>();
+int i = 0;
+for (Tuple tuple : tuples) {
+ JSONObject message = messages.get(i++);
+ Future future = kafkaProducer
+ .send(new ProducerRecord(kafkaTopic,
message.toJSONString()));
--- End diff --
It's effectively a HashMap, since that's what the JSONObject class extends.
I think the writer should basically write out whatever has been passed to it,
meaning that the responsibility would be on the invoking class to sort out
whether the HashMap has been constructed correctly in order to properly
serialize into a JSON string. I've made a conscious decision in this PR to not
make any changes to how we go about managing serialization and
deserialization. Are you worried about security threats, or just garbage data
coming from parsers and enrichments, potentially?
---
[jira] [Commented] (METRON-1597) EC2 10 Node Deployment - New function to check os and update *.cfg control_path
[ https://issues.apache.org/jira/browse/METRON-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500781#comment-16500781 ] ASF GitHub Bot commented on METRON-1597: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1046 @as22323 Really appreciate the effort to get this working across OSes. My concern with this approach is that ansible.cfg is a file necessarily tracked by our Git repo. Any change to it will be reflected in a `git status` command, which is probably not what we want because it will require users to remember to undo the changes anytime they commit. Ideally, we'd be able to defer to Ansible to intelligently choose its pattern based on the OS rather than specifying it externally. > EC2 10 Node Deployment - New function to check os and update *.cfg > control_path > --- > > Key: METRON-1597 > URL: https://issues.apache.org/jira/browse/METRON-1597 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: ClearOS release 7.4.0, CentOS 6.9, Ubuntu 18.04, Mac OS > X 10.13.3 >Reporter: ashah >Assignee: ashah >Priority: Minor > Fix For: 0.5.0 > > > This PR updates run.sh to include a function (changeAnsibleControlPath) that > will check for the OS that the run.sh file is run on. It will update the > control_path in ansible.cfg accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1046: METRON-1597 EC2 10 Node Deployment - New function to che...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1046 @as22323 Really appreciate the effort to get this working across OSes. My concern with this approach is that ansible.cfg is a file necessarily tracked by our Git repo. Any change to it will be reflected in a `git status` command, which is probably not what we want because it will require users to remember to undo the changes anytime they commit. Ideally, we'd be able to defer to Ansible to intelligently choose its pattern based on the OS rather than specifying it externally. ---
[jira] [Updated] (METRON-1600) Solr should be using threat.triage.score everywhere instead of threat:triage:score (back end)
[ https://issues.apache.org/jira/browse/METRON-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-1600: -- Issue Type: Sub-task (was: Bug) Parent: METRON-1416 > Solr should be using threat.triage.score everywhere instead of > threat:triage:score (back end) > - > > Key: METRON-1600 > URL: https://issues.apache.org/jira/browse/METRON-1600 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Priority: Major > > See > [https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-interface/metron-alerts/src/app/utils/constants.ts#L30] > This results in alerts in the UI with valid threat.triage.score (e.g. a Snort > alert with score 10) having a "-" for score. > We should make sure this doesn't happen elsewhere throughout the code (e.g. > metaalerts aggregates things by threat:triage:score, but there is a variable > for it, but we should check anyway). > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1587) Make collection utility work for HDP search
[ https://issues.apache.org/jira/browse/METRON-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500715#comment-16500715 ] ASF GitHub Bot commented on METRON-1587: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1043 @merrimanr There was a bit of discussion around this in #934, and it looks like the conclusion was that since HDP Search didn't support Solr 6 at the time, that we would hold off on it. Given that HDP Search 3.0 does support Solr 6, it might be worth revisiting that discussion. Personally, I'm okay with doing full dev off HDP Search (and having Ambari be able to setup config as needed automagically), as long as indexing can still be configured to work with a non HDP Solr instance. Re: this ticket in particular, I'd do that as a separate PR. Would we still want to use these scripts as part of that? If we still want the scripts, I'd want them documented as part of this ticket (even if it's not in the perfect place). There was also a hesitance on relying on any mpack (internal or external) explicitly in that discussion. I'm unsure if that objection extended to a full dev or not. @cestella Can you expand on your POV? > Make collection utility work for HDP search > --- > > Key: METRON-1587 > URL: https://issues.apache.org/jira/browse/METRON-1587 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Collection scripts need to be improved so that they use the Solr REST api > instead for collection management. Kerberos support should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1587) Make collection utility work for HDP search
[ https://issues.apache.org/jira/browse/METRON-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500727#comment-16500727 ] ASF GitHub Bot commented on METRON-1587: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1043 Ok I'll hold off on that for now and will also hold off on documenting anything Ambari related since it's going to depend on what we decide to do with HDP Search. For this PR I will document standalone usage of the collection scripts. Will that work? > Make collection utility work for HDP search > --- > > Key: METRON-1587 > URL: https://issues.apache.org/jira/browse/METRON-1587 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Collection scripts need to be improved so that they use the Solr REST api > instead for collection management. Kerberos support should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1043: METRON-1587: Make collection utility work for HDP search
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1043 Ok I'll hold off on that for now and will also hold off on documenting anything Ambari related since it's going to depend on what we decide to do with HDP Search. For this PR I will document standalone usage of the collection scripts. Will that work? ---
[jira] [Commented] (METRON-1598) NoClassDefFoundError when running with Elasticsearch X-Pack
[
https://issues.apache.org/jira/browse/METRON-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500710#comment-16500710
]
ASF GitHub Bot commented on METRON-1598:
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1048
METRON-1598 NoClassDefFoundError when running with Elasticsearch X-Pack
In the directions for setting up Metron to work with the X-Pack, we just
have one exclusion too many. Simple fix.
## Testing
1. Launch a development environment.
1. Follow the existing instructions to setup the X-Pack.
1. Execute the queries outlined in the JIRA and ensure that valid responses
are received for both.
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1598
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1048.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1048
commit 635a7efb1ca5c388ee100a6ab71d5719493db4fa
Author: Nick Allen
Date: 2018-06-04T18:43:46Z
METRON-1598 NoClassDefFoundError when running with Elasticsearch X-Pack
> NoClassDefFoundError when running with Elasticsearch X-Pack
> ---
>
> Key: METRON-1598
> URL: https://issues.apache.org/jira/browse/METRON-1598
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running on Elasticsearch with X-Pack enabled, the following exception is
> thrown in the Metron REST logs.
> {code}
> 2018-05-23T07:43:16.000 ERROR
> [org.apache.metron.elasticsearch.utils.ElasticsearchUtils] - Failed to
> convert search request to JSON java.lang.NoClassDefFoundError:
> com/fasterxml/jackson/dataformat/smile/SmileParser at
> org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:123)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:62)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:52)
> at
> org.apache.metron.elasticsearch.utils.ElasticsearchUtils.toJSON(ElasticsearchUtils.java:277)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchRequestSubmitter.submitSearch(ElasticsearchRequestSubmitter.java:58)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:172)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:403)
> at
> org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:83)
> at
> org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
> {code}
> This does not happen with all queries, just some queries. For example, the
> following query completes successfully.
> {code}
> {
> "indices" : [],
> "query" : "*",
> "size" : 100,
> "from" : 0,
> "sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
> "fields" : null,
> "facetFields" : null
> }
> {code}
> On the other hand, this query fails with the exception shown above.
> {code}
> {
> "indices" : [],
> "query" : "*",
> "size" : 100,
> "from" : 0,
> "sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
> "fields" : null,
> "facetFields" : []
> }
>
[jira] [Created] (METRON-1600) Solr should be using threat.triage.score everywhere instead of threat:triage:score (back end)
Ryan Merriman created METRON-1600: - Summary: Solr should be using threat.triage.score everywhere instead of threat:triage:score (back end) Key: METRON-1600 URL: https://issues.apache.org/jira/browse/METRON-1600 Project: Metron Issue Type: Bug Reporter: Ryan Merriman See [https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-interface/metron-alerts/src/app/utils/constants.ts#L30] This results in alerts in the UI with valid threat.triage.score (e.g. a Snort alert with score 10) having a "-" for score. We should make sure this doesn't happen elsewhere throughout the code (e.g. metaalerts aggregates things by threat:triage:score, but there is a variable for it, but we should check anyway). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1043: METRON-1587: Make collection utility work for HDP search
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1043 @merrimanr There was a bit of discussion around this in #934, and it looks like the conclusion was that since HDP Search didn't support Solr 6 at the time, that we would hold off on it. Given that HDP Search 3.0 does support Solr 6, it might be worth revisiting that discussion. Personally, I'm okay with doing full dev off HDP Search (and having Ambari be able to setup config as needed automagically), as long as indexing can still be configured to work with a non HDP Solr instance. Re: this ticket in particular, I'd do that as a separate PR. Would we still want to use these scripts as part of that? If we still want the scripts, I'd want them documented as part of this ticket (even if it's not in the perfect place). There was also a hesitance on relying on any mpack (internal or external) explicitly in that discussion. I'm unsure if that objection extended to a full dev or not. @cestella Can you expand on your POV? ---
[GitHub] metron pull request #1048: METRON-1598 NoClassDefFoundError when running wit...
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1048 METRON-1598 NoClassDefFoundError when running with Elasticsearch X-Pack In the directions for setting up Metron to work with the X-Pack, we just have one exclusion too many. Simple fix. ## Testing 1. Launch a development environment. 1. Follow the existing instructions to setup the X-Pack. 1. Execute the queries outlined in the JIRA and ensure that valid responses are received for both. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1598 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1048.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1048 commit 635a7efb1ca5c388ee100a6ab71d5719493db4fa Author: Nick Allen Date: 2018-06-04T18:43:46Z METRON-1598 NoClassDefFoundError when running with Elasticsearch X-Pack ---
[jira] [Commented] (METRON-1599) Allow user to define global property 'source.type.field' in Ambari
[
https://issues.apache.org/jira/browse/METRON-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500681#comment-16500681
]
ASF GitHub Bot commented on METRON-1599:
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1047
METRON-1599 Allow user to define global property 'source.type.field' in
Ambari
The user can specify the name of the message field containing the source
type in the global configuration. This is used by the Alerts UI.
Currently this necessitates a user using the CLI to add the field. This
property should be exposed to the user via Ambari. The user should be able to
define this property directly in Ambari.
## Testing
1. Launch the development environment.
1. Open the REPL and validate the current value of the global property
'source.type.field'.
```
[root@node1 ~]# source /etc/default/metron
[root@node1 ~]# $METRON_HOME/bin/stellar -z $ZOOKEEPER
Stellar, Go!
...
[Stellar]>>> globals := CONFIG_GET("GLOBAL")
{
...
"source.type.field" : "source.type",
...
}
```
1. Change the value in Ambari, then restart the Alerts UI using Ambari.
1. Open the REPL and validate that the value of the global property
'source.type.field' changed in the global config.
```
[Stellar]>>> globals := CONFIG_GET("GLOBAL")
{
...
"source.type.field" : "source:type",
...
}
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1599
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1047.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1047
commit 20df52f4b1a7cbd44bd647552d1737b4ec8d40c6
Author: Nick Allen
Date: 2018-06-04T18:07:36Z
METRON-1599 Allow user to define global property 'source.type.field' in
Ambari
> Allow user to define global property 'source.type.field' in Ambari
> ---
>
> Key: METRON-1599
> URL: https://issues.apache.org/jira/browse/METRON-1599
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> The user can specify the name of the message field containing the source type
> in the global configuration. This is used by the Alerts UI.
> Currently this necessitates a user using the CLI to add the field. This
> property should be exposed to the user via Ambari. The user should be able
> to define this property directly in Ambari.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1047: METRON-1599 Allow user to define global property ...
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1047
METRON-1599 Allow user to define global property 'source.type.field' in
Ambari
The user can specify the name of the message field containing the source
type in the global configuration. This is used by the Alerts UI.
Currently this necessitates a user using the CLI to add the field. This
property should be exposed to the user via Ambari. The user should be able to
define this property directly in Ambari.
## Testing
1. Launch the development environment.
1. Open the REPL and validate the current value of the global property
'source.type.field'.
```
[root@node1 ~]# source /etc/default/metron
[root@node1 ~]# $METRON_HOME/bin/stellar -z $ZOOKEEPER
Stellar, Go!
...
[Stellar]>>> globals := CONFIG_GET("GLOBAL")
{
...
"source.type.field" : "source.type",
...
}
```
1. Change the value in Ambari, then restart the Alerts UI using Ambari.
1. Open the REPL and validate that the value of the global property
'source.type.field' changed in the global config.
```
[Stellar]>>> globals := CONFIG_GET("GLOBAL")
{
...
"source.type.field" : "source:type",
...
}
```
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1599
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1047.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1047
commit 20df52f4b1a7cbd44bd647552d1737b4ec8d40c6
Author: Nick Allen
Date: 2018-06-04T18:07:36Z
METRON-1599 Allow user to define global property 'source.type.field' in
Ambari
---
[jira] [Commented] (METRON-1587) Make collection utility work for HDP search
[ https://issues.apache.org/jira/browse/METRON-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500675#comment-16500675 ] ASF GitHub Bot commented on METRON-1587: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1043 As I update the documentation in the Solr README it occurs to me that we should consider installing HDP Search 3 in full dev with our Ansible scripts. I have a feeling this will end up being the recommendation for using Solr with Metron. The documentation in the README is a little outdated so I think it's worth figuring this out before I update it so it's accurate and complete. Do you agree with that? Would you be ok doing it in this PR or would you prefer a separate PR? If separate, it might be easier to update the documentation in the next PR because we'll have a clear installation path at that point (and the README will have to be rewritten anyways). My preference would be to just do it here but I understand if you want to review them separately. I guess my point is updating the documentation once makes more sense to me. > Make collection utility work for HDP search > --- > > Key: METRON-1587 > URL: https://issues.apache.org/jira/browse/METRON-1587 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Collection scripts need to be improved so that they use the Solr REST api > instead for collection management. Kerberos support should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1043: METRON-1587: Make collection utility work for HDP search
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1043 As I update the documentation in the Solr README it occurs to me that we should consider installing HDP Search 3 in full dev with our Ansible scripts. I have a feeling this will end up being the recommendation for using Solr with Metron. The documentation in the README is a little outdated so I think it's worth figuring this out before I update it so it's accurate and complete. Do you agree with that? Would you be ok doing it in this PR or would you prefer a separate PR? If separate, it might be easier to update the documentation in the next PR because we'll have a clear installation path at that point (and the README will have to be rewritten anyways). My preference would be to just do it here but I understand if you want to review them separately. I guess my point is updating the documentation once makes more sense to me. ---
[jira] [Created] (METRON-1599) Allow user to define global property 'source.type.field' in Ambari
Nick Allen created METRON-1599: -- Summary: Allow user to define global property 'source.type.field' in Ambari Key: METRON-1599 URL: https://issues.apache.org/jira/browse/METRON-1599 Project: Metron Issue Type: Improvement Reporter: Nick Allen Assignee: Nick Allen The user can specify the name of the message field containing the source type in the global configuration. This is used by the Alerts UI. Currently this necessitates a user using the CLI to add the field. This property should be exposed to the user via Ambari. The user should be able to define this property directly in Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1598) NoClassDefFoundError when running with Elasticsearch X-Pack
[
https://issues.apache.org/jira/browse/METRON-1598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1598:
---
Description:
When running on Elasticsearch with X-Pack enabled, the following exception is
thrown in the Metron REST logs.
{code}
2018-05-23T07:43:16.000 ERROR
[org.apache.metron.elasticsearch.utils.ElasticsearchUtils] - Failed to convert
search request to JSON java.lang.NoClassDefFoundError:
com/fasterxml/jackson/dataformat/smile/SmileParser at
org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:123)
at
org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:62)
at
org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:52)
at
org.apache.metron.elasticsearch.utils.ElasticsearchUtils.toJSON(ElasticsearchUtils.java:277)
at
org.apache.metron.elasticsearch.dao.ElasticsearchRequestSubmitter.submitSearch(ElasticsearchRequestSubmitter.java:58)
at
org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:172)
at
org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:403)
at
org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:83)
at
org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
{code}
This does not happen with all queries, just some queries. For example, the
following query completes successfully.
{code}
{
"indices" : [],
"query" : "*",
"size" : 100,
"from" : 0,
"sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
"fields" : null,
"facetFields" : null
}
{code}
On the other hand, this query fails with the exception shown above.
{code}
{
"indices" : [],
"query" : "*",
"size" : 100,
"from" : 0,
"sort" : [ \{ "field" : "timestamp", "sortOrder" : "DESC" } ],
"fields" : null,
"facetFields" : []
}
{code}
was:
When running on Elasticsearch with X-Pack enabled, the following exception is
thrown in the Metron REST logs.
{code}
2018-05-23T07:43:16.000 ERROR
[org.apache.metron.elasticsearch.utils.ElasticsearchUtils] - Failed to convert
search request to JSON java.lang.NoClassDefFoundError:
com/fasterxml/jackson/dataformat/smile/SmileParser at
org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:123)
at
org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:62)
at
org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:52)
at
org.apache.metron.elasticsearch.utils.ElasticsearchUtils.toJSON(ElasticsearchUtils.java:277)
at
org.apache.metron.elasticsearch.dao.ElasticsearchRequestSubmitter.submitSearch(ElasticsearchRequestSubmitter.java:58)
at
org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:172)
at
org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:403)
at
org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:83)
at
org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
{code}
This does not happen with all queries, just some queries. For example, the
following query completes successfully.
{code}
{ "indices" : [], "query" : "*", "size" : 100, "from" : 0, "sort" : [ \{
"field" : "timestamp", "sortOrder" : "DESC" } ], "fields" : null, "facetFields"
: null }
{code}
On the other hand, this query fails with the exception shown above.
{code}
{ "indices" : [], "query" : "*", "size" : 100, "from" : 0, "sort" : [ \{
"field" : "timestamp", "sortOrder" : "DESC" } ], "fields" : null, "facetFields"
: [] }
{code}
> NoClassDefFoundError when running with Elasticsearch X-Pack
> ---
>
> Key: METRON-1598
> URL: https://issues.apache.org/jira/browse/METRON-1598
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running on Elasticsearch with X-Pack enabled, the following exception is
> thrown in the Metron REST logs.
> {code}
> 2018-05-23T07:43:16.000 ERROR
> [org.apache.metron.elasticsearch.utils.ElasticsearchUtils] - Failed to
> convert search request to JSON java.lang.NoClassDefFoundError:
> com/fasterxml/jackson/dataformat/smile/SmileParser at
> org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:123)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:62)
> at
> org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:52)
> at
> org.apache.metron.elasticsearch.utils.ElasticsearchUtils.toJSON(ElasticsearchUtils.java:277)
> at
>
[jira] [Created] (METRON-1598) NoClassDefFoundError when running with Elasticsearch X-Pack
Nick Allen created METRON-1598:
--
Summary: NoClassDefFoundError when running with Elasticsearch
X-Pack
Key: METRON-1598
URL: https://issues.apache.org/jira/browse/METRON-1598
Project: Metron
Issue Type: Bug
Affects Versions: 0.4.2
Reporter: Nick Allen
Assignee: Nick Allen
When running on Elasticsearch with X-Pack enabled, the following exception is
thrown in the Metron REST logs.
{code}
2018-05-23T07:43:16.000 ERROR
[org.apache.metron.elasticsearch.utils.ElasticsearchUtils] - Failed to convert
search request to JSON java.lang.NoClassDefFoundError:
com/fasterxml/jackson/dataformat/smile/SmileParser at
org.elasticsearch.common.xcontent.XContentFactory.contentBuilder(XContentFactory.java:123)
at
org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:62)
at
org.elasticsearch.action.support.ToXContentToBytes.buildAsBytes(ToXContentToBytes.java:52)
at
org.apache.metron.elasticsearch.utils.ElasticsearchUtils.toJSON(ElasticsearchUtils.java:277)
at
org.apache.metron.elasticsearch.dao.ElasticsearchRequestSubmitter.submitSearch(ElasticsearchRequestSubmitter.java:58)
at
org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:172)
at
org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:403)
at
org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:83)
at
org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
{code}
This does not happen with all queries, just some queries. For example, the
following query completes successfully.
{code}
{ "indices" : [], "query" : "*", "size" : 100, "from" : 0, "sort" : [ \{
"field" : "timestamp", "sortOrder" : "DESC" } ], "fields" : null, "facetFields"
: null }
{code}
On the other hand, this query fails with the exception shown above.
{code}
{ "indices" : [], "query" : "*", "size" : 100, "from" : 0, "sort" : [ \{
"field" : "timestamp", "sortOrder" : "DESC" } ], "fields" : null, "facetFields"
: [] }
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1547) Solr Comment Fields
[ https://issues.apache.org/jira/browse/METRON-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500597#comment-16500597 ] ASF GitHub Bot commented on METRON-1547: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1037 I agree with you @justinleet, we should drop the optional index from the query. > Solr Comment Fields > --- > > Key: METRON-1547 > URL: https://issues.apache.org/jira/browse/METRON-1547 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Right now the Solr schemas don't have comment fields defined. It'll get > caught by the catch all with ignored type not multivalued. > ES just handles this correctly out of the box, but we'll need to take care of > it in Solr and document the schema restriction. > This actually is probably fairly problematic in comparison to ES. Solr > doesn't support an easy way of doing a complex structure without doing > something a bit weird (like parsing a string representation) or miserable > (nested document). > This will be incompatible with the current comment update system (just using > the patch() functionality). Preferably we need to add a new REST endpoint for > comments specifically so that we can handle it without the frontend knowing > the backend system. This also involves adjusting the UI to use the new REST > API, along with testing for both ES and Solr. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1037: METRON-1547: Solr Comment Fields
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1037 I agree with you @justinleet, we should drop the optional index from the query. ---
[jira] [Commented] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500486#comment-16500486 ] ASF GitHub Bot commented on METRON-1594: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1045 @ottobackwards I believe this should already be handled by the acking strategy. Anything not acked will be replayed since we're leveraging at least once message processing semantics. > KafkaWriter is asynchronous and may lose data on node failure > - > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1045: METRON-1594: KafkaWriter is asynchronous and may lose da...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1045 @ottobackwards I believe this should already be handled by the acking strategy. Anything not acked will be replayed since we're leveraging at least once message processing semantics. ---
[jira] [Commented] (METRON-1593) Setting Metron rest additional classpath removes HBase and Hadoop configs from classpath
[
https://issues.apache.org/jira/browse/METRON-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500246#comment-16500246
]
ASF GitHub Bot commented on METRON-1593:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1044
> Setting Metron rest additional classpath removes HBase and Hadoop configs
> from classpath
>
>
> Key: METRON-1593
> URL: https://issues.apache.org/jira/browse/METRON-1593
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> There is a bug in the metron-rest.sh start script where HBase and Hadoop
> configuration directories get left off the classpath when the
> METRON_REST_CLASSPATH environment variable is populated initially. This
> happens when the "Metron rest additional classpath" setting in Ambari >
> Metron > Advanced > {color:#33}Advanced metron-rest-env is update with a
> value.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1044: METRON-1593: Setting Metron rest additional class...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1044 ---
[GitHub] metron issue #1036: METRON-1584 Indexing Topology Crashes with Invalid Messa...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1036 I ran this up in full dev and everything worked as expected. +1 pending approval from other reviewers. ---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500226#comment-16500226
]
ASF GitHub Bot commented on METRON-1584:
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/1036
I ran this up in full dev and everything worked as expected. +1 pending
approval from other reviewers.
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> 2018-05-24 09:21:22.236 o.a.s.util Thread-9-indexingBolt-executor[3 3]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Shutting down worker
> random_access_indexing-24-1527147389 703b5bf7-6c9d-46f3-8136-0c4877a69375 6700
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Terminating messaging
> context
> 2018-05-24 09:21:22.238 o.a.s.d.worker Thread-20 [INFO] Shutting down
> executors
> 2018-05-24 09:21:22.238 o.a.s.d.executor Thread-20 [INFO] Shutting down
> executor
> __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
> 2018-05-24
[jira] [Commented] (METRON-1589) '/api/v1/search/search' fails when 'Solr Zookeeper Urls' has comma separated multiple zookeeper urls
[
https://issues.apache.org/jira/browse/METRON-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500210#comment-16500210
]
ASF GitHub Bot commented on METRON-1589:
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/1040
@cestella Before I merge this in, are you okay with this PR doing this for
the existing constant?
https://github.com/apache/metron/pull/1040/commits/c04b9462669bbe26de0ee1f6c7d0d39a348e865b#diff-47dd38427181750f7a4d42b813b28f43R57
I wanted to keep the existing stuff consistent and avoid too much digging
into something not entirely related.
Having said that, it seems like we could probably clean up some of
configuration constants in a more holistic way, but I'd argue that's outside
the scope of this PR.
> '/api/v1/search/search' fails when 'Solr Zookeeper Urls' has comma separated
> multiple zookeeper urls
>
>
> Key: METRON-1589
> URL: https://issues.apache.org/jira/browse/METRON-1589
> Project: Metron
> Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
>
> http://metronnode:8082/api/v1/search/search with below payload fails with
> internal server error when 'Solr Zookeeper Urls' has comma separated multiple
> zookeeper urls
> {code:java}
> {
> "indices": [],
> "facetFields": [],
> "query": "*",
> "from": 0,
> "size": 25
> }
> {code}
> {code:java}
> {"responseCode":500,"message":"Cannot connect to cluster at
> ctr-e138-1518143905142-328005-01-06.hwx.site:2181/solr,ctr-e138-1518143905142-328005-01-05.hwx.site:2181/solr:
> cluster not found/not ready","fullMessage":"SolrException: Cannot connect to
> cluster at
> ctr-e138-1518143905142-328005-01-06.hwx.site:2181/solr,ctr-e138-1518143905142-328005-01-05.hwx.site:2181/solr:
> cluster not found/not ready"}
> {code}
>
> This appears to result from here
> [SolrDao.java#L137-L144|https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrDao.java#L137-L144]
> In `getSolrClient`, `withZkHost` should be called multiple times after
> splitting the comma delimited string, per
> [CloudSolrClient.Builder#withZkHost|https://lucene.apache.org/solr/6_5_0/solr-solrj/org/apache/solr/client/solrj/impl/CloudSolrClient.Builder.html#withZkHost-java.lang.String-].
> Overall, this could probably be done a couple ways, either to just split the
> String directly, or rearrange the methods to pass around Lists and have
> `getZkHost` be changed to `getZkHosts` and take care of it in a more
> contained manner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1040: METRON-1589: '/api/v1/search/search' fails when 'Solr Zo...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1040 @cestella Before I merge this in, are you okay with this PR doing this for the existing constant? https://github.com/apache/metron/pull/1040/commits/c04b9462669bbe26de0ee1f6c7d0d39a348e865b#diff-47dd38427181750f7a4d42b813b28f43R57 I wanted to keep the existing stuff consistent and avoid too much digging into something not entirely related. Having said that, it seems like we could probably clean up some of configuration constants in a more holistic way, but I'd argue that's outside the scope of this PR. ---
[jira] [Commented] (METRON-1588) Migrate storm-kafka-client to 1.2.1
[
https://issues.apache.org/jira/browse/METRON-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499975#comment-16499975
]
ASF GitHub Bot commented on METRON-1588:
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/metron/pull/1039#discussion_r192684452
--- Diff:
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java
---
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang.Validate;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+/**
+ * Implementation of {@link KafkaSpoutRetryService} using the exponential
backoff formula. The time of the nextRetry is set as follows:
+ * nextRetry = failCount == 1 ? currentTime + initialDelay : currentTime +
delayPeriod*2^(failCount-1)where failCount = 1, 2, 3, ...
+ * nextRetry = Min(nextRetry, currentTime + maxDelay)
+ */
+public class KafkaSpoutRetryExponentialBackoff implements
KafkaSpoutRetryService {
--- End diff --
@ottobackwards
Got it. Thanks for referring. Just updated the PR.
> Migrate storm-kafka-client to 1.2.1
> ---
>
> Key: METRON-1588
> URL: https://issues.apache.org/jira/browse/METRON-1588
> Project: Metron
> Issue Type: Improvement
>Reporter: Jungtaek Lim
>Priority: Critical
>
> Storm community defines storm-kafka-client 1.1.0 to be "unstable" and says
> 1.2.0 to be stabled one, because Storm community resolved 40 issues including
> critical and blocker from 1.2.0.
> There're still couple of issues after 1.2.0 so better to sync up the version
> to the latest, so I suggest Metron to upgrade the version to the latest,
> 1.2.1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1039: METRON-1588 Migrate storm-kafka-client to 1.2.1
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/metron/pull/1039#discussion_r192684452
--- Diff:
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java
---
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.TopicPartition;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang.Validate;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+
+/**
+ * Implementation of {@link KafkaSpoutRetryService} using the exponential
backoff formula. The time of the nextRetry is set as follows:
+ * nextRetry = failCount == 1 ? currentTime + initialDelay : currentTime +
delayPeriod*2^(failCount-1)where failCount = 1, 2, 3, ...
+ * nextRetry = Min(nextRetry, currentTime + maxDelay)
+ */
+public class KafkaSpoutRetryExponentialBackoff implements
KafkaSpoutRetryService {
--- End diff --
@ottobackwards
Got it. Thanks for referring. Just updated the PR.
---
