[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/518 I've had 2 successful runs: 1) Full dev + kerberos. 2) EC2 will full stack installed prior to enabling kerberos. Both ran without error. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #532: METRON-634 Mpack bug fixes and improvements, no...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/532 +1, ran it up in full-dev and on EC2 (Centos7) and worked like a champ. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #527: METRON-848: MPack's metron_theme.json is malfor...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/527 If there's no objection to including this in 0.4, I'd like to do that as well. @mattf-horton - are you okay with my committing to master and 0.4? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/510 Looks like there was a small defect in [PR-522](https://github.com/apache/incubator-metron/pull/522). The build is no longer being built against the HDP-2.5.0.0 profile, so there is no org.apache.kafka.common.protocol.SecurityProtocol.PLAINTEXTSASL. I've opened [METRON-856](https://issues.apache.org/jira/browse/METRON-856). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #527: METRON-848: MPack's metron_theme.json is...
GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/527 METRON-848: MPack's metron_theme.json is malformed ## Contributor Comments Unfortunately, this bug doesn't present itself using full or quick dev as an error. The 'Index Setting' tab is missing on master. Apply this patch and verify that full-dev has the 'Index Setting' tab on the Metron config screen. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron (Incubating). Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [N/A] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book bin/generate-md.sh mvn site:site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-848 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/527.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #527 commit 0d9f88aa27e9220e3363c2b3fbb703d984cd0004 Author: David Lyle Date: 2017-04-12T12:50:09Z METRON-848: MPack's metron_theme.json is malformed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #522: METRON-839: RPM build should happen after archi...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/522 +1 - looks good, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #500: METRON-795: Install Metron REST with Amb...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/500#discussion_r110661559 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_master.py --- @@ -0,0 +1,83 @@ +""" +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +""" + +from resource_management.core.exceptions import ComponentIsNotRunning +from resource_management.core.exceptions import ExecutionFailed +from resource_management.core.resources.system import File +from resource_management.core.source import Template +from resource_management.libraries.functions.format import format +from resource_management.libraries.script import Script +from resource_management.core.resources.system import Execute + +from rest_commands import RestCommands + + +class RestMaster(Script): +def get_component_name(self): +# TODO add this at some point - currently will cause problems with hdp-select +# return "parser-master" --- End diff -- Please remove. (I know they are in parser-master, but they probably shouldn't be.) :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #500: METRON-795: Install Metron REST with Amb...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/500#discussion_r110661298
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py
---
@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute, File
+
+# Wrap major operations and functionality in this class
+class RestCommands:
+__params = None
+
+def __init__(self, params):
+if params is None:
+raise ValueError("params argument is required for
initialization")
+self.__params = params
+
+def setup_repo(self):
+def local_repo():
+Logger.info("Setting up local repo")
+Execute("yum -y install createrepo")
+Execute("createrepo /localrepo")
+Execute("chmod -R o-w+r /localrepo")
+Execute("echo \"[METRON-${metron.version}]\n"
+"name=Metron ${metron.version} packages\n"
+"baseurl=file:///localrepo\n"
+"gpgcheck=0\n"
+"enabled=1\" > /etc/yum.repos.d/local.repo")
+
+def remote_repo():
+print('Using remote repo')
+
+yum_repo_types = {
+'local': local_repo,
+'remote': remote_repo
+}
+repo_type = self.__params.yum_repo_type
+if repo_type in yum_repo_types:
+yum_repo_types[repo_type]()
+else:
+raise ValueError("Unsupported repo type
'{0}'".format(repo_type))
+
--- End diff --
Could you merge with current master and remove the repo setup stuff? It's
been deprecated in favor of repoinfo.xml.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110411427
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
---
@@ -0,0 +1,74 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import os.path
+from resource_management.core.source import Template
+from resource_management.core.resources.system import Directory, File
+from resource_management.core import global_lock
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute
+from resource_management.libraries.functions import format as ambari_format
+
+
+# Convenience function for ensuring home dirs are setup consistently.
+def storm_security_setup(params):
+if params.security_enabled:
+# I don't think there's an Ambari way to get a user's local home
dir , so have Python perform tilde expansion.
+# Ambari's Directory doesn't do tilde expansion.
+metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
+metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
+Directory(metron_storm_dir,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+File(ambari_format('{client_jaas_path}'),
+ content=Template('client_jaas.conf.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
+File(metron_storm_dir + '/storm.yaml',
+ content=Template('storm.yaml.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
--- End diff --
I think I can change it by setting up Storm HA. I think that's allowable
via Ambari. But good deal that it populates it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110409879
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
---
@@ -0,0 +1,74 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import os.path
+from resource_management.core.source import Template
+from resource_management.core.resources.system import Directory, File
+from resource_management.core import global_lock
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute
+from resource_management.libraries.functions import format as ambari_format
+
+
+# Convenience function for ensuring home dirs are setup consistently.
+def storm_security_setup(params):
+if params.security_enabled:
+# I don't think there's an Ambari way to get a user's local home
dir , so have Python perform tilde expansion.
+# Ambari's Directory doesn't do tilde expansion.
+metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
+metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
+Directory(metron_storm_dir,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+File(ambari_format('{client_jaas_path}'),
+ content=Template('client_jaas.conf.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
+File(metron_storm_dir + '/storm.yaml',
+ content=Template('storm.yaml.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
--- End diff --
Just to confirm, the shadowed nimbus.seeds is changed when I change the
value for Storm via Ambari?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/518 Yeah, reading is FUNdamental. I didn't see that it was the Storm service check that was failing. I kind of made up in my head that it was the Metron service check, because that would make sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #518: METRON-799: The MPack should function in a kerb...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/518 I have a (likely wrong) idea about the service check. A couple of things to check- can you verify that Ambari runs the service check as the storm user, because that's just weird, isn't it? If it does, make sure the user identity it's using is in nimbus.admins. If that doesn't work, maybe try adding the metron user to nimbus.admins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110387708
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
---
@@ -0,0 +1,74 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import os.path
+from resource_management.core.source import Template
+from resource_management.core.resources.system import Directory, File
+from resource_management.core import global_lock
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute
+from resource_management.libraries.functions import format as ambari_format
+
+
+# Convenience function for ensuring home dirs are setup consistently.
+def storm_security_setup(params):
+if params.security_enabled:
+# I don't think there's an Ambari way to get a user's local home
dir , so have Python perform tilde expansion.
+# Ambari's Directory doesn't do tilde expansion.
+metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
+metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
+Directory(metron_storm_dir,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+File(ambari_format('{client_jaas_path}'),
+ content=Template('client_jaas.conf.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
+File(metron_storm_dir + '/storm.yaml',
+ content=Template('storm.yaml.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
--- End diff --
So, if we switch on HA Storm after Metron kerberization, there's an issue?
I'm a bit uncomfortable with all the all the additional management required
to use AutoTGT rather than Auto[HDFS|HBase]TGT. I didn't deliberately choose
those two when I started (I just happened to land on those docs first), but
that seems less complex than what we're currently doing. Are there benefits I'm
missing? Because managing a jaas file and storm.yaml from outside of the Ambari
Storm Service seems to add difficulty and risk.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110386021
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
---
@@ -72,55 +93,46 @@ def remote_repo():
raise ValueError("Unsupported repo type
'{0}'".format(repo_type))
def init_kafka_topics(self):
-Logger.info('Creating Kafka topics')
-command_template = """{0}/kafka-topics.sh \
---zookeeper {1} \
---create \
---topic {2} \
---partitions {3} \
---replication-factor {4} \
---config retention.bytes={5}"""
-num_partitions = 1
-replication_factor = 1
-retention_gigabytes = int(self.__params.metron_topic_retention)
-retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
-Logger.info("Creating topics for indexing")
-
-Logger.info("Creating topic'{0}'".format(self.__indexing))
-Execute(command_template.format(self.__params.kafka_bin_dir,
-self.__params.zookeeper_quorum,
-self.__indexing,
-num_partitions,
-replication_factor,
-retention_bytes))
-Logger.info("Done creating Kafka topics")
+Logger.info('Creating Kafka topics for indexing')
+metron_service.init_kafka_topics(self.__params, [self.__indexing])
+
+def init_kafka_acls(self):
+Logger.info('Creating Kafka ACLs')
+# Indexed topic names matches the group
+metron_service.init_kafka_acls(self.__params, [self.__indexing],
[self.__indexing])
def init_hdfs_dir(self):
-Logger.info('Creating HDFS indexing directory')
+Logger.info('Setting up HDFS indexing directory')
+
+# Non Kerberized Metron runs under 'storm', requiring write under
the 'hadoop' group.
+# Kerberized Metron runs under it's own user.
+ownership = 0755 if self.__params.security_enabled else 0775
+Logger.info('HDFS indexing directory ownership is: ' +
str(ownership))
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
group=self.__params.hadoop_group,
--- End diff --
Seems reasonable to me too, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110385802
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
---
@@ -0,0 +1,74 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import os.path
+from resource_management.core.source import Template
+from resource_management.core.resources.system import Directory, File
+from resource_management.core import global_lock
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute
+from resource_management.libraries.functions import format as ambari_format
+
+
+# Convenience function for ensuring home dirs are setup consistently.
+def storm_security_setup(params):
+if params.security_enabled:
+# I don't think there's an Ambari way to get a user's local home
dir , so have Python perform tilde expansion.
+# Ambari's Directory doesn't do tilde expansion.
+metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
+metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
+Directory(metron_storm_dir,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+File(ambari_format('{client_jaas_path}'),
+ content=Template('client_jaas.conf.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
--- End diff --
Won't the supervisors need to read/write to HBase/HDFS? How are they
authenticated?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110384328
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
---
@@ -131,47 +153,35 @@ def init_geo(self):
self.set_geo_configured()
def init_kafka_topics(self):
-Logger.info('Creating Kafka topics')
-command_template = """{0}/kafka-topics.sh \
---zookeeper {1} \
---create \
---topic {2} \
---partitions {3} \
---replication-factor {4} \
---config retention.bytes={5}"""
-num_partitions = 1
-replication_factor = 1
-retention_gigabytes = int(self.__params.metron_topic_retention)
-retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
-
-Logger.info("Creating topics for enrichment")
-topics = [self.__enrichment_topic]
-for topic in topics:
-Logger.info("Creating topic'{0}'".format(topic))
-Execute(command_template.format(self.__params.kafka_bin_dir,
-self.__params.zookeeper_quorum,
-topic,
-num_partitions,
-replication_factor,
-retention_bytes))
-
-Logger.info("Done creating Kafka topics")
+Logger.info('Creating Kafka topics for enrichment')
+# All errors go to indexing topics, so create it here if it's not
already
+metron_service.init_kafka_topics(self.__params,
[self.__enrichment_topic, self.__params.metron_error_topic])
self.set_kafka_configured()
+def init_kafka_acls(self):
+Logger.info('Creating Kafka topics')
--- End diff --
Did you mean something like "Setting ACL" or even to omit this line?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110384006
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
---
@@ -72,55 +93,46 @@ def remote_repo():
raise ValueError("Unsupported repo type
'{0}'".format(repo_type))
def init_kafka_topics(self):
-Logger.info('Creating Kafka topics')
-command_template = """{0}/kafka-topics.sh \
---zookeeper {1} \
---create \
---topic {2} \
---partitions {3} \
---replication-factor {4} \
---config retention.bytes={5}"""
-num_partitions = 1
-replication_factor = 1
-retention_gigabytes = int(self.__params.metron_topic_retention)
-retention_bytes = retention_gigabytes * 1024 * 1024 * 1024
-Logger.info("Creating topics for indexing")
-
-Logger.info("Creating topic'{0}'".format(self.__indexing))
-Execute(command_template.format(self.__params.kafka_bin_dir,
-self.__params.zookeeper_quorum,
-self.__indexing,
-num_partitions,
-replication_factor,
-retention_bytes))
-Logger.info("Done creating Kafka topics")
+Logger.info('Creating Kafka topics for indexing')
+metron_service.init_kafka_topics(self.__params, [self.__indexing])
+
+def init_kafka_acls(self):
+Logger.info('Creating Kafka ACLs')
+# Indexed topic names matches the group
+metron_service.init_kafka_acls(self.__params, [self.__indexing],
[self.__indexing])
def init_hdfs_dir(self):
-Logger.info('Creating HDFS indexing directory')
+Logger.info('Setting up HDFS indexing directory')
+
+# Non Kerberized Metron runs under 'storm', requiring write under
the 'hadoop' group.
+# Kerberized Metron runs under it's own user.
+ownership = 0755 if self.__params.security_enabled else 0775
+Logger.info('HDFS indexing directory ownership is: ' +
str(ownership))
self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir,
type="directory",
action="create_on_execute",
owner=self.__params.metron_user,
group=self.__params.hadoop_group,
--- End diff --
Did you intend to leave this as hadoop group ownership or change it to the
metron group?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110383817
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
---
@@ -0,0 +1,74 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import os.path
+from resource_management.core.source import Template
+from resource_management.core.resources.system import Directory, File
+from resource_management.core import global_lock
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute
+from resource_management.libraries.functions import format as ambari_format
+
+
+# Convenience function for ensuring home dirs are setup consistently.
+def storm_security_setup(params):
+if params.security_enabled:
+# I don't think there's an Ambari way to get a user's local home
dir , so have Python perform tilde expansion.
+# Ambari's Directory doesn't do tilde expansion.
+metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
+metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
+Directory(metron_storm_dir,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+File(ambari_format('{client_jaas_path}'),
+ content=Template('client_jaas.conf.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
--- End diff --
Does this file end up on all the supervisor nodes too? Does it need to?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #518: METRON-799: The MPack should function in...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/518#discussion_r110383752
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_security.py
---
@@ -0,0 +1,74 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import os.path
+from resource_management.core.source import Template
+from resource_management.core.resources.system import Directory, File
+from resource_management.core import global_lock
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute
+from resource_management.libraries.functions import format as ambari_format
+
+
+# Convenience function for ensuring home dirs are setup consistently.
+def storm_security_setup(params):
+if params.security_enabled:
+# I don't think there's an Ambari way to get a user's local home
dir , so have Python perform tilde expansion.
+# Ambari's Directory doesn't do tilde expansion.
+metron_storm_dir_tilde = '~' + params.metron_user + '/.storm'
+metron_storm_dir = os.path.expanduser(metron_storm_dir_tilde)
+Directory(metron_storm_dir,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+File(ambari_format('{client_jaas_path}'),
+ content=Template('client_jaas.conf.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
+File(metron_storm_dir + '/storm.yaml',
+ content=Template('storm.yaml.j2'),
+ owner=params.metron_user,
+ group=params.metron_group,
+ mode=0755
+ )
+
--- End diff --
Does this override the storm.yaml that is at /etc/conf and managed by
Ambari?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #515: METRON-826: Ambari MPack should utilize ...
GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/515 METRON-826: Ambari MPack should utilize service specific repos ## Contributor Comments This change moves us away from the (as of [AMBARI-15538|https://issues-test.apache.org/jira/browse/AMBARI-15538]) non-standard practice of creating repo files in the install methods. To test, run this up on full dev- in addition to a running Metron, you should see the following repos created in /etc/yum.repos.d - METRON.repo - ELASTICSEARCH.repo - CURATOR.repo - KIBANA.repo ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron (Incubating). Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [X] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book bin/generate-md.sh mvn site:site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-826 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/515.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #515 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #499: METRON-196: Deployment Fails Without Ansible 2....
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/499 @JonZeolla - That looks like data was sent before the templates were loaded. Is that possible? I had clean data on all my test runs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/486#discussion_r108835519
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
---
@@ -177,7 +177,7 @@ indexing.executors=0
# Kafka #
kafka.zk={{ zookeeper_quorum }}
kafka.broker={{ kafka_brokers }}
-kafka.start=WHERE_I_LEFT_OFF
+kafka.start=UNCOMMITTED_EARLIEST
# Indexing #
--- End diff --
Little late to the party, but I think this is causing an error in Quick
Dev. The basis assumption is that the MPack is unchanged for a given quick dev
image, so kafka.start remains WHERE_I_LEFT_OFF right now but that's not part of
the enum, so it throws. I think the easy thing is to create a new image. What
do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #499: METRON-196: Deployment Fails Without Ans...
GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/499 METRON-196: Deployment Fails Without Ansible 2.0.0.2 ## Contributor Comments This changeset allows deployment with Ansible 2.0.0.2 and 2.2.2.0. I also added a version check as a pre-task to the top of the two entry playbooks, so deployment will now fail with a clearer error message if it is run on untested Ansible versions. @JonZeolla - this is the branch that (I hope) takes care of your issue. Please update Ansible to 2.2.2.0 and give it a whirl when you can. I setup a virtualenv to test it out, I'd recommend you do the same. **Testing** For those wishing to remain on 2.0.0.2, nothing should change, simply run the deployment of your choice. For those wishing to test on 2.2.2.0, do the following: 1. Setup virtual environment for testing. ``` pip install virtualenv cd ~ virtualenv -p /usr/bin/python2.7 test_ansible source ~/test_ansible/bin/activate pip install ansible pip install boto ``` After this, your prompt will begin with (test_ansible) indicating you're in the virtual Python environment. 2. Run deployment of your choosing. 3. Type _deactivate_ to return to your normal Python environment. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron (Incubating). Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [N/A] Have you included steps to reproduce the behavior or problem that is being changed or addressed. - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [N/A] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? (Full Dev and EC2) ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book bin/generate-md.sh mvn site:site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-196 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/499.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #499 commit 75aa5091d0bbf1d0e3e912477e2d05fb755c4610 Author: David Lyle Date: 2017-03-30T01:45:42Z METRON-196: Deployment Fails Without Ansible 2.0.0.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featur
[GitHub] incubator-metron issue #496: METRON-808: Fix Issue where EC2 deployment step...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/496 +1, thanks for the fix. Looks like this was introduced sometime 2.1.x.x+ in Ansible. Definitely happens in 2.2.0.0. I'm running up some other changes that will make our install work with 2.2.0.0. This helps. Thanks again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #494: METRON-806 set posix
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/494 Since all assemblies require this, would it make sense to configure it in pluginManagement in the top level POM? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #488: METRON-796: Mpack uses wrong group for owning H...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/488 I don't have an objection. I made that change because there was some issue with user_group. Unfortunately, I can't remember exactly what that was. In it's current state, what's the failure mode? I didn't see any errors in any of the final runs I performed in full dev or ec2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #487: METRON-792: Quick Dev should remove/repl...
GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/487 METRON-792: Quick Dev should remove/replace RPM packages ## Contributor Comments While building out METRON-671, I inadvertently omitted recreation of the RPM database for Quick Dev. As a result, stale RPMs are installed so, Quick Dev removes and then re-installs the same bits. This fixes that. Now it clears out /localrepo and re-runs createrepo. To test: - Merge this PR - Run _vagrant up_ from _project directory/metron-deployment/vagrant/quick-dev-platform_. - Once deployment is complete, _vagrant ssh_ to node1. _project directory/metron-deployment/vagrant/quick-dev-platform_. - Once deployment is complete, _vagrant ssh_ to node1. - Verify that the RPMs in /localrepo are timestamped the same as those in _project directory/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch_ ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron (Incubating). Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [N/A] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [N/A] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book bin/generate-md.sh mvn site:site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-792 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/487.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #487 commit f8e8e999096aa6963fa1ca96e186c14320afbc43 Author: David Lyle Date: 2017-03-22T16:46:48Z METRON-792: Quick Dev should remove/replace RPM packages --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #480: METRON-770 Unable to Launch Fastcapa Test Envir...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/480 +1, sorry about missing that inventory file earlier, NIck. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Thanks for all the help! I intend to merge this in tomorrow afternoon. @ottobackwards, @justinleet, I wanted to make sure you two were good. I also want to make sure there's no other feedback I've missed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Thanks @mmiklavc! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @mmiklavc - Exactly. We were wanting to make sure that @nickwallen was having environmental issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @justinleet - Accepted. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @justinleet - that last commit adds the quotation requiremnt to the tool tip. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @nickwallen - I pushed up a changeset that will address both your points. For 1, I added a test to ambari-setup to fail if FATAL appears in stdout. Making sure the EC2 build doesn't run the quick_dev role addresses the second. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Good news, I think I found the issue with the failing tests. The Maven reported "duplicated" dependencies weren't. I've replaced them. Travis will tell. @nickwallen - I did see the error you're talking about in your first point above. I think your memory is correct, it's one of those transients that we see sometimes. There's not much that can be done, but I am testing a patch that looks for "FATAL" in the ambari-setup stdout so at least we'll fail where the problem occurs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @nickwallen - For your first issue, I think we're hitting a transient issue with Ambari where ambari-setup -s completes successfully but Ambari won't actually start. I'll see if I can get some better diagnosis. On the second, did Ambari report that the install succeeded? Or did it fail and I didn't catch it? In other news- good news, bad news. Good news: I am able to replicate the integration test failure by running them in my local environment. Bad news: it's not in the code I touched and I'm completely flummoxed. Help would be much appreciated. Once I can get past these EC2 issues, I can diff master, but like I said. Help? Appreciated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @nickwallen - I had run EC2 testing a bunch and it worked post at least as well as it did prior (sometimes AWS zigs when it should zag). But, I have made a quite few changes since my last EC2 run. I'll spin it up and see if I get it too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Travis failure. Upsetting. This came in with my latest Master merge. Don't fail locally with Travis command line. Log below. It's in unrelated code. Any ideas? > -- > T E S T S > --- > Running org.apache.metron.common.stellar.maas.StellarMaaSIntegrationTest > Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.359 sec - in org.apache.metron.common.stellar.maas.StellarMaaSIntegrationTest > Running org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest > 2017-03-07 21:54:03 ERROR ServiceDiscoverer:171 - instance must be started before calling this method > java.lang.IllegalStateException: instance must be started before calling this method > at com.google.common.base.Preconditions.checkState(Preconditions.java:176) > at org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:379) > at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.queryForNames(ServiceDiscoveryImpl.java:276) > at org.apache.metron.maas.discovery.ServiceDiscoverer.updateState(ServiceDiscoverer.java:130) > at org.apache.metron.maas.discovery.ServiceDiscoverer.lambda$new$0(ServiceDiscoverer.java:94) > at org.apache.metron.maas.discovery.ServiceDiscoverer$$Lambda$4/1527152775.childEvent(Unknown Source) > at org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:685) > at org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:679) > at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) > at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) > at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:84) > at org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:678) > at org.apache.curator.framework.recipes.cache.TreeCache.access$1400(TreeCache.java:69) > at org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:790) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Formatting using clusterid: testClusterID > Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.249 sec <<< FAILURE! - in org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest > org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolverIntegrationTest Time elapsed: 1.249 sec <<< ERROR! > java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.server.AuthenticationFilter.constructSecretProvider(Ljavax/servlet/ServletContext;Ljava/util/Properties;Z)Lorg/apache/hadoop/security/authentication/util/SignerSecretProvider; > at org.apache.hadoop.http.HttpServer2.constructSecretProvider(HttpServer2.java:447) > at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:340) > at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:114) > at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:290) > at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:126) > at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:752) > at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:638) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:811) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:795) > at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488) > at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:) > at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:982) > at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:811) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:471) > at org.apa
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @ottobackwards - thanks for that! Let's do make sure the end result is something you can use. Also remember that this is steps 1 and 2 of an at least 4 step improvement, so perhaps once we get this to a working state, it be a good time to re-check the underlying assumptions to make sure we're still on a working path? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @nickwallen - last commit fixed deployment report for Quick Dev. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/459 +1 on this. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104698233 --- Diff: metron-deployment/roles/kibana/README.md --- @@ -1,35 +0,0 @@ -Kibana 4 - - -This role installs Kibana along with the default Metron Dashboard. - -### FAQ - - How do I change Metron's default dashboard? --- End diff -- Yeah, I'm good either way, I thought it may help you. If it doesn't, no point to rushing it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @ottobackwards - I don't agree that automation breaks with this. It is different though. If you're building automation now, you'd probably want to leverage Ambari Blueprints. Unfortunately, as we go down this path, I don't see a way around checking Ambari. In reality, that was always the case. If the Hadoop deployment failed, that was where you looked. If the probes failed to start, you checked Monit. Ansible alerted you to the fact of the failure, but anything but the simplest required poking around Ambari/Monit/Command Line. It sounds to me like you'd like to revisit the decision to reduce reliance on Ansible. If that's the case, we can certainly do that. I thought we had consensus but maybe we don't. Maybe we need to go back to the discuss thread? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 That's good feedback @ottobackwards. Post build errors will now surface in Ambari. I don't know what to do about the build. Any errors would eventually pop up on your screen. Is there any visual cue that would help? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @nickwallen - glad of that, I did see that I was still using a stale inventory for quick-dev which worked, but without the report. I've got a fix for that in test now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 There's a checkbox in the widget config:  That needs to be checked and the rest auto-populated:  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Chill guys, I'm trying to get a screenshot. ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104676140 --- Diff: metron-deployment/roles/kibana/README.md --- @@ -1,35 +0,0 @@ -Kibana 4 - - -This role installs Kibana along with the default Metron Dashboard. - -### FAQ - - How do I change Metron's default dashboard? --- End diff -- That's a really good catch. I'd recommend opening a separate Jira to use the same pickle file that the Ambari deployment uses. Preferably before METRON-745. With a duplicate file, we run the risk of someone like me accidentally desyncing the dashboards. Thoughts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104673176
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
- kafka-broker:
log.dirs: '{{ kafka_log_dirs }}'
delete.topic.enable: "true"
+ - metron-env:
+ parsers: "bro,snort"
+ - elastic-site:
+ index_number_of_shards: 1
+ index_number_of_replicas: 0
+ zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+ gateway_recover_after_data_nodes: 1
+ network_host: "_lo_,_eth0_,_eth1_"
+ masters_also_are_datanodes: "1"
--- End diff --
Heh, you know, that's EXACTLY where I was with it. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @nickwallen - it did need to change. The map didn't work. It does now. If @justinleet finds it easier to duplicate that change in his work, he can do so, otherwise, he should build on the dashboard.p file in this PR as to not undo what I've done. wrt the failure, it's strange- Ansible agrees with you that Ambari is running. I'll spin it up and see what I can see. Can you curl to Ambari? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104667214
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
# limitations under the License.
#
---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
- ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
+ ambari_service_state:
host: "{{ groups.ambari_master[0] }}"
port: "{{ ambari_port }}"
username: "{{ ambari_user }}"
password: "{{ ambari_password }}"
cluster_name: "{{ cluster_name }}"
-cluster_state: started
-wait_for_complete: False
+state: deleted
+component_name: "{{ item }}"
+component_host: "{{ inventory_hostname }}"
+ with_items:
+- METRON_ENRICHMENT_MASTER
+- METRON_INDEXING
+- METRON_PARSERS
+
+- name: Remove the Metron packages
+ package:
+name: "{{ item }}"
+state: absent
+ with_items:
+- metron-common
+- metron-data-management
+- metron-parsers
+- metron-enrichment
+- metron-indexing
+- metron-elasticsearch
+
+- name: Re-install the Metron Packages via Ambari
--- End diff --
No, I want them to exist there and they do. It simulates the old
run_enrichment_role task by re-installing the Metron rpms and reloads configs,
but leaves everything else alone.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104666299
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
- kafka-broker:
log.dirs: '{{ kafka_log_dirs }}'
delete.topic.enable: "true"
+ - metron-env:
+ parsers: "bro,snort"
+ - elastic-site:
+ index_number_of_shards: 1
+ index_number_of_replicas: 0
+ zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+ gateway_recover_after_data_nodes: 1
+ network_host: "_lo_,_eth0_,_eth1_"
+ masters_also_are_datanodes: "1"
--- End diff --
Scratch the scratch. It needs to be "1" or ""true"" for now. It's the same
ES problem we talked about above. If you want me to look at move the "translate
True to ""true"" " test to params, I can do that, but it's going to need to
exist somewhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104666805
--- Diff: metron-deployment/roles/ambari_master/tasks/main.yml ---
@@ -38,6 +38,16 @@
register: ambari_server_setup
failed_when: ambari_server_setup.stderr
+- name: Copy MPack to Ambari Host
+ copy:
+src: "{{ playbook_dir
}}/../packaging/ambari/metron-mpack/target/metron_mpack-0.3.1.0.tar.gz"
--- End diff --
Sure, good suggestion. In test now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @justinleet - wrt the Dashboard changes, the map shows up. If you've saved off a bunch of changes, you may want to clobber mine, but make sure the map shows up when you're done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104663778 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml --- @@ -27,6 +27,14 @@ Cluster name identifies your cluster +masters_also_are_datanodes +"false" --- End diff -- Just looked at it again, it was a quick hack to overcome a silly bug in ES where False != false. Ambari does some tricks under the hood with bools where they come out as True/False causing ES to fail. I can struggle with it some more if it's important to you, but I think it's going to end with > if False then value = "false" else value = "true" somewhere in the code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104662631
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
- kafka-broker:
log.dirs: '{{ kafka_log_dirs }}'
delete.topic.enable: "true"
+ - metron-env:
+ parsers: "bro,snort"
+ - elastic-site:
+ index_number_of_shards: 1
+ index_number_of_replicas: 0
+ zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+ gateway_recover_after_data_nodes: 1
+ network_host: "_lo_,_eth0_,_eth1_"
+ masters_also_are_datanodes: "1"
--- End diff --
Scratch that, that's not an ES setting (they have some weirdness with
proper booleans). I can try it and get back to you. Fwiw, 1 does work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104662413
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
- kafka-broker:
log.dirs: '{{ kafka_log_dirs }}'
delete.topic.enable: "true"
+ - metron-env:
+ parsers: "bro,snort"
+ - elastic-site:
+ index_number_of_shards: 1
+ index_number_of_replicas: 0
+ zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+ gateway_recover_after_data_nodes: 1
+ network_host: "_lo_,_eth0_,_eth1_"
+ masters_also_are_datanodes: "1"
--- End diff --
1 works, ""true"" will work. It will fail on a proper boolean, so I left it
as 1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104662215 --- Diff: metron-deployment/extra_modules/ambari_service_state.py --- @@ -0,0 +1,352 @@ +#!/usr/bin/python +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +DOCUMENTATION = ''' +--- +module: ambari_service_state +version_added: "2.1" +author: Apache Metron (Incubating : https://github.com/apache/incubator-metron ) +short_description: Start/Stop/Change Service or Component State +description: +- Start/Stop/Change Service or Component State +options: + host: +description: + The hostname for the ambari web server + port: +description: + The port for the ambari web server + username: +description: + The username for the ambari web server + password: +description: + The name of the cluster in web server +required: yes + cluster_name: +description: + The name of the cluster in ambari +required: yes + service_name: +description: + The name of the service to alter +required: no + component_name: +description: + The name of the component to alter +required: no + component_host: +description: + The host running the targeted component. Required when component_name is used. +required: no + state: +description: + The desired service/component state. + wait_for_complete: +description: + Whether to wait for the request to complete before returning. Default is False. +required: no + requirements: [ 'requests'] +''' + +EXAMPLES = ''' +# must use full relative path to any files in stored in roles/role_name/files/ +- name: Create a new ambari cluster +ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: present + blueprint_var: roles/my_role/files/blueprint.yml + blueprint_name: hadoop + wait_for_complete: True +- name: Start the ambari cluster + ambari_cluster_state: +host: localhost +port: 8080 +username: admin +password: admin +cluster_name: my_cluster +cluster_state: started +wait_for_complete: True +- name: Stop the ambari cluster + ambari_cluster_state: +host: localhost +port: 8080 +username: admin +password: admin +cluster_name: my_cluster +cluster_state: stopped +wait_for_complete: True +- name: Delete the ambari cluster + ambari_cluster_state: +host: localhost +port: 8080 +username: admin +password: admin +cluster_name: my_cluster +cluster_state: absent +''' + +RETURN = ''' +results: +description: The content of the requests object returned from the RESTful call +returned: success +type: string +''' + +__author__ = 'apachemetron' + +import json + +try: +import requests +except ImportError: +REQUESTS_FOUND = False +else: +REQUESTS_FOUND = True + + +def main(): + +argument_spec = dict( +host=dict(type='str', default=None, required=True), +port=dict(type='int', default=None, required=True), +username=dict(type='str', default=None, required=True), +password=dict(type='str', default=None, required=True), +cluster_name=dict(type='str', default=None, required=True), +state=dict(type='str', default=None, required=True, + choices=['started
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104662261 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml --- @@ -27,6 +27,14 @@ Cluster name identifies your cluster +masters_also_are_datanodes +"false" --- End diff -- No. ES will fail on a bool. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104660014 --- Diff: metron-deployment/roles/kibana/README.md --- @@ -1,35 +0,0 @@ -Kibana 4 - - -This role installs Kibana along with the default Metron Dashboard. - -### FAQ - - How do I change Metron's default dashboard? --- End diff -- Oh, hey, thanks! Maybe at the end of the deployment readme? I dunno, weirdly enough, I really struggled with a sensible place to put it. Co-located with the module seemed too buried (though, I hope you found the command line help, well, helpful) and way up in deployment seemed mis-located. I was about to put a wiki post together, but then there was some talk about not having one... Anyway, maybe @ottobackwards has a suggestion, he pointed out the lack of documentation in the first place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Resolved the merge conflicts from Master and addressed review comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104565691
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
# limitations under the License.
#
---
-hdp_repo_def:
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo";
+- name: Build Deployment Artifacts
+ local_action: shell cd {{ metron_build_dir }} && mvn clean package
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --
I think I'd rather check in Maven and fail better earlier. What would you
think about handling that as a standalone Jira?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104565422 --- Diff: metron-deployment/roles/quick_dev/tasks/main.yml --- @@ -15,23 +15,50 @@ # limitations under the License. # --- -# -# Workaround for Kafka not starting -# Fire off async start followed by -# Sync start -execution will pause until -# final start completes. -# -- name: Start the ambari cluster - no wait - ambari_cluster_state: +- name: Delete the Metron Components from Ambari --- End diff -- Exactly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104548141
--- Diff: metron-deployment/roles/kafka-broker/tasks/main.yml ---
@@ -1,41 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-- name: Retrieve HDP repository definition
- get_url:
-url: "{{ hdp_repo_def }}"
-dest: /etc/yum.repos.d/hdp.repo
-mode: 0644
-
-- name: Install kafka
- yum: name={{item}}
- with_items:
-- java-1.8.0-openjdk
-- kafka
-- zookeeper-server
-
-- name: Create pid directories
- file: path={{ item }} state=directory mode=0755
- with_items:
-- /var/run/zookeeper
-- /var/run/kafka
-
-- name: Start zookeeper
- shell: /usr/hdp/current/zookeeper-server/bin/zookeeper-server start
-
-- name: Start kafka
--- End diff --
Oops. I'll put it back with the next push.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104546536 --- Diff: metron-deployment/packaging/packer-build/.gitignore --- @@ -1,6 +1,6 @@ .bundle/ iso -*.box +builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box --- End diff -- Urm, no. I must have dumped my copy buffer by mistake while IntelliJ was open. It'll be gone in the next changeset. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/436#discussion_r104546225
--- Diff: metron-deployment/vagrant/full-dev-platform/Vagrantfile ---
@@ -53,7 +53,7 @@ hosts = [{
Vagrant.configure(2) do |config|
# all hosts built on centos 6
- config.vm.box = "bento/centos-6.7"
+ config.vm.box = "metron/centos_base"
--- End diff --
Exactly that for exactly that reason. That and I can control the swap space
size now. I get a much more reliable (not perfect, but better) full dev
experience with this image. Something that I wanted to do for quite some time,
but I finally got it done because I was having trouble differentiating between
errors I had introduced and pre-existing errors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104545807 --- Diff: metron-deployment/roles/quick_dev/tasks/main.yml --- @@ -15,23 +15,50 @@ # limitations under the License. # --- -# -# Workaround for Kafka not starting -# Fire off async start followed by -# Sync start -execution will pause until -# final start completes. -# -- name: Start the ambari cluster - no wait - ambari_cluster_state: +- name: Delete the Metron Components from Ambari --- End diff -- It bridges the gap between the quick-dev image and a fully functioning instance. To wit: - Removes the stale Metron that ships with the image from Ambari - Removes the stale Metron RPMs - Re-installs fresh Metron packages - Starts the cluster We only require these activities on a quick-dev image. If you think of Ansible as declarative, it's not that weird. We apply the quick-dev role to the quick-dev host. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r104544895 --- Diff: metron-deployment/roles/kibana/README.md --- @@ -1,35 +0,0 @@ -Kibana 4 - - -This role installs Kibana along with the default Metron Dashboard. - -### FAQ - - How do I change Metron's default dashboard? --- End diff -- There's a Python [module](https://github.com/apache/incubator-metron/tree/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard) that I wrote as part of the MPack that does the work there. I have an outstanding item to write up it's basic usage, but can't find a good place in the code. Ideas? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 I'm working on those merge conflicts right now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @JonZeolla - makes sense, I've pushed that change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 From @JonZeolla - "Looks like Docker is a dependency now which rules out CentOS 6 if we strictly follow the docker documentation. Was that expected or would you assume people would opt for the workaround steps?" It is, to integrate with the MPack we have to build RPMs and (right now anyway), so that means Docker. It was always implicit in this ticket, but I did try to underline it over the weekend. I wasn't aware that Docker and CentOS6 were problematic. I used to use those two together a little while back, but it was forever ago in Dockertime. Is that a showstopper for anyone? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @ottobackwards - sure, I used [this](https://cwiki.apache.org/confluence/display/AMBARI/API+usage+scenarios%2C+troubleshooting%2C+and+other+FAQs) as a guide. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 This is ready for review. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 @ottobackwards - that's correct. At that time the new boxes hadn't been prepped. Sorry for the confusion, I had left that box uncheck in an effort to communicate that. It should be good now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/468 Oh, hey, that stack trace looks familiar. I think Justin fixed this here: https://github.com/apache/incubator-metron/pull/461. -D... On Thu, Mar 2, 2017 at 2:08 PM, Michael Miklavcic wrote: > +1 tested this in Vagrant quick-dev. > > @cestella <https://github.com/cestella> I'm good with waiting on the > default dir. I was also able to run this through e2e with your latest > version. I also ran into issues with vagrant up not loading the geo > enrichment. I get the following exception. (I'll add a Jira.) > > 2017-03-01 14:59:24.634 o.a.m.e.a.g.GeoLiteDatabase [ERROR] [Metron] Unable to open new database file /apps/metron/geo/default/GeoLite2-City.mmdb.gz > java.io.FileNotFoundException: File does not exist: /apps/metron/geo/default/GeoLite2-City.mmdb.gz > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) > at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1860) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1831) > at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1744) > at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:693) > at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:373) > at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_77] > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_77] > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_77] > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_77] > at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) ~[stormjar.jar:?] > at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1242) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1227) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1215) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:303) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:269) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:261) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) ~[stormjar.jar:?] > at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[stormjar.jar:?] > at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312) ~[stormjar.jar:?] > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) ~[stormjar.jar:?] > at org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:93) [stormjar.jar:?] > at org.apache.metron.enrichment.bolt.Threat
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Sounds good. The important thing to remember is the overall goal of reducing reliance on Ansible. I had to add a build task because installation will fail without it, but I fully expect that task to disappear sooner rather than later, so adding additional dependencies on it will need to be scrutinized carefully. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 No. It's not meant for that. If you want to build standalone, Maven is the supported way. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Yeah, that seems like a good thing to try. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Sure. It's vagrant up or ./run.sh. :) I run it as part of running full dev or ec2. What error are you getting? Btw, to only build the rpms, you may can still do an mvn package -DskipTests -Pbuild-rpms from maven-deploy. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 Yes, that works on my rig. Can you tell me a bit more about what you're experiencing? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/436 I am, for both full dev and ec2. What are you trying to do? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r103241052 --- Diff: metron-deployment/roles/java_jdk/defaults/main.yml --- @@ -15,4 +15,4 @@ # limitations under the License. # --- --- End diff -- I used Ambari's default so I don't have to worry about any introduced non-compatibility. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r103240842 --- Diff: metron-deployment/roles/kibana/README.md --- @@ -1,35 +0,0 @@ -Kibana 4 --- End diff -- Ah, it is, but not well. I'll make sure I get it as part of this effort. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r103235875 --- Diff: metron-deployment/roles/kibana/README.md --- @@ -1,35 +0,0 @@ -Kibana 4 --- End diff -- Is what documented? I just removed the role as it's deprecated with the MPack. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #422: METRON-670 Monit Incorrectly Reports Status
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/422 It creates a bunch of rebase conflicts for me when I bring it in. Fwiw, I've got one more hard problem + a review until I'm done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #422: METRON-670 Monit Incorrectly Reports Status
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/422 All of these files will go away as a result of [PR-436](https://github.com/apache/incubator-metron/pull/436). Would you guys be willing to hold off until I can get it completed and through the process? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #446: METRON-707: Correct ansible to execute threat i...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/446 I'm +1 on this (travis pending). As a side note, do you think we should revisit our strategy of not running threat intel on vagrant builds? IIRC, we didn't have sufficient headroom to run MR reliably, but maybe that's changed now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/436 METRON-671: Refactor existing Ansible deployment to use Ambari MPack **WORK IN PROGRESS - DO NOT MERGE** This is the first set of changes to enable Ansible installation to use the Ambari MPack. It currently works (in my environment) with full-dev using sensor-stubs. It will not work with Quick-Dev or EC2 at this time. It's (well past - sorry) starting to get large, so I wanted to push it out for feedback while I'm working issues with the distributed install. Some points of interest: - As discussed, components (ES, Kibana, Metron Topologies) that are managed by Ambari are removed from Monit - I had added a notion of 'required configurations' to the blueprint and ambari_cluster_state script so blueprint verification would pass. - I've refactored our Ansible host groupings (see inventory/full-dev-platform/hosts). The Metron group is now the host that we use as the Metron services master. - I cleaned up the very minimum number of issues that @mattf-horton found in [METRON-609](https://issues.apache.org/jira/browse/METRON-609) required to make single-node deployment work with the MPack. It's my intention to work to get his changes incorporated on top of the final version of this changeset. - I removed the Solr role from Ansible. My current intention is to open a Jira to add a Solr Indexing component to the MPack. Immediate next steps: - [ ] Get the EC2 distributed install working. - [ ] Adjust readmes and docs where required. - [ ] Merge in Matt's changes and make required PRs on his branch (if any). - [ ] Document test plan I also hope to receive and respond to feedback. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-671 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/436.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #436 commit 01cd4216754861c7eaf58c04472fb2a8125a6f28 Author: David Lyle Date: 2017-01-23T20:47:14Z METRON-671: Refactor existing Ansible deployment to use Ambari MPack --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #425: METRON 609 Enhance Mpack to handle single-node ...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/425 I think I can collapse both templates into 1, at least that's what I was thinking. Just so you're aware @mattf-horton (and other interested parties). I've got a [METRON-671](https://issues.apache.org/jira/browse/METRON-671) PR coming out in the few days. It addresses a small subset of this changeset (METRON-641) and a few templating issues that will allow us to run on 1 node or 3+ nodes. It turns out, those were the only working configurations under Ansible automation. Immediately after that, I want to get this change incorporated, so I'll be testing it on the branch that deploys with Ansible using the MPack. I'm using that order so that we can accelerate the path to using the MPack in Ansible. Otherwise, we'll keep having changes in the Ansible aren't reflected in the MPack. Please let me know if that approach makes sense or if there are any concerns. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/405 Here's a fun fact- quick-dev uses 2.6.6. Had to make this exact change while unifying Ansible with the MPack, so if nothing else, we'll get it there. Still I'd like @DimDroll to get the deserved credit, so I'd be happy for this one to get there first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #421: METRON-283 Migrate Geo Enrichment outside of My...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/421 Still +1 @justinleet - thanks for checking. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/405 Yup, please change the other 3 and we'll get this in, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #421: METRON-283 Migrate Geo Enrichment outside of My...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/421 Ditto +1. Thanks for all the hard work! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #425: METRON 609 Enhance Mpack to handle single-node ...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/425 Hi @mattf-horton - thanks for putting this up! It's got my "have to haves" plus a bunch more good stuff. I'll get working with it today. If I discover any small changes, I'll put a PR on your branch. Thanks again, this is great stuff. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/421#discussion_r97839650
--- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
---
@@ -317,6 +316,8 @@ This package installs the Metron Profiler %{metron_home}
#
~~
%changelog
+* Thu Jan 19 2017 Justin Leet - 0.3.0
--- End diff --
I think that's where I'm at with it. How about we let @justinleet's mentor
thread confirm/deny that it's okay to keep it and then handle it as a directed
effort after?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/421#discussion_r97838057
--- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
---
@@ -317,6 +316,8 @@ This package installs the Metron Profiler %{metron_home}
#
~~
%changelog
+* Thu Jan 19 2017 Justin Leet - 0.3.0
--- End diff --
But so we're clear- with the changelog in place you can interrogate the rpm
directly to find out what's changed "rpm -q --changelog". We'd lose that
capability. I'm still good with omitting it, but I wanted to make sure everyone
was aware of that consequence.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/421#discussion_r97836561
--- Diff: metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
---
@@ -317,6 +316,8 @@ This package installs the Metron Profiler %{metron_home}
#
~~
%changelog
+* Thu Jan 19 2017 Justin Leet - 0.3.0
--- End diff --
I'd just yank that section or leave it blank if it's required. No point in
figuring out how to keep it if we're agreed we don't need it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/421#discussion_r97830250
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBolt.java
---
@@ -149,9 +154,10 @@ public JSONObject load(CacheKey key) throws Exception {
cache = CacheBuilder.newBuilder().maximumSize(maxCacheSize)
.expireAfterWrite(maxTimeRetain, TimeUnit.MINUTES)
.build(loader);
+
GeoLiteDatabase.INSTANCE.update((String)getConfigurations().getGlobalConfig().get(GeoLiteDatabase.GEO_HDFS_FILE));
boolean success = adapter.initializeAdapter();
--- End diff --
@justinleet - I think that's a great solution, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #408: METRON-608 Mpack to install a single-node test ...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/408 @mattf-horton - I need an MPack that works for 1-N nodes in order to complete [METRON-671](https://issues.apache.org/jira/browse/METRON-671). I suspect I'm working through stuff you've already fixed. Would it make sense to push out a PR with the unified branch and let me test it? Then I could use finish METRON-671 on top of your changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #421: METRON-283 Migrate Geo Enrichment outsid...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/421#discussion_r97334025
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBolt.java
---
@@ -161,6 +167,7 @@ protected void initializeStellar() {
stellarContext = new Context.Builder()
.with(Context.Capabilities.ZOOKEEPER_CLIENT, ()
-> client)
.with(Context.Capabilities.GLOBAL_CONFIG, () ->
getConfigurations().getGlobalConfig())
+ .with(Context.Capabilities.GEO_IP, () ->
GeoLiteDatabase.INSTANCE)
--- End diff --
Also- why is this in the GenericEnrichment bolt? Am I mistaken, or wouldn't
this be better located in the GEO bolt?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
