[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 The only issue with it, is if we did that in the old version, then this would be a regression. Other than that, I'm just waiting to review the docs ;) ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 Whew, that's great news! Hahahahaha, that's the default colors from the Kibana widgets... I usually refrain from bringing it up, but I'm partially colorblind (even though I used to do a fair amount of Photoshop graphic design work on the side, but it's a bit more effort for me). Anyhow, Kibana had no easy way to pick color names or supply RGB values, so I punted on it for now because I figured that's an easy change. If you would be willing to take a screenshot or the xy coordinate of colors in the color picker that you'd prefer, I'd be more than happy to change it. ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @ottobackwards You should be good to try it now. ---
[GitHub] metron issue #824: METRON-1289: Alert fields are lost when a MetaAlert is cr...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/824 The latest commits address some outstanding tasks and address a couple other issues: - disabling updates to meta alert objects means we can't update a meta alert name or comments (or other fields that may be added in the future). The patch implementation was added to the ElasticsearchMetaAlertDao to allow updates to fields other than `alert` and `status` which require special handling and have dedicated endpoints - a `getAllLatest` implementation was added to the HBaseDao along with an integration test - a bug was discovered where the number of search results returned defaults to 10 meaning changes to alerts are only propagated to the first 10 meta alerts and vice-versa. This was corrected in the ElasticsearchDao.searchByGuids by explicitly setting the size and fixed in the alert/meta alert lookup queries by paging through all the results if necessary. A test was added to the integration test to catch this case in the future. I feel like this PR is in pretty good shape from a functional perspective. Still working on javadocs/documentation but it's ready for functional testing/code review. ---
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150336506
--- Diff:
metron-platform/metron-indexing/src/main/scripts/start_hdfs_topology.sh ---
@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+TOPOLOGY_JAR=metron-elasticsearch-$METRON_VERSION-uber.jar
+storm jar $METRON_HOME/lib/$TOPOLOGY_JAR org.apache.storm.flux.Flux
--remote $METRON_HOME/flux/indexing/batch/remote.yaml --filter
$METRON_HOME/config/hdfs.properties
--- End diff --
Very true. I think you're right, @ottobackwards
---
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150317133
--- Diff:
metron-platform/metron-indexing/src/main/scripts/start_hdfs_topology.sh ---
@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+TOPOLOGY_JAR=metron-elasticsearch-$METRON_VERSION-uber.jar
+storm jar $METRON_HOME/lib/$TOPOLOGY_JAR org.apache.storm.flux.Flux
--remote $METRON_HOME/flux/indexing/batch/remote.yaml --filter
$METRON_HOME/config/hdfs.properties
--- End diff --
I almost think that if it is going to be hdfs, we should call it hdfs. If
we have multiple topologies based on the batch indexer, each will end up with a
unique name anyways ( S3 indexer etc ).
---
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150316808
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
---
@@ -162,34 +164,77 @@ def start_indexing_topology(self, env):
self.__params.metron_principal_name,
execute_user=self.__params.metron_user)
-start_cmd_template =
"""{0}/bin/start_elasticsearch_topology.sh \
--s {1} \
--z {2}"""
-start_cmd =
start_cmd_template.format(self.__params.metron_home,
- self.__indexing_topology,
-
self.__params.zookeeper_quorum)
+start_cmd_template = """{0}/bin/start_hdfs_topology.sh"""
+start_cmd =
start_cmd_template.format(self.__params.metron_home)
Execute(start_cmd, user=self.__params.metron_user, tries=3,
try_sleep=5, logoutput=True)
else:
-Logger.info('Indexing topology already running')
+Logger.info('Batch Indexing topology already running')
-Logger.info('Finished starting indexing topology')
+Logger.info('Finished starting batch indexing topology')
-def stop_indexing_topology(self, env):
-Logger.info('Stopping ' + self.__indexing_topology)
+def start_random_access_indexing_topology(self, env):
+Logger.info('Starting ' + self.__random_access_indexing_topology)
--- End diff --
I like this a lot. I would like to see the indexing be like the parses,
where you start with a name parameter. I think the flux files make it
difficult though. I would be +1 exploring this approach though, maybe a
feature branch?
---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 I got full_dev failure on connecting to metron web. Can you post when you think full dev is ready and I'll try again then? ---
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/831#discussion_r150315275 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-indexing-env.xml --- @@ -73,18 +98,31 @@ Indexing Update Column Family -indexing_workers +ra_indexing_workers +Number of Indexing Topology Workers --- End diff -- Actually, I see the display name is different. Doh. But here we are using the terms "Elasticsearch" and "HDFS", rather than "Random Access" and "Batch". We should probably stick with one nomenclature. ---
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150314839
--- Diff:
metron-platform/metron-indexing/src/main/scripts/start_hdfs_topology.sh ---
@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+TOPOLOGY_JAR=metron-elasticsearch-$METRON_VERSION-uber.jar
+storm jar $METRON_HOME/lib/$TOPOLOGY_JAR org.apache.storm.flux.Flux
--remote $METRON_HOME/flux/indexing/batch/remote.yaml --filter
$METRON_HOME/config/hdfs.properties
--- End diff --
In the MPack we refer to this as the "batch" topology. But in this script
we refer to it as the "HDFS" topology. We should probably stick with one
nomenclature.
---
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/831#discussion_r150314308 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-indexing-env.xml --- @@ -73,18 +98,31 @@ Indexing Update Column Family -indexing_workers +ra_indexing_workers +Number of Indexing Topology Workers --- End diff -- With the name and description the same for the random access and batch topologies; can the user distinguish which is which? ---
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r150314117
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
---
@@ -162,34 +164,77 @@ def start_indexing_topology(self, env):
self.__params.metron_principal_name,
execute_user=self.__params.metron_user)
-start_cmd_template =
"""{0}/bin/start_elasticsearch_topology.sh \
--s {1} \
--z {2}"""
-start_cmd =
start_cmd_template.format(self.__params.metron_home,
- self.__indexing_topology,
-
self.__params.zookeeper_quorum)
+start_cmd_template = """{0}/bin/start_hdfs_topology.sh"""
+start_cmd =
start_cmd_template.format(self.__params.metron_home)
Execute(start_cmd, user=self.__params.metron_user, tries=3,
try_sleep=5, logoutput=True)
else:
-Logger.info('Indexing topology already running')
+Logger.info('Batch Indexing topology already running')
-Logger.info('Finished starting indexing topology')
+Logger.info('Finished starting batch indexing topology')
-def stop_indexing_topology(self, env):
-Logger.info('Stopping ' + self.__indexing_topology)
+def start_random_access_indexing_topology(self, env):
+Logger.info('Starting ' + self.__random_access_indexing_topology)
--- End diff --
First off, I think we definitely need to make this happen. Each index
destination is going to have very different performance characteristics that
need to be tuned in isolation. I think this is a step in the right direction.
As I read this we have effectively hard-coded two indexing topologies;
random access and batch. This is definitely the most logical way to get to
separate topologies based on our existing code base. But I am wondering if we
might think about this in a slightly different way.
What I really like about indexing is that we have the idea of multiple,
independent destinations. For example, my indexing configuration could look
like this.
```
{
"elasticsearch": {
"index": "foo",
"enabled" : true
},
"hdfs": {
"index": "foo",
"batchSize": 1,
"enabled" : true
}
}
```
What if we introduced logic that consumes the indexing configuration,
determines that it needs to launch 2 topologies in this case, and then launches
those 2 separate topologies? If I had 3 destinations configured, then it would
launch 3 topologies; one for each destination?
I can definitely see the extra complexity in doing this. You have to make
sure the user can independently configure each of the topologies. You have to
respond to configuration changes made by the user. And probably a few other
complications.
But these are already complications that we need to deal with in Parsing.
A user can define 1 to N Parsing topologies. It seems like if we can solve
these challenges for Parsing, we can do the same for Indexing.
Anywho, I can totally see this PR as a near-term solution to the immediate
problem, which might lead towards a longer-term solution like I propose. I
just wanted to see if anyone had related thoughts.
---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 Ok, looks like 2 more minor issues. 1. params.py missing the kibana_server_host, so it's not making its way to kibana.yml 2. The default index mapping template that ES creates when I index the Kibana configuration is not compatible with Kibana. I'm done fixing 1 already and have the solution for 2. I'm going to blow away the .kibana index, let Kibana recreate it with the expected template, and then extract that into a template file that's used during the Kibana server install. The only hitch is that this is one more thing that potentially needs to be modified/updated on future upgrades of ES and Kibana. ---
[GitHub] metron issue #837: METRON-1309: Change metron-deployment to pull the plugin ...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/837 +1 LGTM. Thanks! ---
[GitHub] metron issue #839: METRON-1311 Service Check Should Check Elasticsearch Inde...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/839 I ran this through another full test after merge with master. Everything looks good to me. ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Thanks @mmiklavc, those changes look good. Running up in full dev ---
[GitHub] metron pull request #810: METRON-1270 fix for warnings missing @return tag a...
Github user dbist closed the pull request at: https://github.com/apache/metron/pull/810 ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150282327 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Heh, yep. That extra import broke the Kibana install hard. I've made most of your requested changes and will have another commit coming shortly. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150281496
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExportTest.java
---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.integration.utils.TestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ElasticsearchImportExportTest {
+
+
--- End diff --
Yeah, I think you're right. I really like having data inline so that you
can see the full context of your tests without having to follow multiple
branches to piece together what actually happens. I think it's ok for the
system under test, but the tests themselves should be much more intuitive and
easy to follow. So I think the right answer here is that I've included a lot of
extra noise that is unnecessary for the test. I'm going to prune the "_source"
object, which will make this very lean and much easier to read. Really good
catch @ottobackwards.
---
[GitHub] metron issue #839: METRON-1311 Service Check Should Check Elasticsearch Inde...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/839 Thanks @anandsubbu . I resolved the merge conflicts from master. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150274541 --- Diff: metron-platform/elasticsearch-shaded/pom.xml --- @@ -99,7 +148,15 @@ --- End diff -- Yes, absolutely agreed. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150274108 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Hm, I'm not sure where that came from. I suspect IntelliJ added that in the middle of my edits. ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @mraliagha @cestella agreed on timestamp. ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Can you call the curator stuff out in your checklist above? ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Ok, killing my vagrant up then, until your commit ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @ottobackwards Oh, afa the data pruner, I believe we can use Curator now as a built-in mechanism for pruning data. Steps will be added to the documentation. https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html ---
[GitHub] metron issue #839: METRON-1311 Service Check Should Check Elasticsearch Inde...
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/839
+1 (non-binding) works fine, and a much needed fix as well ! If for some
reason, we were not able to install the templates during the startup, it would
be caught by the service check.
Validated this on a full dev environment. For a successful service check, I
see this now:
```
2017-11-10 15:48:06,150 - Checking Elasticsearch templates for Indexing
2017-11-10 15:48:06,151 - Execute['curl -s -XGET
"http://node1:9200/_template/yaf_index; | grep -o yaf_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
yaf_index
yaf_index
2017-11-10 15:48:06,343 - Execute['curl -s -XGET
"http://node1:9200/_template/error_index; | grep -o error_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
error_index
error_index
2017-11-10 15:48:06,385 - Execute['curl -s -XGET
"http://node1:9200/_template/snort_index; | grep -o snort_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
snort_index
snort_index
2017-11-10 15:48:06,433 - Execute['curl -s -XGET
"http://node1:9200/_template/bro_index; | grep -o bro_index'] {'logoutput':
True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3,
'user': 'metron', 'try_sleep': 5}
bro_index
bro_index
2017-11-10 15:48:06,491 - Execute['curl -s -XGET
"http://node1:9200/_template/metaalert_index; | grep -o metaalert_index']
{'logoutput': True, 'path': ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'],
'tries': 3, 'user': 'metron', 'try_sleep': 5}
metaalert_index
```
And the service check fails when I delete the ES template file.
```
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/service_check.py",
line 78, in
ServiceCheck().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 280, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/service_check.py",
line 51, in service_check
indexing_cmds.service_check(env)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_commands.py",
line 264, in service_check
self.check_elasticsearch_templates()
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_commands.py",
line 179, in check_elasticsearch_templates
metron_service.execute(cmd, user=self.__params.metron_user,
err_msg=err_msg)
File
"/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/metron_service.py",
line 257, in execute
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Missing Elasticsearch index
template: name=yaf_index
```
@nickwallen - I see two conflicting files for the commit. Guess this would
be need to be resolved first?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150269845
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/slave.py
---
@@ -48,19 +51,30 @@ def slave():
content=InlineTemplate(params.elastic_env_sh_template)
)
-configurations = params.config['configurations']['elastic-site']
-
-File("{0}/elasticsearch.yml".format(params.conf_dir),
+elastic_site = params.config['configurations']['elastic-site']
+path = "{0}/elasticsearch.yml".format(params.conf_dir)
+Logger.info("Cre")
--- End diff --
Hm, adding to my list of cleanup tasks.
---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 @ottobackwards thanks for the review! The templates themselves changed, but the deployment of those templates is the same. I discovered that with a host change in Kibana, I broke the Ambari Blueprint for running up full dev. I found the issue and am running it up now and should have that committed shortly. ---
[GitHub] metron issue #810: METRON-1270 fix for warnings missing @return tag argument...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/810 It's merged. The commit message looks acceptable too. Not sure why it didn't automatically close. Please close @dbist ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/840 Are there any required changes to the ansible roles? I thought we injected the templates from there. Or has the indexing service injection landed? ---
[GitHub] metron issue #810: METRON-1270 fix for warnings missing @return tag argument...
Github user dbist commented on the issue: https://github.com/apache/metron/pull/810 @justinleet thanks for looking into this, as soon as I get a confirmation, I'll close it. ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150264674 --- Diff: metron-platform/elasticsearch-shaded/pom.xml --- @@ -99,7 +148,15 @@ --- End diff -- Maybe a comment as to why we are doing relocations will help future maintainers ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150262722
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/slave.py
---
@@ -48,19 +51,30 @@ def slave():
content=InlineTemplate(params.elastic_env_sh_template)
)
-configurations = params.config['configurations']['elastic-site']
-
-File("{0}/elasticsearch.yml".format(params.conf_dir),
+elastic_site = params.config['configurations']['elastic-site']
+path = "{0}/elasticsearch.yml".format(params.conf_dir)
+Logger.info("Cre")
--- End diff --
I don't think "Cre" is what you wanted here, typo?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150263736 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/5.6.2/package/scripts/kibana_master.py --- @@ -24,6 +24,7 @@ from ambari_commons.os_check import OSCheck from ambari_commons.os_family_impl import OsFamilyFuncImpl, OsFamilyImpl --- End diff -- Is the ansible module something we are going to distribute? ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150264904
--- Diff: metron-platform/metron-data-management/pom.xml ---
@@ -231,11 +231,11 @@
httpclient
${httpcore.version}
--- End diff --
Why not just remove these?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150259383 --- Diff: dependencies_with_url.csv --- @@ -310,6 +310,29 @@ org.springframework.security.kerberos:spring-security-kerberos-core:jar:1.0.1.RE org.springframework.kafka:spring-kafka:jar:1.1.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-kafka ch.hsr:geohash:jar:1.3.0:compile,ASLv2,https://github.com/kungfoo/geohash-java org.locationtech.spatial4j:spatial4j:jar:0.6:compile,ASLv2,https://github.com/locationtech/spatial4j --- End diff -- can we remove any of the older dependencies? ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r150266275
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExportTest.java
---
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.bulk;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import java.io.File;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.integration.utils.TestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ElasticsearchImportExportTest {
+
+
--- End diff --
Is there a tipping point where this should be a file vs. multiline?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150265577 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java --- @@ -139,13 +141,17 @@ protected SearchResponse search(SearchRequest searchRequest, QueryBuilder queryB searchRequest.getSort().forEach(sortField -> searchSourceBuilder.sort(sortField.getField(), getElasticsearchSortOrder(sortField.getSortOrder(; Optionalfields = searchRequest.getFields(); if (fields.isPresent()) { --- End diff -- why not just remove? ---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r150265417 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/ElasticsearchImportExport.java --- @@ -0,0 +1,76 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.elasticsearch.bulk; + +import com.fasterxml.jackson.core.type.TypeReference; +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.io.FileReader; +import java.io.FileWriter; +import java.io.IOException; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import org.apache.metron.common.utils.JSONUtils; + --- End diff -- javadoc ---
[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user cestella commented on the issue: https://github.com/apache/metron/pull/840 @mraliagha Yeah, absolutely. This task purposefully leaves out major changes that we could do to improve things (e.g. moving back to `.` separated fields instead of `:`). The attempt is to touch as few things as possible. Your ask makes a lot of sense and we should add it as a follow-on task. ---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150239607
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/MetaAlertController.java
---
@@ -60,5 +63,37 @@
) throws RestException {
return new ResponseEntity<>(metaAlertService.create(createRequest),
HttpStatus.OK);
}
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/add/alert", method = RequestMethod.POST)
+ ResponseEntity addAlertsToMetaAlert(
+ @ApiParam(name = "request", value = "Meta Alert Create Request",
required = true)
+ @RequestBody final MetaAlertAddRemoveRequest
metaAlertAddRemoveRequest
+ ) throws RestException {
+return new
ResponseEntity<>(metaAlertService.addAlertsToMetaAlert(metaAlertAddRemoveRequest),
HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/remove/alert", method = RequestMethod.POST)
+ ResponseEntity removeAlertsFromMetaAlert(
+ @ApiParam(name = "request", value = "Meta Alert Create Request",
required = true)
+ @RequestBody final MetaAlertAddRemoveRequest
metaAlertAddRemoveRequest
+ ) throws RestException {
+return new
ResponseEntity<>(metaAlertService.removeAlertsFromMetaAlert(metaAlertAddRemoveRequest),
HttpStatus.OK);
+ }
+
+ @ApiOperation(value = "Create a meta alert")
+ @ApiResponse(message = "Created meta alert", code = 200)
+ @RequestMapping(value = "/update/status/{guid}/{status}", method =
RequestMethod.POST)
+ ResponseEntity updateMetaAlertStatus(
+ final @ApiParam(name = "guid", value = "Kafka topic name", required
= true)
+ @PathVariable String guid,
+ final @ApiParam(name = "status", value = "Kafka topic name",
required = true)
+ @PathVariable String status) throws RestException {
+return new
ResponseEntity<>(metaAlertService.updateMetaAlertStatus(guid,
+MetaAlertStatus.valueOf(status.toUpperCase())), HttpStatus.OK);
+ }
--- End diff --
I think we'll also need to be able to add comments and a name to the
metaalert as a whole, to support work in
https://github.com/apache/metron/pull/803/
@iraghumitra Is there anything else needed in terms of updates on meta
alerts themselves (underlying alerts aren't affected by this stuff)?
---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150239332
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
---
@@ -256,59 +256,91 @@ public Document getLatest(final String guid, final
String sensorType) throws IOE
return ret.orElse(null);
}
+ @Override
+ public Iterable getAllLatest(final Collection guids,
final Collection sensorTypes) throws IOException {
+List documents = searchByGuids(
+guids
+, sensorTypes
+, hit -> {
+ Long ts = 0L;
+ String doc = hit.getSourceAsString();
+ String sourceType =
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+ try {
+return Optional.of(new Document(doc, hit.getId(), sourceType,
ts));
+ } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve latest: "
+ e.getMessage(), e);
+ }
+}
+
+);
+return documents;
+ }
+
+ Optional searchByGuid(String guid, String sensorType,
+ Function callback) {
+Collection sensorTypes = sensorType != null ?
Collections.singleton(sensorType) : null;
+List results = searchByGuids(Collections.singleton(guid),
sensorTypes, callback);
+if (results.size() > 0) {
+ return Optional.of(results.get(0));
+} else {
+ return Optional.empty();
+}
+ }
+
/**
* Return the search hit based on the UUID and sensor type.
* A callback can be specified to transform the hit into a type T.
* If more than one hit happens, the first one will be returned.
*/
- Optional searchByGuid(String guid, String sensorType,
+ List searchByGuids(Collection guids, Collection
sensorTypes,
Function callback) {
QueryBuilder query;
-if (sensorType != null) {
- query = QueryBuilders.idsQuery(sensorType + "_doc").ids(guid);
+if (sensorTypes != null) {
+ String[] types = sensorTypes.stream().map(sensorType -> sensorType +
"_doc").toArray(String[]::new);
+ query = QueryBuilders.idsQuery(types).ids(guids);
} else {
- query = QueryBuilders.idsQuery().ids(guid);
+ query = QueryBuilders.idsQuery().ids(guids);
}
SearchRequestBuilder request = client.prepareSearch()
--- End diff --
Good catch. I will add a fix and test case.
---
[GitHub] metron pull request #824: METRON-1289: Alert fields are lost when a MetaAler...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/824#discussion_r150230733
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
---
@@ -256,59 +256,91 @@ public Document getLatest(final String guid, final
String sensorType) throws IOE
return ret.orElse(null);
}
+ @Override
+ public Iterable getAllLatest(final Collection guids,
final Collection sensorTypes) throws IOException {
+List documents = searchByGuids(
+guids
+, sensorTypes
+, hit -> {
+ Long ts = 0L;
+ String doc = hit.getSourceAsString();
+ String sourceType =
Iterables.getFirst(Splitter.on("_doc").split(hit.getType()), null);
+ try {
+return Optional.of(new Document(doc, hit.getId(), sourceType,
ts));
+ } catch (IOException e) {
+throw new IllegalStateException("Unable to retrieve latest: "
+ e.getMessage(), e);
+ }
+}
+
+);
+return documents;
+ }
+
+ Optional searchByGuid(String guid, String sensorType,
+ Function callback) {
+Collection sensorTypes = sensorType != null ?
Collections.singleton(sensorType) : null;
+List results = searchByGuids(Collections.singleton(guid),
sensorTypes, callback);
+if (results.size() > 0) {
+ return Optional.of(results.get(0));
+} else {
+ return Optional.empty();
+}
+ }
+
/**
* Return the search hit based on the UUID and sensor type.
* A callback can be specified to transform the hit into a type T.
* If more than one hit happens, the first one will be returned.
*/
- Optional searchByGuid(String guid, String sensorType,
+ List searchByGuids(Collection guids, Collection
sensorTypes,
Function callback) {
QueryBuilder query;
-if (sensorType != null) {
- query = QueryBuilders.idsQuery(sensorType + "_doc").ids(guid);
+if (sensorTypes != null) {
+ String[] types = sensorTypes.stream().map(sensorType -> sensorType +
"_doc").toArray(String[]::new);
+ query = QueryBuilders.idsQuery(types).ids(guids);
} else {
- query = QueryBuilders.idsQuery().ids(guid);
+ query = QueryBuilders.idsQuery().ids(guids);
}
SearchRequestBuilder request = client.prepareSearch()
--- End diff --
This is only going to return 10 max results, from the default size on the
query. This was fine before (since we only returned one), but it's going to be
incomplete with the refactor.
---
