[GitHub] metron pull request #1198: METRON-1780: Fix broken website images
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1198 METRON-1780: Fix broken website images ## Contributor Comments Noticed that a couple images have been broken through at least the 0.5.0 release, but didn't want to hold up the 0.6.0 release. This makes the manual changes necessary to get it working again in `generate-md.sh`. Please check out https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge for instructions on how to spin it up. Just click through all the pages. No errors should occur on the command line related to a file not found. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron site_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1198.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1198 commit f240a266b9ca0fc3d50ef0f72e445a067b651787 Author: Justin Leet Date: 2018-09-13T20:49:59Z Fixing site images ---
[jira] [Commented] (METRON-1780) Fix broken website images
[ https://issues.apache.org/jira/browse/METRON-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614057#comment-16614057 ] ASF GitHub Bot commented on METRON-1780: GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1198 METRON-1780: Fix broken website images ## Contributor Comments Noticed that a couple images have been broken through at least the 0.5.0 release, but didn't want to hold up the 0.6.0 release. This makes the manual changes necessary to get it working again in `generate-md.sh`. Please check out https://cwiki.apache.org/confluence/display/METRON/Website+PR+Merge for instructions on how to spin it up. Just click through all the pages. No errors should occur on the command line related to a file not found. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron site_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1198.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1198 commit f240a266b9ca0fc3d50ef0f72e445a067b651787 Author: Justin Leet Date: 2018-09-13T20:49:59Z Fixing site images > Fix broken website images > - > > Key: METRON-1780 > URL: https://issues.apache.org/jira/browse/METRON-1780 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Some of the images on our site are broken. They need to be added manually to > the generate-md script to be moved and rewritten appropriately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1780) Fix broken website images
Justin Leet created METRON-1780: --- Summary: Fix broken website images Key: METRON-1780 URL: https://issues.apache.org/jira/browse/METRON-1780 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Some of the images on our site are broken. They need to be added manually to the generate-md script to be moved and rewritten appropriately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613973#comment-16613973 ] ASF GitHub Bot commented on METRON-1476: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1096 > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1096: METRON-1476: Update to Angular 6.1.3
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1096 ---
[GitHub] metron pull request #1195: METRON-1776: Update public web site to point at 0...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1195 ---
[jira] [Commented] (METRON-1776) Update public web site to point at 0.6.0 new release
[ https://issues.apache.org/jira/browse/METRON-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613955#comment-16613955 ] ASF GitHub Bot commented on METRON-1776: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1195 > Update public web site to point at 0.6.0 new release > > > Key: METRON-1776 > URL: https://issues.apache.org/jira/browse/METRON-1776 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1188: METRON-1769: Script creation of a release candidate
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1188 > @justinleet: that's because we tar up the rc and just rename the tar.gz file when it gets promoted. It's expected (and occurs on our previous releases). To add onto that, I don't know what other projects actually do about that (they may not, in which case we might want to modify what we do). I can't think of projects that keep the `-rcN`. Just to be sure I checked HBase, Phoenix, and Nifi and none of them seem to do it. We can fix it later though. ---
[jira] [Commented] (METRON-1769) Script creation of a release candidate
[ https://issues.apache.org/jira/browse/METRON-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613917#comment-16613917 ] ASF GitHub Bot commented on METRON-1769: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1188 > @justinleet: that's because we tar up the rc and just rename the tar.gz file when it gets promoted. It's expected (and occurs on our previous releases). To add onto that, I don't know what other projects actually do about that (they may not, in which case we might want to modify what we do). I can't think of projects that keep the `-rcN`. Just to be sure I checked HBase, Phoenix, and Nifi and none of them seem to do it. We can fix it later though. > Script creation of a release candidate > -- > > Key: METRON-1769 > URL: https://issues.apache.org/jira/browse/METRON-1769 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Our process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process is fairly > error prone. There's a lot of typing out versions and other tedious busywork. > In particular, this JIRA is for steps 4/5, as they're the most egregious. > This script should be able to produce both the core artifacts as well as the > bro kafka plugin artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1769) Script creation of a release candidate
[ https://issues.apache.org/jira/browse/METRON-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613824#comment-16613824 ] ASF GitHub Bot commented on METRON-1769: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1188 To add onto that, I don't know what other projects actually do about that (they may not, in which case we might want to modify what we do). E.g. create the folder without the suffix and rename the resulting tar.gz to have the suffix. > Script creation of a release candidate > -- > > Key: METRON-1769 > URL: https://issues.apache.org/jira/browse/METRON-1769 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Our process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process is fairly > error prone. There's a lot of typing out versions and other tedious busywork. > In particular, this JIRA is for steps 4/5, as they're the most egregious. > This script should be able to produce both the core artifacts as well as the > bro kafka plugin artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1188: METRON-1769: Script creation of a release candidate
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1188 To add onto that, I don't know what other projects actually do about that (they may not, in which case we might want to modify what we do). E.g. create the folder without the suffix and rename the resulting tar.gz to have the suffix. ---
[jira] [Commented] (METRON-1769) Script creation of a release candidate
[ https://issues.apache.org/jira/browse/METRON-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613817#comment-16613817 ] ASF GitHub Bot commented on METRON-1769: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1188 @nickwallen that's because we tar up the rc and just rename the tar.gz file when it gets promoted. It's expected (and occurs on our previous releases) > Script creation of a release candidate > -- > > Key: METRON-1769 > URL: https://issues.apache.org/jira/browse/METRON-1769 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Our process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process is fairly > error prone. There's a lot of typing out versions and other tedious busywork. > In particular, this JIRA is for steps 4/5, as they're the most egregious. > This script should be able to produce both the core artifacts as well as the > bro kafka plugin artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1188: METRON-1769: Script creation of a release candidate
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1188 @nickwallen that's because we tar up the rc and just rename the tar.gz file when it gets promoted. It's expected (and occurs on our previous releases) ---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613793#comment-16613793
]
ASF GitHub Bot commented on METRON-1771:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217462174
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --
No problem
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613791#comment-16613791
]
ASF GitHub Bot commented on METRON-1771:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217462087
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/InMemoryMetaAlertRetrieveLatestDao.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing;
+
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.metaalert.MetaAlertRetrieveLatestDao;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.io.IOException;
+import java.util.List;
+
+public class InMemoryMetaAlertRetrieveLatestDao implements
MetaAlertRetrieveLatestDao {
--- End diff --
Sure no problem.
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613789#comment-16613789
]
ASF GitHub Bot commented on METRON-1771:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217461869
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --
Ah, gotcha. Could you add that to the javadoc?
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217462174
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --
No problem
---
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217462087
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/InMemoryMetaAlertRetrieveLatestDao.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing;
+
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.metaalert.MetaAlertRetrieveLatestDao;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.io.IOException;
+import java.util.List;
+
+public class InMemoryMetaAlertRetrieveLatestDao implements
MetaAlertRetrieveLatestDao {
--- End diff --
Sure no problem.
---
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217461869
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --
Ah, gotcha. Could you add that to the javadoc?
---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613778#comment-16613778
]
ASF GitHub Bot commented on METRON-1771:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217459943
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String
sensorType) throws IOExcepti
public List getIndices() {
return indices;
}
+
+ private Document getDocument(List documentContainers)
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+ if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+ }
+ else {
+if(dc.getDocument().isPresent()) {
+ Document d = dc.getDocument().get();
+ if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+ }
+}
+ }
+}
--- End diff --
I like it. Will make that change.
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613777#comment-16613777
]
ASF GitHub Bot commented on METRON-1771:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217459852
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --
Check out UpdateDaoTest. As I created tests for ElasticsearchUpdateDao,
SolrUpdateDao, and HBaseDao I found myself copy/pasting the various tests. So
I consolidated them into a single test. I believe there is value in having a
single sets of tests for the UpdateDao interface methods that all the Daos must
satisfy. This class sets up it's Dao implementation to be used in the tests.
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217459943
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String
sensorType) throws IOExcepti
public List getIndices() {
return indices;
}
+
+ private Document getDocument(List documentContainers)
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+ if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+ }
+ else {
+if(dc.getDocument().isPresent()) {
+ Document d = dc.getDocument().get();
+ if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+ }
+}
+ }
+}
--- End diff --
I like it. Will make that change.
---
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217459852
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --
Check out UpdateDaoTest. As I created tests for ElasticsearchUpdateDao,
SolrUpdateDao, and HBaseDao I found myself copy/pasting the various tests. So
I consolidated them into a single test. I believe there is value in having a
single sets of tests for the UpdateDao interface methods that all the Daos must
satisfy. This class sets up it's Dao implementation to be used in the tests.
---
[jira] [Commented] (METRON-1769) Script creation of a release candidate
[ https://issues.apache.org/jira/browse/METRON-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613730#comment-16613730 ] ASF GitHub Bot commented on METRON-1769: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1188 @justinleet One thing I just noticed is that when you download the release tarball from a mirror the file `apache-metron-0.6.0.tar.gz` contains a directory called `apache-metron-0.6.0-rc1`. I would expect it to just be just `apache-metron-0.6.0`. I am not sure if this is something you have scripted here or if it is a manual step in the release process. Either way, not a big deal, just thought I would mention it. > Script creation of a release candidate > -- > > Key: METRON-1769 > URL: https://issues.apache.org/jira/browse/METRON-1769 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Our process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process is fairly > error prone. There's a lot of typing out versions and other tedious busywork. > In particular, this JIRA is for steps 4/5, as they're the most egregious. > This script should be able to produce both the core artifacts as well as the > bro kafka plugin artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1188: METRON-1769: Script creation of a release candidate
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1188 @justinleet One thing I just noticed is that when you download the release tarball from a mirror the file `apache-metron-0.6.0.tar.gz` contains a directory called `apache-metron-0.6.0-rc1`. I would expect it to just be just `apache-metron-0.6.0`. I am not sure if this is something you have scripted here or if it is a manual step in the release process. Either way, not a big deal, just thought I would mention it. ---
[jira] [Commented] (METRON-1776) Update public web site to point at 0.6.0 new release
[ https://issues.apache.org/jira/browse/METRON-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613720#comment-16613720 ] ASF GitHub Bot commented on METRON-1776: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1195 +1 Looks good. Validated according to the Wiki. > Update public web site to point at 0.6.0 new release > > > Key: METRON-1776 > URL: https://issues.apache.org/jira/browse/METRON-1776 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1195: METRON-1776: Update public web site to point at 0.6.0 ne...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1195 +1 Looks good. Validated according to the Wiki. ---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613695#comment-16613695
]
ASF GitHub Bot commented on METRON-1771:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217441951
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/InMemoryMetaAlertRetrieveLatestDao.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing;
+
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.metaalert.MetaAlertRetrieveLatestDao;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.io.IOException;
+import java.util.List;
+
+public class InMemoryMetaAlertRetrieveLatestDao implements
MetaAlertRetrieveLatestDao {
--- End diff --
Would you mind adding this comment to the javadoc for this class? I can
imagine I will ask myself that question a few years down the road.
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217441951
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/InMemoryMetaAlertRetrieveLatestDao.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing;
+
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.metaalert.MetaAlertRetrieveLatestDao;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.io.IOException;
+import java.util.List;
+
+public class InMemoryMetaAlertRetrieveLatestDao implements
MetaAlertRetrieveLatestDao {
--- End diff --
Would you mind adding this comment to the javadoc for this class? I can
imagine I will ask myself that question a few years down the road.
---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613694#comment-16613694
]
ASF GitHub Bot commented on METRON-1771:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217441386
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String
sensorType) throws IOExcepti
public List getIndices() {
return indices;
}
+
+ private Document getDocument(List documentContainers)
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+ if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+ }
+ else {
+if(dc.getDocument().isPresent()) {
+ Document d = dc.getDocument().get();
+ if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+ }
+}
+ }
+}
--- End diff --
Do we gain anything from a slight restructuring of your if/else chain that
adds a sanity check?
```
if(dc.getException().isPresent()) {
..
} else if(dc.getDocument.isPresent()) {
..
} else {
throw new IllegalStateException("Expected either document or exception;
got neither");
}
```
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217441386
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -282,4 +276,27 @@ public Document getLatest(final String guid, String
sensorType) throws IOExcepti
public List getIndices() {
return indices;
}
+
+ private Document getDocument(List documentContainers)
throws IOException {
+Document ret = null;
+List error = new ArrayList<>();
+for(DocumentContainer dc : documentContainers) {
+ if(dc.getException().isPresent()) {
+Throwable e = dc.getException().get();
+error.add(e.getMessage() + "\n" + ExceptionUtils.getStackTrace(e));
+ }
+ else {
+if(dc.getDocument().isPresent()) {
+ Document d = dc.getDocument().get();
+ if(ret == null || ret.getTimestamp() < d.getTimestamp()) {
+ret = d;
+ }
+}
+ }
+}
--- End diff --
Do we gain anything from a slight restructuring of your if/else chain that
adds a sanity check?
```
if(dc.getException().isPresent()) {
..
} else if(dc.getDocument.isPresent()) {
..
} else {
throw new IllegalStateException("Expected either document or exception;
got neither");
}
```
---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613687#comment-16613687
]
ASF GitHub Bot commented on METRON-1771:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217439809
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -101,48 +103,59 @@ public void batchUpdate(Map> updates) throws IOExcept
}
@Override
- public void addCommentToAlert(CommentAddRemoveRequest request) throws
IOException {
+ public Document addCommentToAlert(CommentAddRemoveRequest request)
throws IOException {
Document latest = getLatest(request.getGuid(),
request.getSensorType());
-addCommentToAlert(request, latest);
+return addCommentToAlert(request, latest);
}
-
+ /**
+ * Adds comments to an alert. Updates are written to each Dao in
parallel with the assumption that all updates
+ * are identical. The first update to be applied is returned as the
current version of the alert with comments added.
+ * @param request Request to add comments
+ * @param latest The latest version of the alert the comments will be
added to.
+ * @return The complete alert document with comments added.
+ * @throws IOException
+ */
@Override
- public void addCommentToAlert(CommentAddRemoveRequest request, Document
latest) throws IOException {
-List exceptions =
-indices.parallelStream().map(dao -> {
- try {
-dao.addCommentToAlert(request, latest);
-return null;
- } catch (Throwable e) {
-return dao.getClass() + ": " + e.getMessage() + "\n" +
ExceptionUtils.getStackTrace(e);
- }
-}).filter(Objects::nonNull).collect(Collectors.toList());
-if (exceptions.size() > 0) {
- throw new IOException(Joiner.on("\n").join(exceptions));
-}
+ public Document addCommentToAlert(CommentAddRemoveRequest request,
Document latest) throws IOException {
+List output =
+indices.parallelStream().map(dao -> {
+ try {
+return new
DocumentContainer(dao.addCommentToAlert(request, latest));
+ } catch (Throwable e) {
+return new DocumentContainer(e);
+ }
+}).collect(Collectors.toList());
--- End diff --
Nice! I like your approach.
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217439809
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
---
@@ -101,48 +103,59 @@ public void batchUpdate(Map> updates) throws IOExcept
}
@Override
- public void addCommentToAlert(CommentAddRemoveRequest request) throws
IOException {
+ public Document addCommentToAlert(CommentAddRemoveRequest request)
throws IOException {
Document latest = getLatest(request.getGuid(),
request.getSensorType());
-addCommentToAlert(request, latest);
+return addCommentToAlert(request, latest);
}
-
+ /**
+ * Adds comments to an alert. Updates are written to each Dao in
parallel with the assumption that all updates
+ * are identical. The first update to be applied is returned as the
current version of the alert with comments added.
+ * @param request Request to add comments
+ * @param latest The latest version of the alert the comments will be
added to.
+ * @return The complete alert document with comments added.
+ * @throws IOException
+ */
@Override
- public void addCommentToAlert(CommentAddRemoveRequest request, Document
latest) throws IOException {
-List exceptions =
-indices.parallelStream().map(dao -> {
- try {
-dao.addCommentToAlert(request, latest);
-return null;
- } catch (Throwable e) {
-return dao.getClass() + ": " + e.getMessage() + "\n" +
ExceptionUtils.getStackTrace(e);
- }
-}).filter(Objects::nonNull).collect(Collectors.toList());
-if (exceptions.size() > 0) {
- throw new IOException(Joiner.on("\n").join(exceptions));
-}
+ public Document addCommentToAlert(CommentAddRemoveRequest request,
Document latest) throws IOException {
+List output =
+indices.parallelStream().map(dao -> {
+ try {
+return new
DocumentContainer(dao.addCommentToAlert(request, latest));
+ } catch (Throwable e) {
+return new DocumentContainer(e);
+ }
+}).collect(Collectors.toList());
--- End diff --
Nice! I like your approach.
---
[jira] [Commented] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[
https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613685#comment-16613685
]
ASF GitHub Bot commented on METRON-1771:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217438881
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --
What is this test for? There seems to be no actual tests. Maybe you
forgot to push a commit?
> Update REST endpoints to support eventually consistent UI updates
> -
>
> Key: METRON-1771
> URL: https://issues.apache.org/jira/browse/METRON-1771
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Priority: Major
>
> Currently the REST endpoints that perform document updates either return
> true/false or nothing. This puts the responsibility of retrieving the
> updated state of the object on the client in a separate call or
> optimistically applying the changes and reverting when an update fails. This
> can be problematic if a client attempts to get the current state immediately
> after an update and the change isn't visible yet in the back end.
> Ideally they should return the updated state of the object, eliminating the
> need to look up the updated state in a separate call.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1190: METRON-1771: Update REST endpoints to support eve...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1190#discussion_r217438881
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDaoTest.java
---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.UpdateDaoTest;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.elasticsearch.client.transport.TransportClient;
+import org.junit.Before;
+
+import static org.mockito.Mockito.mock;
+
+public class ElasticsearchUpdateDaoTest extends UpdateDaoTest {
--- End diff --
What is this test for? There seems to be no actual tests. Maybe you
forgot to push a commit?
---
[jira] [Commented] (METRON-1778) Out-of-order timestamps may delay flush in Storm Profiler
[ https://issues.apache.org/jira/browse/METRON-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613672#comment-16613672 ] ASF GitHub Bot commented on METRON-1778: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1197 METRON-1778 Out-of-order timestamps may delay flush in Storm Profiler In the Storm Profiler, when timestamps are received out-of-order there can be cases where a flush signal can be either delayed or occur prematurely. The smaller the profile period is, the more likely this is to impact the results. This is more likely to impact things like integration tests that run with small profile period values. I would not expect this to greatly impact the results of the Profiler under normal usage. ## Changes The previous implementation of `FixedFrequencyFlushSignal` set the flush time based only on the first timestamp it sees. With out-of-order timestamps the flush time can effectively change as additional data arrives. * See `FixedFrequencyFlushSignalTest.testOutOfOrderTimestamps` for an example of how this logic can cause a problem * The fix in this PR tracks the min and max timestamps that have been seen. If max > min + flushFrequency, then it is time to flush. Otherwise, it is not time to flush. * I added a fair number of unit tests and described in the comments what should happen. ## Testing There is not an easy way to manually test this as it requires a very specific sequence of timestamps to trigger. I feel the updates that were made to the unit tests, along with regression testing the environment as described in the Profiler README, is sufficient. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1778 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1197.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1197 commit afa7e7cde4cf2335b5491c7e39a46bb03f4eb643 Author: Nick Allen Date: 2018-09-13T14:20:06Z METRON-1778 Out-of-order timestamps may delay flush in Spark Profiler commit 0e708b07a9e2f061d92717463618908532d476f7 Author: Nick Allen Date: 2018-09-13T15:39:54Z Added back check for out-of-order timestamps > Out-of-order timestamps may delay flush in Storm Profiler > - > > Key: METRON-1778 > URL: https://issues.apache.org/jira/browse/METRON-1778 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > When timestamps are received out-of-order there can be cases where a flush > signal should have been signalled, but is not. The flush signal can be > either delayed slightly or occur prematurely. > The smaller the profile period is, the more likely this is to impact the > results. I would not expect this to greatly impact the results of the > Profiler under normal usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1197: METRON-1778 Out-of-order timestamps may delay flu...
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1197 METRON-1778 Out-of-order timestamps may delay flush in Storm Profiler In the Storm Profiler, when timestamps are received out-of-order there can be cases where a flush signal can be either delayed or occur prematurely. The smaller the profile period is, the more likely this is to impact the results. This is more likely to impact things like integration tests that run with small profile period values. I would not expect this to greatly impact the results of the Profiler under normal usage. ## Changes The previous implementation of `FixedFrequencyFlushSignal` set the flush time based only on the first timestamp it sees. With out-of-order timestamps the flush time can effectively change as additional data arrives. * See `FixedFrequencyFlushSignalTest.testOutOfOrderTimestamps` for an example of how this logic can cause a problem * The fix in this PR tracks the min and max timestamps that have been seen. If max > min + flushFrequency, then it is time to flush. Otherwise, it is not time to flush. * I added a fair number of unit tests and described in the comments what should happen. ## Testing There is not an easy way to manually test this as it requires a very specific sequence of timestamps to trigger. I feel the updates that were made to the unit tests, along with regression testing the environment as described in the Profiler README, is sufficient. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1778 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1197.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1197 commit afa7e7cde4cf2335b5491c7e39a46bb03f4eb643 Author: Nick Allen Date: 2018-09-13T14:20:06Z METRON-1778 Out-of-order timestamps may delay flush in Spark Profiler commit 0e708b07a9e2f061d92717463618908532d476f7 Author: Nick Allen Date: 2018-09-13T15:39:54Z Added back check for out-of-order timestamps ---
[jira] [Assigned] (METRON-1756) REST tests should use Embedded LDAP in metron-security
[ https://issues.apache.org/jira/browse/METRON-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Elliston Ball reassigned METRON-1756: --- Assignee: Ryan Merriman > REST tests should use Embedded LDAP in metron-security > -- > > Key: METRON-1756 > URL: https://issues.apache.org/jira/browse/METRON-1756 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Tests in metron-rest are still using JDBC-based authentication. These tests > (and all security-related tests) should use the testing infrastructure > introduced in METRON-1665. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1779) Front-end authentication simulation for development environments
Simon Elliston Ball created METRON-1779: --- Summary: Front-end authentication simulation for development environments Key: METRON-1779 URL: https://issues.apache.org/jira/browse/METRON-1779 Project: Metron Issue Type: Sub-task Reporter: Simon Elliston Ball Assignee: Simon Elliston Ball When working on the UI components, it is common to use ng serve on a local machine pointing the proxy conf to a full-dev hosted instance of metron-rest. In order to make this work we need a development profile with test users for the full-dev instance, and an injector to simulate authentication in the absence of an SSO setup. This could be simulated with Basic auth injection against the test LDAP credentials as part of a development environment configuration on the front-end. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1778) Out-of-order timestamps may delay flush in Storm Profiler
[ https://issues.apache.org/jira/browse/METRON-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1778: --- Summary: Out-of-order timestamps may delay flush in Storm Profiler (was: Out-of-order timestamps may delay flush in Spark Profiler) > Out-of-order timestamps may delay flush in Storm Profiler > - > > Key: METRON-1778 > URL: https://issues.apache.org/jira/browse/METRON-1778 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > When timestamps are received out-of-order there can be cases where a flush > signal should have been signalled, but is not. The flush signal can be > either delayed slightly or occur prematurely. > The smaller the profile period is, the more likely this is to impact the > results. I would not expect this to greatly impact the results of the > Profiler under normal usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1778) Out-of-order timestamps may delay flush in Spark Profiler
Nick Allen created METRON-1778: -- Summary: Out-of-order timestamps may delay flush in Spark Profiler Key: METRON-1778 URL: https://issues.apache.org/jira/browse/METRON-1778 Project: Metron Issue Type: Bug Reporter: Nick Allen Assignee: Nick Allen When timestamps are received out-of-order there can be cases where a flush signal should have been signalled, but is not. The flush signal can be either delayed slightly or occur prematurely. The smaller the profile period is, the more likely this is to impact the results. I would not expect this to greatly impact the results of the Profiler under normal usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1775) Transient exception could prevent expired profiles from being flushed
[ https://issues.apache.org/jira/browse/METRON-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613444#comment-16613444 ] ASF GitHub Bot commented on METRON-1775: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1194 > Transient exception could prevent expired profiles from being flushed > - > > Key: METRON-1775 > URL: https://issues.apache.org/jira/browse/METRON-1775 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > > In the Storm Profiler, the ProfileBuilderBolt uses a separate thread to flush > expired profiles. Expired profiles are those that have not received a > message for an extended period of time. That period of time is called the > profile time-to-live (profiler.ttl). This is done to avoid potential memory > leaks. > This separate thread is scheduled using > java.util.concurrent.ScheduledExecutorService#scheduleAtFixedRate. The > documentation for scheduleAtFixedRate says that if an exception is thrown, > subsequent executions will be suppressed. We do not want this to happen. If > an exception occurs, we need to log it and continue, ensuring that expired > profiles will continue to flush. > I have no direct proof that this has caused a problem, but it certainly is a > potential issue that should be addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
