[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206130#comment-16206130 ] ASF GitHub Bot commented on METRON-1226: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/793 > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206015#comment-16206015 ] ASF GitHub Bot commented on METRON-1226: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/793 +1 by inspection > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200307#comment-16200307 ] ASF GitHub Bot commented on METRON-1226: Github user cestella commented on the issue: https://github.com/apache/metron/pull/793 I'm +1 by inspection but will hold it until @mmiklavc gives his. > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200211#comment-16200211 ] ASF GitHub Bot commented on METRON-1226: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/793 @cestella @mmiklavc How is this looking? > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197288#comment-16197288 ] ASF GitHub Bot commented on METRON-1226: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/793 This is looking pretty good by inspection. Good minor refactorings and javadoc additions as well. I'd like to run it up in full dev when I get a moment. > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[
https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197042#comment-16197042
]
ASF GitHub Bot commented on METRON-1226:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/793#discussion_r143481097
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
---
@@ -45,15 +49,30 @@ public static SimpleDateFormat
getIndexFormat(Map globalConfig)
return DATE_FORMAT_CACHE.get().computeIfAbsent(format,
SimpleDateFormat::new);
}
+ /**
+ * Builds the name of an Elasticsearch index.
+ * @param sensorType The sensor type; bro, yaf, snort, ...
+ * @param indexPostfix The index postfix; most often a formatted date.
+ * @param configurations User-defined configuration for the writers.
+ */
public static String getIndexName(String sensorType, String
indexPostfix, WriterConfiguration configurations) {
String indexName = sensorType;
if (configurations != null) {
indexName = configurations.getIndex(sensorType);
}
-indexName = indexName + "_index_" + indexPostfix;
+indexName = indexName + getIndexDelimiter() + "_" + indexPostfix;
return indexName;
}
+ /**
+ * Returns the delimiter that is appended to the user-defined index name
to separate
+ * the index's date postfix. For example, if the user-defined index
name is 'bro'
+ * and the delimiter is '_index_', then one likely index name is
'bro_index_2017.10.03.19'.
+ */
+ public static String getIndexDelimiter() {
--- End diff --
Honestly, I have no idea. I went down a dangerous rabbit hole making tons
of changes, then backed myself out to just what is in this PR. It might have
been for changes that I backed out.
Making it a `static final String` certainly would be more expected. I'll
fix it.
> Searching Can Errantly Query the Wrong Indices
> --
>
> Key: METRON-1226
> URL: https://issues.apache.org/jira/browse/METRON-1226
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The ElasticsearchDAO can errantly query the wrong indices under certain
> conditions. This can occur when there are at least two different sensors
> that have been configured with a similar name prefix.
> For example, if you have have configured both a "snort" and "snort_v2"
> sensor, the query generated by the ElasticsearchDAO will hit both of those
> indices, when in fact only one or the other is desired.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[
https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197005#comment-16197005
]
ASF GitHub Bot commented on METRON-1226:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/793#discussion_r143473906
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
---
@@ -45,15 +49,30 @@ public static SimpleDateFormat
getIndexFormat(Map globalConfig)
return DATE_FORMAT_CACHE.get().computeIfAbsent(format,
SimpleDateFormat::new);
}
+ /**
+ * Builds the name of an Elasticsearch index.
+ * @param sensorType The sensor type; bro, yaf, snort, ...
+ * @param indexPostfix The index postfix; most often a formatted date.
+ * @param configurations User-defined configuration for the writers.
+ */
public static String getIndexName(String sensorType, String
indexPostfix, WriterConfiguration configurations) {
String indexName = sensorType;
if (configurations != null) {
indexName = configurations.getIndex(sensorType);
}
-indexName = indexName + "_index_" + indexPostfix;
+indexName = indexName + getIndexDelimiter() + "_" + indexPostfix;
return indexName;
}
+ /**
+ * Returns the delimiter that is appended to the user-defined index name
to separate
+ * the index's date postfix. For example, if the user-defined index
name is 'bro'
+ * and the delimiter is '_index_', then one likely index name is
'bro_index_2017.10.03.19'.
+ */
+ public static String getIndexDelimiter() {
--- End diff --
I'm more curious than opposed to this, but is there a strong reason why
this isn't just a `public static final String`?
> Searching Can Errantly Query the Wrong Indices
> --
>
> Key: METRON-1226
> URL: https://issues.apache.org/jira/browse/METRON-1226
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The ElasticsearchDAO can errantly query the wrong indices under certain
> conditions. This can occur when there are at least two different sensors
> that have been configured with a similar name prefix.
> For example, if you have have configured both a "snort" and "snort_v2"
> sensor, the query generated by the ElasticsearchDAO will hit both of those
> indices, when in fact only one or the other is desired.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197004#comment-16197004 ] ASF GitHub Bot commented on METRON-1226: Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/793#discussion_r143473489 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java --- @@ -154,9 +134,9 @@ protected SearchResponse search(SearchRequest searchRequest, QueryBuilder queryB .size(searchRequest.getSize()) .from(searchRequest.getFrom()) .query(queryBuilder) - .trackScores(true); -searchRequest.getSort().forEach(sortField -> searchSourceBuilder.sort(sortField.getField(), getElasticsearchSortOrder(sortField.getSortOrder(;Optional> fields = searchRequest.getFields(); +searchRequest.getSort().forEach(sortField -> searchSourceBuilder.sort(sortField.getField(), getElasticsearchSortOrder(sortField.getSortOrder(; --- End diff -- haha, thanks for cleaning this up. :) > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196991#comment-16196991 ] ASF GitHub Bot commented on METRON-1226: Github user cestella commented on the issue: https://github.com/apache/metron/pull/793 cool, just making sure :) > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196987#comment-16196987 ] ASF GitHub Bot commented on METRON-1226: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/793 @cestella Not that I am aware of. I launched Full Dev and was able to submit queries and use the Alerts UI as before. I will add that to the "Test" section for completeness. > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196977#comment-16196977 ] ASF GitHub Bot commented on METRON-1226: Github user cestella commented on the issue: https://github.com/apache/metron/pull/793 Does this cause any issues in the alerts UI? > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1226) Searching Can Errantly Query the Wrong Indices
[ https://issues.apache.org/jira/browse/METRON-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16196924#comment-16196924 ] ASF GitHub Bot commented on METRON-1226: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/793 METRON-1226 Searching Can Errantly Query the Wrong Indices The ElasticsearchDAO can errantly query the wrong indices under certain conditions. This can occur when there are at least two different sensors that have been configured with a similar name prefix. For example, consider the case when the user has created two different sensors; "snort" and "snort_v2". When the user attempts to query the "snort" index in Elasticsearch, the query generated by the ElasticsearchDAO will hit both of these indices which is unexpected. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1226 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/793.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #793 commit 4ab87f55d30b584c7943edfb8bfc5cbca89eabde Author: Nick Allen Date: 2017-10-03T22:10:00Z METRON-1226 Searching Can Errantly Query the Wrong Indices > Searching Can Errantly Query the Wrong Indices > -- > > Key: METRON-1226 > URL: https://issues.apache.org/jira/browse/METRON-1226 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The ElasticsearchDAO can errantly query the wrong indices under certain > conditions. This can occur when there are at least two different sensors > that have been configured with a similar name prefix. > For example, if you have have configured both a "snort" and "snort_v2" > sensor, the query generated by the ElasticsearchDAO will hit both of those > indices, when in fact only one or the other is desired. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
