[GitHub] ottobackwards commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14
ottobackwards commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14 URL: https://github.com/apache/metron/pull/1296#issuecomment-446398244 I'm sorry, I've made a mess of this. I should not have mentioned it. @merrimanr is right, this should be done in a separate pr This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merrimanr commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14
merrimanr commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14 URL: https://github.com/apache/metron/pull/1296#issuecomment-446391551 I thought we put that in contrib so that it didn't need to be maintained? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240815917
##
File path:
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.html
##
@@ -61,7 +61,7 @@
{{
hasScore(alert.source) ? getScore(alert.source) : '-' }}
- {{ alert.source['name'] ? alert.source['name'] :
alert.id | centerEllipses:20:cell }}
+ {{
alert.source['name'] ? alert.source['name'] : alert.source['guid'] |
centerEllipses:20:cell }}
Review comment:
Displays the GUID, if a user has not renamed a meta alert.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240815380
##
File path:
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.ts
##
@@ -218,7 +215,7 @@ export class TableViewComponent implements OnInit,
OnChanges, OnDestroy {
}
addFilter(field: string, value: string) {
-field = (field === 'id') ? 'guid' : field;
+field = (field === 'id') ? '_id' : field;
Review comment:
Allows a user to filter on the document ID, if they choose to add it to
their view/table.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240815495
##
File path:
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.ts
##
@@ -141,14 +141,14 @@ export class TableViewComponent implements OnInit,
OnChanges, OnDestroy {
onSort(sortEvent: SortEvent) {
let sortOrder = (sortEvent.sortOrder === Sort.ASC ? 'asc' : 'desc');
-let sortBy = sortEvent.sortBy === 'id' ? 'guid' : sortEvent.sortBy;
+let sortBy = sortEvent.sortBy === 'id' ? '_uid' : sortEvent.sortBy;
Review comment:
Allows a user to sort by the document ID, if they choose to add it to their
view/table.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240815380
##
File path:
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.ts
##
@@ -218,7 +215,7 @@ export class TableViewComponent implements OnInit,
OnChanges, OnDestroy {
}
addFilter(field: string, value: string) {
-field = (field === 'id') ? 'guid' : field;
+field = (field === 'id') ? '_id' : field;
Review comment:
Allows a user to filter on the document ID, if they choose to add it to the
table.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240814413
##
File path:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchRetrieveLatestDao.java
##
@@ -102,54 +92,47 @@ public Document getLatest(String guid, String sensorType)
throws IOException {
if (guids == null || guids.isEmpty()) {
return Collections.emptyList();
}
-QueryBuilder query = null;
-IdsQueryBuilder idsQuery;
-if (sensorTypes != null) {
- String[] types = sensorTypes.stream().map(sensorType -> sensorType +
"_doc")
- .toArray(String[]::new);
- idsQuery = QueryBuilders.idsQuery(types);
-} else {
- idsQuery = QueryBuilders.idsQuery();
-}
-for (String guid : guids) {
- query = idsQuery.addIds(guid);
+// should match any of the guids
+// the 'guid' field must be of type 'keyword' or this term query will not
match
+BoolQueryBuilder guidQuery = boolQuery().must(termsQuery(Constants.GUID,
guids));
Review comment:
We can no longer use an id query, since the doc ID is no longer the same as
the GUID.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240813591
##
File path:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
##
@@ -226,18 +257,30 @@ public void init(AccessConfig config) {
@Override
public Document getLatest(final String guid, String sensorType) throws
IOException {
-List output =
-indices.parallelStream().map(dao -> {
- try {
-return new DocumentContainer(dao.getLatest(guid, sensorType));
- } catch (Throwable e) {
-return new DocumentContainer(e);
- }
-}).collect(Collectors.toList());
-
+List output = indices
+.parallelStream()
+.map(dao -> getLatest(dao, guid, sensorType))
+.collect(Collectors.toList());
return getLatestDocument(output);
}
+ private DocumentContainer getLatest(IndexDao indexDao, String guid, String
sensorType) {
+DocumentContainer container;
+try {
+ Document document = indexDao.getLatest(guid, sensorType);
+ container = new DocumentContainer(document);
+ LOG.debug("Found latest document; indexDao={}, guid={}, sensorType={},
document={}",
+ ClassUtils.getShortClassName(indexDao.getClass()), guid,
sensorType, document);
+
+} catch (Throwable e) {
+ container = new DocumentContainer(e);
+ LOG.error("Unable to find latest document; indexDao={}, error={}",
Review comment:
We had no logging to indicate when a particular IndexDao fails. I ran into
this and it took some effort to trace since we lacked logging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240813659
##
File path:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
##
@@ -117,18 +124,30 @@ public Document
addCommentToAlert(CommentAddRemoveRequest request) throws IOExce
*/
@Override
public Document addCommentToAlert(CommentAddRemoveRequest request, Document
latest) throws IOException {
-List output =
-indices.parallelStream().map(dao -> {
- try {
-return new DocumentContainer(dao.addCommentToAlert(request,
latest));
- } catch (Throwable e) {
-return new DocumentContainer(e);
- }
-}).collect(Collectors.toList());
-
+List output = indices
+.parallelStream()
+.map(dao -> addCommentToAlert(dao, request, latest))
+.collect(Collectors.toList());
return getLatestDocument(output);
}
+ private DocumentContainer addCommentToAlert(IndexDao indexDao,
CommentAddRemoveRequest request, Document latest) {
+DocumentContainer container;
+try {
+ Document document = indexDao.addCommentToAlert(request, latest);
+ container = new DocumentContainer(document);
+ LOG.debug("Added comment to alert; indexDao={}, guid={}, sensorType={},
document={}",
+ ClassUtils.getShortClassName(indexDao.getClass()),
document.getGuid(), document.getSensorType(), document);
+
+} catch (Throwable e) {
+ container = new DocumentContainer(e);
+ LOG.error("Unable to add comment to alert; indexDao={}, error={}",
Review comment:
We had no logging to indicate when a particular IndexDao fails. I ran into
this and it took some effort to trace since we lacked logging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240813632
##
File path:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/MultiIndexDao.java
##
@@ -145,18 +164,30 @@ public Document
removeCommentFromAlert(CommentAddRemoveRequest request) throws I
*/
@Override
public Document removeCommentFromAlert(CommentAddRemoveRequest request,
Document latest) throws IOException {
-List output =
-indices.parallelStream().map(dao -> {
- try {
-return new
DocumentContainer(dao.removeCommentFromAlert(request, latest));
- } catch (Throwable e) {
-return new DocumentContainer(e);
- }
-}).collect(Collectors.toList());
-
+List output = indices
+.parallelStream()
+.map(dao -> removeCommentFromAlert(dao, request, latest))
+.collect(Collectors.toList());
return getLatestDocument(output);
}
+ private DocumentContainer removeCommentFromAlert(IndexDao indexDao,
CommentAddRemoveRequest request, Document latest) {
+DocumentContainer container;
+try {
+ Document document = indexDao.removeCommentFromAlert(request, latest);
+ container = new DocumentContainer(document);
+ LOG.debug("Removed comment from alert; indexDao={}, guid={},
sensorType={}, document={}",
+ ClassUtils.getShortClassName(indexDao.getClass()),
document.getGuid(), document.getSensorType(), document);
+
+} catch (Throwable e) {
+ container = new DocumentContainer(e);
+ LOG.error("Unable to remove comment from alert; indexDao={}, error={}",
Review comment:
We had no logging to indicate when a particular IndexDao fails. I ran into
this and it took some effort to trace since we lacked logging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] mmiklavc edited a comment on issue #1275: METRON-1878: Add Metron as a Knox service
mmiklavc edited a comment on issue #1275: METRON-1878: Add Metron as a Knox service URL: https://github.com/apache/metron/pull/1275#issuecomment-446383790 @merrimanr Just summing up a few things I think we should have come out of this PR per our offline convo earlier: 1. Update README's with some explanation of the multiple UI and REST API communication architecture - e.g. I had expected there was a way to have the REST application and Angular application assets be served up by (or have the REST calls handled through) the same web container context, but that is not how this is setup. The pre-Knox existing architecture uses ExpressJS within the Angular app hosting container to do a very straightforward proxying of requests to the REST API as opposed to having a heavier weight facade (e.g. a Spring J2EE application) for an abstraction managing those requests to REST. This is fine, but now that we've added Knox to the mix, we should be very clear about what requests and responses go to which hosts/servers and why. 2. Developer documentation - simple diagram and/or write up that expresses what the request/response lifecycle looks like with and without Knox. This relates to number 1, but is more about showing the before/after of enabling Knox. 3. Explanation of the choice to NOT use Knox's URL rewriting capabilities. Per our convo earlier, the choice is between setting a base root that's used by Knox for routing requests (similar to the function of a controller in MVC architecture) to the appropriate host, or leaving the REST application alone and leveraging Knox's regular expression pattern matching to handle rewriting URL's that show up in the client browser. 4. EDIT - Fix logout bug 5. EDIT - Fix Travis Cypress test failure 6. I'm going to make a PR against your PR to add the Management UI to this. Here's the basic request/response lifecycle as I understand it: **Static asset call for Angular application** This is for javascript, html, and images. I realize "static" may be a misnomer for Angular javascript, but it's static insofar as the web container is concerned: ``` Client request -> knox server -> angular web server Server response <- angular web server <- knox server <- client ``` **Dynamic/REST content call.** Knox is now replacing ExpressJS entirely as the proxy mechanism. Note, ExpressJS didn't perform any function for static assets, only for calls to 3rd parties like the REST app. This is necessary because CORS restricts a browser from making the calls directly to REST (or any other non-origin host that's different from the original host that served up the client-side Angular app). This could theoretically be handled by a proper server side web application, but my understanding is that this would be superfluous given the API contract we've enacted with the REST API, i.e. it handles requests/responses in JSON that an AngularJS application can process easily. ``` Client request -> knox server -> rest api server Server response <- rest api server <- knox server <- client ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mmiklavc edited a comment on issue #1275: METRON-1878: Add Metron as a Knox service
mmiklavc edited a comment on issue #1275: METRON-1878: Add Metron as a Knox service URL: https://github.com/apache/metron/pull/1275#issuecomment-446383790 @merrimanr Just summing up a few things I think we should have come out of this PR per our offline convo earlier: 1. Update README's with some explanation of the multiple UI and REST API communication architecture - e.g. I had expected there was a way to have the REST application and Angular application assets be served up by (or have the REST calls handled through) the same web container context, but that is not how this is setup. The pre-Knox existing architecture uses ExpressJS within the Angular app hosting container to do a very straightforward proxying of requests to the REST API as opposed to having a heavier weight facade (e.g. a Spring J2EE application) for an abstraction managing those requests to REST. This is fine, but now that we've added Knox to the mix, we should be very clear about what requests and responses go to which hosts/servers and why. 2. Developer documentation - simple diagram and/or write up that expresses what the request/response lifecycle looks like with and without Knox. This relates to #1, but is more about showing the before/after of enabling Knox. 3. Explanation of the choice to NOT use Knox's URL rewriting capabilities. Per our convo earlier, the choice is between setting a base root that's used by Knox for routing requests (similar to the function of a controller in MVC architecture) to the appropriate host, or leaving the REST application alone and leveraging Knox's regular expression pattern matching to handle rewriting URL's that show up in the client browser. 4. EDIT - Fix logout bug 5. EDIT - Fix Travis Cypress test failure 6. I'm going to make a PR against your PR to add the Management UI to this. Here's the basic request/response lifecycle as I understand it: **Static asset call for Angular application** This is for javascript, html, and images. I realize "static" may be a misnomer for Angular javascript, but it's static insofar as the web container is concerned: ``` Client request -> knox server -> angular web server Server response <- angular web server <- knox server <- client ``` **Dynamic/REST content call.** Knox is now replacing ExpressJS entirely as the proxy mechanism. Note, ExpressJS didn't perform any function for static assets, only for calls to 3rd parties like the REST app. This is necessary because CORS restricts a browser from making the calls directly to REST (or any other non-origin host that's different from the original host that served up the client-side Angular app). This could theoretically be handled by a proper server side web application, but my understanding is that this would be superfluous given the API contract we've enacted with the REST API, i.e. it handles requests/responses in JSON that an AngularJS application can process easily. ``` Client request -> knox server -> rest api server Server response <- rest api server <- knox server <- client ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] nickwallen commented on a change in pull request #1269: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
nickwallen commented on a change in pull request #1269: METRON-1879 Allow
Elasticsearch to Auto-Generate the Document ID
URL: https://github.com/apache/metron/pull/1269#discussion_r240813000
##
File path:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/update/UpdateDao.java
##
@@ -73,23 +74,28 @@ default Document patch(RetrieveLatestDao
retrieveLatestDao, PatchRequest request
}
default Document getPatchedDocument(RetrieveLatestDao retrieveLatestDao,
PatchRequest request,
- Optional timestamp
+ Optional optionalTimestamp
) throws OriginalNotFoundException, IOException {
-Map latest = request.getSource();
-if (latest == null) {
- Document latestDoc = retrieveLatestDao.getLatest(request.getGuid(),
request.getSensorType());
- if (latestDoc != null && latestDoc.getDocument() != null) {
-latest = latestDoc.getDocument();
+String guid = request.getGuid();
+String sensorType = request.getSensorType();
+String documentID = null;
+Long timestamp = optionalTimestamp.orElse(System.currentTimeMillis());
+
+Map originalSource = request.getSource();
+if (originalSource == null) {
+ // no document source provided, lookup the latest
+ Document toPatch = retrieveLatestDao.getLatest(guid, sensorType);
+ if(toPatch != null && toPatch.getDocument() != null) {
+originalSource = toPatch.getDocument();
+documentID = toPatch.getDocumentID().orElse(null);
Review comment:
We need to set the doc ID when patching an already indexed document.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] mmiklavc commented on issue #1275: METRON-1878: Add Metron as a Knox service
mmiklavc commented on issue #1275: METRON-1878: Add Metron as a Knox service URL: https://github.com/apache/metron/pull/1275#issuecomment-446383790 @merrimanr Just summing up a few things I think we should have come out of this PR per our offline convo earlier: 1. Update README's with some explanation of the multiple UI and REST API communication architecture - e.g. I had expected there was a way to have the REST application and Angular application assets be served up by (or have the REST calls handled through) the same web container context, but that is not how this is setup. The pre-Knox existing architecture uses ExpressJS within the Angular app hosting container to do a very straightforward proxying of requests to the REST API as opposed to having a heavier weight facade (e.g. a Spring J2EE application) for an abstraction managing those requests to REST. This is fine, but now that we've added Knox to the mix, we should be very clear about what requests and responses go to which hosts/servers and why. 2. Developer documentation - simple diagram and/or write up that expresses what the request/response lifecycle looks like with and without Knox. This relates to #1, but is more about showing the before/after of enabling Knox. 3. Explanation of the choice to NOT use Knox's URL rewriting capabilities. Per our convo earlier, the choice is between setting a base root that's used by Knox for routing requests (similar to the function of a controller in MVC architecture) to the appropriate host, or leaving the REST application alone and leveraging Knox's regular expression pattern matching to handle rewriting URL's that show up in the client browser. 4. I'm going to make a PR against your PR to add the Management UI to this. Here's the basic request/response lifecycle as I understand it: **Static asset call for Angular application** This is for javascript, html, and images. I realize "static" may be a misnomer for Angular javascript, but it's static insofar as the web container is concerned: ``` Client request -> knox server -> angular web server Server response <- angular web server <- knox server <- client ``` **Dynamic/REST content call.** Knox is now replacing ExpressJS entirely as the proxy mechanism. Note, ExpressJS didn't perform any function for static assets, only for calls to 3rd parties like the REST app. This is necessary because CORS restricts a browser from making the calls directly to REST (or any other non-origin host that's different from the original host that served up the client-side Angular app). This could theoretically be handled by a proper server side web application, but my understanding is that this would be superfluous given the API contract we've enacted with the REST API, i.e. it handles requests/responses in JSON that an AngularJS application can process easily. ``` Client request -> knox server -> rest api server Server response <- rest api server <- knox server <- client ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] JonZeolla commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14
JonZeolla commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14 URL: https://github.com/apache/metron/pull/1296#issuecomment-446380583 Why not? It makes sense to me that it should be kept in sync. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ottobackwards commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14
ottobackwards commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14 URL: https://github.com/apache/metron/pull/1296#issuecomment-446377677 actually, I don't think you should touch it. although there might be a follow on jira for that stuff This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] JonZeolla commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14
JonZeolla commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.14 URL: https://github.com/apache/metron/pull/1296#issuecomment-446373696 @ottobackwards Is [this](https://github.com/apache/metron/pull/1296/commits/b47d1808785a12a78d94739c69ec66fde508bb8e#diff-efd719d636b235ed83e2ded856682e93R17) what you would expect to upgrade docker? I'll have to spin it up later. Looks like that got missed in the initial move to 5.6. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] JonZeolla commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.13
JonZeolla commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.13 URL: https://github.com/apache/metron/pull/1296#issuecomment-446370224 I didn't expect that would need an update. In looking at it, they've released a 5.6.14 which I'm going to move things to. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ottobackwards commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.13
ottobackwards commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.13 URL: https://github.com/apache/metron/pull/1296#issuecomment-446362951 are you going to update the metron-contrib/docker ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-1849) Elasticsearch Index Write Functionality Should be Shared
[ https://issues.apache.org/jira/browse/METRON-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-1849: --- Issue Type: Improvement (was: Bug) > Elasticsearch Index Write Functionality Should be Shared > > > Key: METRON-1849 > URL: https://issues.apache.org/jira/browse/METRON-1849 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > The index write functionality is currently duplicated between the > ElasticsearchWriter and the ElasticsearchUpdateDao. This functionality > should be de-duplicated and shared between the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1925) Provide Verbose View of Profile Results in REPL
[
https://issues.apache.org/jira/browse/METRON-1925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated METRON-1925:
---
Issue Type: Improvement (was: Bug)
> Provide Verbose View of Profile Results in REPL
> ---
>
> Key: METRON-1925
> URL: https://issues.apache.org/jira/browse/METRON-1925
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When viewing profile measurements in the REPL using PROFILE_GET, you simply
> get back a list of values. It is not easy to determine from which time period
> the measurements were taken.
> For example, are the following values all sequential? Are there any gaps in
> the measurements taken over the past 30 minutes? When was the first
> measurement taken?
> {code:java}
> [Stellar]>>> PROFILE_GET("hello-world","global",PROFILE_FIXED(30, "MINUTES"))
> [2655, 1170, 1185, 1170, 1185, 1215, 1200, 1170]{code}
> The `PROFILE_GET` function was designed to return values that serve as input
> to other functions. It was not designed to return values in a human-readable
> form that can be easily understood.
> We need another way to query for profile measurements in the REPL that gives
> me a better understanding of the profile measurements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit closed pull request #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared
asfgit closed pull request #1254: METRON-1849 Elasticsearch Index Write
Functionality Should be Shared
URL: https://github.com/apache/metron/pull/1254
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
new file mode 100644
index 00..34f543ec88
--- /dev/null
+++
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.bulk;
+
+import org.apache.metron.indexing.dao.update.Document;
+
+/**
+ * Writes documents to an index in bulk.
+ *
+ * @param The type of document to write.
+ */
+public interface BulkDocumentWriter {
+
+/**
+ * Add a document to the batch.
+ * @param document The document to write.
+ * @param index The name of the index to write to.
+ */
+void addDocument(D document, String index);
+
+/**
+ * @return The number of documents waiting to be written.
+ */
+int size();
+
+/**
+ * Write all documents in the batch.
+ */
+BulkDocumentWriterResults write();
+}
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
new file mode 100644
index 00..90e5ce31f1
--- /dev/null
+++
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.bulk;
+
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * The result of writing documents in bulk using a {@link BulkDocumentWriter}.
+ * @param The type of documents to write.
+ */
+public class BulkDocumentWriterResults {
+
+ private List> successes;
+ private List> failures;
+
+ public BulkDocumentWriterResults() {
+this.successes = new ArrayList<>();
+this.failures = new ArrayList<>();
+ }
+
+ public void add(WriteSuccess success) {
+this.successes.add(success);
+ }
+
+ public void addSuccess(D success) {
+add(new WriteSuccess(success));
+ }
+
+ public void addSuccesses(List successes) {
+for(D success: successes) {
+ addSuccess(success);
+}
+ }
+
+ public List> getSuccesses() {
+return successes;
+ }
+
+ public void add(WriteFailure failure) {
+this.failures.add(failure);
+ }
+
+ public void addFailure(D document, Throwable cause, String message) {
+add(new WriteFailure(document, cause, message));
+ }
+
+ public List> getFailures() {
+return failures;
+ }
+}
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/ElasticsearchBulkDocumentWriter.java
[jira] [Created] (METRON-1934) Stellar should built without error prone messages
Otto Fowler created METRON-1934: --- Summary: Stellar should built without error prone messages Key: METRON-1934 URL: https://issues.apache.org/jira/browse/METRON-1934 Project: Metron Issue Type: Improvement Reporter: Otto Fowler We should fix as many error prone messages as we can to build clean. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ottobackwards commented on issue #1298: METRON-1934 fix errorprone messages, remove logical unreachable statements
ottobackwards commented on issue #1298: METRON-1934 fix errorprone messages, remove logical unreachable statements URL: https://github.com/apache/metron/pull/1298#issuecomment-446343063 @cestella This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ottobackwards opened a new pull request #1298: METRON-1934 fix errorprone messages, remove logical unreachable statements
ottobackwards opened a new pull request #1298: METRON-1934 fix errorprone
messages, remove logical unreachable statements
URL: https://github.com/apache/metron/pull/1298
Build and tests should work
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be
created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [-] Have you written or updated unit tests and or integration tests to
verify your changes?
- [-] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [-] Have you verified the basic functionality of the build by building and
running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [-] Have you ensured that format looks appropriate for the output in which
it is rendered by building and verifying the site-book? If not then run the
following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for
your personal repository such that your branches are built there before
submitting a pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] JonZeolla opened a new pull request #1296: METRON-1932: Update ES and Kibana to 5.6.13
JonZeolla opened a new pull request #1296: METRON-1932: Update ES and Kibana to
5.6.13
URL: https://github.com/apache/metron/pull/1296
## Contributor Comments
This upgrades ES and Kibana from 5.6.2 to 5.6.13, specifically with the goal
to fix [this](https://github.com/elastic/elasticsearch/pull/27069) which was
backported to 5.6.4. Thanks to the great work in #1242 this was a piece of
cake. I spun it up and poked around in ES, Kibana, the Alerts UI, and did some
digging through logs looking for errors or warnings. Found some things that
can be improved for the future, but don't appear to have been introduced by
this PR.
### Testing
Spin up full-dev, ensure ES/Kibana is healthy/working as expected, and is
version 5.6.13.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be
created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and
running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered by building and verifying the site-book? If not then run the
following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for
your personal repository such that your branches are built there before
submitting a pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] JonZeolla closed pull request #1296: METRON-1932: Update ES and Kibana to 5.6.13
JonZeolla closed pull request #1296: METRON-1932: Update ES and Kibana to 5.6.13 URL: https://github.com/apache/metron/pull/1296 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv index 0a6392246d..4f97736be7 100644 --- a/dependencies_with_url.csv +++ b/dependencies_with_url.csv @@ -434,16 +434,16 @@ io.netty:netty-resolver:jar:4.1.13.Final:compile,ASLv2,http://netty.io/ io.netty:netty-transport:jar:4.1.13.Final:compile,ASLv2,http://netty.io/ joda-time:joda-time:jar:2.9.5:compile,ASLv2,https://github.com/JodaOrg/joda-time net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile,The MIT License,http://jopt-simple.sourceforge.net -org.elasticsearch.client:elasticsearch-rest-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.client:transport:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch:elasticsearch:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.client:elasticsearch-rest-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.client:transport:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch:elasticsearch:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt org.elasticsearch:jna:jar:4.4.0-1:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:lang-mustache-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:parent-join-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:percolator-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:reindex-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:transport-netty3-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:transport-netty4-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:lang-mustache-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:parent-join-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:percolator-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:reindex-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:transport-netty3-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:transport-netty4-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt org.elasticsearch:securesm:jar:1.1:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt org.hdrhistogram:HdrHistogram:jar:2.1.9:compile,BSD,https://github.com/HdrHistogram/HdrHistogram/blob/master/LICENSE.txt com.trendmicro:tlsh:jar:3.7.1:compile,ASLv2,https://github.com/trendmicro/tlsh @@ -487,6 +487,9 @@ org.sonatype.sisu:sisu-inject-plexus:jar:2.2.2:compile com.zaxxer:HikariCP:jar:2.7.8:compile,ASLv2,https://github.com/brettwooldridge/HikariCP org.hibernate.validator:hibernate-validator:jar:6.0.9.Final:compile,ASLv2,https://github.com/hibernate/hibernate-validator com.github.palindromicity:simple-syslog-5424:jar:0.0.9:compile,ASLv2,https://github.com/palindromicity/simple-syslog-5424 -org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:aggs-matrix-stats-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:aggs-matrix-stats-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt org.fusesource.jansi:jansi:jar:1.16:compile,ASLv2,https://github.com/fusesource/jansi/blob/master/license.txt +com.vividsolutions:jts:jar:1.13:compile +joda-time:joda-time:jar:2.10:compile
[GitHub] JonZeolla commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.13
JonZeolla commented on issue #1296: METRON-1932: Update ES and Kibana to 5.6.13 URL: https://github.com/apache/metron/pull/1296#issuecomment-446316414 This should be ready for review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] JonZeolla opened a new pull request #1296: METRON-1932: Update ES and Kibana to 5.6.13
JonZeolla opened a new pull request #1296: METRON-1932: Update ES and Kibana to
5.6.13
URL: https://github.com/apache/metron/pull/1296
## Contributor Comments
This upgrades ES and Kibana from 5.6.2 to 5.6.13, specifically with the goal
to fix [this](https://github.com/elastic/elasticsearch/pull/27069) which was
backported to 5.6.4. Thanks to the great work in #1242 this was a piece of
cake. I spun it up and poked around in ES, Kibana, the Alerts UI, and did some
digging through logs looking for errors or warnings. Found some things that
can be improved for the future, but don't appear to have been introduced by
this PR.
### Testing
Spin up full-dev, ensure ES/Kibana is healthy/working as expected, and is
version 5.6.13.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be
created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and
running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered by building and verifying the site-book? If not then run the
following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for
your personal repository such that your branches are built there before
submitting a pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] JonZeolla closed pull request #1296: METRON-1932: Update ES and Kibana to 5.6.13
JonZeolla closed pull request #1296: METRON-1932: Update ES and Kibana to 5.6.13 URL: https://github.com/apache/metron/pull/1296 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv index 0a6392246d..4f97736be7 100644 --- a/dependencies_with_url.csv +++ b/dependencies_with_url.csv @@ -434,16 +434,16 @@ io.netty:netty-resolver:jar:4.1.13.Final:compile,ASLv2,http://netty.io/ io.netty:netty-transport:jar:4.1.13.Final:compile,ASLv2,http://netty.io/ joda-time:joda-time:jar:2.9.5:compile,ASLv2,https://github.com/JodaOrg/joda-time net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile,The MIT License,http://jopt-simple.sourceforge.net -org.elasticsearch.client:elasticsearch-rest-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.client:transport:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch:elasticsearch:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.client:elasticsearch-rest-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.client:transport:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch:elasticsearch:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt org.elasticsearch:jna:jar:4.4.0-1:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:lang-mustache-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:parent-join-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:percolator-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:reindex-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:transport-netty3-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:transport-netty4-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:lang-mustache-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:parent-join-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:percolator-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:reindex-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:transport-netty3-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:transport-netty4-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt org.elasticsearch:securesm:jar:1.1:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt org.hdrhistogram:HdrHistogram:jar:2.1.9:compile,BSD,https://github.com/HdrHistogram/HdrHistogram/blob/master/LICENSE.txt com.trendmicro:tlsh:jar:3.7.1:compile,ASLv2,https://github.com/trendmicro/tlsh @@ -487,6 +487,9 @@ org.sonatype.sisu:sisu-inject-plexus:jar:2.2.2:compile com.zaxxer:HikariCP:jar:2.7.8:compile,ASLv2,https://github.com/brettwooldridge/HikariCP org.hibernate.validator:hibernate-validator:jar:6.0.9.Final:compile,ASLv2,https://github.com/hibernate/hibernate-validator com.github.palindromicity:simple-syslog-5424:jar:0.0.9:compile,ASLv2,https://github.com/palindromicity/simple-syslog-5424 -org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt -org.elasticsearch.plugin:aggs-matrix-stats-client:jar:5.6.2:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt +org.elasticsearch.plugin:aggs-matrix-stats-client:jar:5.6.13:compile,ASLv2,https://github.com/elastic/elasticsearch/blob/master/LICENSE.txt org.fusesource.jansi:jansi:jar:1.16:compile,ASLv2,https://github.com/fusesource/jansi/blob/master/license.txt +com.vividsolutions:jts:jar:1.13:compile +joda-time:joda-time:jar:2.10:compile
[GitHub] nickwallen edited a comment on issue #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared
nickwallen edited a comment on issue #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared URL: https://github.com/apache/metron/pull/1254#issuecomment-446297383 Thanks for the feedback @justinleet and @mmiklavc . It turned out better for it. I will merge once the latest CI build completes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] nickwallen commented on issue #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared
nickwallen commented on issue #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared URL: https://github.com/apache/metron/pull/1254#issuecomment-446297383 Thanks for the feedback @justinleet and @mmiklavc . It turned out better for it. Will merge ones that latest CI build completes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mmiklavc commented on issue #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared
mmiklavc commented on issue #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared URL: https://github.com/apache/metron/pull/1254#issuecomment-446295318 +1 lgtm with the latest changes, thanks for the submission @nickwallen! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mmiklavc commented on a change in pull request #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared
mmiklavc commented on a change in pull request #1254: METRON-1849 Elasticsearch
Index Write Functionality Should be Shared
URL: https://github.com/apache/metron/pull/1254#discussion_r240720635
##
File path:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
##
@@ -196,7 +196,7 @@ public ElasticsearchDao
withRefreshPolicy(WriteRequest.RefreshPolicy refreshPoli
}
protected Optional getIndexName(String guid, String sensorType)
throws IOException {
-return updateDao.getIndexName(guid, sensorType);
+return updateDao.findIndexNameByGUID(guid, sensorType);
Review comment:
That's fine @nickwallen. I wasn't clear on the difference between the 2
functions and wanted to call it out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] sardell commented on issue #1294: METRON-1930: Update webpack-dev-server in Alerts UI
sardell commented on issue #1294: METRON-1930: Update webpack-dev-server in Alerts UI URL: https://github.com/apache/metron/pull/1294#issuecomment-446289981 @nickwallen I just spun-up Full Dev with no problems. Thanks again to you and @ruffle1986 for reviewing my work! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-1890) Metron Vagrant should disable audio
[ https://issues.apache.org/jira/browse/METRON-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1890: Fix Version/s: 0.7.0 > Metron Vagrant should disable audio > --- > > Key: METRON-1890 > URL: https://issues.apache.org/jira/browse/METRON-1890 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 0.7.0 > > > our vagrant files should disable audio, so that the vm doesn't hit the mic > and audio. This sets off micro snitch and other applications that alert for > microphone access. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nickwallen commented on a change in pull request #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared
nickwallen commented on a change in pull request #1254: METRON-1849
Elasticsearch Index Write Functionality Should be Shared
URL: https://github.com/apache/metron/pull/1254#discussion_r240687913
##
File path:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriterTest.java
##
@@ -18,170 +18,241 @@
package org.apache.metron.elasticsearch.writer;
-import static org.junit.Assert.assertEquals;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.writer.BulkWriterResponse;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.tuple.Tuple;
+import org.json.simple.JSONObject;
+import org.junit.Before;
+import org.junit.Test;
-import com.google.common.collect.ImmutableList;
+import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
+import java.util.List;
import java.util.Map;
-import org.apache.metron.common.writer.BulkWriterResponse;
-import org.apache.storm.tuple.Tuple;
-import org.elasticsearch.action.bulk.BulkItemResponse;
-import org.elasticsearch.action.bulk.BulkResponse;
-import org.junit.Test;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
public class ElasticsearchWriterTest {
-@Test
-public void testSingleSuccesses() throws Exception {
-Tuple tuple1 = mock(Tuple.class);
-BulkResponse response = mock(BulkResponse.class);
-when(response.hasFailures()).thenReturn(false);
+Map stormConf;
+TopologyContext topologyContext;
+WriterConfiguration writerConfiguration;
-BulkWriterResponse expected = new BulkWriterResponse();
-expected.addSuccess(tuple1);
+@Before
+public void setup() {
+topologyContext = mock(TopologyContext.class);
-ElasticsearchWriter esWriter = new ElasticsearchWriter();
-BulkWriterResponse actual =
esWriter.buildWriteReponse(ImmutableList.of(tuple1), response);
+writerConfiguration = mock(WriterConfiguration.class);
+when(writerConfiguration.getGlobalConfig()).thenReturn(globals());
-assertEquals("Response should have no errors and single success",
expected, actual);
+stormConf = new HashMap();
}
@Test
-public void testMultipleSuccesses() throws Exception {
-Tuple tuple1 = mock(Tuple.class);
-Tuple tuple2 = mock(Tuple.class);
-
-BulkResponse response = mock(BulkResponse.class);
-when(response.hasFailures()).thenReturn(false);
+public void shouldWriteSuccessfully() {
+// create a writer where all writes will be successful
+float probabilityOfSuccess = 1.0F;
+ElasticsearchWriter esWriter = new ElasticsearchWriter();
+esWriter.setDocumentWriter( new
BulkDocumentWriterStub<>(probabilityOfSuccess));
+esWriter.init(stormConf, topologyContext, writerConfiguration);
-BulkWriterResponse expected = new BulkWriterResponse();
-expected.addSuccess(tuple1);
-expected.addSuccess(tuple2);
+// create a tuple and a message associated with that tuple
+List tuples = createTuples(1);
+List messages = createMessages(1);
-ElasticsearchWriter esWriter = new ElasticsearchWriter();
-BulkWriterResponse actual =
esWriter.buildWriteReponse(ImmutableList.of(tuple1, tuple2), response);
+BulkWriterResponse response = esWriter.write("bro",
writerConfiguration, tuples, messages);
-assertEquals("Response should have no errors and two successes",
expected, actual);
+// response should only contain successes
+assertFalse(response.hasErrors());
+assertTrue(response.getSuccesses().contains(tuples.get(0)));
}
@Test
-public void testSingleFailure() throws Exception {
-Tuple tuple1 = mock(Tuple.class);
-
-BulkResponse response = mock(BulkResponse.class);
-when(response.hasFailures()).thenReturn(true);
-
-Exception e = new IllegalStateException();
-BulkItemResponse itemResponse = buildBulkItemFailure(e);
-
when(response.iterator()).thenReturn(ImmutableList.of(itemResponse).iterator());
+public void shouldWriteManySuccessfully() {
+// create a writer where all writes will be successful
+float probabilityOfSuccess = 1.0F;
Review comment:
FYI - With the latest commits, I was able to completely remove this stub and
just use a mock. I couldn't figure out how to do this with mocks when I had
the callbacks, but since that was
[GitHub] justinleet commented on issue #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared
justinleet commented on issue #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared URL: https://github.com/apache/metron/pull/1254#issuecomment-446267363 +1, pending Travis and @mmiklavc. Thanks a lot, it's great to see this cleaned up and refactored! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-1714) Create RPM Packaging for the Batch Profiler
[ https://issues.apache.org/jira/browse/METRON-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1714: Fix Version/s: 0.7.0 > Create RPM Packaging for the Batch Profiler > --- > > Key: METRON-1714 > URL: https://issues.apache.org/jira/browse/METRON-1714 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > Create RPM packaging for the Batch Profiler -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1705) Create ProfilePeriod Using Period ID
[ https://issues.apache.org/jira/browse/METRON-1705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1705: Fix Version/s: 0.7.0 > Create ProfilePeriod Using Period ID > > > Key: METRON-1705 > URL: https://issues.apache.org/jira/browse/METRON-1705 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > Need to be able to create a ProfilePeriod using the Period ID. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1787) Input Time Constraints for Batch Profiler
[ https://issues.apache.org/jira/browse/METRON-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1787: Fix Version/s: (was: 0.6.1) 0.7.0 > Input Time Constraints for Batch Profiler > - > > Key: METRON-1787 > URL: https://issues.apache.org/jira/browse/METRON-1787 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > A user should be able to constrain the input telemetry based on time. The > user might optionally provide a --begin and/or --end command line argument > that defines the time window. > > This was originally [discussed on the mailing > list|https://lists.apache.org/thread.html/39b5636c9ae264611aa8aecc426b649682efaf1bd0755848a90ee6b9@%3Cdev.metron.apache.org%3E]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nickwallen commented on a change in pull request #1254: METRON-1849 Elasticsearch Index Write Functionality Should be Shared
nickwallen commented on a change in pull request #1254: METRON-1849
Elasticsearch Index Write Functionality Should be Shared
URL: https://github.com/apache/metron/pull/1254#discussion_r240683821
##
File path:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/writer/BulkDocumentWriterStub.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.writer;
+
+import org.apache.metron.elasticsearch.bulk.BulkDocumentWriter;
+import org.apache.metron.elasticsearch.bulk.BulkDocumentWriterResults;
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+
+/**
+ * A {@link BulkDocumentWriter} stub that can pretend that all documents
+ * have been written successfully, that all documents have failed, or something
+ * in between those two extremes.
+ *
+ * @param The type of {@link Document} to write.
+ */
+public class BulkDocumentWriterStub implements
BulkDocumentWriter {
+
+private float probabilityOfSuccess;
+private Exception exception;
+private List documents;
+
+/**
+ * @param probabilityOfSuccess The probability that a write will succeed.
+ * If 1.0, all writes succeed. If 0.0, all
writes fail.
+ */
+public BulkDocumentWriterStub(float probabilityOfSuccess) {
+this.probabilityOfSuccess = probabilityOfSuccess;
+this.exception = new IllegalStateException("Exception created by a
stub for testing");
+this.documents = new ArrayList<>();
+}
+
+@Override
+public void addDocument(D document, String index) {
+documents.add(document);
+}
+
+@Override
+public BulkDocumentWriterResults write() {
+BulkDocumentWriterResults results = new
BulkDocumentWriterResults<>();
+Random random = new Random();
Review comment:
I was able to remove the stub completely and just use a mock. Much simpler.
Thanks for the suggestion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (METRON-1760) Kill PCAP job should prompt for confirmation
[ https://issues.apache.org/jira/browse/METRON-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1760: Fix Version/s: 0.7.0 > Kill PCAP job should prompt for confirmation > > > Key: METRON-1760 > URL: https://issues.apache.org/jira/browse/METRON-1760 > Project: Metron > Issue Type: Improvement >Reporter: Tamas Fodor >Assignee: Tamas Fodor >Priority: Major > Fix For: 0.7.0 > > > Since killing a PCAP job is a disruptive operation, it would be good to > prompt for user confirmation before carrying out the operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1874) Create a Parser Debugger
[ https://issues.apache.org/jira/browse/METRON-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1874: Fix Version/s: 0.7.0 > Create a Parser Debugger > > > Key: METRON-1874 > URL: https://issues.apache.org/jira/browse/METRON-1874 > Project: Metron > Issue Type: New Feature >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > Users need a way to test their parser configurations in a controlled > environment. This would allow a user to tweak and test the parser > configuration before deploying it into a live Metron cluster. The user could > also use the REPL to understand why a particular message failed to parse > correctly. > This could follow the pattern of the [Profiler > Debugger|https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles] > and [Threat Triage > Simulator|https://github.com/apache/metron/tree/master/metron-platform/metron-management#simulate-threat-triage-rules]. > This allows a user to apply canned messages against a configuration and > inspect the output, all within the REPL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1888) Default Topology Settings in MPack Cause Profiler to Stall
[ https://issues.apache.org/jira/browse/METRON-1888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1888: Fix Version/s: 0.7.0 > Default Topology Settings in MPack Cause Profiler to Stall > -- > > Key: METRON-1888 > URL: https://issues.apache.org/jira/browse/METRON-1888 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > The default max spout pending set by the Mpack for the Profiler topology is > 300. This value is too low and causes the topology to stall and stop > consuming messages. > > The Profiler topology is the only Metron topology that uses Storm's windowing > support. Storm will queue up messages for the window length (by default 30 > seconds) and process each of those windows. > > This causes Storm to need to "hang on" to tuples longer than the other > topologies and this requires a much greater max spout pending value. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1869) Unable to Sort an Escalated Meta Alert
[
https://issues.apache.org/jira/browse/METRON-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1869:
Fix Version/s: 0.7.0
> Unable to Sort an Escalated Meta Alert
> --
>
> Key: METRON-1869
> URL: https://issues.apache.org/jira/browse/METRON-1869
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: Can Sort on Metaalert Before Escalation.png, Cannot View
> Metaalert Sorted by Status After Escalation.png
>
>
> Steps to Replicate:
> 1. Create a meta-alert.
> 2. Escalate the meta-alert.
> 3. Submit another search that filters the results to only show meta-alerts.
> 4. Sort the results by “alert_status” field.
> 5. The search will not return any meta-alerts in the results.
> 6. The REST log will contain the exception shown above.
> {code:java}
> 18/11/12 22:25:50 ERROR dao.ElasticsearchRequestSubmitter: Shard search
> failure [1/1]; reason=ElasticsearchException: Elasticsearch exception
> [type=illegal_argument_exception, reason=Fielddata is disabled on text fields
> by default. Set fielddata=true on [alert_status] in order to load fielddata
> in memory by uninverting the inverted index. Note that this can however use
> significant memory. Alternatively use a keyword field instead.],
> index=metaalert_index, shard=0, status=INTERNAL_SERVER_ERROR,
> nodeId=uuG7-lZoSiaowJK_cUD4IQ
> ElasticsearchException[Elasticsearch exception
> [type=illegal_argument_exception, reason=Fielddata is disabled on text fields
> by default. Set fielddata=true on [alert_status] in order to load fielddata
> in memory by uninverting the inverted index. Note that this can however use
> significant memory. Alternatively use a keyword field instead.]]
> at
> org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:490)
> at
> org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:406)
> at
> org.elasticsearch.action.search.ShardSearchFailure.fromXContent(ShardSearchFailure.java:205)
> at
> org.elasticsearch.action.search.SearchResponse.fromXContent(SearchResponse.java:297)
> at
> org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:526)
> at
> org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$2(RestHighLevelClient.java:382)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:413)
> at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:382)
> at
> org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:323)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchRequestSubmitter.submitSearch(ElasticsearchRequestSubmitter.java:62)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchSearchDao.search(ElasticsearchSearchDao.java:128)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchDao.search(ElasticsearchDao.java:191)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertSearchDao.search(ElasticsearchMetaAlertSearchDao.java:81)
> at
> org.apache.metron.elasticsearch.dao.ElasticsearchMetaAlertDao.search(ElasticsearchMetaAlertDao.java:209)
> at
> org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:92)
> at
> org.apache.metron.rest.controller.SearchController.search(SearchController.java:54){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1845) Correct Test Data Load in Elasticsearch Integration Tests
[ https://issues.apache.org/jira/browse/METRON-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1845: Fix Version/s: 0.7.0 > Correct Test Data Load in Elasticsearch Integration Tests > - > > Key: METRON-1845 > URL: https://issues.apache.org/jira/browse/METRON-1845 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > The Elasticsearch integration tests use the legacy Transport client to load > test data into the search indexes before running the tests. Loading the test > data like this does not accurately reflect how the indices will appear in a > production environment. > This should be changed to use our existing ElasticsearchUpdateDao to write > the test data. This ensures that any changes made to the 'write' portion of > our Elasticsearch code will function correctly with the 'read' portion. This > ensures that telemetry written into Elasticsearch by 'Indexing' can be read > correctly by the Alerts UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1867) Remove `/api/v1/update/replace` endpoint
[ https://issues.apache.org/jira/browse/METRON-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1867: Fix Version/s: 0.7.0 > Remove `/api/v1/update/replace` endpoint > > > Key: METRON-1867 > URL: https://issues.apache.org/jira/browse/METRON-1867 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > This is dead code and no longer used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1784) Re-allow remote ssh and scp in Centos full dev
[ https://issues.apache.org/jira/browse/METRON-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1784: Fix Version/s: 0.7.0 > Re-allow remote ssh and scp in Centos full dev > -- > > Key: METRON-1784 > URL: https://issues.apache.org/jira/browse/METRON-1784 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.0 > > > Centos full dev changed such that remote ssh and scp are disabled by default > unless you change to the playbook directory and use "vagrant ssh" directly. > This modifies the sshd_config to set "PasswordAuthentication yes" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1933) Improve build-utils helper scripts
Jon Zeolla created METRON-1933: -- Summary: Improve build-utils helper scripts Key: METRON-1933 URL: https://issues.apache.org/jira/browse/METRON-1933 Project: Metron Issue Type: Bug Reporter: Jon Zeolla Assignee: Jon Zeolla Currently generate_license.py doesn't handle errors properly, and the README doesn't mention you should have run `mvn clean install -DskipTests` beforehand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1770) Add Docs for Running the Profiler with Spark on YARN
[ https://issues.apache.org/jira/browse/METRON-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1770: Fix Version/s: 0.7.0 > Add Docs for Running the Profiler with Spark on YARN > > > Key: METRON-1770 > URL: https://issues.apache.org/jira/browse/METRON-1770 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > Add some docs to get help a user get started running the Profiler using Spark > on YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1703) Make Core Profiler Components Serializable
[ https://issues.apache.org/jira/browse/METRON-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1703: Fix Version/s: 0.7.0 > Make Core Profiler Components Serializable > -- > > Key: METRON-1703 > URL: https://issues.apache.org/jira/browse/METRON-1703 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1931) Update dev utilities to support new repo location
[ https://issues.apache.org/jira/browse/METRON-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1931: Fix Version/s: 0.7.0 > Update dev utilities to support new repo location > - > > Key: METRON-1931 > URL: https://issues.apache.org/jira/browse/METRON-1931 > Project: Metron > Issue Type: Task >Reporter: Roy Lenferink >Priority: Major > Fix For: 0.7.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Because of the move from git-wip-us to gitbox, the location of the > repositories changed. This needs to be changed in the current metron scripts > as well. > E.g. > * dev-utilities/committer-utils/metron-committer-common > * dev-utilities/release-utils/prepare-release-candidate > > Related mailing list thread: > https://lists.apache.org/thread.html/98d20625e740adb2e2992b48c9118f2e1c33f7bd9a3fb16f517e8c6b@%3Cdev.metron.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1706) HbaseClient.mutate should return the number of mutations
[ https://issues.apache.org/jira/browse/METRON-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1706: Fix Version/s: 0.7.0 > HbaseClient.mutate should return the number of mutations > > > Key: METRON-1706 > URL: https://issues.apache.org/jira/browse/METRON-1706 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > HbaseClient.mutate should return the number of mutations -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1736) Enhance Batch Profiler Integration Test
[ https://issues.apache.org/jira/browse/METRON-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1736: Fix Version/s: 0.7.0 > Enhance Batch Profiler Integration Test > --- > > Key: METRON-1736 > URL: https://issues.apache.org/jira/browse/METRON-1736 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > The integration test for the Batch Profiler should use the Profiler Client > API and `PROFILE_GET` to validate the values that are produced. This is more > effective end-to-end validation that the Batch Profiler is working. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1708) Run the Batch Profiler in Spark
[ https://issues.apache.org/jira/browse/METRON-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1708: Fix Version/s: 0.7.0 > Run the Batch Profiler in Spark > --- > > Key: METRON-1708 > URL: https://issues.apache.org/jira/browse/METRON-1708 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > Create tooling to allow the user to run the Batch Profiler from the command > line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1748) Improve Storm Profiler Integration Test
[ https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1748: Fix Version/s: 0.7.0 > Improve Storm Profiler Integration Test > --- > > Key: METRON-1748 > URL: https://issues.apache.org/jira/browse/METRON-1748 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > We should use the Profiler Client, like PROFILE_GET, to validate the output > of the Storm Profiler Integration Test. This is better validation that > things are working end-to-end. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1741) Move REPL Port of Profiler to Separate Project
[ https://issues.apache.org/jira/browse/METRON-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1741: Fix Version/s: 0.7.0 > Move REPL Port of Profiler to Separate Project > -- > > Key: METRON-1741 > URL: https://issues.apache.org/jira/browse/METRON-1741 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1794) Include User Details When Escalating Alerts
[ https://issues.apache.org/jira/browse/METRON-1794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1794: Fix Version/s: (was: Next + 1) 0.7.0 > Include User Details When Escalating Alerts > --- > > Key: METRON-1794 > URL: https://issues.apache.org/jira/browse/METRON-1794 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > When an alert is escalated in the Alerts UI, the escalation should be > attributable to a user. The username and a timestamp should be appended to > the escalation message that is pushed to the escalation topic in Kafka. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1807) Auto populate the recommended values to some of the metron config parameters
[ https://issues.apache.org/jira/browse/METRON-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1807: Fix Version/s: 0.7.0 > Auto populate the recommended values to some of the metron config parameters > - > > Key: METRON-1807 > URL: https://issues.apache.org/jira/browse/METRON-1807 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Major > Fix For: 0.7.0 > > > Auto populate the recommended values to below metron config parameters > Storm Rest Server Address > Zeppelin Server Address > Solr Zookeeper Urls -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1717) Relocate Storm Profiler Code
[ https://issues.apache.org/jira/browse/METRON-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1717: Fix Version/s: 0.7.0 > Relocate Storm Profiler Code > > > Key: METRON-1717 > URL: https://issues.apache.org/jira/browse/METRON-1717 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > The Storm port of the Profiler currently lives in > metron-analytics/metron-profiler. This should be moved to > metron-analytics/metron-profiler-storm. This would mirror the project names > for the Spark port (metron-profiler-spark) and the REPL port > (metron-profiler-repl). > The package name for the Storm port of the Profiler should be changed to > org.apache.metron.profiler.storm. This would mimic the package name used for > Spark; org.apache.metron.profiler.spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1761:
Fix Version/s: 0.7.0
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
> Fix For: 0.7.0
>
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1809) Support Column Oriented Input with Batch Profiler
[ https://issues.apache.org/jira/browse/METRON-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1809: Fix Version/s: 0.7.0 > Support Column Oriented Input with Batch Profiler > - > > Key: METRON-1809 > URL: https://issues.apache.org/jira/browse/METRON-1809 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > The Batch Profiler currently only accepts input formats that can be directly > serialized to JSON. This should be enhanced to accept a wider variety of > input formats. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1775) Transient exception could prevent expired profiles from being flushed
[ https://issues.apache.org/jira/browse/METRON-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1775: Fix Version/s: 0.7.0 > Transient exception could prevent expired profiles from being flushed > - > > Key: METRON-1775 > URL: https://issues.apache.org/jira/browse/METRON-1775 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Minor > Fix For: 0.7.0 > > > In the Storm Profiler, the ProfileBuilderBolt uses a separate thread to flush > expired profiles. Expired profiles are those that have not received a > message for an extended period of time. That period of time is called the > profile time-to-live (profiler.ttl). This is done to avoid potential memory > leaks. > This separate thread is scheduled using > java.util.concurrent.ScheduledExecutorService#scheduleAtFixedRate. The > documentation for scheduleAtFixedRate says that if an exception is thrown, > subsequent executions will be suppressed. We do not want this to happen. If > an exception occurs, we need to log it and continue, ensuring that expired > profiles will continue to flush. > I have no direct proof that this has caused a problem, but it certainly is a > potential issue that should be addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1776) Update public web site to point at 0.6.0 new release
[ https://issues.apache.org/jira/browse/METRON-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1776: Fix Version/s: 0.7.0 > Update public web site to point at 0.6.0 new release > > > Key: METRON-1776 > URL: https://issues.apache.org/jira/browse/METRON-1776 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1811) Alert Search Fails When Sorting by Alert Status
[
https://issues.apache.org/jira/browse/METRON-1811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1811:
Fix Version/s: 0.7.0
> Alert Search Fails When Sorting by Alert Status
> ---
>
> Key: METRON-1811
> URL: https://issues.apache.org/jira/browse/METRON-1811
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.7.0
>
>
> Searching for alerts does not work when sorting by Alert Status. When this
> happens, no error is indicated in the UI, but the REST calls fails.
> Request:
> {{{"indices":[],"facetFields":[],"query":"*","from":0,"size":25,"sort":[\{"field":"alert_status","sortOrder":"desc"}]}
> }}
> Response:
> {{500 Internal Server Error }}
> The following is logged in the REST logs @ /var/log/metron/metron-rest.log
> {{18/09/26 20:38:24 ERROR controller.RestExceptionHandler: Encountered error:
> Failed to execute search; error='IllegalArgumentException: Fielddata is
> disabled on text fields by default. Set fielddata=true on [__anonymous_text]
> in order to load fielddata in memory by uninverting the inverted index. Note
> that this can however use significant memory. Alternatively use a keyword
> field instead.',
> search='\{"from":0,"size":25,"query":{"constant_score":{"filter":{"bool":{"must":[{"bool":{"should":[{"query_string":{"query":"*","fields":[],"use_dis_max":true,"tie_breaker":0.0,"default_operator":"or","auto_generate_phrase_queries":false,"max_determinized_states":1,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"split_on_whitespace":true,"boost":1.0}},\{"nested":{"query":{"query_string":{"query":"*","fields":[],"use_dis_max":true,"tie_breaker":0.0,"default_operator":"or","auto_generate_phrase_queries":false,"max_determinized_states":1,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"split_on_whitespace":true,"boost":1.0}},"path":"metron_alert","ignore_unmapped":false,"score_mode":"none","boost":1.0}}],"disable_coord":false,"adjust_pure_negative":true,"boost":1.0}},\{"bool":{"should":[{"term":{"status":{"value":"active","boost":1.0}}},\{"bool":{"must_not":[{"exists":{"field":"status","boost":1.0}}],"disable_coord":false,"adjust_pure_negative":true,"boost":1.0}}],"disable_coord":false,"adjust_pure_negative":true,"boost":1.0}}],"must_not":[\{"exists":{"field":"metaalerts","boost":1.0}}],"disable_coord":false,"adjust_pure_negative":true,"boost":1.0}},"boost":1.0}},"_source":\{"includes":[],"excludes":[]},"sort":[\{"alert_status":{"order":"desc","missing":"_last","unmapped_type":"text"}}],"track_scores":true,"aggregations":\{"source:type_count":{"terms":{"field":"source:type","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},\{"_term":"asc"}]}},"ip_src_addr_count":\{"terms":{"field":"ip_src_addr","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},\{"_term":"asc"}]}},"ip_dst_addr_count":\{"terms":{"field":"ip_dst_addr","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},\{"_term":"asc"}]}},"enrichments:geo:ip_dst_addr:country_count":\{"terms":{"field":"enrichments:geo:ip_dst_addr:country","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},\{"_term":"asc"}]'
> org.apache.metron.rest.RestException: Failed to execute search;
> error='IllegalArgumentException: Fielddata is disabled on text fields by
> default. Set fielddata=true on [__anonymous_text] in order to load fielddata
> in memory by uninverting the inverted index. Note that this can however use
> significant memory. Alternatively use a keyword field instead.',
>
[GitHub] nickwallen commented on issue #1294: METRON-1930: Update webpack-dev-server in Alerts UI
nickwallen commented on issue #1294: METRON-1930: Update webpack-dev-server in Alerts UI URL: https://github.com/apache/metron/pull/1294#issuecomment-446262595 @sardell mentioned to me that this only impacts development dependencies. This will not impact the production dependencies that are packaged and shipped in an RPM/DEB. +1 Pending a successful spin-up of Full Dev just to confirm that these dependencies do not break packaging or deployment. Thanks @sardell ! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1476: Fix Version/s: 0.6.0 > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > Fix For: 0.7.0 > > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1476) Update to Angular 6.1.3
[ https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1476: Fix Version/s: (was: 0.6.0) 0.7.0 > Update to Angular 6.1.3 > --- > > Key: METRON-1476 > URL: https://issues.apache.org/jira/browse/METRON-1476 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Major > Fix For: 0.7.0 > > > Update angular to speed up development -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1781) Fix RPM Spec File
[ https://issues.apache.org/jira/browse/METRON-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1781: Fix Version/s: 0.7.0 > Fix RPM Spec File > - > > Key: METRON-1781 > URL: https://issues.apache.org/jira/browse/METRON-1781 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > The RPM spec is pointing to the wrong path for the Storm Profiler tarball. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1777) Fix Elasticsearch X-Pack sample pom in documentation
[ https://issues.apache.org/jira/browse/METRON-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1777: Fix Version/s: 0.7.0 > Fix Elasticsearch X-Pack sample pom in documentation > > > Key: METRON-1777 > URL: https://issues.apache.org/jira/browse/METRON-1777 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1759) PCAP UI: Removing wrong Input annotations from pcap panel component
[ https://issues.apache.org/jira/browse/METRON-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1759: Fix Version/s: 0.7.0 > PCAP UI: Removing wrong Input annotations from pcap panel component > --- > > Key: METRON-1759 > URL: https://issues.apache.org/jira/browse/METRON-1759 > Project: Metron > Issue Type: Improvement >Reporter: Tibor Meller >Priority: Major > Fix For: 0.7.0 > > > The following fields are annotated as inputs of pcap-panel, but actually, > they don't. > > @Input pdml:Pdml=null; > @Input pcapRequest:PcapRequest; > @Input resetPaginationForSearch:boolean; > > The current state of the code is misleading and makes harder to understand > the actual behaviour of the panel component. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1772) Support alternative input formats in the Batch Profiler
[ https://issues.apache.org/jira/browse/METRON-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1772: Fix Version/s: 0.7.0 > Support alternative input formats in the Batch Profiler > --- > > Key: METRON-1772 > URL: https://issues.apache.org/jira/browse/METRON-1772 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > Additional formats, like ORC, should be supported when reading the input > telemetry in the Batch Profiler. The user should be able to customize the > Batch Profiler based on how they have chosen to archive their telemetry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1774) Allow user to configure JAAS client in Ambari
[ https://issues.apache.org/jira/browse/METRON-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1774: Fix Version/s: 0.7.0 > Allow user to configure JAAS client in Ambari > - > > Key: METRON-1774 > URL: https://issues.apache.org/jira/browse/METRON-1774 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > Right now, the user can only configure the client JAAS in limited ways. We > should build the client JAAS from a template, so that the user has more > control over this. > For example, I cannot easily turn on debugging today. If I make a change to > `$METRON_HOME/client_jaas.conf` to turn on debugging, Ambari will write over > my change when any of the topologies are restarted. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1829) Large Error Message Causes Slow Search Performance
[ https://issues.apache.org/jira/browse/METRON-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1829: Fix Version/s: 0.7.0 > Large Error Message Causes Slow Search Performance > -- > > Key: METRON-1829 > URL: https://issues.apache.org/jira/browse/METRON-1829 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.0 > > > Errors that occur during batch writes in the index topologies (batch and RA) > are written to Elasticsearch as a single, large error message, with a field > for each failed message. For example, if the batch size is 5000, a single > error message will be created with 5000 fields `raw_message_0`, > `raw_message_1`, .., `raw_message_4999`. With such large messages, searching > the error index in Elasticsearch is excessively slow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1508) In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1508:
Fix Version/s: (was: Next + 1)
0.7.0
> In Ubuntu14 Dev Indexing Fails to Write to Elasticsearch
>
>
> Key: METRON-1508
> URL: https://issues.apache.org/jira/browse/METRON-1508
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
> Fix For: 0.7.0
>
>
> When spinning up the Ubuntu 14 development environment, the indexing topology
> fails to write to Elasticsearch. This does not appear to be caused by
> resource constraints or OOM conditions. The indexing topology reports the
> following error.
> {code:java}
> 2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3]
> [ERROR]
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the
> configured nodes are available:
> [{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
> ~[stormjar.jar:?]
> at
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
> ~[stormjar.jar:?]
> at
> org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at
> org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1786) Pcap Topology Status Incorrect
[
https://issues.apache.org/jira/browse/METRON-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1786:
Fix Version/s: 0.7.0
> Pcap Topology Status Incorrect
> --
>
> Key: METRON-1786
> URL: https://issues.apache.org/jira/browse/METRON-1786
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Mohan
>Priority: Major
> Fix For: 0.7.0
>
>
> After running for some period of time, the pcap topology status in Ambari
> goes to "red" indicating that it is not running, when it actually is running.
> The following exception is thrown in the Ambari Agent logs.
> {code:java}
> INFO 2018-09-20 16:48:08,199 PythonReflectiveExecutor.py:67 - Reflective
> command failed with exception:
> Traceback (most recent call last):
> File
> "/usr/lib/python2.6/site-packages/ambari_agent/PythonReflectiveExecutor.py",
> line 59, in run_file
> imp.load_source('__main__', script)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_master.py",
> line 105, in
> Pcap().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 375, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_master.py",
> line 93, in status
> commands = PcapCommands(status_params)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.6.0/package/scripts/pcap_commands.py",
> line 51, in __init__
> self.__pcap_topology = params.metron_pcap_topology
> AttributeError: 'module' object has no attribute 'metron_pcap_topology'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1709) Add controls to start / stop the PCAP topology from Ambari.
[ https://issues.apache.org/jira/browse/METRON-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1709: Fix Version/s: 0.7.0 > Add controls to start / stop the PCAP topology from Ambari. > --- > > Key: METRON-1709 > URL: https://issues.apache.org/jira/browse/METRON-1709 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Major > Fix For: 0.7.0 > > > We currently don't have a mechanism to start/stop the PCAP topology from > Ambari -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1804) Update version to 0.6.1
[ https://issues.apache.org/jira/browse/METRON-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1804: Fix Version/s: 0.7.0 > Update version to 0.6.1 > --- > > Key: METRON-1804 > URL: https://issues.apache.org/jira/browse/METRON-1804 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.0 > > > Post release version bump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1699) Create Batch Profiler
[ https://issues.apache.org/jira/browse/METRON-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1699: Fix Version/s: (was: Next + 1) 0.7.0 > Create Batch Profiler > - > > Key: METRON-1699 > URL: https://issues.apache.org/jira/browse/METRON-1699 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > Attachments: Screen Shot 2018-07-27 at 10.55.27 AM.png, Screen Shot > 2018-07-27 at 11.07.33 AM.png, Screen Shot 2018-07-27 at 11.10.16 AM.png > > > Create a Batch Profiler that satisfies the following use cases. > h3. Use Cases > * As a Security Data Scientist, I want to understand the historical > behaviors and trends of a profile that I have created so that I can determine > if I have created a feature set that has predictive value for model building. > * As a Security Data Scientist, I want to understand the historical > behaviors and trends of a profile that I have created so that I can determine > if I have defined the profile correctly and created a feature set that > matches reality. > * As a Security Platform Engineer, I want to generate a profile using > archived telemetry when I deploy a new model to production so that models > depending on that profile can function on day 1. > h3. Goal > * Currently, a profile can only be generated from the telemetry consumed > *after* the profile was created. > * The goal would be to enable “profile seeding” which allows profiles to be > populated from a time *before* the profile was created. > * A profile would be seeded using the telemetry that has been archived by > Metron in HDFS. > * A profile consumer should not be able to distinguish the “seeded” portion > of a profile. > !Screen Shot 2018-07-27 at 10.55.27 AM.png! > h3. Current State > * There are currently two ports of the Profiler; the Streaming Profiler that > handles streaming data in Storm and the other that runs in the REPL and > allows a user to manually build, test, and debug profiles. > * These ports largely share a common code base in > metron-analytics/metron-profiler-common. > * A smaller set of “orchestration” logic is required to maintain each port; > one for Storm, another for the REPL. > * Both Profiler ports supports both system time and event time processing. > !Screen Shot 2018-07-27 at 11.07.33 AM.png! > h3. Approach > * Create a third port of the Profiler; the Batch Profiler. > * The Batch Profiler will be built to run in Spark so that the telemetry can > be consumed in batch. > * Allows a user to seed profiles using the JSON telemetry that is archived > in HDFS by Metron Indexing. > * Only generates the profile data stored in HBase, not the messages that are > produced for Threat Triage and Kafka. > * Any number of profiles can be generated at once, but no dependencies > between the profiles are supported. A dependency is where one profile is a > consumer of the profile generated by another. > * The Batch Profiler must use the timestamps contained within the telemetry; > it runs on event time. Luckily the Profiler already supports event time. > * Enable a pluggable mechanism so that telemetry stored in different formats > can be consumed by the Batch Profiler. For example, the Profiler should be > able to consume telemetry stored as raw JSON or in other formats like ORC or > Parquet. > !Screen Shot 2018-07-27 at 11.10.16 AM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1782) Add Kafka Partition and Offset to Profiler Debug Logs
[ https://issues.apache.org/jira/browse/METRON-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1782: Fix Version/s: (was: Next + 1) 0.7.0 > Add Kafka Partition and Offset to Profiler Debug Logs > - > > Key: METRON-1782 > URL: https://issues.apache.org/jira/browse/METRON-1782 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > I was troubleshooting an issue that a user was running into with the > Profiler. The logs made it appear that a second batch of messages were being > sent after a period of time, when none actually were. > We found that the second batch was actually just Storm timing out the first > batch of messages and re-sending them. This would have been simple to > uncover if we had the Kafka topic, partition, and offset details of each > message logged at a DEBUG level. > I made those code change while debugging the issue. I thought this might be > generally useful to roll into Apache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1750) Create Parser for Syslog RFC 5424 Messages
[ https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1750: Fix Version/s: 0.7.0 > Create Parser for Syslog RFC 5424 Messages > -- > > Key: METRON-1750 > URL: https://issues.apache.org/jira/browse/METRON-1750 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 0.7.0 > > > Create a Metron parser for working with valid RFC 5424 syslog messages, > including support for structured data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1758) Add support for Ansible 2.6 in dev
[ https://issues.apache.org/jira/browse/METRON-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1758: Fix Version/s: 0.7.0 > Add support for Ansible 2.6 in dev > -- > > Key: METRON-1758 > URL: https://issues.apache.org/jira/browse/METRON-1758 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > Fix For: 0.7.0 > > > Add support for Ansible 2.6 in dev -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1791) Add GUID to Messages Produced by Profiler
[ https://issues.apache.org/jira/browse/METRON-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1791: Fix Version/s: 0.7.0 > Add GUID to Messages Produced by Profiler > - > > Key: METRON-1791 > URL: https://issues.apache.org/jira/browse/METRON-1791 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > If a profile defines a triage result, when that profile is flushed the > Profiler will generate a message containing the measurement data so that it > can be used for Threat Triage. The message that is generated does not > contain a GUID. All messages in Metron should contain a GUID. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1798) Add mpack support for parser aggregation
[ https://issues.apache.org/jira/browse/METRON-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1798: Fix Version/s: (was: 0.6.1) 0.7.0 > Add mpack support for parser aggregation > > > Key: METRON-1798 > URL: https://issues.apache.org/jira/browse/METRON-1798 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.7.0 > > > Support spawning of storm topologies if a user specifies an aggregated parser > configuration at: > Ambari -> Metron -> Configs -> Parsers -> "Metron Parsers" > > For example, specifying the following: > "bro,snort,yaf", "snort,yaf", yaf > should spawn an aggregated topology for first two, and a regular topology for > the 'yaf'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1771) Update REST endpoints to support eventually consistent UI updates
[ https://issues.apache.org/jira/browse/METRON-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1771: Fix Version/s: 0.7.0 > Update REST endpoints to support eventually consistent UI updates > - > > Key: METRON-1771 > URL: https://issues.apache.org/jira/browse/METRON-1771 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Priority: Major > Fix For: 0.7.0 > > > Currently the REST endpoints that perform document updates either return > true/false or nothing. This puts the responsibility of retrieving the > updated state of the object on the client in a separate call or > optimistically applying the changes and reverting when an update fails. This > can be problematic if a client attempts to get the current state immediately > after an update and the change isn't visible yet in the back end. > Ideally they should return the updated state of the object, eliminating the > need to look up the updated state in a separate call. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1695: Fix Version/s: 0.7.0 > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.7.0 > > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1806) Upgrade Maven Shade Plugin version
[ https://issues.apache.org/jira/browse/METRON-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1806: Fix Version/s: 0.7.0 > Upgrade Maven Shade Plugin version > -- > > Key: METRON-1806 > URL: https://issues.apache.org/jira/browse/METRON-1806 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1792) Simplify Profile Definitions in Integration Tests
[ https://issues.apache.org/jira/browse/METRON-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1792: Fix Version/s: 0.7.0 > Simplify Profile Definitions in Integration Tests > - > > Key: METRON-1792 > URL: https://issues.apache.org/jira/browse/METRON-1792 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.0 > > > It would be easier to read the ProfilerIntegrationTest if the profile > definitions were defined 'inline' rather than in separate configuration > files. > > This was feedback that was offered as part of another code review; > https://github.com/apache/metron/pull/1174#discussion_r215456000 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1813) Stellar REPL Not Initialized with Client JAAS
[
https://issues.apache.org/jira/browse/METRON-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1813:
Fix Version/s: 0.7.0
> Stellar REPL Not Initialized with Client JAAS
> -
>
> Key: METRON-1813
> URL: https://issues.apache.org/jira/browse/METRON-1813
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.7.0
>
>
> Running a function like `KAFKA_GET` in the Stellar REPL does not work in a
> kerberized environment.
> h3. Steps to Replicate
> 1. Deploy Metron in a kerberized environment.
> 2. Launch the REPL.
> {code}
> source /etc/default/metron
> $METRON_HOME/bin/stellar -z $ZOOKEEPER
> {code}
> 3. Attempt to get a message from Kafka.
> {code}
> [Stellar]>>> conf := \{
> "group.id":"bro_parser","security.protocol":"SASL_PLAINTEXT" }
> \{security.protocol=SASL_PLAINTEXT, group.id=bro_parser} [Stellar]>>>
> KAFKA_GET("bro", 10, conf) [!] Unable to parse: KAFKA_GET("bro", 10, conf)
> due to: Failed to construct kafka consumer with relevant variables
> conf=\{security.protocol=SASL_PLAINTEXT, group.id=bro_parser}
> org.apache.metron.stellar.dsl.ParseException: Unable to parse:
> KAFKA_GET("bro", 10, conf) due to: Failed to construct kafka consumer with
> relevant variables conf=\{security.protocol=SASL_PLAINTEXT,
> group.id=bro_parser} at
> org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:173)
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154)
> at
> org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:405)
> at
> org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)
> at
> org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:357)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) Caused by:
> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:702)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:587)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:569)
> at
> org.apache.metron.management.KafkaFunctions$KafkaGet.getMessages(KafkaFunctions.java:227)
> at
> org.apache.metron.management.KafkaFunctions$KafkaGet.apply(KafkaFunctions.java:209)
> at
> org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
> at
> org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
> at
> org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
> ... 7 more Caused by: org.apache.kafka.common.KafkaException:
> org.apache.kafka.common.KafkaException: Jaas configuration not found at
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
> at
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:71)
> at
> org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:623)
> ... 14 more Caused by: org.apache.kafka.common.KafkaException: Jaas
> configuration not found at
> org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:299)
> at
> org.apache.kafka.common.security.kerberos.KerberosLogin.configure(KerberosLogin.java:103)
> at
> org.apache.kafka.common.security.authenticator.LoginManager.(LoginManager.java:45)
> at
> org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
> at
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
> ... 17 more Caused by: java.io.IOException: Could not find a 'KafkaClient'
> entry in this configuration. at
> org.apache.kafka.common.security.JaasUtils.jaasConfig(JaasUtils.java:50) at
> org.apache.kafka.common.security.kerberos.KerberosLogin.getServiceName(KerberosLogin.java:297)
> ... 21 more
> [Stellar]>>>
> {code}
> h3. Root Cause
> When the Stellar REPL is launched in a Kerberized environment, it needs to
> have the Client JAAS passed to it so that Stellar functions can access
> resources like Kafka. The JVM running the REPL never gets passed the
> "-Djava.security.auth.login.config=/usr/hcp/current/metron/client_jaas.conf"
> JVM arg.
[jira] [Updated] (METRON-1769) Script creation of a release candidate
[ https://issues.apache.org/jira/browse/METRON-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1769: Fix Version/s: 0.7.0 > Script creation of a release candidate > -- > > Key: METRON-1769 > URL: https://issues.apache.org/jira/browse/METRON-1769 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.0 > > > Our process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process is fairly > error prone. There's a lot of typing out versions and other tedious busywork. > In particular, this JIRA is for steps 4/5, as they're the most egregious. > This script should be able to produce both the core artifacts as well as the > bro kafka plugin artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1812) Fix dependencies_with_url.csv
[ https://issues.apache.org/jira/browse/METRON-1812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1812: Fix Version/s: 0.7.0 > Fix dependencies_with_url.csv > - > > Key: METRON-1812 > URL: https://issues.apache.org/jira/browse/METRON-1812 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.0 > > > Some bad content snuck in during a merge somewhere along the way. Just > cleaning this up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1799) Remove outdated bylaws from site.
[ https://issues.apache.org/jira/browse/METRON-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1799: Fix Version/s: 0.7.0 > Remove outdated bylaws from site. > - > > Key: METRON-1799 > URL: https://issues.apache.org/jira/browse/METRON-1799 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.0 > > > Our bylaws are at > https://cwiki.apache.org/confluence/display/METRON/Apache+Metron+Bylaws, not > http://metron.apache.org/develop/bylaws.html > I believe the whole '/develop' in general can be dropped. To the best of my > knowledge, it's just orphaned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1820) Update to new Simple-Syslog-5424 version to support error handling
[ https://issues.apache.org/jira/browse/METRON-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1820: Fix Version/s: 0.7.0 > Update to new Simple-Syslog-5424 version to support error handling > -- > > Key: METRON-1820 > URL: https://issues.apache.org/jira/browse/METRON-1820 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 0.7.0 > > > Now that we have an api to support handling per line errors in metron, > upgrade to the newest version of the simple-syslog5424 library which also > supports this -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1805) Provide a default value for the Storm topology.max.spout.pending setting
[ https://issues.apache.org/jira/browse/METRON-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1805: Fix Version/s: 0.7.0 > Provide a default value for the Storm topology.max.spout.pending setting > > > Key: METRON-1805 > URL: https://issues.apache.org/jira/browse/METRON-1805 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.0 > > > Should we provide a default value for this property? Which topologies should > that apply to? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1821) Align prepare-release-candidate with documentation
[ https://issues.apache.org/jira/browse/METRON-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1821: Fix Version/s: 0.7.0 > Align prepare-release-candidate with documentation > -- > > Key: METRON-1821 > URL: https://issues.apache.org/jira/browse/METRON-1821 > Project: Metron > Issue Type: Task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1805) Provide a default value for the Storm topology.max.spout.pending setting
[ https://issues.apache.org/jira/browse/METRON-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1805: --- Assignee: Ryan Merriman > Provide a default value for the Storm topology.max.spout.pending setting > > > Key: METRON-1805 > URL: https://issues.apache.org/jira/browse/METRON-1805 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.0 > > > Should we provide a default value for this property? Which topologies should > that apply to? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1681: --- Assignee: Ryan Merriman > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.0 > > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1681) Decouple the ParserBolt from the Parse execution logic
[ https://issues.apache.org/jira/browse/METRON-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1681: Fix Version/s: 0.7.0 > Decouple the ParserBolt from the Parse execution logic > -- > > Key: METRON-1681 > URL: https://issues.apache.org/jira/browse/METRON-1681 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.0 > > > Per discussion on https://github.com/apache/metron/pull/1099, there are > concerns about the ParserBolt needed some refactoring. The discussion didn't > hold the PR up, but it was generally agreed that we should decouple some of > the initialization and execution logic. > This also aids us in integrating with other systems such as NiFi or Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1816) Date format Stellar function
[ https://issues.apache.org/jira/browse/METRON-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1816: --- Assignee: Ryan Merriman > Date format Stellar function > > > Key: METRON-1816 > URL: https://issues.apache.org/jira/browse/METRON-1816 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.0 > > > It would be useful to have a Stellar function that exposes the Java > SimpleDateFormat capability. This will allow us to correctly format date > fields for storage platforms that require a specific date format (Solr for > example). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
