[jira] [Commented] (METRON-1696) Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster
[
https://issues.apache.org/jira/browse/METRON-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575764#comment-16575764
]
ASF GitHub Bot commented on METRON-1696:
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1134
@mmiklavc - in my setup, I have seen that non-kerb requires storm:hadoop
permissions, while kerberized needs metron:hadoop for PCAP to write into HDFS.
> Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster
> -
>
> Key: METRON-1696
> URL: https://issues.apache.org/jira/browse/METRON-1696
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> pcap parser fails to write the pcap sequence files to hdfs directory due to
> insufficient privileges to hdfs folder for 'metron' user
> {code:java}
> 2018-07-25 10:15:50.035 o.a.m.s.p.HDFSWriterCallback
> Thread-9-kafkaSpout-executor[3 3] [ERROR] Permission denied: user=metron,
> access=WRITE,
> inode="/apps/metron/pcap/pcap_pcap_1532513746365022000_0_pcap-20-1532414055":hdfs:hdfs:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1134 @mmiklavc - in my setup, I have seen that non-kerb requires storm:hadoop permissions, while kerberized needs metron:hadoop for PCAP to write into HDFS. ---
[GitHub] metron pull request #1157: METRON-1732: Fix job status liveness bug and para...
GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/1157 METRON-1732: Fix job status liveness bug and parallelize finalizer file writing ## Contributor Comments This still needs to have the # of finalizer threads option exposed for the REST application, but since it's multi-threaded code I wanted to get the review process started while I finish that part up. Test plan and more detailed description to follow. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron parallel-hdfs-write Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1157.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1157 commit 0b887fb64f9e4f2682e454e69b42b0b1014f3f4d Author: Michael Miklavcic Date: 2018-08-10T05:11:59Z Parallelize finalizer writing ---
[jira] [Commented] (METRON-1732) Fix job status liveness bug and parallelize finalizer file writing
[ https://issues.apache.org/jira/browse/METRON-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575759#comment-16575759 ] ASF GitHub Bot commented on METRON-1732: GitHub user mmiklavc opened a pull request: https://github.com/apache/metron/pull/1157 METRON-1732: Fix job status liveness bug and parallelize finalizer file writing ## Contributor Comments This still needs to have the # of finalizer threads option exposed for the REST application, but since it's multi-threaded code I wanted to get the review process started while I finish that part up. Test plan and more detailed description to follow. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron parallel-hdfs-write Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1157.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1157 commit 0b887fb64f9e4f2682e454e69b42b0b1014f3f4d Author: Michael Miklavcic Date: 2018-08-10T05:11:59Z Parallelize finalizer writing > Fix job status liveness bug and parallelize finalizer file writing > -- > > Key: METRON-1732 > URL: https://issues.apache.org/jira/browse/METRON-1732 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1732) Fix job status liveness bug and parallelize finalizer file writing
Michael Miklavcic created METRON-1732: - Summary: Fix job status liveness bug and parallelize finalizer file writing Key: METRON-1732 URL: https://issues.apache.org/jira/browse/METRON-1732 Project: Metron Issue Type: Sub-task Reporter: Michael Miklavcic Assignee: Michael Miklavcic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1156: METRON-1702: Reload a running job in the UI
Github user sardell commented on the issue: https://github.com/apache/metron/pull/1156 @merrimanr I reviewed and tested on full dev. I have the same thought as @ruffle1986 regarding the cancelling of the subscription. We should add that line inside the ngOnDestroy lifecycle hook. Besides that, tested on full dev and everything works as described. This is really great work and will significantly improve the user experience. +1 once the subscription cancellation is added. ---
[jira] [Commented] (METRON-1702) Reload a running job in the UI
[ https://issues.apache.org/jira/browse/METRON-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575497#comment-16575497 ] ASF GitHub Bot commented on METRON-1702: Github user sardell commented on the issue: https://github.com/apache/metron/pull/1156 @merrimanr I reviewed and tested on full dev. I have the same thought as @ruffle1986 regarding the cancelling of the subscription. We should add that line inside the ngOnDestroy lifecycle hook. Besides that, tested on full dev and everything works as described. This is really great work and will significantly improve the user experience. +1 once the subscription cancellation is added. > Reload a running job in the UI > -- > > Key: METRON-1702 > URL: https://issues.apache.org/jira/browse/METRON-1702 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > When a user reloads the UI, the state of a running pcap job is lost. We need > a way to look up and load a running job when a page is created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1722) PcapCLI should print progress to stdout
[ https://issues.apache.org/jira/browse/METRON-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575470#comment-16575470 ] ASF GitHub Bot commented on METRON-1722: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1138 +1 > PcapCLI should print progress to stdout > --- > > Key: METRON-1722 > URL: https://issues.apache.org/jira/browse/METRON-1722 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > It would be helpful if the Pcap CLI reported job status included state and > percent complete. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1722) PcapCLI should print progress to stdout
[ https://issues.apache.org/jira/browse/METRON-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575468#comment-16575468 ] ASF GitHub Bot commented on METRON-1722: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1138#discussion_r209095504 --- Diff: metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java --- @@ -166,12 +166,12 @@ public void runs_fixed_pcap_filter_job_with_full_argument_list_and_default_datef PcapOptions.START_TIME_MS.put(config, 500L); PcapOptions.END_TIME_MS.put(config, 1000L); PcapOptions.NUM_RECORDS_PER_FILE.put(config, 1000); +PcapOptions.PRINT_JOB_STATUS.put(config, true); --- End diff -- Ah, I see. Missed that with the diff only showing the option line in the latest commit. Full diff shows the extra option in the cli options arg list. > PcapCLI should print progress to stdout > --- > > Key: METRON-1722 > URL: https://issues.apache.org/jira/browse/METRON-1722 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > It would be helpful if the Pcap CLI reported job status included state and > percent complete. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1138: METRON-1722: PcapCLI should print progress to stdout
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1138 +1 ---
[GitHub] metron pull request #1138: METRON-1722: PcapCLI should print progress to std...
Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1138#discussion_r209095504 --- Diff: metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java --- @@ -166,12 +166,12 @@ public void runs_fixed_pcap_filter_job_with_full_argument_list_and_default_datef PcapOptions.START_TIME_MS.put(config, 500L); PcapOptions.END_TIME_MS.put(config, 1000L); PcapOptions.NUM_RECORDS_PER_FILE.put(config, 1000); +PcapOptions.PRINT_JOB_STATUS.put(config, true); --- End diff -- Ah, I see. Missed that with the diff only showing the option line in the latest commit. Full diff shows the extra option in the cli options arg list. ---
[jira] [Commented] (METRON-1558) Enable paging through Pcap result sets
[ https://issues.apache.org/jira/browse/METRON-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575414#comment-16575414 ] Michael Miklavcic commented on METRON-1558: --- Paging was added as a capability when async pcap jobs were introduced. > Enable paging through Pcap result sets > -- > > Key: METRON-1558 > URL: https://issues.apache.org/jira/browse/METRON-1558 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Michael Miklavcic >Priority: Major > > Due to the volume of pcap data and the potential for very large query > results, the Pcap query process should write results in manageable chunks or > files, so that a user can page through the results in an efficient manner. > Page size should be configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1641) Enable Pcap jobs to be submitted asynchronously
[ https://issues.apache.org/jira/browse/METRON-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-1641: -- Issue Type: Sub-task (was: New Feature) Parent: METRON-1554 > Enable Pcap jobs to be submitted asynchronously > --- > > Key: METRON-1641 > URL: https://issues.apache.org/jira/browse/METRON-1641 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1558) Enable paging through Pcap result sets
[ https://issues.apache.org/jira/browse/METRON-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-1558: - Assignee: Michael Miklavcic > Enable paging through Pcap result sets > -- > > Key: METRON-1558 > URL: https://issues.apache.org/jira/browse/METRON-1558 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Michael Miklavcic >Priority: Major > > Due to the volume of pcap data and the potential for very large query > results, the Pcap query process should write results in manageable chunks or > files, so that a user can page through the results in an efficient manner. > Page size should be configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1722) PcapCLI should print progress to stdout
[ https://issues.apache.org/jira/browse/METRON-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575406#comment-16575406 ] ASF GitHub Bot commented on METRON-1722: Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1138#discussion_r209083961 --- Diff: metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java --- @@ -166,12 +166,12 @@ public void runs_fixed_pcap_filter_job_with_full_argument_list_and_default_datef PcapOptions.START_TIME_MS.put(config, 500L); PcapOptions.END_TIME_MS.put(config, 1000L); PcapOptions.NUM_RECORDS_PER_FILE.put(config, 1000); +PcapOptions.PRINT_JOB_STATUS.put(config, true); --- End diff -- This needs to be there since the print status command line argument is set. > PcapCLI should print progress to stdout > --- > > Key: METRON-1722 > URL: https://issues.apache.org/jira/browse/METRON-1722 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > It would be helpful if the Pcap CLI reported job status included state and > percent complete. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1138: METRON-1722: PcapCLI should print progress to std...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/1138#discussion_r209083961 --- Diff: metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java --- @@ -166,12 +166,12 @@ public void runs_fixed_pcap_filter_job_with_full_argument_list_and_default_datef PcapOptions.START_TIME_MS.put(config, 500L); PcapOptions.END_TIME_MS.put(config, 1000L); PcapOptions.NUM_RECORDS_PER_FILE.put(config, 1000); +PcapOptions.PRINT_JOB_STATUS.put(config, true); --- End diff -- This needs to be there since the print status command line argument is set. ---
[jira] [Commented] (METRON-1722) PcapCLI should print progress to stdout
[ https://issues.apache.org/jira/browse/METRON-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575393#comment-16575393 ] ASF GitHub Bot commented on METRON-1722: Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/1138#discussion_r209080362 --- Diff: metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java --- @@ -166,12 +166,12 @@ public void runs_fixed_pcap_filter_job_with_full_argument_list_and_default_datef PcapOptions.START_TIME_MS.put(config, 500L); PcapOptions.END_TIME_MS.put(config, 1000L); PcapOptions.NUM_RECORDS_PER_FILE.put(config, 1000); +PcapOptions.PRINT_JOB_STATUS.put(config, true); --- End diff -- Probably remove these as they're not used anymore by the test. > PcapCLI should print progress to stdout > --- > > Key: METRON-1722 > URL: https://issues.apache.org/jira/browse/METRON-1722 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > It would be helpful if the Pcap CLI reported job status included state and > percent complete. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[ https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575386#comment-16575386 ] ASF GitHub Bot commented on METRON-1725: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1153 lgtm, thanks @merrimanr. +1 via inspection. > Add ability to specify YARN queue for pcap jobs > --- > > Key: METRON-1725 > URL: https://issues.apache.org/jira/browse/METRON-1725 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > There should be a way to configure which YARN queue a pcap query job is > submitted to. This option should be available for both the CLI and REST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[ https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575387#comment-16575387 ] ASF GitHub Bot commented on METRON-1725: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1153 (pending Travis) > Add ability to specify YARN queue for pcap jobs > --- > > Key: METRON-1725 > URL: https://issues.apache.org/jira/browse/METRON-1725 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > There should be a way to configure which YARN queue a pcap query job is > submitted to. This option should be available for both the CLI and REST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1153: METRON-1725: Add ability to specify YARN queue for pcap ...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1153 (pending Travis) ---
[GitHub] metron issue #1153: METRON-1725: Add ability to specify YARN queue for pcap ...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1153 lgtm, thanks @merrimanr. +1 via inspection. ---
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[
https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575382#comment-16575382
]
ASF GitHub Bot commented on METRON-1725:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1153#discussion_r209078217
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -112,7 +120,24 @@ public void
runs_fixed_pcap_filter_job_with_default_argument_list() throws Excep
return new TypeSafeMatcher>() {
@Override
protected boolean matchesSafely(Map item) {
-return item.entrySet().containsAll(map.entrySet());
+for(K key: map.keySet()) {
+ if (key.equals(PcapOptions.HADOOP_CONF.getKey())) {
+Configuration itemConfiguration = (Configuration)
item.get(PcapOptions.HADOOP_CONF.getKey());
+Map mapConfiguration = (Map)
map.get(PcapOptions.HADOOP_CONF.getKey());
+for(String setting: mapConfiguration.keySet()) {
+ if
(!mapConfiguration.get(setting).equals(itemConfiguration.get(setting, ""))) {
+return false;
+ }
+}
+ } else {
+V itemValue = item.get(key);
+V mapValue = map.get(key);
+if (itemValue != null ? !itemValue.equals(mapValue) : mapValue
== null) {
--- End diff --
Fixed
> Add ability to specify YARN queue for pcap jobs
> ---
>
> Key: METRON-1725
> URL: https://issues.apache.org/jira/browse/METRON-1725
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> There should be a way to configure which YARN queue a pcap query job is
> submitted to. This option should be available for both the CLI and REST.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1153: METRON-1725: Add ability to specify YARN queue fo...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1153#discussion_r209078217
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -112,7 +120,24 @@ public void
runs_fixed_pcap_filter_job_with_default_argument_list() throws Excep
return new TypeSafeMatcher>() {
@Override
protected boolean matchesSafely(Map item) {
-return item.entrySet().containsAll(map.entrySet());
+for(K key: map.keySet()) {
+ if (key.equals(PcapOptions.HADOOP_CONF.getKey())) {
+Configuration itemConfiguration = (Configuration)
item.get(PcapOptions.HADOOP_CONF.getKey());
+Map mapConfiguration = (Map)
map.get(PcapOptions.HADOOP_CONF.getKey());
+for(String setting: mapConfiguration.keySet()) {
+ if
(!mapConfiguration.get(setting).equals(itemConfiguration.get(setting, ""))) {
+return false;
+ }
+}
+ } else {
+V itemValue = item.get(key);
+V mapValue = map.get(key);
+if (itemValue != null ? !itemValue.equals(mapValue) : mapValue
== null) {
--- End diff --
Fixed
---
[jira] [Commented] (METRON-1722) PcapCLI should print progress to stdout
[ https://issues.apache.org/jira/browse/METRON-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575378#comment-16575378 ] ASF GitHub Bot commented on METRON-1722: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1138 Let me know what you think of the latest commit. > PcapCLI should print progress to stdout > --- > > Key: METRON-1722 > URL: https://issues.apache.org/jira/browse/METRON-1722 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > It would be helpful if the Pcap CLI reported job status included state and > percent complete. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1138: METRON-1722: PcapCLI should print progress to stdout
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1138 Let me know what you think of the latest commit. ---
[jira] [Commented] (METRON-1731) PCAP - Escape colons in output dir names
[ https://issues.apache.org/jira/browse/METRON-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575317#comment-16575317 ] ASF GitHub Bot commented on METRON-1731: Github user mmiklavc closed the pull request at: https://github.com/apache/metron/pull/1155 > PCAP - Escape colons in output dir names > > > Key: METRON-1731 > URL: https://issues.apache.org/jira/browse/METRON-1731 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1731) PCAP - Escape colons in output dir names
[ https://issues.apache.org/jira/browse/METRON-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575318#comment-16575318 ] ASF GitHub Bot commented on METRON-1731: GitHub user mmiklavc reopened a pull request: https://github.com/apache/metron/pull/1155 METRON-1731: PCAP - Escape colons in output dir names ## Contributor Comments https://issues.apache.org/jira/browse/METRON-1731 This is an improvement on handling existing PCAP output directory writing. The PcapJob is extremely permissive in values it accepts for filtering packets, however the fixed/query filter parameters are used for constructing the output directory of the MapReduce job. When the query/field value contains a colon ":", this causes the hdfs Path object to throw a URIException. **Includes:** - token replace colons with underscores in output dir name - consolidate redundant unit test classes for Fixed and Query PCAP filters. - create an output directory formatter class to enable testing in isolation from the rest of pcapjob - consolidates the char escape convention used by fixed and query filter dir naming. Currently handle spaces, periods, apostrophes, and colons. **Testing** This doesn't technically require test data to validate, since it's an error that occurs prior to the job starting, though you're certainly welcome to. Run the following from the CLI and verify no exception is thrown. ``` $METRON_HOME/bin/pcap_query.sh fixed -bp /apps/metron/pcap/input -df "-MM-dd-HH-mm" -st 2018-07-26-00-00 -et 2018-07-26-00-10 -rpf 500 -p "Protocol: ICMP (1)" ``` ## Pull Request Checklist ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: n/a Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron output-dir-formatter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1155 commit 16ffcc16bef169d9082151b9b2fa381512bc813a Author: Michael Miklavcic Date: 2018-08-08T20:56:10Z Extract output directory formatter, add tests, consolidate redundant pcap filter unit test classes. commit 256a21dfe92f95e466b0a9488427331f176ab5d8 Author: Michael Miklavcic Date: 2018-08-09T15:34:47Z Merge branch 'feature/METRON-1554-pcap-query-panel' into output-dir-formatter > PCAP - Escape colons in output dir names > > > Key: METRON-1731 > URL: https://issues.apache.org/jira/browse/METRON-1731 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1731) PCAP - Escape colons in output dir names
[ https://issues.apache.org/jira/browse/METRON-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575316#comment-16575316 ] ASF GitHub Bot commented on METRON-1731: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1155 Kicking Travis - there appears to be an unrelated integration test in metron-common that failed. > PCAP - Escape colons in output dir names > > > Key: METRON-1731 > URL: https://issues.apache.org/jira/browse/METRON-1731 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1155: METRON-1731: PCAP - Escape colons in output dir n...
GitHub user mmiklavc reopened a pull request: https://github.com/apache/metron/pull/1155 METRON-1731: PCAP - Escape colons in output dir names ## Contributor Comments https://issues.apache.org/jira/browse/METRON-1731 This is an improvement on handling existing PCAP output directory writing. The PcapJob is extremely permissive in values it accepts for filtering packets, however the fixed/query filter parameters are used for constructing the output directory of the MapReduce job. When the query/field value contains a colon ":", this causes the hdfs Path object to throw a URIException. **Includes:** - token replace colons with underscores in output dir name - consolidate redundant unit test classes for Fixed and Query PCAP filters. - create an output directory formatter class to enable testing in isolation from the rest of pcapjob - consolidates the char escape convention used by fixed and query filter dir naming. Currently handle spaces, periods, apostrophes, and colons. **Testing** This doesn't technically require test data to validate, since it's an error that occurs prior to the job starting, though you're certainly welcome to. Run the following from the CLI and verify no exception is thrown. ``` $METRON_HOME/bin/pcap_query.sh fixed -bp /apps/metron/pcap/input -df "-MM-dd-HH-mm" -st 2018-07-26-00-00 -et 2018-07-26-00-10 -rpf 500 -p "Protocol: ICMP (1)" ``` ## Pull Request Checklist ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: n/a Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mmiklavc/metron output-dir-formatter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1155 commit 16ffcc16bef169d9082151b9b2fa381512bc813a Author: Michael Miklavcic Date: 2018-08-08T20:56:10Z Extract output directory formatter, add tests, consolidate redundant pcap filter unit test classes. commit 256a21dfe92f95e466b0a9488427331f176ab5d8 Author: Michael Miklavcic Date: 2018-08-09T15:34:47Z Merge branch 'feature/METRON-1554-pcap-query-panel' into output-dir-formatter ---
[GitHub] metron pull request #1155: METRON-1731: PCAP - Escape colons in output dir n...
Github user mmiklavc closed the pull request at: https://github.com/apache/metron/pull/1155 ---
[GitHub] metron issue #1155: METRON-1731: PCAP - Escape colons in output dir names
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1155 Kicking Travis - there appears to be an unrelated integration test in metron-common that failed. ---
[jira] [Commented] (METRON-1702) Reload a running job in the UI
[ https://issues.apache.org/jira/browse/METRON-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575309#comment-16575309 ] ASF GitHub Bot commented on METRON-1702: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1156 Thanks @tiborm and @ruffle1986. I merged in the latest feature branch which included the onDestroy code. I also fixed a regression introduced by https://github.com/apache/metron/pull/1142 and added test to catch it in the future. Let me know what you think. > Reload a running job in the UI > -- > > Key: METRON-1702 > URL: https://issues.apache.org/jira/browse/METRON-1702 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > When a user reloads the UI, the state of a running pcap job is lost. We need > a way to look up and load a running job when a page is created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[
https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575201#comment-16575201
]
ASF GitHub Bot commented on METRON-1725:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1153#discussion_r209019921
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -112,7 +120,24 @@ public void
runs_fixed_pcap_filter_job_with_default_argument_list() throws Excep
return new TypeSafeMatcher>() {
@Override
protected boolean matchesSafely(Map item) {
-return item.entrySet().containsAll(map.entrySet());
+for(K key: map.keySet()) {
+ if (key.equals(PcapOptions.HADOOP_CONF.getKey())) {
+Configuration itemConfiguration = (Configuration)
item.get(PcapOptions.HADOOP_CONF.getKey());
+Map mapConfiguration = (Map)
map.get(PcapOptions.HADOOP_CONF.getKey());
+for(String setting: mapConfiguration.keySet()) {
+ if
(!mapConfiguration.get(setting).equals(itemConfiguration.get(setting, ""))) {
+return false;
+ }
+}
+ } else {
+V itemValue = item.get(key);
+V mapValue = map.get(key);
+if (itemValue != null ? !itemValue.equals(mapValue) : mapValue
== null) {
--- End diff --
@merrimanr, sorry I provided a bad example - I was just trying to rewrite
the conditional in a different way with less "nots", rather than suggest that
exact change. More precisely, this looks correct to me:
```
if (itemValue != null ? !itemValue.equals(mapValue) : mapValue != null) {
...
```
as this will treat instances where BOTH values are `null` as equal.
> Add ability to specify YARN queue for pcap jobs
> ---
>
> Key: METRON-1725
> URL: https://issues.apache.org/jira/browse/METRON-1725
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> There should be a way to configure which YARN queue a pcap query job is
> submitted to. This option should be available for both the CLI and REST.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1153: METRON-1725: Add ability to specify YARN queue fo...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1153#discussion_r209019921
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -112,7 +120,24 @@ public void
runs_fixed_pcap_filter_job_with_default_argument_list() throws Excep
return new TypeSafeMatcher>() {
@Override
protected boolean matchesSafely(Map item) {
-return item.entrySet().containsAll(map.entrySet());
+for(K key: map.keySet()) {
+ if (key.equals(PcapOptions.HADOOP_CONF.getKey())) {
+Configuration itemConfiguration = (Configuration)
item.get(PcapOptions.HADOOP_CONF.getKey());
+Map mapConfiguration = (Map)
map.get(PcapOptions.HADOOP_CONF.getKey());
+for(String setting: mapConfiguration.keySet()) {
+ if
(!mapConfiguration.get(setting).equals(itemConfiguration.get(setting, ""))) {
+return false;
+ }
+}
+ } else {
+V itemValue = item.get(key);
+V mapValue = map.get(key);
+if (itemValue != null ? !itemValue.equals(mapValue) : mapValue
== null) {
--- End diff --
@merrimanr, sorry I provided a bad example - I was just trying to rewrite
the conditional in a different way with less "nots", rather than suggest that
exact change. More precisely, this looks correct to me:
```
if (itemValue != null ? !itemValue.equals(mapValue) : mapValue != null) {
...
```
as this will treat instances where BOTH values are `null` as equal.
---
[jira] [Commented] (METRON-1722) PcapCLI should print progress to stdout
[
https://issues.apache.org/jira/browse/METRON-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575163#comment-16575163
]
ASF GitHub Bot commented on METRON-1722:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1138
@merrimanr - That's not quite what I suggested - it's still not using the
common config we pass in
> Can we add a param for the CLI like
PcapOptions.PRINT_JOB_STATUS.put(commonConfig, true)
If that monitorAndPrintJob() is a blocking call, it probably belongs in the
`get()` itself. Rather than creating a one-off custom method invoked by the
client, we already have configuration facilities for providing these options
that we use for all other options. e.g. this section
https://github.com/apache/metron/blob/f17d50fbfed4fd14e5cdd5a718cc973f51d2c4bc/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java#L215
And then it's just using that property like we use the others.
https://github.com/apache/metron/blob/f17d50fbfed4fd14e5cdd5a718cc973f51d2c4bc/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java#L464
```
public Pageable get() throws JobException, InterruptedException {
if (PcapOptions.PRINT_JOB_STATUS.get(configuration, Boolean.class)) {
mrJob.monitorAndPrintJob();
}
for (; ; ) {
JobStatus status = getStatus();
if (status.getState() == State.SUCCEEDED
|| status.getState() == State.KILLED
|| status.getState() == State.FAILED) {
return getFinalResults();
}
Thread.sleep(completeCheckInterval);
}
}
```
The only reservation I have about the monitorAndPrintJob (versus using the
pcapjob's TimerTask method) is that users cannot print status in a non-blocking
way. Honestly I don't see a circumstance where you would do that with stdout
anyhow. It's through pcapcli, where we want the process to return only after
fully completing, so I think this is fine.
> PcapCLI should print progress to stdout
> ---
>
> Key: METRON-1722
> URL: https://issues.apache.org/jira/browse/METRON-1722
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> It would be helpful if the Pcap CLI reported job status included state and
> percent complete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1138: METRON-1722: PcapCLI should print progress to stdout
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1138
@merrimanr - That's not quite what I suggested - it's still not using the
common config we pass in
> Can we add a param for the CLI like
PcapOptions.PRINT_JOB_STATUS.put(commonConfig, true)
If that monitorAndPrintJob() is a blocking call, it probably belongs in the
`get()` itself. Rather than creating a one-off custom method invoked by the
client, we already have configuration facilities for providing these options
that we use for all other options. e.g. this section
https://github.com/apache/metron/blob/f17d50fbfed4fd14e5cdd5a718cc973f51d2c4bc/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java#L215
And then it's just using that property like we use the others.
https://github.com/apache/metron/blob/f17d50fbfed4fd14e5cdd5a718cc973f51d2c4bc/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java#L464
```
public Pageable get() throws JobException, InterruptedException {
if (PcapOptions.PRINT_JOB_STATUS.get(configuration, Boolean.class)) {
mrJob.monitorAndPrintJob();
}
for (; ; ) {
JobStatus status = getStatus();
if (status.getState() == State.SUCCEEDED
|| status.getState() == State.KILLED
|| status.getState() == State.FAILED) {
return getFinalResults();
}
Thread.sleep(completeCheckInterval);
}
}
```
The only reservation I have about the monitorAndPrintJob (versus using the
pcapjob's TimerTask method) is that users cannot print status in a non-blocking
way. Honestly I don't see a circumstance where you would do that with stdout
anyhow. It's through pcapcli, where we want the process to return only after
fully completing, so I think this is fine.
---
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[ https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575095#comment-16575095 ] ASF GitHub Bot commented on METRON-1725: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1153 The latest commit adds the YARN queue property to Ambari. > Add ability to specify YARN queue for pcap jobs > --- > > Key: METRON-1725 > URL: https://issues.apache.org/jira/browse/METRON-1725 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > There should be a way to configure which YARN queue a pcap query job is > submitted to. This option should be available for both the CLI and REST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1153: METRON-1725: Add ability to specify YARN queue for pcap ...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1153 The latest commit adds the YARN queue property to Ambari. ---
[jira] [Commented] (METRON-1728) Handle null values in config in Pcap backend more gracefully
[ https://issues.apache.org/jira/browse/METRON-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574999#comment-16574999 ] ASF GitHub Bot commented on METRON-1728: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1151 Closing as merged > Handle null values in config in Pcap backend more gracefully > > > Key: METRON-1728 > URL: https://issues.apache.org/jira/browse/METRON-1728 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1728) Handle null values in config in Pcap backend more gracefully
[ https://issues.apache.org/jira/browse/METRON-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575000#comment-16575000 ] ASF GitHub Bot commented on METRON-1728: Github user mmiklavc closed the pull request at: https://github.com/apache/metron/pull/1151 > Handle null values in config in Pcap backend more gracefully > > > Key: METRON-1728 > URL: https://issues.apache.org/jira/browse/METRON-1728 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1151: METRON-1728: Handle null values in config in Pcap backen...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1151 Closing as merged ---
[GitHub] metron pull request #1151: METRON-1728: Handle null values in config in Pcap...
Github user mmiklavc closed the pull request at: https://github.com/apache/metron/pull/1151 ---
[jira] [Commented] (METRON-1730) Update steps to run pycapa on Centos 6
[ https://issues.apache.org/jira/browse/METRON-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574995#comment-16574995 ] ASF GitHub Bot commented on METRON-1730: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1152 > Update steps to run pycapa on Centos 6 > -- > > Key: METRON-1730 > URL: https://issues.apache.org/jira/browse/METRON-1730 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1152: METRON-1730: Update steps to run pycapa on Centos...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1152 ---
[jira] [Commented] (METRON-1702) Reload a running job in the UI
[
https://issues.apache.org/jira/browse/METRON-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574979#comment-16574979
]
ASF GitHub Bot commented on METRON-1702:
Github user ruffle1986 commented on a diff in the pull request:
https://github.com/apache/metron/pull/1156#discussion_r208968690
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts
---
@@ -15,22 +15,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import { Component, Input } from '@angular/core';
+import { Component, OnInit, Input } from '@angular/core';
import { PcapService } from '../service/pcap.service';
import { PcapStatusResponse } from '../model/pcap-status-response';
import { PcapRequest } from '../model/pcap.request';
import { Pdml } from '../model/pdml';
import { Subscription } from 'rxjs/Rx';
import { PcapPagination } from '../model/pcap-pagination';
-import {RestError} from "../../model/rest-error";
+import { RestError } from "../../model/rest-error";
@Component({
selector: 'app-pcap-panel',
templateUrl: './pcap-panel.component.html',
styleUrls: ['./pcap-panel.component.scss']
})
-export class PcapPanelComponent {
+export class PcapPanelComponent implements OnInit {
--- End diff --
I've already done it here in this commit:
https://github.com/apache/metron/commit/14dcb2d90581835d8206c65918c24e8cb04bfd06
It's already merged back into the feature branch and it implements the
destroy interface and I also refactored it a bit to make it more maintainable.
We both touched the same lines so these conflicts have to be resolved before
merging it back.
> Reload a running job in the UI
> --
>
> Key: METRON-1702
> URL: https://issues.apache.org/jira/browse/METRON-1702
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> When a user reloads the UI, the state of a running pcap job is lost. We need
> a way to look up and load a running job when a page is created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1156: METRON-1702: Reload a running job in the UI
Github user ruffle1986 commented on a diff in the pull request:
https://github.com/apache/metron/pull/1156#discussion_r208968690
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts
---
@@ -15,22 +15,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import { Component, Input } from '@angular/core';
+import { Component, OnInit, Input } from '@angular/core';
import { PcapService } from '../service/pcap.service';
import { PcapStatusResponse } from '../model/pcap-status-response';
import { PcapRequest } from '../model/pcap.request';
import { Pdml } from '../model/pdml';
import { Subscription } from 'rxjs/Rx';
import { PcapPagination } from '../model/pcap-pagination';
-import {RestError} from "../../model/rest-error";
+import { RestError } from "../../model/rest-error";
@Component({
selector: 'app-pcap-panel',
templateUrl: './pcap-panel.component.html',
styleUrls: ['./pcap-panel.component.scss']
})
-export class PcapPanelComponent {
+export class PcapPanelComponent implements OnInit {
--- End diff --
I've already done it here in this commit:
https://github.com/apache/metron/commit/14dcb2d90581835d8206c65918c24e8cb04bfd06
It's already merged back into the feature branch and it implements the
destroy interface and I also refactored it a bit to make it more maintainable.
We both touched the same lines so these conflicts have to be resolved before
merging it back.
---
[jira] [Commented] (METRON-1695) Expose pcap properties through Ambari
[ https://issues.apache.org/jira/browse/METRON-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574949#comment-16574949 ] ASF GitHub Bot commented on METRON-1695: Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Sure @merrimanr , sounds good. @MohanDV has already begun work on METRON-1709. I will wait for him to complete and then submit this pull request so it would be a natural fit. > Expose pcap properties through Ambari > - > > Key: METRON-1695 > URL: https://issues.apache.org/jira/browse/METRON-1695 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > > Currently, the $METRON_HOME/config/pcap.properties file is hardcoded with the > defaults. One has to hand edit the file before deploying the PCAP topology. > These properties should be configurable via Ambari. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1132: METRON-1695: Expose pcap properties through Ambari
Github user anandsubbu commented on the issue: https://github.com/apache/metron/pull/1132 Sure @merrimanr , sounds good. @MohanDV has already begun work on METRON-1709. I will wait for him to complete and then submit this pull request so it would be a natural fit. ---
[jira] [Commented] (METRON-1702) Reload a running job in the UI
[
https://issues.apache.org/jira/browse/METRON-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574894#comment-16574894
]
ASF GitHub Bot commented on METRON-1702:
Github user ruffle1986 commented on a diff in the pull request:
https://github.com/apache/metron/pull/1156#discussion_r208946955
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts
---
@@ -15,22 +15,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import { Component, Input } from '@angular/core';
+import { Component, OnInit, Input } from '@angular/core';
import { PcapService } from '../service/pcap.service';
import { PcapStatusResponse } from '../model/pcap-status-response';
import { PcapRequest } from '../model/pcap.request';
import { Pdml } from '../model/pdml';
import { Subscription } from 'rxjs/Rx';
import { PcapPagination } from '../model/pcap-pagination';
-import {RestError} from "../../model/rest-error";
+import { RestError } from "../../model/rest-error";
@Component({
selector: 'app-pcap-panel',
templateUrl: './pcap-panel.component.html',
styleUrls: ['./pcap-panel.component.scss']
})
-export class PcapPanelComponent {
+export class PcapPanelComponent implements OnInit {
--- End diff --
@merrimanr You should also implement the OnDestroy interface and
unsubscribe from all the observables in the `ngOnDestroy` method. Especially
from the polling observable. By doing this, you can make sure that the
component won't continue the polling for the job's status when the user
navigates somewhere else from the pcap tab. If you unsubscribe in the
destructor method, it'll stop the pending http requests immediately as well
when the route changes. Not to mention freeing up the memory when the component
is not rendered anymore.
> Reload a running job in the UI
> --
>
> Key: METRON-1702
> URL: https://issues.apache.org/jira/browse/METRON-1702
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> When a user reloads the UI, the state of a running pcap job is lost. We need
> a way to look up and load a running job when a page is created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1156: METRON-1702: Reload a running job in the UI
Github user ruffle1986 commented on a diff in the pull request:
https://github.com/apache/metron/pull/1156#discussion_r208946955
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts
---
@@ -15,22 +15,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import { Component, Input } from '@angular/core';
+import { Component, OnInit, Input } from '@angular/core';
import { PcapService } from '../service/pcap.service';
import { PcapStatusResponse } from '../model/pcap-status-response';
import { PcapRequest } from '../model/pcap.request';
import { Pdml } from '../model/pdml';
import { Subscription } from 'rxjs/Rx';
import { PcapPagination } from '../model/pcap-pagination';
-import {RestError} from "../../model/rest-error";
+import { RestError } from "../../model/rest-error";
@Component({
selector: 'app-pcap-panel',
templateUrl: './pcap-panel.component.html',
styleUrls: ['./pcap-panel.component.scss']
})
-export class PcapPanelComponent {
+export class PcapPanelComponent implements OnInit {
--- End diff --
@merrimanr You should also implement the OnDestroy interface and
unsubscribe from all the observables in the `ngOnDestroy` method. Especially
from the polling observable. By doing this, you can make sure that the
component won't continue the polling for the job's status when the user
navigates somewhere else from the pcap tab. If you unsubscribe in the
destructor method, it'll stop the pending http requests immediately as well
when the route changes. Not to mention freeing up the memory when the component
is not rendered anymore.
---
[jira] [Commented] (METRON-1723) PCAP UI - Unable to select/copy from packets details in PCAP query panel
[ https://issues.apache.org/jira/browse/METRON-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574885#comment-16574885 ] ASF GitHub Bot commented on METRON-1723: Github user sardell closed the pull request at: https://github.com/apache/metron/pull/1139 > PCAP UI - Unable to select/copy from packets details in PCAP query panel > > > Key: METRON-1723 > URL: https://issues.apache.org/jira/browse/METRON-1723 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Attachments: expand-collapse-details-PCAP-query.mp4 > > > Upon query completion, a user can click on a row and drill down (up to 2 > levels) for a given packet. > A SOC analyst might find it useful to select/copy specific information from > the 2nd level granular details (E.g. MAC address, IP address, timestamps > etc.) to be used subsequently for analysis/investigation. > Currently, selecting or clicking on nested content fires an event that > expands/collapses the information. See the attached video to view the > behavior. > It would be better if target for the collapse/expand event only applied to > the top of the details section. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1139: METRON-1723: PCAP UI - Unable to select/copy from...
Github user sardell closed the pull request at: https://github.com/apache/metron/pull/1139 ---
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574880#comment-16574880 ] ASF GitHub Bot commented on METRON-1713: Github user tiborm closed the pull request at: https://github.com/apache/metron/pull/1143 > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Tibor Meller >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1143: METRON-1713: PCAP UI - Add a way to kill a pcap j...
Github user tiborm closed the pull request at: https://github.com/apache/metron/pull/1143 ---
[jira] [Commented] (METRON-1730) Update steps to run pycapa on Centos 6
[
https://issues.apache.org/jira/browse/METRON-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574849#comment-16574849
]
ASF GitHub Bot commented on METRON-1730:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208932022
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
Yeah, that's exactly what I was looking for.
I did see the FAIL/ERROR messages, but I had the same experience where it
ran as expected. I suspected it was due to initial connection setup, because
once it started writing, it worked as expected.
> Update steps to run pycapa on Centos 6
> --
>
> Key: METRON-1730
> URL: https://issues.apache.org/jira/browse/METRON-1730
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1152: METRON-1730: Update steps to run pycapa on Centos...
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1152#discussion_r208932022
--- Diff: metron-sensors/pycapa/README.md ---
@@ -76,8 +79,83 @@ General notes on the installation of Pycapa.
python setup.py install
```
+### Centos 6
+
+* These instructions can be used directly on CentOS 6 - useful for
developers using the Full Dev Vagrant test box.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed,
should install Python 2.7 within a virtual environment and then run Pycapa from
within the virtual environment.
+
+1. Set up a couple environment variables.
+
+```
+PYCAPA_HOME=/opt/pycapa
+PYTHON27_HOME=/opt/rh/python27/root
+```
+
+1. Install required packages.
+
+```
+for item in epel-release centos-release-scl "@Development tools"
python27 python27-scldevel python27-python-virtualenv libpcap-devel
libselinux-python; do yum install -y $item; done
+```
+
+1. Setup Pycapa directory.
+
+```
+mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME
+```
+
+1. Create the virtualenv.
+
+```
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+cd $PYCAPA_HOME
+${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv
+```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+```
+export PREFIX=/usr
+wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz
-O - | tar -xz
--- End diff --
Yeah, that's exactly what I was looking for.
I did see the FAIL/ERROR messages, but I had the same experience where it
ran as expected. I suspected it was due to initial connection setup, because
once it started writing, it worked as expected.
---
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574843#comment-16574843 ] ASF GitHub Bot commented on METRON-1713: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1143 This is merged. Can you close @tiborm? > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Tibor Meller >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1143: METRON-1713: PCAP UI - Add a way to kill a pcap job
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1143 This is merged. Can you close @tiborm? ---
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574832#comment-16574832 ] ASF GitHub Bot commented on METRON-1713: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1143 +1, thanks for this, it's super nice to have this ability. > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Tibor Meller >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1728) Handle null values in config in Pcap backend more gracefully
[ https://issues.apache.org/jira/browse/METRON-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574829#comment-16574829 ] ASF GitHub Bot commented on METRON-1728: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1151 +1 by inspection. Thanks for the improvement! > Handle null values in config in Pcap backend more gracefully > > > Key: METRON-1728 > URL: https://issues.apache.org/jira/browse/METRON-1728 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1702) Reload a running job in the UI
[
https://issues.apache.org/jira/browse/METRON-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574823#comment-16574823
]
ASF GitHub Bot commented on METRON-1702:
Github user tiborm commented on a diff in the pull request:
https://github.com/apache/metron/pull/1156#discussion_r208919679
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts
---
@@ -44,7 +44,23 @@ export class PcapPanelComponent {
savedPcapRequest: {};
errorMsg: string;
- constructor(private pcapService: PcapService) { }
+ constructor(private pcapService: PcapService ) { }
--- End diff --
There is an extra space after the type. Is it intentional?
> Reload a running job in the UI
> --
>
> Key: METRON-1702
> URL: https://issues.apache.org/jira/browse/METRON-1702
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> When a user reloads the UI, the state of a running pcap job is lost. We need
> a way to look up and load a running job when a page is created.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1156: METRON-1702: Reload a running job in the UI
Github user tiborm commented on a diff in the pull request:
https://github.com/apache/metron/pull/1156#discussion_r208919679
--- Diff:
metron-interface/metron-alerts/src/app/pcap/pcap-panel/pcap-panel.component.ts
---
@@ -44,7 +44,23 @@ export class PcapPanelComponent {
savedPcapRequest: {};
errorMsg: string;
- constructor(private pcapService: PcapService) { }
+ constructor(private pcapService: PcapService ) { }
--- End diff --
There is an extra space after the type. Is it intentional?
---
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[ https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574820#comment-16574820 ] ASF GitHub Bot commented on METRON-1725: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1153 Sure why not? I will add the Ambari changes. > Add ability to specify YARN queue for pcap jobs > --- > > Key: METRON-1725 > URL: https://issues.apache.org/jira/browse/METRON-1725 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > There should be a way to configure which YARN queue a pcap query job is > submitted to. This option should be available for both the CLI and REST. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1153: METRON-1725: Add ability to specify YARN queue for pcap ...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1153 Sure why not? I will add the Ambari changes. ---
[jira] [Commented] (METRON-1725) Add ability to specify YARN queue for pcap jobs
[
https://issues.apache.org/jira/browse/METRON-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574819#comment-16574819
]
ASF GitHub Bot commented on METRON-1725:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1153#discussion_r208921801
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
---
@@ -112,7 +120,24 @@ public void
runs_fixed_pcap_filter_job_with_default_argument_list() throws Excep
return new TypeSafeMatcher>() {
@Override
protected boolean matchesSafely(Map item) {
-return item.entrySet().containsAll(map.entrySet());
+for(K key: map.keySet()) {
+ if (key.equals(PcapOptions.HADOOP_CONF.getKey())) {
+Configuration itemConfiguration = (Configuration)
item.get(PcapOptions.HADOOP_CONF.getKey());
+Map mapConfiguration = (Map)
map.get(PcapOptions.HADOOP_CONF.getKey());
+for(String setting: mapConfiguration.keySet()) {
+ if
(!mapConfiguration.get(setting).equals(itemConfiguration.get(setting, ""))) {
+return false;
+ }
+}
+ } else {
+V itemValue = item.get(key);
+V mapValue = map.get(key);
+if (itemValue != null ? !itemValue.equals(mapValue) : mapValue
== null) {
--- End diff --
If any key/value pair doesn't match, they we want to return false. If all
key/value pairs match, then we return true. With your change, true will be
returned the first time a key/value pair matches and all others are skipped.
> Add ability to specify YARN queue for pcap jobs
> ---
>
> Key: METRON-1725
> URL: https://issues.apache.org/jira/browse/METRON-1725
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> There should be a way to configure which YARN queue a pcap query job is
> submitted to. This option should be available for both the CLI and REST.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1704) Message Timestamp Logic Should be Shared
[
https://issues.apache.org/jira/browse/METRON-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574802#comment-16574802
]
ASF GitHub Bot commented on METRON-1704:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1146#discussion_r208916394
--- Diff:
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/DefaultMessageRouter.java
---
@@ -86,20 +96,24 @@ public DefaultMessageRouter(Context context) {
* @param profile The profile that may need the message.
* @return A MessageRoute if the message is needed by the profile.
*/
- private Optional routeToProfile(JSONObject message,
ProfileConfig profile) {
+ private Optional routeToProfile(JSONObject message,
ProfileConfig profile, Clock clock) {
Optional route = Optional.empty();
// allow the profile to access the fields defined within the message
@SuppressWarnings("unchecked")
final Map state = (Map) message;
-
try {
// is this message needed by this profile?
if (executor.execute(profile.getOnlyif(), state, Boolean.class)) {
-// what is the name of the entity in this message?
-String entity = executor.execute(profile.getForeach(), state,
String.class);
-route = Optional.of(new MessageRoute(profile, entity));
+// what time is is? could be either system or event time
--- End diff --
The timestamp logic now lives here. This allows it to be shared by all of
the Profiler ports; Storm, REPL, and Spark.
> Message Timestamp Logic Should be Shared
>
>
> Key: METRON-1704
> URL: https://issues.apache.org/jira/browse/METRON-1704
> Project: Metron
> Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> The message timestamp logic is currently duplicated across both ports of the
> Profiler; the REPL and Storm. This should be pulled into
> metron-profiler-common so that the logic can be shared and also used by the
> Spark port.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1146: METRON-1704 Message Timestamp Logic Should be Sha...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1146#discussion_r208916394
--- Diff:
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/DefaultMessageRouter.java
---
@@ -86,20 +96,24 @@ public DefaultMessageRouter(Context context) {
* @param profile The profile that may need the message.
* @return A MessageRoute if the message is needed by the profile.
*/
- private Optional routeToProfile(JSONObject message,
ProfileConfig profile) {
+ private Optional routeToProfile(JSONObject message,
ProfileConfig profile, Clock clock) {
Optional route = Optional.empty();
// allow the profile to access the fields defined within the message
@SuppressWarnings("unchecked")
final Map state = (Map) message;
-
try {
// is this message needed by this profile?
if (executor.execute(profile.getOnlyif(), state, Boolean.class)) {
-// what is the name of the entity in this message?
-String entity = executor.execute(profile.getForeach(), state,
String.class);
-route = Optional.of(new MessageRoute(profile, entity));
+// what time is is? could be either system or event time
--- End diff --
The timestamp logic now lives here. This allows it to be shared by all of
the Profiler ports; Storm, REPL, and Spark.
---
[jira] [Commented] (METRON-1722) PcapCLI should print progress to stdout
[ https://issues.apache.org/jira/browse/METRON-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574796#comment-16574796 ] ASF GitHub Bot commented on METRON-1722: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1138 Are you good @mmiklavc? > PcapCLI should print progress to stdout > --- > > Key: METRON-1722 > URL: https://issues.apache.org/jira/browse/METRON-1722 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > It would be helpful if the Pcap CLI reported job status included state and > percent complete. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1704) Message Timestamp Logic Should be Shared
[
https://issues.apache.org/jira/browse/METRON-1704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574794#comment-16574794
]
ASF GitHub Bot commented on METRON-1704:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1146#discussion_r208915112
--- Diff:
metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileSplitterBolt.java
---
@@ -162,15 +156,9 @@ private void doExecute(Tuple input) throws
ParseException, UnsupportedEncodingEx
// ensure there is a valid profiler configuration
ProfilerConfig config = getProfilerConfig();
if(config != null && config.getProfiles().size() > 0) {
+ routeMessage(input, message, config);
- // what time is it?
--- End diff --
Here I removed the timestamp logic from Storm. The `MessageRouter` does
this for us now.
> Message Timestamp Logic Should be Shared
>
>
> Key: METRON-1704
> URL: https://issues.apache.org/jira/browse/METRON-1704
> Project: Metron
> Issue Type: Sub-task
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> The message timestamp logic is currently duplicated across both ports of the
> Profiler; the REPL and Storm. This should be pulled into
> metron-profiler-common so that the logic can be shared and also used by the
> Spark port.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574795#comment-16574795 ] ASF GitHub Bot commented on METRON-1713: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1143 @tiborm can you merge the latest feature branch in? @justinleet are there any other outstanding requests or are you satisfied? > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Tibor Meller >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1138: METRON-1722: PcapCLI should print progress to stdout
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1138 Are you good @mmiklavc? ---
[GitHub] metron issue #1143: METRON-1713: PCAP UI - Add a way to kill a pcap job
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1143 @tiborm can you merge the latest feature branch in? @justinleet are there any other outstanding requests or are you satisfied? ---
[GitHub] metron pull request #1146: METRON-1704 Message Timestamp Logic Should be Sha...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1146#discussion_r208915112
--- Diff:
metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/ProfileSplitterBolt.java
---
@@ -162,15 +156,9 @@ private void doExecute(Tuple input) throws
ParseException, UnsupportedEncodingEx
// ensure there is a valid profiler configuration
ProfilerConfig config = getProfilerConfig();
if(config != null && config.getProfiles().size() > 0) {
+ routeMessage(input, message, config);
- // what time is it?
--- End diff --
Here I removed the timestamp logic from Storm. The `MessageRouter` does
this for us now.
---
[jira] [Commented] (METRON-1723) PCAP UI - Unable to select/copy from packets details in PCAP query panel
[ https://issues.apache.org/jira/browse/METRON-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574788#comment-16574788 ] ASF GitHub Bot commented on METRON-1723: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1139 This has been merge in the the feature branch. Can you close? > PCAP UI - Unable to select/copy from packets details in PCAP query panel > > > Key: METRON-1723 > URL: https://issues.apache.org/jira/browse/METRON-1723 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Attachments: expand-collapse-details-PCAP-query.mp4 > > > Upon query completion, a user can click on a row and drill down (up to 2 > levels) for a given packet. > A SOC analyst might find it useful to select/copy specific information from > the 2nd level granular details (E.g. MAC address, IP address, timestamps > etc.) to be used subsequently for analysis/investigation. > Currently, selecting or clicking on nested content fires an event that > expands/collapses the information. See the attached video to view the > behavior. > It would be better if target for the collapse/expand event only applied to > the top of the details section. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1139: METRON-1723: PCAP UI - Unable to select/copy from packet...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/1139 This has been merge in the the feature branch. Can you close? ---
[jira] [Assigned] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1713: Assignee: Shane Ardell > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1713: Assignee: Tibor Meller (was: Shane Ardell) > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Tibor Meller >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1713) PCAP UI - Add a way to kill a pcap job
[ https://issues.apache.org/jira/browse/METRON-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1713: Assignee: Shane Ardell (was: Tibor Meller) > PCAP UI - Add a way to kill a pcap job > -- > > Key: METRON-1713 > URL: https://issues.apache.org/jira/browse/METRON-1713 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Attachments: Screen Shot 2018-08-01 at 4.04.58 PM.png > > > As a user, I would like to be able to cancel my ongoing PCAP query request. > In order to do that, I would like to see a cancel button beside the progress > bar of the query. > An endpoint for this already exists in REST. > Attached image is from Swagger after building full-dev from the feature > branch from [METRON-1554|https://issues.apache.org/jira/browse/METRON-1554]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1676) PCAP UI - Add data range selector to the filter bar
[ https://issues.apache.org/jira/browse/METRON-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1676: Assignee: Tibor Meller (was: Shane Ardell) > PCAP UI - Add data range selector to the filter bar > --- > > Key: METRON-1676 > URL: https://issues.apache.org/jira/browse/METRON-1676 > Project: Metron > Issue Type: Sub-task >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > > As a user, I like to set a date range for my PCAP query request. > * Strat time (From) and end time (To) selectors should appear as the first > two elements of the filter bar > * Time selectors should have default values (end time: now, start time: now > - 5 days) > * Due to these fields are mandatory, the user should not be able to set time > selectors to an invalid state (eg. missing value, null value or any value > other than time) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1676) PCAP UI - Add data range selector to the filter bar
[ https://issues.apache.org/jira/browse/METRON-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1676: Assignee: Shane Ardell > PCAP UI - Add data range selector to the filter bar > --- > > Key: METRON-1676 > URL: https://issues.apache.org/jira/browse/METRON-1676 > Project: Metron > Issue Type: Sub-task >Reporter: Tibor Meller >Assignee: Shane Ardell >Priority: Major > > As a user, I like to set a date range for my PCAP query request. > * Strat time (From) and end time (To) selectors should appear as the first > two elements of the filter bar > * Time selectors should have default values (end time: now, start time: now > - 5 days) > * Due to these fields are mandatory, the user should not be able to set time > selectors to an invalid state (eg. missing value, null value or any value > other than time) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1662) PCAP UI - Downloading PCAP page files
[ https://issues.apache.org/jira/browse/METRON-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1662: Assignee: Shane Ardell > PCAP UI - Downloading PCAP page files > - > > Key: METRON-1662 > URL: https://issues.apache.org/jira/browse/METRON-1662 > Project: Metron > Issue Type: Sub-task >Reporter: Tibor Meller >Assignee: Shane Ardell >Priority: Major > > As a user, I like to be able to download the filtered PCAP query output in > PCAP binary format. > As a user, when I click the download button below the result grid, I like to > get the result page selected by the paginator component. > > *Acceptance criteria* > Given > * the user is logged in > * and navigated to the PCAP panel > * submitted a PCAP query > * the valid result appeared on the UI > When > * the user clicking the download button below the result grid > Then > * the selected page of the result should download in PCAP binary format -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1662) PCAP UI - Downloading PCAP page files
[ https://issues.apache.org/jira/browse/METRON-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1662: Assignee: Tibor Meller (was: Shane Ardell) > PCAP UI - Downloading PCAP page files > - > > Key: METRON-1662 > URL: https://issues.apache.org/jira/browse/METRON-1662 > Project: Metron > Issue Type: Sub-task >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > > As a user, I like to be able to download the filtered PCAP query output in > PCAP binary format. > As a user, when I click the download button below the result grid, I like to > get the result page selected by the paginator component. > > *Acceptance criteria* > Given > * the user is logged in > * and navigated to the PCAP panel > * submitted a PCAP query > * the valid result appeared on the UI > When > * the user clicking the download button below the result grid > Then > * the selected page of the result should download in PCAP binary format -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1723) PCAP UI - Unable to select/copy from packets details in PCAP query panel
[ https://issues.apache.org/jira/browse/METRON-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1723: Assignee: Shane Ardell > PCAP UI - Unable to select/copy from packets details in PCAP query panel > > > Key: METRON-1723 > URL: https://issues.apache.org/jira/browse/METRON-1723 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Attachments: expand-collapse-details-PCAP-query.mp4 > > > Upon query completion, a user can click on a row and drill down (up to 2 > levels) for a given packet. > A SOC analyst might find it useful to select/copy specific information from > the 2nd level granular details (E.g. MAC address, IP address, timestamps > etc.) to be used subsequently for analysis/investigation. > Currently, selecting or clicking on nested content fires an event that > expands/collapses the information. See the attached video to view the > behavior. > It would be better if target for the collapse/expand event only applied to > the top of the details section. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1675) PCAP UI - Introduce the paging capability
[ https://issues.apache.org/jira/browse/METRON-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1675: Assignee: Shane Ardell > PCAP UI - Introduce the paging capability > - > > Key: METRON-1675 > URL: https://issues.apache.org/jira/browse/METRON-1675 > Project: Metron > Issue Type: Sub-task >Reporter: Tibor Meller >Assignee: Shane Ardell >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1683) PCAP UI - Fix the download progress bar
[ https://issues.apache.org/jira/browse/METRON-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Ardell reassigned METRON-1683: Assignee: Shane Ardell > PCAP UI - Fix the download progress bar > --- > > Key: METRON-1683 > URL: https://issues.apache.org/jira/browse/METRON-1683 > Project: Metron > Issue Type: Sub-task >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > > The download progress bar currently does not appear when a PCAP search is > submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
